The Hugging Face platform hosts thousands of models compatible with llama.cpp:
You can use any llama.cpp-compatible model from Hugging Face using this CLI argument: -hf <user>/<model>[:quant]. For example:
llama cli -hf ggml-org/gemma-3-1b-it-GGUFYou can use the same CLI invocation to download from other sites, by pointing the MODEL_ENDPOINT environment variable to an endpoint compatible with the Hugging Face API.
llama.cpp can also run models you have downloaded locally to your filesystem.
After downloading a model, use the CLI tools to run it locally - see below.
llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.
To learn more about model quantization, read this documentation
The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:
- Use the GGUF-my-repo space to convert to GGUF format and quantize model weights to smaller sizes
- Use the GGUF-my-LoRA space to convert LoRA adapters to GGUF format (more info: ggml-org#10123)
- Use the GGUF-editor space to edit GGUF meta data in the browser (more info: ggml-org#9268)
- Use the Inference Endpoints to directly host
llama.cppin the cloud (more info: ggml-org#9669)
For models with a per-layer embedding (PLE) table, export the backbone and table from the same checkpoint:
python3 convert_hf_to_gguf.py /path/to/model --no-ple --outfile model-no-ple.gguf
python3 convert_hf_to_gguf.py /path/to/model --ple-only --outfile model-ple.gguf
llama-cli -m model-no-ple.gguf --shard model-ple.gguf--no-ple excludes only per_layer_token_embd.weight and keeps the PLE metadata and projection weights. --ple-only exports only that table, without a tokenizer. Each export supports --outtype, so the table and backbone can use different precisions. Default output names start with no-ple- and ple- respectively.
These modes are mutually exclusive and require a model with a PLE table, such as Qwen3.8 Flash Next or Gemma 4. They cannot be combined with vocabulary, multimodal projector, MTP-only, or DSpark-only export. --no-ple can be combined with --no-mtp.
--ple is an alias for --shard. Extra shards can be comma separated or supplied with repeated options, for example -m backbone.gguf --shard /fast/ple.gguf,/other/token-embd.gguf. Paths may be on different drives. Tensor names must be unique across all files, except for the PLE case below.
A numbered backbone is auto-discovered and checked as usual; explicitly supplied paths outside that numbered set are appended as extra shards. Their split numbering is ignored, and the loader uses the actual combined tensor count. Paths already in the discovered set are loaded once. If the main filename does not follow the numbered naming scheme, an explicit backbone list must still match its declared split count and order.
If an extra shard holds only per_layer_token_embd.weight, it replaces the PLE table of the model. If the model already has the table, the extra shard overrides it. A backbone split that holds only that tensor is skipped, and it can be absent from disk. This lets one PLE file be shared across quants and releases, for example -m model-Q4_K_M-00001-of-00003.gguf --shard /fast/ple-Q8_0.gguf.