llama.cpp — Build, CLI Reference and Small-GPU Optimisation

Last reviewed: 2026-10-09

Purpose: Build llama.cpp on Linux and macOS, understand the CLI switches that actually matter, and get the best performance out of a small GPU.

Overview

llama.cpp is a C/C++ inference engine for large language models stored in GGUF format. It runs the same model on a laptop CPU, an Apple Silicon GPU, a consumer NVIDIA card, an Intel iGPU or an AMD card, without a Python runtime in the loop. The project ships a reusable llama library plus a set of tools — llama-cli for interactive use, llama-server for an OpenAI-compatible HTTP API, llama-bench for measurement, and several llama-* utilities for conversion and quantisation.

It matters for three reasons:

It runs where cloud models cannot — local, offline, on modest hardware, with no per-token cost.
It exposes the knobs. Almost everything that decides speed and memory — how many layers sit on the GPU, how the KV cache is stored, how large the batch is — is a command-line switch you control.
It is the engine under many other tools, so understanding the flags pays off twice.

[!NOTE] llama.cpp moves fast: switches are added, renamed and removed between releases. The authoritative list for your build is llama-server --help (or --usage). This article reflects the master branch as of October 2026 — the tables below name the current flag form and flag anything recently renamed.

Architecture and Concepts

Component	Role
`llama` library	Core inference engine; C API in `include/llama.h`
`llama-cli`	Interactive prompt / one-shot generation, chat mode
`llama-server`	HTTP server with an OpenAI-compatible API and a built-in Web UI
`llama-bench`	Reproducible benchmarks (prompt processing and generation)
`ggml`	The tensor library underneath; its backends are what you enable at build time

The backend is chosen at compile time and is what makes the GPU work:

Backend	Hardware	CMake switch
CPU	Anything (always built)	default
Metal	Apple Silicon and Intel Macs	`-DGGML_METAL=ON` (default on macOS)
CUDA	NVIDIA	`-DGGML_CUDA=ON`
HIP	AMD (ROCm)	`-DGGML_HIP=ON`
Vulkan	AMD, Intel, NVIDIA — no vendor SDK needed	`-DGGML_VULKAN=ON`
SYCL	Intel GPUs	`-DGGML_SYCL=ON`
OpenCL	Adreno and other OpenCL devices	`-DGGML_OPENCL=ON`

Three concepts explain most of the flags:

Offloading — layers of the model live in VRAM (-ngl); the rest stays in system RAM and is computed on the CPU.
The KV cache — the memory holding the conversation's attention keys and values. It grows linearly with context length and is frequently the reason a small GPU runs out of memory.
Quantisation — the model weights are stored at reduced precision (Q4_K_M, Q8_0, …). Quantising the KV cache is a separate, independent lever.

Build on Linux

Prerequisites

# Debian / Ubuntu
sudo apt-get update && sudo apt-get install -y build-essential cmake git curl libcurl4-openssl-dev

CPU Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8

The binaries land in build/bin/. Notes that save real time:

-j parallelises the compile. Without it a full build is slow; with it, minutes.
Ninja is faster than make: add -G Ninja to the configure step.
ccache (install ccache) makes repeated rebuilds far cheaper.
Static binaries: add -DBUILD_SHARED_LIBS=OFF at configure time.
HTTPS support (needed to download models directly from Hugging Face) requires the OpenSSL development headers — install libssl-dev on Debian/Ubuntu, openssl-devel on Fedora/RHEL, openssl on Arch. Without them the project still builds, but without SSL.

# portable build covering all CUDA GPUs, not just the one present at build time
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF

GPU Backends

# NVIDIA (CUDA toolkit must be installed)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

# AMD (ROCm)
cmake -B build -DGGML_HIP=ON

# a GPU with no vendor SDK, or mixed hardware (works on AMD, Intel and NVIDIA)
cmake -B build -DGGML_VULKAN=ON

# Intel GPU
cmake -B build -DGGML_SYCL=ON

[!TIP] Vulkan is the pragmatic choice for an old or unusual GPU. It needs no CUDA toolkit and no ROCm install, works across all three vendors, and gives most of the speedup of a vendor backend on a small card. Verify it picked up the device by running with -ngl 99 and reading the first lines of output — a Vulkan build prints the detected device.

BLAS and CPU Acceleration

BLAS only accelerates prompt processing (batched ingest of the input); it does not change generation speed. Useful when a long prompt is fed to a CPU-only or CPU-heavy setup.

BLAS	Vendor switch	Note
OpenBLAS	`-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS`	General-purpose, easiest to install
Intel oneMKL	`-DGGML_BLAS_VENDOR=Intel10_64lp`	Best on recent Intel CPUs
BLIS / AOCL-BLAS	`-DGGML_BLAS_VENDOR=FLAME` / `AOCL`	AMD-optimised variants
Accelerate	`-DGGML_BLAS_VENDOR=Apple`	macOS only, ships with the OS

Build on macOS

Metal is enabled by default — on macOS the standard build already computes on the GPU, so no extra flag is required:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8

Optional refinements:

# Apple's vectorised BLAS for faster prompt processing
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple

# Xcode project instead of makefiles (better for debugging in the IDE)
cmake -B build -G "Xcode"
cmake --build build --config Release

# a CPU-only build, if you ever need to compare or debug
cmake -B build -DGGML_METAL=OFF

Two macOS specifics:

You can disable GPU use at runtime without rebuilding: --n-gpu-layers 0.
Apple Silicon has unified memory — GPU and CPU share the same RAM pool. "Offloading to VRAM" therefore does not create a separate budget; the discipline is to leave headroom for the OS and the app, and to size the context rather than chase a VRAM ceiling.

Homebrew remains the quickest way to a working binary (brew install llama.cpp), at the cost of lagging the master branch. Build from source when a fix or a new flag matters.

Quick Start

Run from the Hugging Face Hub

-hf resolves the repository, picks a quantisation (default Q4_K_M), downloads it, and starts:

llama-cli -hf <user>/<model>-GGUF:Q4_K_M

Start an OpenAI-Compatible Server

llama-server \
  -hf <user>/<model>-GGUF:Q4_K_M \
  -c 8192 \
  -ngl all \
  --host 127.0.0.1 \
  --port 8080

[!WARNING] The server's default port is 9931, and it binds to 127.0.0.1 only. Set --host 0.0.0.0 deliberately, and never expose it beyond your network without --api-key. The official Docker image examples bind 8080, which is why so many guides assume that port — pass --port explicitly.

Call the API

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Explain the KV cache in one paragraph."}],
    "max_tokens": 256
  }'

Health check and machine-readable model info:

curl -s http://localhost:8080/health
curl -s http://localhost:8080/props | head -40

CLI Reference

All common switches behave identically in llama-cli and llama-server. Every parameter can also be supplied through an environment variable (LLAMA_ARG_*); the command-line argument wins when both are set, which is what makes the same container image configurable.

Model Loading and Download

Switch	Purpose	Notes
`-m, --model FNAME`	Path to a local GGUF file	The classic entry point
`-hf, --hf-repo <user>/<model>[:quant]`	Download and run straight from Hugging Face	Quant defaults to `Q4_K_M`; multimodal projectors are fetched automatically
`-hff, --hf-file FILE`	Pin an exact file instead of a quant label	Use when the repo has unusual filenames
`-hft, --hf-token TOKEN`	Hugging Face token for gated repos	Or set `HF_TOKEN`
`-lm, --load-mode MODE`	How the model is mapped into memory: `auto`, `none`, `mmap`, `mlock`, `mmap+mlock`, `dio`	Replaces the older `--mlock` and `--no-mmap` switches, which no longer exist. `mlock` pins the model in RAM (no swapping); `dio` uses DirectIO
`-lzm, --lazy-mode MODE`	Read large tensors on demand instead of keeping them resident	Saves RAM for per-layer embedding models; needs mmap
`-cl, --cache-list`	List models already in the local cache	Check before re-downloading
`--offline`	Force cache use, forbid network access	Air-gapped and reproducible runs
`--check-tensors`	Validate tensor data for invalid values	Diagnoses a corrupt download
`--override-kv KEY=TYPE:VALUE`	Override model metadata	E.g. forcing `add_bos_token=false`
`-a, --alias STRING`	Model name exposed by the API	Cosmetic but useful for clients

Context and Compute Sizing

Switch	Purpose	Notes
`-c, --ctx-size N`	Prompt context window in tokens	Default `0` = take it from the model. The single most important memory dial
`-n, --predict N`	Tokens to generate	Default `-1` = unlimited
`-b, --batch-size N`	Logical batch size	Default 2048; larger helps prompt throughput
`-ub, --ubatch-size N`	Physical micro-batch	Default 512; lowering it shrinks peak compute-buffer VRAM
`--keep N`	Tokens kept from the initial prompt	Default 0; `-1` keeps everything
`--rope-scaling`	RoPE scaling method: `none`, `linear`, `yarn`	To run beyond the trained context length
`--rope-scale N`	Multiplies the context window	Pair with the method above
`--yarn-orig-ctx N`	YaRN: original training context	The reference point for the scaling

GPU Offload and Memory

Switch	Purpose	Notes
`-ngl, --gpu-layers N`	Layers stored in VRAM	Accepts an exact number, `auto`, or `all`. Default `auto`
`--list-devices`	Print available devices and exit	First thing to run on new hardware
`-dev, --device LIST`	Restrict offload to specific devices	Comma-separated
`-fit, --fit on / off`	Auto-adjust unset arguments to fit device memory	Default `on` — the modern answer to "what `-ngl` should I use"
`-fitt, --fit-target MiB`	Safety margin per device for `--fit`	Default 1024 MiB; raise it if you still hit OOM
`-fitc, --fit-ctx N`	Minimum context `--fit` may reduce to	Default 4096
`-ot, --override-tensor PATTERN=TYPE`	Force specific tensors onto a buffer type	Surgical placement, e.g. keep attention on GPU
`-cmoe, --cpu-moe`	Keep all MoE expert weights on the CPU	For MoE models too large for the card
`-ncmoe, --n-cpu-moe N`	Keep the MoE experts of the first N layers on the CPU	The graded version of the switch above
`--moe-cache-mib N`	GPU cache for CPU-resident MoE experts	Default 0 (disabled)
`-ncffn, --n-cpu-ffn N`	Keep the dense FFN weights of the first N layers on the CPU	Dense models only
`-sm, --split-mode MODE`	Multi-GPU split: `none`, `layer`, `row`, `tensor`	Default `layer`; irrelevant with one GPU
`-ts, --tensor-split N0,N1`	Proportion of the model per GPU	For uneven multi-GPU setups
`-mg, --main-gpu INDEX`	Which GPU holds the model and KV	Default 0

KV Cache and Attention

Switch	Purpose	Notes
`-fa, --flash-attn MODE`	Flash Attention: `on`, `off`, `auto`	Default `auto`. Enables the KV-cache savings below and cuts compute-buffer memory
`-ctk, --cache-type-k TYPE`	Data type of the K cache	Default `f16`; accepts `q8_0`, `q4_0`, `q4_1`, `iq4_nl`, `q5_0`, `q5_1`, `bf16`, `f32`
`-ctv, --cache-type-v TYPE`	Data type of the V cache	Same values; quantised V requires Flash Attention
`-kvo, --kv-offload`	Keep the KV cache in VRAM	Default enabled — leave it
`--swa-full`	Use a full-size sliding-window cache	Default off; only relevant for SWA models
`--no-host`	Bypass the host buffer to allow extra buffers	Niche; can free a little memory or break things

CPU Threads and Affinity

Switch	Purpose	Notes
`-t, --threads N`	Threads for generation	Default `-1` = auto. Start at physical cores, not hyperthreads
`-tb, --threads-batch N`	Threads for prompt processing	Defaults to `--threads`
`-C, --cpu-mask` / `-Cr, --cpu-range`	Pin the process to specific cores	Reduces migration on NUMA or busy hosts
`--cpu-strict`	Strict placement	Default 0
`--prio N`	Thread priority: `low`, `normal`, `medium`, `high`, `realtime`
`--poll N`	Busy-poll level 0–100	Default 50; higher can cut latency at the cost of CPU
`--numa TYPE`	NUMA strategy: `distribute`, `isolate`, `numactl`	Only on multi-socket machines

Server Parameters

Switch	Purpose	Notes
`--host HOST`	Addresses to listen on	Default `127.0.0.1`
`--port PORT`	Listening port	Default 9931
`-np, --parallel N`	Number of server slots	Default `-1` = auto. Each slot multiplies the KV requirement
`-cb, --cont-batching`	Continuous (dynamic) batching	Default enabled — keep it
`-kvu, --kv-unified`	One KV buffer shared across slots	Default enabled when slots are auto
`-cram, --cache-ram N`	Maximum prompt cache size in MiB	Default 8192; `-1` unlimited, `0` disabled
`-ctxcp, --ctx-checkpoints N`	Context checkpoints per slot	Default 32
`-cms, --checkpoint-min-step N`	Minimum spacing between checkpoints	Default 8192 tokens
`--cache-prompt`	Cache the prompt between requests	Default enabled
`--cache-reuse N`	Minimum chunk size reused from the cache via KV shifting	Default 0 (disabled); a real win for RAG with shared prefixes
`--api-key KEY`	Require authentication	Multiple keys comma-separated; set this on anything not localhost
`--ssl-key-file` / `--ssl-cert-file`	Serve HTTPS directly	Must be built with SSL support
`--chat-template NAME`	Force a built-in chat template	E.g. `chatml`, `llama3`, `mistral-v7`, `qwen`-family names
`--jinja`	Use the model's Jinja chat template	Default enabled in current builds; required for tool calling
`--reasoning-format FORMAT`	How thinking traces are returned: `none`, `deepseek`, `deepseek-legacy`	Default `auto`
`--reasoning-budget N`	Token budget for thinking	`-1` unrestricted, `0` ends immediately
`--embedding` / `--rerank`	Restrict the server to embeddings or reranking	Use with dedicated models
`--metrics`	Prometheus-compatible metrics endpoint	Default disabled — enable it if you monitor
`--slots`	Slots monitoring endpoint	Default enabled
`--sleep-idle-seconds N`	Unload from GPU after N idle seconds	Useful on a shared or small GPU
`-to, --timeout N`	Read/write timeout in seconds	Default 3600
`--ui` / `--no-ui`	Built-in Web UI	Default enabled

[!WARNING] The agent-oriented switches — --tools, --agent, --mcp-servers-config — let the server read files, write files and execute shell commands on the host. They are experimental and the documentation says plainly not to enable them in untrusted environments. Leave them off unless you are deliberately building a sandboxed lab.

Sampling Parameters

Switch	Purpose	Default
`--temp, --temperature N`	Randomness	0.80
`--top-k N`	Keep the K most likely tokens	40
`--top-p N`	Nucleus sampling	0.95
`--min-p N`	Minimum-probability floor	0.05
`--repeat-penalty N`	Penalise repetition	1.00 (disabled)
`--repeat-last-n N`	Window for the repetition penalty	64
`--presence-penalty` / `--frequency-penalty`	OpenAI-style penalties	0.00
`--samplers LIST`	Exact sampler chain, semicolon-separated	`penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature`
`-s, --seed N`	RNG seed	`-1` (random)
`-j, --json-schema SCHEMA`	Constrain output to a JSON schema	Disabled
`--grammar` / `--grammar-file`	Constrain output with a BNF grammar	Disabled
`--mirostat N`	Mirostat sampling	0 (disabled)

Logging and Diagnostics

Switch	Purpose	Notes
`-h, --help, --usage`	Print the full argument list	The authority for your build
`--version`	Version and build info	Includes compiled backends
`-v, --verbose`	Maximum verbosity	Confirms which backend and device were used
`-lv, --verbosity N`	Log threshold: 0 generic, 1 error, 2 warning, 3 info, 4 trace, 5 debug	Default 3
`--log-file FNAME` / `--log-jsonl`	File logging, or one JSON object per line on stdout	JSONL suits log shippers
`--perf`	Internal performance timings	Quick profiling without a separate tool
`--completion-bash`	Emit a sourceable bash completion script	Quality of life

Small-GPU Optimisation

A small GPU (4–8 GB) is not a dead end: it changes what you run and how you size it. The whole game is one budget.

The VRAM Budget

VRAM needed  =  model weights  +  KV cache  +  compute buffer  +  (multimodal projector)

Three of those four terms are under your control, and people almost always attack the wrong one — they shrink the model when the KV cache was the problem.

The KV Cache Is Usually the Problem

The KV cache grows linearly with context length, and its cost depends on the model's layer count, KV-head count and head dimension. For a typical 8B model with grouped-query attention (32 layers, 8 KV heads, head dimension 128), one token costs:

KV data type	Per token	4k context	8k context	16k context	32k context
`f16` (default)	128 KiB	0.50 GiB	1.00 GiB	2.00 GiB	4.00 GiB
`q8_0`	68 KiB	0.27 GiB	0.53 GiB	1.06 GiB	2.12 GiB
`q4_0`	36 KiB	0.14 GiB	0.28 GiB	0.56 GiB	1.12 GiB

Read that table twice: at 32k context the KV cache alone eats 4 GiB, which is the entire VRAM of a small card — before a single model weight is loaded. Quantising it to q8_0 halves the cost, and to q4_0 quarters it.

Models without grouped-query attention are far worse: a 32-KV-head model costs 512 KiB per token in f16 (4 GiB at 8k context, 16 GiB at 32k). Prefer a GQA model on a small GPU — it is a bigger decision than the quantisation level.

Levers in Order of Impact

Pick a model that fits with room to spare. A quality 3B–8B model at Q4_K_M beats a 14B model that spills into system RAM. Weights at Q4_K_M are roughly 0.6 GB per billion parameters.
Set -c to what you actually need. Halving the context halves the KV cache. An 8k window is plenty for chat and most coding assistance; 32k is a deliberate choice, not a default.
Turn on Flash Attention (-fa on). It reduces the compute buffer and is the prerequisite for quantised KV.
Quantise the KV cache — -ctk q8_0 -ctv q8_0 for a near-free halving, q4_0 on both when you are desperate for space. Expect a small quality cost on long contexts; test it on your workload.
Let --fit do the layer maths, then verify. It is on by default and adjusts unset arguments to fit. If you prefer to be explicit, raise -ngl step by step from 0 and watch VRAM; raise -fitt if it still OOMs.
Lower -ub (try 128 or 256). The micro-batch size drives the peak compute buffer; small values trade a little prompt-processing speed for a lot of headroom.
For MoE models, offload the experts to the CPU with -ncmoe N (start with the layer count divided by two) or -cmoe. MoE models give you far more parameters per gigabyte of VRAM than dense models, and this switch is what makes them usable on a small card.
Keep one slot (-np 1). Every parallel slot reserves its own KV space; a single user does not need slots.
Do not disable KV offloading. --no-kv-offload moves the cache to system RAM; it frees VRAM but usually costs more speed than it buys, and it is the mistake people reach for first.
Consider n-gram speculative decoding (--spec-type ngram-simple) for repetitive or structured output such as code edits or JSON — it needs no draft model and therefore no extra VRAM.

Worked Configurations by VRAM

Indicative starting points (weights plus q8_0 KV, compute buffer included, leaving OS headroom):

VRAM	Model	Context	Flags to start with
4 GB	3B at `Q4_K_M`	16k	`-ngl all -fa on -ctk q8_0 -ctv q8_0 -ub 256`
6 GB	8B at `Q4_K_M`	8k	`-ngl all -fa on -ctk q8_0 -ctv q8_0`
8 GB	8B at `Q5_K_M`	16k	`-ngl all -fa on -ctk q8_0 -ctv q8_0`
8 GB	MoE 30B-class at `Q4_K_M`	8k	`-ncmoe 16 -ngl all -fa on -ctk q8_0 -ctv q8_0`
12 GB	14B at `Q4_K_M`	16k	`-ngl all -fa on -ctk q8_0 -ctv q8_0`

[!TIP] An 8 GB card running an 8B Q4_K_M model with an 8k context and q8_0 KV has roughly 2 GB of slack. Spend it on context length if your work needs long inputs, or on a higher-precision quant (Q5_K_M, Q6_K) if it needs accuracy — not on raising -ngl, which is already at its maximum.

Measuring Instead of Guessing

Every tuning claim above should be checked on your own hardware, because PCIe bandwidth, driver versions and quantisation choices shift the balance:

# prompt processing and generation speed, repeated for stability
llama-bench -m model-Q4_K_M.gguf -ngl all -p 512 -n 128

# same model, KV quantised - compare both numbers
llama-bench -m model-Q4_K_M.gguf -ngl all -ctk q8_0 -ctv q8_0 -p 512 -n 128

While serving, --metrics exposes the same figures to Prometheus, and --perf prints internal timings per request. Watch nvidia-smi (or the Vulkan/Metal equivalent) during a run: if the generation speed collapses after the prompt is ingested, model weights are spilling to system RAM and -ngl is too high.

Traps

Symptom	Real cause	Fix
Out of memory at load	KV cache reserved for the full context	Lower `-c`, or quantise the KV
Out of memory mid-generation	Compute buffer too large	Lower `-ub`, add `-fa on`
Very slow generation, VRAM "full"	Weights spilling into system RAM	Lower `-ngl`, or quantise the model further
Fast prompt, slow tokens	Normal CPU/GPU split — prompt is batched, generation is not	Move more layers to the GPU, or accept it
Slow with a long prompt only	BLAS not enabled, or `-tb` too low	Rebuild with BLAS, raise `-tb`
Prompts re-processed every request	Prompt caching or reuse disabled	Keep `--cache-prompt` on, set `--cache-reuse`
Second parallel user doubles memory	One KV per slot	`-np 1` unless you truly serve concurrently

Troubleshooting

Problem	Likely cause and fix
`unknown argument` after an upgrade	The flag was renamed. Re-check `--help`; `--mlock` and `--no-mmap` are now `--load-mode`
CUDA build fails on an unusual GPU	Build with `-DGGML_NATIVE=OFF`, or set the compute capability explicitly
Model downloads fail with an SSL error	The build lacks OpenSSL development headers; rebuild with them
`ggml_vulkan` does not detect the card	Wrong driver/Vulkan loader; test with `vulkaninfo`
Metal unused on macOS	Rebuild without `-DGGML_METAL=OFF`; confirm with `-v` and check `-ngl` is not 0
Server unreachable from another machine	It binds `127.0.0.1` by default — set `--host 0.0.0.0`
Client expects port 8080	The default is 9931; pass `--port 8080` or point the client at 9931
Chat template errors or raw tokens in output	Force the template with `--chat-template`, and keep `--jinja` enabled
Garbled or repetitive output	Sampling mismatch, or a KV quantisation the model dislikes — retest with `-ctk f16 -ctv f16`