"Will it fit?" is the first question anyone with a graphics card asks about a new model, and the answers on forums range from helpful to wildly wrong. The correct answer is a short sum. This article walks through it, and every number in the tables below was produced by a script we ship alongside it, calc-vram.mjs, rather than by mental arithmetic.
Part one: the weights
A model's parameters are its learned numbers. Store each in 16 bits and the file is parameters × 2 bytes. The llama.cpp project's quantisation guide gives the reference point: Llama 3.1 8B is 32.1 GB at original size, and 4.9 GB once quantised to Q4_K_M. The 70B model goes from 280.9 GB to 43.1 GB. The same page notes that, for fully loaded models, memory and disk requirements are the same, so the file size on disk is a good first estimate of what you must fit.
"4-bit" is a family name, not a promise. The measured average sits above the headline number. The same README reports bits per weight on Llama 3.1 8B:
| F16 | 16.0005 |
|---|---|
| Q8_0 | 8.5008 |
| Q6_K | 6.5633 |
| Q5_K_M | 5.7036 |
| Q4_K_M | 4.8944 |
Those figures were measured by the llama.cpp maintainers on one model. Other models land close but not identically, because the quantiser has options to treat individual tensors differently (the README documents several). We apply them to other models as an approximation and say so. Note also the unit: the project quotes sizes in GiB in its measurement tables and GB elsewhere, which is why the maths below uses GiB (1024³ bytes) consistently and the figures can sit a few per cent away from marketing GB.
We ran the sum for three open models whose sizes and architecture are published on their Hugging Face model cards: Qwen3-8B (8.2B parameters), Qwen3-14B (14.8B) and Qwen3-32B (32.8B).
Weights only, in GiB
| Qwen3-8B | F16 / BF16: 15.3; Q8_0: 8.1; Q6_K: 6.3; Q5_K_M: 5.4; Q4_K_M: 4.7 |
|---|---|
| Qwen3-14B | F16 / BF16: 27.6; Q8_0: 14.6; Q6_K: 11.3; Q5_K_M: 9.8; Q4_K_M: 8.4 |
| Qwen3-32B | F16 / BF16: 61.1; Q8_0: 32.5; Q6_K: 25.1; Q5_K_M: 21.8; Q4_K_M: 18.7 |
This is the table people stop at, and it is why the next two parts matter.
Part two: the KV cache
When a model writes a word, each layer of the network looks back over everything already in the conversation. To avoid recomputing the past every time, the runtime stores two vectors per token per layer, the keys and the values. That store is the KV cache.
NVIDIA's technical blog on inference optimisation gives the size per token as 2 × layers × heads × head dimension × bytes per number, and walks through Llama 2 7B at 16-bit precision: with a 4,096-token sequence the cache is about 2 GB. Our script reproduces that figure (2.0 GiB) as a sanity check before it does anything else.
One refinement matters a great deal for modern models. Many use grouped-query attention, in which several query heads share each key/value head, so the "heads" in the formula is the number of KV heads, not the number of attention heads. NVIDIA's article notes that sharing key/value heads reads less data from memory and shrinks the cache. The Qwen3 model cards state it plainly: Qwen3-8B has 32 query heads but only 8 KV heads, and the 32B has 64 query heads and 8 KV heads. The models' config.json files give a head dimension of 128. That is a large saving over the older layout.
KV cache at 16-bit precision, in GiB
| Qwen3-8B | KiB per token: 144; 4k tokens: 0.6; 32k tokens: 4.5; 128k tokens: 18.0 |
|---|---|
| Qwen3-14B | KiB per token: 160; 4k tokens: 0.6; 32k tokens: 5.0; 128k tokens: 20.0 |
| Qwen3-32B | KiB per token: 256; 4k tokens: 1.0; 32k tokens: 8.0; 128k tokens: 32.0 |
A caution on the last column. The Qwen3 cards give 32,768 tokens of native context, and 131,072 only with YaRN, a context-extension technique. We include the 128k row because it shows how the cost scales, not because every setup should use it.
Two things reduce the real figure for some models. Hugging Face's documentation notes that for models using sliding-window attention, the cache stops growing once those layers reach the window size, so the linear formula overstates such models. Our three Qwen3 examples are treated as full-attention here. And the same documentation describes offloading the cache to the CPU as a fallback when a small GPU runs out of memory, trading speed for fit.
Part three: the total
Add the weights, the cache at your chosen context length, and an allowance for the runtime's own buffers. We use 1.5 GiB. Treat that as a round planning number of our own; the real overhead depends on the runtime, the batch size and how it allocates compute buffers.
Total at Q4_K_M with a 16-bit KV cache, in GiB
| Qwen3-8B | 4k context: 6.7; 32k context: 10.7; 128k context: 24.2 |
|---|---|
| Qwen3-14B | 4k context: 10.6; 32k context: 14.9; 128k context: 29.9 |
| Qwen3-32B | 4k context: 21.2; 32k context: 28.2; 128k context: 52.2 |
And at the higher-quality Q8_0 with 32k of context: 14.1 GiB for the 8B, 21.1 GiB for the 14B and 42.0 GiB for the 32B.
Read across the rows and the pattern is clear. A 32B model at Q4_K_M and short context needs about 21 GiB, so it is out of reach of a 16GB card, a squeeze on a 24GB one once you ask for longer context, and comfortable on a 32GB card. Push it to 32k tokens and it wants 28 GiB. For the 8B, 128k tokens of 16-bit cache costs more than three times as much as the weights it sits beside.
What to do when it does not fit
You have several honest options, in rough order of pain.
- Shorten the context. It is the cheapest lever, and the one most people forget.
- Drop a quantisation step. Q8_0 to Q6_K or Q5_K_M saves roughly a quarter to a third of the weights, at some quality cost that depends on the model and task.
- Choose a model with fewer KV heads for its size. Architecture matters as much as parameter count.
- Split across CPU and GPU. llama.cpp supports hybrid inference for models bigger than your VRAM, and it works, but layers left in system memory are far slower.
- Buy more memory, or a machine where the memory is shared. Our piece on choosing a mini PC for home AI covers that choice.
Mixture-of-experts: a caveat
Everything above assumes a dense model, where every weight is used for every token. Mixture-of-experts models activate only a fraction of their weights per token, which helps speed, but you still generally need room to hold all the weights. Count total parameters for capacity, not active parameters.
Run it yourself
The script is calc-vram.mjs, published with this article. It needs only Node, has no dependencies, and prints every table above. To check a different model, add its parameter count, layer count, KV-head count and head dimension (all in the model's config.json and card) to the models list. If the KV-head count is wrong the answer will be wrong, so read it from the config rather than guessing.
What this does not tell you
Memory is only half of local performance. The sum above says whether a model fits, not whether it will feel quick, which depends largely on how fast your hardware can read memory. That is the subject of our guide to CPU, GPU and NPU inference. We have not measured any runtime's real overhead for this article, which is why that figure is labelled as an assumption.
