How much VRAM does Llama 3.1 8B need?
Llama 3.1 8B has 8.03 billion parameters. What that costs in GPU memory depends entirely on the quantization you run it at.
5.2 GB
Running Q4 with a 4K token context — the setup most people actually use.
VRAM by quantization
| Quantization | Weights | KV cache | Total |
|---|---|---|---|
| F16 | 15.0 GB | 0.5 GB | 16.0 GB |
| Q8 | 7.5 GB | 0.5 GB | 8.5 GB |
| Q5 | 5.1 GB | 0.5 GB | 6.1 GB |
| Q4 | 4.2 GB | 0.5 GB | 5.2 GB |
Figures are for a 4K token context. The KV cache grows linearly with context length, which is why a model that fits comfortably at 4K can stop fitting at 32K without the weights changing at all.
Cards that fit it at Q4
RTX 308010 GB
RTX 306012 GB
RTX 4070 Super12 GB
RTX 507012 GB
RX 6900 XT16 GB
RX 7800 XT16 GB
Smallest first. The cheapest card the model fits on is more useful than the largest one that also fits.
Another context length, another number
This page fixes the context at 4K. Set your own context and quantization and the calculator recomputes the whole breakdown.
Open the calculator