How much VRAM does Mixtral 8x7B need?
Mixtral 8x7B has 46.7 billion parameters. What that costs in GPU memory depends entirely on the quantization you run it at.
25.5 GB
Running Q4 with a 4K token context — the setup most people actually use.
VRAM by quantization
| Quantization | Weights | KV cache | Total |
|---|---|---|---|
| F16 | 87.0 GB | 0.5 GB | 88.0 GB |
| Q8 | 43.5 GB | 0.5 GB | 44.5 GB |
| Q5 | 29.9 GB | 0.5 GB | 30.9 GB |
| Q4 | 24.5 GB | 0.5 GB | 25.5 GB |
Figures are for a 4K token context. The KV cache grows linearly with context length, which is why a model that fits comfortably at 4K can stop fitting at 32K without the weights changing at all.
Cards that fit it at Q4
RTX 509032 GB
Radeon PRO W790048 GB
RTX 6000 Ada48 GB
RTX A600048 GB
A100 PCIe80 GB
H100 PCIe80 GB
Smallest first. The cheapest card the model fits on is more useful than the largest one that also fits.
Another context length, another number
This page fixes the context at 4K. Set your own context and quantization and the calculator recomputes the whole breakdown.
Open the calculator