LLM Inference
Speed Estimator.
Select an LLM model, quantization format, and GPU to estimate real-time token generation speed. Based on memory bandwidth — the true bottleneck of autoregressive inference.
1. Select LLM Model
Selecciona una familia de modelos
2. Quantization
3. Select GPU
Selecciona una marca de GPU
Configure your estimate
Select a model, quantization, and GPU to see the estimated token generation speed.
Frequently asked questions
addWhat actually determines tokens per second?
Memory bandwidth, far more than raw compute. Generating each token requires reading the whole active model out of memory, so the speed at which a card can move data is the ceiling. This is why two cards with similar compute can differ sharply if one has much faster memory.
addWhy is my card slower than the estimate?
Usually because part of the model is not on it. Once weights spill into system memory the rate collapses to a fraction. Other common causes are a long context, a runtime falling back to an unoptimised path, and thermal or power limits on a small chassis.
addCan I run a model on the processor alone?
You can, and for small models it is perfectly usable. System memory is several times slower than a graphics card's, so expect single-digit tokens per second on anything sizeable — fine for batch work, frustrating for conversation.
addDoes quantization make generation faster as well as smaller?
Yes, and for the same reason it saves memory: there is less data to read per token. Moving from 16-bit to 4-bit typically gives a large speed-up on the same hardware, on top of letting a bigger model fit.
addHow many tokens per second is enough?
People read at roughly 5 to 10 tokens per second, so anything above about 15 feels immediate in conversation. Below 5 it is uncomfortable to sit and watch, though it remains fine for work you leave running.
Keep going
Other tools and reference pages that pick up where this one leaves off.