Running a language model on your own graphics card feels free once the card is paid for. It is not free, and it is not as expensive as people fear either. The cost splits into two parts — the electricity each token burns, and the share of the hardware each token uses up — and they behave very differently. This article prices both for four graphics cards, and ends with the reasons that matter more than price.
The unit: a million tokens
Cloud AI services charge per million tokens, so that is the unit worth using at home too. A token is a fragment of a word; in English a million tokens is roughly 750,000 words, or a dozen long novels. Generating that much locally takes a number of hours set by how fast the card is, and during those hours the machine draws power.
Speed comes from the same arithmetic our inference speed estimator uses: memory bandwidth divided by the size of the model in memory, times 0.75. Power is the graphics card's rated draw plus 105 W for a mainstream processor and 50 W for the rest of the machine — the same way our power supply calculator adds it up. Both models are at 4 bits.
What the electricity costs
| Card and model | Speed | Hours per million tokens | US ($0.18/kWh) | Spain ($0.25/kWh) | Germany ($0.40/kWh) |
|---|---|---|---|---|---|
| RTX 5060 Ti 16GB · Llama 3.1 8B | 84 tok/s | 3.3 h | $0.20 | $0.28 | $0.44 |
| RTX 5060 Ti 16GB · Qwen3 14B | 45 tok/s | 6.2 h | $0.37 | $0.52 | $0.83 |
| RTX 5070 Ti · Llama 3.1 8B | 167 tok/s | 1.7 h | $0.14 | $0.19 | $0.30 |
| RTX 5070 Ti · Qwen3 14B | 91 tok/s | 3.1 h | $0.25 | $0.35 | $0.56 |
| RTX 4090 · Llama 3.1 8B | 188 tok/s | 1.5 h | $0.16 | $0.22 | $0.36 |
| RTX 4090 · Qwen3 14B | 102 tok/s | 2.7 h | $0.30 | $0.41 | $0.66 |
| RTX 5090 · Llama 3.1 8B | 335 tok/s | 0.8 h | $0.11 | $0.15 | $0.24 |
| RTX 5090 · Qwen3 14B | 182 tok/s | 1.5 h | $0.20 | $0.28 | $0.45 |
The fastest card is the cheapest per token
It looks backwards until you see why. The RTX 5090 draws more than twice the power of the RTX 5060 Ti, but it finishes the same million tokens four times faster, so it burns less energy in total. Per token, the most power-hungry card on the table has the lowest electricity cost. Generating text is a job with a fixed amount of reading to do, and doing it faster costs less.
Model size doubles the bill
Moving from an 8-billion to a 14-billion-parameter model roughly doubles the cost on every card, because there are roughly twice as many weights to read for every token. As with speed, the choice of model moves the number more than the choice of card.
What the hardware costs
Electricity turns out to be the small part. The larger one is the card itself, spread across every token it ever generates:
hardware cost per million tokens = price of the card ÷ millions of tokens it generates in its working life
Take a card that cost $800 and is used for three years. Generating a million tokens a day — heavy use, an assistant running all day or a batch job every night — that is about 1,100 million tokens, or roughly $0.73 per million. Generating 100,000 tokens a day, which is a lot of conversation for one person, it is about $7.30 per million. The hardware share falls in direct proportion to how hard you use it.
That is the whole economic argument in one line: local AI is cheap per token only for people who generate a great deal. For occasional use, the card is the cost, and the electricity is a rounding error. Compare the result with the per-million-token price your cloud provider publishes for a model of similar size before deciding either way — and note that a card you already own for gaming has a hardware cost close to zero for this purpose.
The reasons that are not about price
For most people who run models locally, cost is not the reason. Three others usually are:
- Privacy. Nothing leaves the machine. Documents, code and conversations stay on your own disk, which for some work is the only acceptable arrangement.
- Stability. A model you have downloaded does not change. A cloud service can change its model, its features or its behaviour overnight, without an update on your side and without a way back to last week's version. Software installed locally can be frozen on a version that works; a service cannot.
- Availability. No outages, no rate limits, no network. The model answers on a plane.
The trade-off is capability. The models that fit on a consumer card — up to roughly 30 billion parameters at 4 bits on 24 GB — are good and improving, but they are not the largest models cloud services run. Whether that matters depends entirely on the task.
Where these figures can be off
The speeds are estimates from memory bandwidth, not measurements, and real runtimes land somewhat above or below them. Power under text generation is usually below the rated figure, so the electricity column is a ceiling. Idle time costs something too: a machine left on all day waiting for prompts draws power while doing nothing. And tariffs vary widely within countries. To run your own numbers, find the speed of your card and model in the inference speed estimator, check the model fits with the VRAM calculator, and put the draw and your hours into the electricity cost calculator.