AI Inferencing Explained 2026
You train once and pay for inference every day. The same model can cost a few dollars per million tokens for one team and tens of dollars for another, and the difference is not the model — it is batching, quantization, caching, and whose GPU it runs on.
💡 What You Will Learn
You train once and pay for inference every day. The same model can cost a few dollars per million tokens for one team and tens of dollars for another, and the difference is not the model — it is batch
📜 Table of Contents
AI Inference Explained: What a Request Costs and Where to Run It
Training is a one-time capital cost; inference is the daily bill. The same model can cost a few dollars per million tokens for one team and tens for another, and the difference sits in batching, quantization, caching and hardware.
Two phases, two profiles
Prefill runs the whole input in parallel — compute-bound and efficient. Decode emits one token at a time and re-reads the KV cache — memory-bandwidth-bound. The GPU is not short of arithmetic; it cannot move data fast enough. Context length costs VRAM, output length costs time.
Four metrics
TTFT governs whether it feels fast (under a second for chat). TPOT governs reading speed. Throughput governs cost per token. Cost per million tokens is the bill — see official pricing. Optimise latency for interactive products and throughput for batch jobs; they pull in different directions.
Where to run it
| Route | Tools | Cost |
|---|---|---|
| Cloud GPU | vLLM, SGLang, TensorRT-LLM | Per token |
| Local workstation | llama.cpp, Ollama, LM Studio | Hardware up front |
| Phone / edge | llama.cpp mobile, MLC-LLM | Small models only |
| Browser | WebLLM on WebGPU | Weight download |
vLLM is the most mature (paged attention, continuous batching), SGLang leads on structured output and prefix reuse, TensorRT-LLM needs per-GPU engine builds but is fastest once tuned.
Quantization buys memory with accuracy
FP16 is the baseline; INT8 roughly halves VRAM with near-lossless quality on most tasks; 4-bit cuts it to a quarter to a third with visible but acceptable degradation in math and code; 2-3 bit should be a last resort. Start local at 4-bit, and try INT8 or FP8 on servers first.
Cloud or local
Count daily input and output tokens from logs, multiply by official prices, and compare with hardware amortised over 24-36 months plus power and ops time. High volume, simple tasks, latency-tolerant: local usually wins. Bursty traffic and always-newest-model requirements: cloud earns its premium. A local router in front of a hosted flagship is often the healthiest structure.
FAQ
Inference vs fine-tuning? Fine-tuning changes weights; inference uses them.
No GPU? llama.cpp on CPU handles small quantized models for offline work.
Local feels slow? Check memory bandwidth, then whether the model fits in VRAM — spilling to system RAM costs an order of magnitude.
Framework parameters and pricing change; check official docs.
Related
❓ FAQ
Is inferencing the same as fine-tuning?
No. Fine-tuning changes the weights; inferencing uses the weights to predict.
Can I do inferencing without a GPU?
Yes - CPU-only works with small quantized models via llama.cpp or Ollama.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.