The same model can run as a nightly batch job, a millisecond online endpoint, on-device inference, or pay-per-call serverless. The four types have completely different cost structures and failure modes - choosing wrong breaks either the bill or the experience.

💡 What You Will Learn

The same model can run as a nightly batch job, a millisecond online endpoint, on-device inference, or pay-per-call serverless. The four types have completely different cost structures and failure mode

📜 Table of Contents

• 1. Comparison
• 2. Batch: cheapest, but not an online service
• 3. Online: manage both latency and cost
• 4. Edge: privacy in exchange for complexity
• 5. Serverless: for spiky traffic
• 6. Production is usually hybrid
• FAQ

AI Model Deployment Types Explained: Batch, Online, Edge, Serverless

The same model can run as a nightly batch job, a millisecond endpoint, on-device inference or pay-per-call serverless. The cost structures and failure modes differ completely.

1. Comparison

2. Batch: cheapest, but not an online service

Process large datasets without latency pressure. Requirements: resumable jobs, sharding so memory does not blow up, and idempotency so reruns yield identical results. Batch jobs usually die from one malformed record silently stalling the whole pipeline for days.

3. Online: manage both latency and cost

A resident inference service behind a load balancer, the most expensive shape because GPUs stay on. Three levers: request batching within a window, KV cache reuse across turns, and quantization or distillation for high-frequency traffic. P99 matters far more than the average - a 200ms average with a 3s P99 feels like "sometimes it just hangs".

4. Edge: privacy in exchange for complexity

Quantized models in phones, cameras and vehicles. Zero latency, data never leaves the device, works offline. The cost is fleet management: version distribution, effect monitoring, and low-end devices. Put small high-frequency abilities on device and fall back to cloud for hard tasks.

5. Serverless: for spiky traffic

Pay per call, scale to zero. Suits internal tools, campaign apps and extreme peaks. Two constraints: cold starts (model reload can take seconds to tens of seconds) and concurrency limits. Mitigate with smaller models, a minimum warm instance count, or an async submit-and-poll pattern for long jobs.

6. Production is usually hybrid

Batch handles backfills and index refreshes, online serves the product, edge does privacy-sensitive preprocessing, serverless absorbs campaign peaks. Decide with three questions: can it wait, can the data leave, is the traffic steady.

FAQ

Which is cheapest? Batch, by a wide margin. If a workload can be batch, do not serve it online.

Is serverless more expensive than always-on? For steady high utilization, always-on wins; for sparse or spiky traffic, serverless wins. Compare average utilization.

How much extra work is edge? Quantization, format conversion, on-device runtime adaptation and staged distribution - often more work than model development itself.

How to control online cost? Route simple requests to small models, batch within a window, and cap context and output length. Most waste comes from using the biggest model for everything.

Can you switch types later? Yes, and it is common. Wrap the model call in one layer and the transport can change.

Note: check each platform's official docs for billing, cold-start behavior and concurrency limits.

❓ FAQ

Can I mix types?

Yes - most companies run batch for backfill and online for the live product.

Which type is cheapest?

Batch, by far, since you can use spot instances and idle GPUs.