AI Agent Async Processing 2026
An agent sends an email and makes the user wait three seconds. This guide covers when to go async, how to do it with a queue, and the implementation details: state, delivery and idempotency.
💡 What You Will Learn
An agent sends an email and makes the user wait three seconds. This guide covers when to go async, how to do it with a queue, and the implementation details: state, delivery and idempotency.
📜 Table of Contents
Synchronous waiting is the agent's biggest experience hole
A user says "summarise this week's meeting notes and post them to the group". The agent retrieves, summarises, calls the mail or bot API. That is 5 to 30 seconds. Wait synchronously and the user stares at a spinner, browsers and gateways time out, and people click twice.
The idea is simple: return immediately for what the user needs now, push the slow part to the background, notify when done.
When to go async
| Scenario | Recommendation | Why |
|---|---|---|
| Single Q&A, lookups | Sync | 1-3 seconds, a queue adds noise |
| Multi-step retrieval + summary | Async or streaming | 10+ seconds kills the UX |
| 100 documents in a batch | Async | Minutes of work |
| Email, calendar, third-party API | Async | Needs retries, nobody should wait |
| Long code execution | Async plus sandbox | Timeout risk |
| Tasks with visible progress | Streaming | Users want to see steps |
A workable rule: if expected latency exceeds five seconds, or the step can fail and needs retrying, make it async.
Three implementations
| Approach | Mechanism | Strength | Cost |
|---|---|---|---|
| In-process background task | Spawn and return a task id | No dependencies | Lost on restart, not for heavy work |
| Message queue | Redis plus Celery/RQ/arq | Reliable, retryable, scalable | You must run Redis and workers |
| Streaming with stages | SSE or WebSocket | Best UX, progress visible | Long connections to manage |
The common combination: streaming for interactive requests, a queue for batches and side effects.
Implementation essentials
1. Task id and state. The user gets an id, not a result. States: queued, running, done, failed - stored in shared storage, never only in worker memory.
2. Idempotency. Queues redeliver (timeouts, killed workers). Every write path needs a dedupe key, usually the task id.
@app.task(bind=True, max_retries=3, acks_late=True)
def process_request(self, task_id: str, user_input: str):
if already_done(task_id):
return get_result(task_id)
result = agent.run(user_input)
save_result(task_id, result)
notify_user(task_id, result)
return result
3. Timeouts and retries. Hard timeout per task, capped retries with backoff, and retry only retryable errors (network, 429) - never bad parameters.
4. Delivery. Push results out: polling endpoint, WebSocket, or a notification channel. Writing to the database without notifying looks like a lost task.
5. Queue priority. Interactive and batch work in one queue means batches block everything. Split queues or set priorities.
Front-end side
Minimum viable: poll every one to two seconds, fetch on completion. Smoother: SSE so the server pushes state and the UI can show "retrieving", "generating". Cap the polling (stop after five minutes) so closed tabs stop generating requests.
Pitfalls
- Writing the result without notifying, so users retry and the action runs twice
- No idempotency, so retries resend email or charge twice
- State only in worker memory, lost on restart
- No queue timeout, so one stuck task blocks the line forever
- Stuffing large objects into queue messages and hurting throughput
- Ignoring token accounting on streaming paths
FAQ
Do small projects need Redis? Not always. One machine, few tasks, restarts acceptable - the framework's own background tasks are enough. Adopt a queue when you need retries, scaling or restart safety.
How do users see results after going async? Give explicit feedback twice - accepted and completed - and keep a task history page. This is the part most implementations skip.
How to set a timeout for unknown durations? Start at two to three times your P95 latency and adjust. A timeout is not necessarily a failure: you can degrade to returning partial results.
How to debug a stuck task? Queue backlog, then worker health, then the per-task log and external call latency. That narrows it quickly.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
