LLM RAG Observability 2026: Track Tokens

Your RAG app works in testing and misbehaves in production - slow answers, wrong citations, silent failures. LLM observability is the missing layer, and the best tools are open source.

📜 Table of Contents

• Why RAG Apps Fail Silently
• What to Track for RAG
• Langfuse: Traces With Cost
• Arize Phoenix: Evals Built In
• The 2026 Baseline Stack

Why RAG Apps Fail Silently

A RAG pipeline has three failure points a normal APM never sees: retrieval (did the right chunks come back?), generation (did the model actually use them?) and cost (how many tokens did one answer burn?). In 2026 the open source answer is a tool class called LLM observability, and the two names you will meet are Langfuse (32,973 stars) and Arize Phoenix (11,015 stars), both fetched 2026-08-13.

What to Track for RAG

Retrieval metrics: how many documents were returned, their similarity scores, and whether the answer cited the top-1 chunk or something in position 7. A low-position citation usually means bad chunking or a weak retriever.

Generation metrics: input/output tokens per request, time-to-first-token and total latency. When a query answers in 1 second today and 9 seconds after your next model swap, observability tells you which stage regressed.

Quality signals: with RAGAS or manual tracing, you can score faithfulness (did the answer stick to retrieved context?) and answer relevancy per trace. An answer that ignored the retrieved docs is a pipeline bug, not a model quirk.

Langfuse: Traces With Cost

Langfuse instruments LangChain, LlamaIndex and raw OpenAI calls with a few lines. Every trace shows the prompt, the retrieved context, the completion and the token cost. It can run fully self-hosted (Docker Compose) and supports OpenTelemetry, so it slots into existing dashboards.

Arize Phoenix: Evals Built In

Phoenix goes deeper on evaluation: it can run LLM-as-judge evals over your traces, detect drift between your dev and production distributions, and give embeddings a visual UMAP view so you can spot retrieval clusters going wrong. Heavier to set up, but the eval-first workflow pays off for teams that ship weekly.

The 2026 Baseline Stack

1. Instrument the app with OpenTelemetry (vendor-neutral spans).
2. Ship traces to Langfuse for cost and latency dashboards.
3. Run a nightly RAGAS eval on 100 sampled queries.
4. Alert when faithfulness drops below your threshold (start at 0.8).

Start with Langfuse and add Phoenix when you need systematic evals. Both are open source, so the data stays on your server.