How the three layers of LLM evaluation fit together in 2026: eval frameworks, tracing platforms and LLM-as-judge, plus the five metrics to start with.

LLM Evaluation Platforms 2026: Langfuse, Arize Phoenix and Open-Source Evals

The hard moment after launch is changing one prompt and being unable to say whether things got better. LLM apps have no deterministic unit tests, so evaluation has to be layered.

Three layers: eval frameworks define and run metrics (RAGAS, promptfoo, DeepEval); tracing platforms capture production traces, cost and sampled scoring (Langfuse, Arize Phoenix, Helicone); LLM-as-judge scores answers when there is no ground truth.

RAGAS covers the four RAG metrics: faithfulness, answer relevance, context precision, context recall. promptfoo is declarative YAML with assertions and belongs in CI so every prompt change runs a regression. DeepEval is pytest-style with built-in metric templates and the lowest migration cost for Python teams.

Langfuse is the usual starting point: full trace of prompt, retrieved chunks, model and tool calls, latency and cost, plus datasets so you can rerun a changed prompt against real traces. Phoenix is stronger for visual debugging of RAG and embedding drift. Helicone is a gateway: change the base URL and you get call, cost and cache visibility immediately.

For judging: make criteria specific ("cites a number from the source and the number matches"), avoid using the same model family as judge and defendant, write anchors per score band, and hand-check 5-10 percent of samples.

Start with five metrics: correctness, faithfulness, refusal rate, P95 latency and cost per successful answer. Then build a golden set of 50-200 real questions, run it in CI, trace everything in production, and judge 1-5 percent of live traffic with the same criteria so offline and online numbers stay comparable.