Arize Phoenix 2026: Open Source LLM Observability and Evals (10k Stars) - Setup Guide

You traced the problem to production but cannot reproduce it locally. Phoenix captures production traces and turns them into evaluation datasets - closing the loop.

## Arize Phoenix: Observability That Feeds Back Into Your Evals Arize Phoenix (10,850 GitHub stars, Elastic-style license with a permissive core) is the open-source LLM observability platform from Arize AI, the company behind the popular Arize production ML monitoring. Its distinctive idea: production traces are not just for debugging - they are raw material for evaluation. You capture real requests, and with one click turn bad ones into eval cases. ## The Loop: Trace, Detect, Evaluate Phoenix captures spans for every LLM call, retrieval, and tool execution - the same tracing model as Langfuse, but with a stronger evaluation story. When a production response is flagged (by a score, a user report, or a drift alert), you export that trace into an eval dataset, run it against a candidate fix, and verify the fix before redeploying. That loop is the difference between observability as a log viewer and observability as a quality engine. ## Setup ```bash pip install arize-phoenix python -m phoenix.server.main serve # or in-notebook: phoenix.launch_app() ``` Then instrument with the OpenInference auto-instrumentors (one decorator for LangChain, LlamaIndex, or OpenAI calls) and traces appear in the Phoenix UI. It also runs fully in a notebook, which makes it popular for offline eval sessions. ## What You Get - **Traces with embeddings.** Phoenix stores the actual embeddings from traces, so you can cluster and visualize where retrieval goes wrong. - **Evals built in.** LLM-as-judge templates for hallucination, QA correctness, and toxicity - run them on captured traces directly. - **Dataset export.** Bad traces become golden eval examples with one click. - **Local by default.** Data stays on your machine; no cloud required. ## Phoenix vs Langfuse vs Helicone All three do tracing. The split: Langfuse has the richest product polish and prompt management; Helicone is the lightest and cheapest to run at scale (great for cost per trace); Phoenix has the deepest eval-to-production loop and the best notebook workflow. Many teams run Phoenix during development and one of the others in production - or Phoenix everywhere for simplicity. ## FAQ **Is Phoenix free?** The core is open source and self-hostable; Arize sells a hosted enterprise tier. **Does it work with any model?** Yes - it instruments the framework layer, so OpenAI, Anthropic, and local models all trace. **Can it handle production scale?** Yes - it supports Postgres-backed persistence and spans millions of traces. **Do I need to change my code much?** No - one decorator or auto-instrumentor call per framework.