Arize Phoenix 2026: Open Source LLM Observability and Evals (10k Stars) - Setup Guide

You traced the problem to production but cannot reproduce it locally. Phoenix captures production traces and turns them into evaluation datasets - closing the loop.

💡 What You Will Learn

You traced the problem to production but cannot reproduce it locally. Phoenix captures production traces and turns them into evaluation datasets - closing the loop.

📜 Table of Contents

• Arize Phoenix: Observability That Feeds Back Into Your Evals
• The Loop: Trace, Detect, Evaluate
• Setup
• What You Get
• Phoenix vs Langfuse vs Helicone
• FAQ

Arize Phoenix: Observability That Feeds Back Into Your Evals

Arize Phoenix (10,850 GitHub stars, Elastic-style license with a permissive core) is the open-source LLM observability platform from Arize AI, the company behind the popular Arize production ML monitoring. Its distinctive idea: production traces are not just for debugging - they are raw material for evaluation. You capture real requests, and with one click turn bad ones into eval cases.

The Loop: Trace, Detect, Evaluate

Phoenix captures spans for every LLM call, retrieval, and tool execution - the same tracing model as Langfuse, but with a stronger evaluation story. When a production response is flagged (by a score, a user report, or a drift alert), you export that trace into an eval dataset, run it against a candidate fix, and verify the fix before redeploying. That loop is the difference between observability as a log viewer and observability as a quality engine.

Setup

pip install arize-phoenix
python -m phoenix.server.main serve
# or in-notebook: phoenix.launch_app()

Then instrument with the OpenInference auto-instrumentors (one decorator for LangChain, LlamaIndex, or OpenAI calls) and traces appear in the Phoenix UI. It also runs fully in a notebook, which makes it popular for offline eval sessions.

What You Get

• Traces with embeddings. Phoenix stores the actual embeddings from traces, so you can cluster and visualize where retrieval goes wrong.
• Evals built in. LLM-as-judge templates for hallucination, QA correctness, and toxicity - run them on captured traces directly.
• Dataset export. Bad traces become golden eval examples with one click.
• Local by default. Data stays on your machine; no cloud required.

Phoenix vs Langfuse vs Helicone

All three do tracing. The split: Langfuse has the richest product polish and prompt management; Helicone is the lightest and cheapest to run at scale (great for cost per trace); Phoenix has the deepest eval-to-production loop and the best notebook workflow. Many teams run Phoenix during development and one of the others in production - or Phoenix everywhere for simplicity.

FAQ

Is Phoenix free? The core is open source and self-hostable; Arize sells a hosted enterprise tier.

Does it work with any model? Yes - it instruments the framework layer, so OpenAI, Anthropic, and local models all trace.

Can it handle production scale? Yes - it supports Postgres-backed persistence and spans millions of traces.

Do I need to change my code much? No - one decorator or auto-instrumentor call per framework.

❓ FAQ

Is Phoenix free?

The core is open source and self-hostable; Arize sells a hosted enterprise tier.

Does it work with any model?

Yes - it instruments the framework layer, so OpenAI, Anthropic, and local models all trace.

Can it handle production scale?

Yes - it supports Postgres-backed persistence and spans millions of traces.

Do I need to change my code much?

No - one decorator or auto-instrumentor call per framework.