LLMOps with MLflow: Track Prompts, Evaluate Models, and Ship to Production (27k Stars)

You are running three prompt versions and two models and you cannot remember which combination produced the good results last week.

💡 What You Will Learn

You are running three prompt versions and two models and you cannot remember which combination produced the good results last week.

📜 Table of Contents

• LLMOps: What MLflow Does for LLM Applications
• The Four Pieces You Will Actually Use
• A Realistic First Step
• Where It Fits vs Langfuse / Phoenix
• FAQ

LLMOps: What MLflow Does for LLM Applications

MLflow started as a machine-learning experiment tracker and, since the 2.x line, has become the most-used open-source platform for the whole LLM lifecycle. It has 27,318 GitHub stars (August 2026) and an Apache-2.0 license, and it solves one concrete problem: making every prompt-model-data combination reproducible.

The Four Pieces You Will Actually Use

1. Experiment tracking. Every run logs the prompt template, model ID, temperature, and evaluation metrics. Three weeks later you can compare gpt-4o with a condensed prompt against claude with a verbose prompt in a few clicks.

2. Prompt management. Prompts live in MLflow's registry as versioned artifacts. When a prompt changes, you get a new version with a diff - no more prompt_v7_final_REAL files.

3. Model evaluation. MLflow has built-in LLM-as-judge evaluation: you provide a test dataset and it scores correctness, toxicity, and answer relevance across model versions. This turns I think it got better into a number.

4. The Model Registry. Approved prompts and models get promoted through staging to production. Deployments read from the registry, so rollback is changing a tag.

A Realistic First Step

import mlflow

with mlflow.start_run():
    mlflow.log_param("prompt_version", "v3")
    mlflow.log_param("model", "gpt-4o")
    mlflow.log_metrics(evaluate_on_testset(prompt_v3))

Do this for one week on one app. The habit of logging every experiment is worth more than any single MLflow feature.

Where It Fits vs Langfuse / Phoenix

MLflow is the heavyweight lifecycle platform (good if you already run MLflow for classical ML). Langfuse and Arize Phoenix are lighter-weight, LLM-specialized observability tools with deeper tracing. The pragmatic pattern in 2026: MLflow for experiments and the registry, a tracer for production debugging.

FAQ

Is MLflow free? Yes, open source. Databricks offers a hosted version.

Can it run on one server? Yes, the tracking server runs on a single box easily.

Does it support local LLMs? Yes, any model can be registered, and local endpoints work as evaluation targets.

Is it overkill for one app? For one prompt, yes. For teams running multiple apps, it pays for itself in a week.

❓ FAQ

Is MLflow free?

Yes, open source. Databricks offers a hosted version.

Can it run on one server?

Yes, the tracking server runs on a single box easily.

Does it support local LLMs?

Yes, any model can be registered, and local endpoints work as evaluation targets.

Is it overkill for one app?

For one prompt, yes. For teams running multiple apps, it pays for itself in a week.