用MLflow做LLMOps：追踪提示词、评估模型、上线生产

2026-08-02 约 5 分钟阅读

三个提示词版本加两个模型，你已经记不清上周的好结果来自哪组组合。

LLMOps: What MLflow Does for LLM Applications

MLflow started as a machine-learning experiment tracker and, since the 2.x line, has become the most-used open-source platform for the whole LLM lifecycle. It has 27,318 GitHub stars (August 2026) and an Apache-2.0 license, and it solves one concrete problem: making every prompt-model-data combination reproducible.

The Four Pieces You Will Actually Use

1. Experiment tracking. Every run logs the prompt template, model ID, temperature, and evaluation metrics. Three weeks later you can compare gpt-4o with a condensed prompt against claude with a verbose prompt in a few clicks.

2. Prompt management. Prompts live in MLflow's registry as versioned artifacts. When a prompt changes, you get a new version with a diff - no more prompt_v7_final_REAL files.

3. Model evaluation. MLflow has built-in LLM-as-judge evaluation: you provide a test dataset and it scores correctness, toxicity, and answer relevance across model versions. This turns I think it got better into a number.

4. The Model Registry. Approved prompts and models get promoted through staging to production. Deployments read from the registry, so rollback is changing a tag.

A Realistic First Step

import mlflow

with mlflow.start_run():
    mlflow.log_param("prompt_version", "v3")
    mlflow.log_param("model", "gpt-4o")
    mlflow.log_metrics(evaluate_on_testset(prompt_v3))

Do this for one week on one app. The habit of logging every experiment is worth more than any single MLflow feature.

Where It Fits vs Langfuse / Phoenix

MLflow is the heavyweight lifecycle platform (good if you already run MLflow for classical ML). Langfuse and Arize Phoenix are lighter-weight, LLM-specialized observability tools with deeper tracing. The pragmatic pattern in 2026: MLflow for experiments and the registry, a tracer for production debugging.