FastChat (39,515 Stars) 2026

FastChat (39,515 stars) is the open platform for training, serving and evaluating LLMs - including the Vicuna models and a high-performance serving engine. Here is the full guide.

💡 What You Will Learn

FastChat (39,515 stars) is the open platform for training, serving and evaluating LLMs - including the Vicuna models and a high-performance serving engine. Here is the full guide.

📜 Table of Contents

• The short answer
• What you get
• Serve a model with the OpenAI-compatible API
• Multi-model serving with workers
• Practical tips
• FAQ

The short answer

lm-sys/FastChat (39,515 stars, Python) is an open platform from LMSYS (the team behind Chatbot Arena) for training, serving, and evaluating large language models. It includes the Vicuna model family, an OpenAI-compatible serving engine, and evaluation tools - all in one codebase.

What you get

• Models: Vicuna and other fine-tuned chat models
• Serving: high-throughput inference with the FastChat server
• Evaluation: pipeline for comparing model outputs (like arena-style)
• Training: scripts and data for fine-tuning chat models
• Web UI: a ChatGPT-like interface for demos

Serve a model with the OpenAI-compatible API

pip install fschat

# Serve with the CLI
python -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

# Or start the API server (OpenAI-compatible)
python -m fastchat.serve.openai_api_server \n  --model-path lmsys/vicuna-7b-v1.5 --port 8000

Then use any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="vicuna-7b-v1.5",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Multi-model serving with workers

For production, run one controller + multiple model workers to serve several models behind one API endpoint, with load balancing.

Practical tips

• Use the controller/worker architecture for serving multiple models at scale.
• Use the evaluation pipeline to compare fine-tunes before deployment.
• Vicuna checkpoints require the base Llama weights for full functionality.

FAQ

Is it free? Yes - Apache-2.0 open source (model licenses vary).

Who maintains it? LMSYS Org - the team behind Chatbot Arena (lmarena.ai).

Can I use it with GPUs? Yes - it is optimized for GPU inference and supports batching for throughput.

❓ FAQ

Is it free?

Yes - Apache-2.0 open source (model licenses vary).

Who maintains it?

LMSYS Org - the team behind Chatbot Arena (lmarena.ai).

Can I use it with GPUs?

Yes - it is optimized for GPU inference and supports batching for throughput.