AI Model Deployment: From Dev to Production

AI Model Deployment: From Dev to Production

💡 What You Will Learn

AI Model Deployment: From Dev to Production

ollama run qwen2.5:7b
# Auto/Automatic localhost:11434  OpenAI  API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b","messages":[{"role":"user","content":""}]}'

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

FROM vllm/vllm-openai:latest

|:----|:-------:|:-------:|:---:|:--------| | Ollama || ~500 t/s ||| | vLLM || ~2400 t/s |||