vLLM Deployment Guide: 10x Faster Model Inference

Running large model inference with native transformers gives only 20 tokens per second, and a single API call takes 10 seconds to wait. vLLM says: I can make you 10x faster.

💡 What You Will Learn

Running large model inference with native transformers gives only 20 tokens per second, and a single API call takes 10 seconds to wait. vLLM says: I can make you 10x faster.

vLLMWhat Is

pip install vllm

from vllm import LLM, SamplingParams

# 
llm = LLM(model="Qwen/Qwen2.5-7B-Instruct")

# Parameter
params = SamplingParams(temperature=0.7, max_tokens=512)

# 
outputs = llm.generate([""], params)
print(outputs[0].outputs[0].text)

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --port 8000

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": ""}]
)

# vLLMAuto/AutomaticBatch Processing
from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as executor:
    futures = [executor.submit(llm.generate, [prompt], params) 
               for prompt in prompts]

# AWQ
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-AWQ \
    --quantization awq

# Tensor Parallelism
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-72B-Instruct \
    --tensor-parallel-size 4  # 4

|:----|:---:|:----| | 7B | 16GB | RTX 4060 Ti 16GB | | 13B | 24GB | RTX 4090 | | 70B | 4x24GB ||