TensorRT-LLM: NVIDIA Official Inference Optimization Guide

TensorRT-LLM: NVIDIA Official Inference Optimization Guide

💡 What You Will Learn

TensorRT-LLM: NVIDIA Official Inference Optimization Guide

|:--------|:----------------|:-----------|:--------| || 680 | 4250 | 28GB | | vLLM | 1520 | 1850 | 22GB | | TensorRT-LLM FP16 | 1890 | 1420 | 18GB | | TensorRT-LLM FP8 | 2350 | 1080 | 14GB |

# : 
docker run --gpus all -it --rm   -v $(pwd):/workspace   nvidia/cuda:12.4.0-devel-ubuntu22.04   bash

# : Installation/SetupTensorRT-LLM
pip install tensorrt_llm -U

# : (HFFT)
python3 examples/convert_checkpoint.py   --model_dir ./qwen3-7b-instruct   --output_dir ./converted   --dtype bfloat16

# : TensorRT
trtllm-build   --checkpoint_dir ./converted   --output_dir ./trt-engine   --max_batch_size 128   --max_input_len 8192   --max_output_len 2048   --gemm_plugin float16

# : Start
python3 examples/run.py   --engine_dir ./trt-engine   --tokenizer_dir ./qwen3-7b-instruct   --max_output_len 1024

# FP8
python3 examples/quantization/quantize.py   --model_dir ./qwen3-7b-instruct   --dtype float16   --qformat fp8   --output_dir ./trt-fp8-engine   --calib_size 512