AI Model Quantization: How to Run Large Models on a Single GPU

AI Model Quantization: How to Run Large Models on a Single GPU

💡 What You Will Learn

AI Model Quantization: How to Run Large Models on a Single GPU

📜 Table of Contents

• ResultsComparison
• Summary

|:----|:----:|:-----------:|:---------| | Q2_K || 3GB || | Q4_K_M || 4.5GB || | Q5_K_M || 5.5GB || | Q8_0 || 7GB ||

pip install auto-gptq
from auto_gptq import AutoGPTQForCausalLM

pip install awq

# 1. llama.cpp
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
make

# 2. FP16
# HuggingFace

# 3. Q4_K_M
./quantize ./models/original-model.gguf ./models/q4-model.gguf Q4_K_M

ResultsComparison

|:----|:--------:|:----------:|

Summary