AI Agent Semantic Cache 2026

"Beijing's weather" and "What's the temperature in Beijing today" — asked 100 times, hitting the API 100 times. Semantic caching lets similar questions share the same answer.

💡 What You Will Learn

"Beijing's weather" and "What's the temperature in Beijing today" — asked 100 times, hitting the API 100 times. Semantic caching lets similar questions share the same answer.

📜 Table of Contents

• Results
• Summary

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("BAAI/bge-small-zh-v1.5")
cache = {}

def semantic_get(query, threshold=0.92):
    q_vec = model.encode(query)
    for cached_q, response in cache.items():
        c_vec = model.encode(cached_q)
        sim = np.dot(q_vec, c_vec) / (np.linalg.norm(q_vec) * np.linalg.norm(c_vec))
        if sim > threshold:
            return response
    return None

Results

|:--------|:-----:|:----------:| || 0% | 0% | || 15-20% | 15-20% | || 40-60% | 40-60% |

Summary