Semantic Cache: 10x Faster AI Responses

80% of the questions users ask are repetitive — "What's your return and exchange policy?" "Which payment methods do you support?" Every time, we have to call the LLM, which is both slow and expensive. Semantic caching can solve this problem.

💡 What You Will Learn

80% of the questions users ask are repetitive — "What's your return and exchange policy?" "Which payment methods do you support?" Every time, we have to call the LLM, which is both slow and expensive.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
cache = {}  # →

def semantic_cache(question, threshold=0.85):
    # 
    q_vec = model.encode(question, normalize_embeddings=True)

    # Cache
    for cached_q, cached_vec in cache.items():
        similarity = np.dot(q_vec, cached_vec)
        if similarity > threshold:
            return cache[cached_q]["answer"]

    # LLM
    answer = call_llm(question)

    # Cache
    cache[question] = {"vector": q_vec, "answer": answer}
    return answer