AI Agent Memory Systems: Vector Stores Explained and Compared (2026)

Your AI Agent forgets everything between conversations. The fix is **vector databases** — and choosing the right one makes the difference between a demo and a product.

## Why Agents Need Vector Databases LLMs have no native memory. You can stuff everything into the context window, but GPT-4o's 128K tokens runs out fast in a deep conversation — and it gets expensive fast. **Vector databases solve three problems:** 1. **Selective memory** — remember what matters, not everything 2. **Semantic retrieval** — find memories by meaning, not keywords 3. **Infinite scale** — context windows cap at ~100K words; vector DBs store millions --- ## How Vector Databases Work (Plain English) **Step 1:** Every memory becomes a mathematical vector (a few hundred numbers) **Step 2:** Similar memories are close together in vector space ("cat care costs" and "pet expenses" are nearby vectors) **Step 3:** When querying, convert the question to a vector and find the nearest memories This is called **Approximate Nearest Neighbor (ANN)** search. No math degree required — think of it as semantic search for memory. --- ## Three Types of Agent Memory ### 1. Conversation Memory Store the last N turns of the current session. ```python class ConversationMemory: def __init__(self, max_turns=20): self.history = [] self.max_turns = max_turns def add(self, role: str, content: str): self.history.append({"role": role, "content": content}) if len(self.history) > self.max_turns * 2: self.history = self.history[-self.max_turns * 2:] ``` **When to use:** Short Q&A, single-task agents. ### 2. Summary Memory When conversations get long, let the LLM summarize periodically. **When to use:** Long conversations without needing precise recall (therapy bots, companion agents). ### 3. Vector Memory The most important type. Each memory is embedded and stored for semantic retrieval. ```python class VectorMemory: def __init__(self, collection_name="agent_memory"): self.client = chromadb.Client() self.collection = self.client.get_or_create_collection(collection_name) def remember(self, text: str, metadata: dict = None): embedding = get_embedding(text) self.collection.add( embeddings=[embedding], documents=[text], metadatas=[metadata or {}], ids=[str(uuid4())] ) def recall(self, query: str, top_k: int = 5): query_vector = get_embedding(query) results = self.collection.query(query_embeddings=[query_vector], n_results=top_k) return results['documents'][0] ``` --- ## 4 Vector Databases Compared Data from GitHub API, July 2026. | Feature | Chroma | Milvus | Qdrant | Weaviate | |:--------|:-------|:-------|:-------|:---------| | **GitHub Stars** | 28.8K | 45.3K | 33.4K | 16.6K | | **Deployment** | Embedded | Server | Docker | Docker | | **Language** | Python | Go/Java | Rust | Go | | **Learning Curve** | Minimal | Steep | Moderate | Steep | | **Memory Usage** | Low | High | Medium | Medium-High | | **Best For** | Prototyping | Billion-scale | Production | Hybrid search | | **Filtering** | Limited | Rich | Rich (best) | Rich (GraphQL) | ### Selection Guide **Chroma — Prototype in 5 minutes** ```bash pip install chromadb ``` Embeds into your Python process. No server needed. Perfect for hacks and MVPs. Not suitable for production. **Milvus — Enterprise scale** Docker-compose with 2+ services. GPU-accelerated. Use for 1M+ vectors. Overkill for small projects. **Qdrant — Production sweet spot** Written in Rust, excellent performance. Best-in-class filtered search: ```python client.search( collection_name="products", query_vector=vector, query_filter=models.Filter( must=[ models.FieldCondition(key="price", range=models.Range(gte=100, lte=500)), ] ), limit=10 ) ``` **Weaviate — Hybrid search specialist** Combines vector search with BM25 full-text search — 70% semantic + 30% keyword. Great for search applications. --- ## Implementation: Adding Memory to Your Agent API Connecting vector memory to the FastAPI Agent from the previous article: ```python class AIAgentWithMemory(AIAgent): def __init__(self): super().__init__() self.short_term = ConversationMemory(max_turns=20) self.long_term = VectorMemory("agent_working_memory") async def chat(self, message: str) -> str: # 1. Retrieve relevant memories memories = self.long_term.recall(message, top_k=3) # 2. Inject into system prompt enhanced_prompt = self.system_prompt + "\n\nRelevant memories:\n" + "\n".join(memories) # 3. Build context messages = [{"role": "system", "content": enhanced_prompt}] messages += self.short_term.get_context() messages.append({"role": "user", "content": message}) # 4. Call LLM response = await self.call_llm(messages) # 5. Store what matters self.long_term.remember(f"User asked: {message} → Response: {response}") self.short_term.add("user", message) self.short_term.add("assistant", response) return response ``` The **three-tier memory architecture**: Conversation memory (recent) → Vector memory (semantic recall) → Injection (context assembly). --- ## Embedding Model Recommendations The vector DB is just storage — **embedding quality determines memory quality**. | Model | Dimensions | Best For | |:------|:-----------|:---------| | text-embedding-3-small | 512/1536 | General, best value | | text-embedding-3-large | 3072 | Precision, bigger budget | | BAAI/bge-m3 | 1024 | Open-source, multilingual | | intfloat/e5-large | 1024 | Open-source, strong English | For Chinese content: **BAAI/bge-m3** (free, self-hosted) or **text-embedding-3-small** (OpenAI, stable quality). --- ## Summary Don't pick the "best" vector database — pick the right one: - First time → Chroma (5-min setup) - Prototype/MVP → Chroma → migrate to Qdrant later - Production, high concurrency → Qdrant - Billion-scale, GPU acceleration → Milvus - Hybrid search needed → Weaviate **Best practice: three-tier memory.** ``` Short-term (list) → Summary (LLM) → Vector (semantic retrieval) ``` Each layer has its job. Together they're cost-effective and powerful. > 💡 **Bookmark this.** Switching vector databases after going to production is painful. Pick wisely from the start. > > 📤 **Share with friends building RAG/Agent systems.** Most people pick a vector DB without understanding the tradeoffs. This saves them from a costly mistake. Previous: Building AI Agent APIs with FastAPI Next: Building Web Scraping Agents with Playwright