Local AI Document Chat in 2026
AnythingLLM (64,364 stars), Khoj (36,218) and PrivateGPT (57,407) let you chat with your PDFs, notes and code locally - your documents never leave your machine.
💡 What You Will Learn
AnythingLLM (64,364 stars), Khoj (36,218) and PrivateGPT (57,407) let you chat with your PDFs, notes and code locally - your documents never leave your machine.
📜 Table of Contents
The short answer
AnythingLLM (64,364 stars, MIT) is the easiest all-in-one: desktop app, web UI, works with Ollama/OpenAI/anything, and has a built-in document workspace. Khoj (36,218 stars, AGPL-3.0) is a "second brain" - it indexes your notes, PDFs and even your calendar, and answers from them. PrivateGPT (57,407 stars, Apache-2.0) is the developer option: an API layer for RAG on local models with full control.
Fastest start - AnythingLLM desktop
- Download the desktop app (Windows/macOS/Linux) from the GitHub releases.
- Start Ollama (177,825 stars) and pull a model:
ollama pull llama3.1:8b - In AnythingLLM settings, select Ollama as the LLM provider and pick the model.
- Create a workspace, upload PDFs, and start chatting - it builds a vector index automatically.
Comparing the three
| Tool | Stars | Best for | Setup |
|---|---|---|---|
| AnythingLLM | 64,364 | Non-developers, quick start | 10 min |
| Khoj | 36,218 | Personal knowledge base | 20 min |
| PrivateGPT | 57,407 | Developers, custom RAG API | 40 min |
Real numbers
- AnythingLLM embeds documents locally by default (free, private) and supports 10+ vector databases.
- Khoj syncs with Obsidian, Notion, Google Docs and more - chat from your phone too.
- With an 8B model on a 16GB Mac, you get ~20-40 tokens/sec - plenty for document Q&A.
FAQ
Q: Do I need a GPU? A: No - an 8B model runs fine on CPU (slower) or Apple Silicon. Use a 3-4B model for weak laptops.
Q: Which vector database? A: AnythingLLM's built-in LanceDB works out of the box; use Qdrant/Chroma for larger corpora.
Q: Can I use cloud models instead? A: Yes - all three support OpenAI/Anthropic APIs if you prefer cloud, but then documents are sent to the API.
❓ FAQ
Do I need a GPU?
No - an 8B model runs fine on CPU (slower) or Apple Silicon. Use a 3-4B model for weak laptops.
Which vector database?
AnythingLLM's built-in LanceDB works out of the box; use Qdrant/Chroma for larger corpora.
Can I use cloud models instead?
Yes - all three support OpenAI/Anthropic APIs if you prefer cloud, but then documents are sent to the API.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.