Local AI document chat in 2026: a step-by-step AnythingLLM and RAGFlow setup guide, with the four parameters that decide retrieval quality.

AI Document Chat: Talk to Your Files with AnythingLLM and RAGFlow

Document Q&A is the fastest RAG use case to get working: drop in PDFs, ask a question, get an answer with a source. In 2026 the self-hosted options are good enough. AnythingLLM is the quick path, RAGFlow is the deep one.

Pick AnythingLLM first if your files are clean text PDFs. Pick RAGFlow if you have scans, dense tables or two-column layouts. RAGFlow does layout-aware parsing, which is why it can return page-level citations.

AnythingLLM in five steps: install the desktop app or run the Docker image on port 3001 with a mounted storage volume; point the LLM setting at Ollama using host.docker.internal:11434; choose a real embedding model (bge-m3 for Chinese, nomic-embed-text for English); create a workspace, upload files and hit Save and Embed; then check the citation panel on every answer.

RAGFlow runs from docker compose, and the parsing template matters more than any knob: General, Paper, Table, Manual, QA. The wrong template cannot be fixed later with top-k tuning.

Four parameters decide quality: chunk size 512-1024 tokens, overlap 10-20 percent, top-k 4-8, similarity threshold 0.2-0.3, plus reranking on. Set temperature to 0.1-0.3.

When answers are wrong, check in this order: extracted text, chunk boundaries, retrieval hit rate, prompt instruction ("answer only from the provided material"), then model context behaviour.

A ten-minute acceptance test: ask a 100-page annual report for a revenue figure with page number, the top three risk factors, and a verbatim management quote. The three questions test number extraction, summarisation and exact location. Ports, defaults and model lists change between releases, so confirm against official docs.