Building a RAG chatbot is less about connecting an LLM to a vector database and more about creating a dependable information pipeline. This guide explains the core architecture, shows how to estimate retrieval quality, latency, and operating cost with repeatable inputs, and provides a practical evaluation checklist you can revisit as documents, traffic, and model pricing change.
Overview
A retrieval-augmented generation, or RAG, chatbot combines two systems: a retrieval layer that finds relevant content and a generation layer that uses that content to produce an answer. The retrieved passages may come from product documentation, internal policies, support articles, contracts, or another controlled knowledge base.
The basic request flow is:
- A user submits a question.
- The application rewrites or embeds the question.
- A retriever searches indexed document chunks.
- The application filters and formats the best results.
- An LLM receives the question, retrieved context, and response instructions.
- The chatbot returns an answer, ideally with citations or links to the source material.
This architecture can make a knowledge base chatbot more useful than a general-purpose model for domain-specific questions, but retrieval does not automatically make answers accurate. If the source documents are incomplete, outdated, poorly chunked, or incorrectly retrieved, the model may still produce an unhelpful response. Treat the system as a chain of measurable stages rather than a single AI feature.
A practical RAG chatbot architecture usually includes document storage, an ingestion pipeline, text extraction, chunking, metadata management, an embedding service, a vector database or hybrid search index, an orchestration layer, an LLM API, observability tools, and access controls. For a broader deployment view, see the RAG chatbot architecture and deployment guide.
How to estimate
Before selecting infrastructure, define what “good” means for the first release. A useful estimate separates four outcomes:
- Retrieval quality: Does the system find the passages needed to answer the question?
- Answer quality: Is the final response supported, complete, and appropriately cautious?
- Responsiveness: How long does a user wait for the first token and the complete answer?
- Operating cost: What does one conversation and one month of expected usage consume?
Use a small test set before making production assumptions. Create representative questions from real support tickets or approved test scenarios. Label each question with the expected source document, the key facts required, and whether the correct behavior is to answer, ask for clarification, or decline because the knowledge base lacks sufficient evidence.
For a simple cost estimate, calculate monthly usage in stages:
Monthly requests = active users × requests per user per month
Monthly model cost = input tokens ÷ token unit × input rate + output tokens ÷ token unit × output rate
Then add embedding requests, vector search, application hosting, document storage, monitoring, and any optional reranking or speech services. Keep each item separate. A combined monthly total is useful for budgeting, but separate line items show which assumption needs attention when usage changes. A cost calculator can help organize these variables; see the chatbot cost calculator guide and the chatbot hosting cost calculator.
Estimate latency as a sequence rather than a single guess:
Total response time = query processing + embedding + retrieval + reranking + prompt assembly + model generation + network overhead
Measure each stage in a test environment, then repeat under realistic concurrency. A faster model will not fix a slow document query, oversized prompt, or overloaded application server.
Inputs and assumptions
Source content
Inventory the documents before building the index. Record format, language, owner, revision date, access level, and expected update frequency. Remove duplicates and decide how the system handles archived material. A RAG chatbot should not treat every file as equally authoritative; metadata can help prioritize approved sources and expose citations.
Chunking
Chunking determines what the retriever can return. Chunks that are too small may lose the surrounding explanation; chunks that are too large may contain unrelated material and consume more context. Start with boundaries that respect headings, paragraphs, list items, and tables where possible. Store document title, section heading, URL, revision identifier, and permissions alongside each chunk.
Test more than one chunking strategy using the same evaluation questions. Change one variable at a time so that improvements can be attributed to chunk size, overlap, metadata filtering, or another retrieval setting.
Retrieval
Choose between semantic search, keyword search, or a hybrid approach based on the question types in your dataset. Semantic search can help with paraphrased questions, while keyword signals may matter for product codes, error messages, legal terms, and exact names. Configure the number of initial results, then consider a reranking step if the first-pass results contain several plausible matches.
Prompt assembly
Keep the prompt explicit and bounded. Tell the model to use the supplied context, distinguish evidence from inference, say when the context is insufficient, and preserve important qualifications. Include citation instructions that map each claim to a source identifier. Do not place unrestricted user text in a position where it can silently override system instructions; treat retrieved documents and user messages as untrusted input.
Security and operations
Apply document permissions during retrieval, not only in the user interface. Log request identifiers, retrieval results, model configuration, latency, token usage, and refusal or fallback behavior while avoiding unnecessary storage of sensitive conversation content. Add limits for file size, request frequency, context length, and repeated failures. These controls matter whether the bot is hosted on a managed platform or deployed with cloud services from a major provider.
Worked examples
Example 1: Estimating monthly usage
Assume a pilot has 400 active users, with each user sending 12 messages per month. The forecast is:
400 × 12 = 4,800 monthly requests
For a planning model, assign an average input size, retrieved-context size, and output size to each request. Multiply each token category by 4,800, then apply the selected provider’s current rates. Add a separate allowance for failed requests, evaluation runs, indexing new documents, and administrative testing. Do not hide these activities inside the end-user average; they often change during rollout.
If the chatbot uses conversation history, estimate two scenarios: a short-history design and a longer-history design. The second may improve continuity in some conversations but can increase input tokens and latency. Use the comparison to decide whether a summary, session memory, or explicit user confirmation is more appropriate.
Example 2: Evaluating retrieval quality
Suppose a test set contains 50 questions. For each question, mark whether at least one retrieved chunk contains the required evidence. If 42 questions meet that condition, the simple retrieval coverage estimate is:
42 ÷ 50 = 84%
This is not a complete accuracy score. The model may still misunderstand the passage, omit a qualification, or cite the wrong source. For the same 50 questions, separately review answer correctness, citation validity, refusal behavior, and whether the response contains unsupported claims. A spreadsheet with one row per question is enough to begin; include the query, expected source, retrieved sources, answer rating, failure category, and remediation.
Example 3: Comparing architecture choices
Compare two designs using identical questions and traffic assumptions. Design A uses vector search and a single generation call. Design B uses hybrid retrieval, reranking, and a larger context window. Design B may improve evidence selection, but it also adds processing steps and potentially higher token usage. Select it only if the evaluation set shows a meaningful reduction in the failures that matter to the business, such as incorrect policy guidance or missing troubleshooting steps.
When to recalculate
Revisit the estimate whenever an input changes, not only when the chatbot appears to fail. Recalculate after:
- Adding a new document collection, language, or access group.
- Changing chunk size, overlap, embedding model, retrieval count, or reranking.
- Changing the LLM, prompt, output limit, conversation-memory policy, or citation format.
- Moving from a pilot to broader traffic or adding a website, messaging channel, or voice interface.
- Receiving evidence of unsupported answers, missing sources, or permission failures.
- Seeing changes in provider pricing, rate limits, model availability, or infrastructure utilization.
Keep a versioned evaluation set and run it before each substantial release. Review a sample of production conversations as well, because a curated test set can miss new user phrasing and unexpected retrieval behavior. When a result changes, identify whether the cause is the documents, retrieval, prompt assembly, model response, or application logic.
A practical launch checklist is to verify document ownership, permissions, ingestion logs, source citations, fallback responses, rate limits, monitoring, cost alerts, and rollback steps. Start with a narrow knowledge domain, measure it carefully, and expand only when the evidence supports expansion. That approach produces a more maintainable RAG chatbot than adding model capability without improving the underlying data pipeline.
