How I Built a RAG Pipeline That Actually Works in Production

Most tutorials teach you the happy path. Production teaches you everything else.Every RAG tutorial follows the same script: chunk some documents, embed them, throw them into a vector database, retrieve the top 5, and feed them to an LLM. Done. Ship it.Then you deploy it and everything falls apart.Users ask questions that span multiple documents. The retriever pulls back irrelevant chunks. The LLM hallucinates confidently. Your CEO asks why the AI just told a customer their refund policy is "flexible and negotiable."I've built RAG systems that serve 50,000+ documents in production. Here's what I learned the hard way.

Chunking is where most pipelines silently fail

The default advice is "chunk by 512 tokens with 50-token overlap." That's fine for blog posts. It's terrible for everything else.Here's what actually matters:
- Semantic chunking over fixed-size chunking. I use a two-pass approach. First pass: split by natural document boundaries (headings, sections, paragraph breaks). Second pass: merge small chunks that share semantic context and split oversized ones at sentence boundaries.Preserve document structure. Every chunk carries metadata: the document title, section heading hierarchy, page number, and the chunk's position in the document. When the LLM gets a chunk from page 47 of a technical manual, it needs to know it's reading "Section 4.2: Error Handling" under "Chapter 4: API Reference."Overlapping context windows. I don't just overlap tokens. I prepend a one-sentence summary of the previous chunk. This costs a few extra tokens but dramatically reduces the "torn context" problem where a chunk starts mid-thought.For one client's legal document system, switching from fixed 512-token chunks to semantic chunking with structural metadata improved answer accuracy from 71% to 89% overnight.

Retrieval is not just cosine similarity

Vector similarity gets you candidates. It doesn't get you answers.My production pipeline uses a three-stage retrieval process:
- Stage 1: Hybrid search. I combine dense vector search (embeddings) with sparse keyword search (BM25). Some queries are best served by semantic similarity. Others need exact keyword matching. A query like "what's our policy on HIPAA compliance" needs the keyword "HIPAA" to anchor the search, not just semantic similarity to "health regulations."I weight them dynamically: if the query contains domain-specific terms or acronyms, I boost the BM25 score. For conversational queries, I lean on the vector search.Stage 2: Re-ranking. The initial retrieval pulls back 20-30 candidates. A cross-encoder re-ranker (I use a fine-tuned model based on `ms-marco-MiniLM`) scores each candidate against the original query with full attention. This is computationally heavier than cosine similarity, but it catches the nuance that embedding distance misses.Stage 3: Contextual filtering. Not every high-scoring chunk belongs in the final context. I filter based on document recency (for time-sensitive domains), user permissions (some documents are restricted), and redundancy (if three chunks say the same thing, I keep the most comprehensive one).This three-stage approach added about 200ms of latency. It cut hallucination rates by over 60%.

Hallucination detection is not optional

You cannot rely on the LLM to tell you when it's making things up. You need structural guardrails.Citation enforcement. Every claim in the response must reference a specific chunk. I use a structured output format where the LLM generates `[source: chunk_id]` inline. In post-processing, I verify each citation actually supports the claim. If a citation doesn't match, the claim gets flagged.Confidence scoring. I run a separate, smaller model that scores how well the retrieved context supports the generated answer. If the support score drops below a threshold, the system responds with "I found related information but I'm not confident in a specific answer" instead of guessing.Contradiction detection. When multiple source documents disagree, the system flags the contradiction explicitly rather than picking one. "Document A from 2024 states X, but Document B from 2025 states Y. The more recent document may reflect updated policy."Abstention. This is the hardest one culturally. Stakeholders want the AI to always answer. But a system that says "I don't know" when it genuinely doesn't know is infinitely more trustworthy than one that fabricates answers 5% of the time. I set a retrieval relevance threshold: if no chunk scores above 0.7 relevance to the query, the system abstains.

The embedding model matters more than the LLM

Teams spend weeks evaluating GPT-4 vs Claude vs Gemini for generation. They spend zero time evaluating their embedding model.This is backwards.A mediocre LLM with excellent retrieval will outperform an excellent LLM with mediocre retrieval every single time. If the right context lands in the prompt, even a smaller model generates accurate answers. If the wrong context lands in the prompt, no model can save you.I evaluate embedding models on my actual data, not on MTEB benchmarks. I create a test set of 200 query-document pairs from real user questions, then measure recall at different k values. The model that scores highest on public benchmarks is often not the best for domain-specific data.For technical documentation, I've had better results with `e5-large-v2` than with OpenAI's `text-embedding-3-large`, despite the latter scoring higher on general benchmarks. Domain matters.

Evaluation is a system, not a one-time test

I run evaluation continuously, not just before launch.Automated regression tests. A suite of 300 question-answer pairs with expected source documents. Every pipeline change gets tested against this suite. If recall drops by more than 2%, the change doesn't ship.User feedback loops. Every response includes a thumbs up/down. Thumbs-down responses get logged with the full retrieval trace: what was retrieved, what was re-ranked, what was generated. I review these weekly and add the worst failures to the regression suite.Drift detection. As new documents get added, embedding distributions shift. I monitor the average similarity scores over time. If they start dropping, it usually means the new documents introduce vocabulary or concepts that the embedding model handles poorly, and it's time to consider fine-tuning.

The architecture that actually works

Here's the stack I use in production:
- - Document ingestion: Custom pipeline with Apache Tika for parsing, plus format-specific extractors for PDFs with tables.
- - Chunking: Semantic chunker I built that respects document structure. Open-sourcing this soon.
- - Embeddings: Self-hosted e5-large-v2 on a GPU instance. Batched processing with a queue.
- - Vector store: Qdrant. I evaluated Pinecone, Weaviate, and Milvus. Qdrant won on filtering performance and self-hosting simplicity.
- - BM25 index: Elasticsearch, running alongside the vector store.
- - Re-ranker: Cross-encoder model served via a lightweight FastAPI service.
- - Orchestration: LangChain for the initial prototype, then replaced with custom Python code. LangChain's abstractions help you start fast but fight you when you need fine-grained control.
- - Generation: Claude API with structured outputs for citation enforcement.
- - Monitoring: Custom dashboard tracking retrieval relevance, hallucination rates, and user satisfaction.

What I'd do differently

If I were starting over, I'd invest in evaluation infrastructure from day one. I spent the first month building features and the second month debugging them because I had no systematic way to measure quality.I'd also skip the "let's try the simplest thing first" approach for chunking. Fixed-size chunking is not simpler. It's just faster to implement. You pay for it later in debugging sessions that take longer than building semantic chunking would have.RAG is not an AI problem. It's an information retrieval problem with an AI layer on top. Treat it that way and you'll build systems that actually work.