[{"content":" I help engineering teams get LLM inference costs under control — through measurement, routing, and serving efficiency, without giving up reliability.\nPlatform engineer, 12 years. Tech Lead Manager @ Kittl. Based in Dubai.\nRead the blog About ","date":null,"permalink":"https://vbhargava.org/","section":"","summary":"","title":""},{"content":"Writing on the infrastructure layer of LLM systems — inference cost, serving, retrieval, evals, and observability. Mostly what I\u0026rsquo;ve measured, and what surprised me.\n","date":null,"permalink":"https://vbhargava.org/writing/","section":"Blog","summary":"","title":"Blog"},{"content":"","date":null,"permalink":"https://vbhargava.org/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/evals/","section":"Tags","summary":"","title":"Evals"},{"content":"I added hybrid search and a cross-encoder reranker to my RAG service, measured it, and wrote up four findings. Then I ran the one comparison I\u0026rsquo;d been putting off, and three of the four didn\u0026rsquo;t survive.\nThe pipeline I\u0026rsquo;d spent weeks on had contributed nothing measurable. Every gain, every regression, and the entire cost reduction belonged to a second variable — one I hadn\u0026rsquo;t chosen so much as inherited, because adding the reranker is what forced it.\nThat second part is why I\u0026rsquo;m writing this up rather than quietly reverting.\nNote: This carries on from the first post, but you don\u0026rsquo;t need to have read it — I explain the terms as they come up. That post was a walkthrough of the smallest RAG system that still behaves like a real service. This one is less about the pipeline and more about the harder question underneath it: how do you actually know whether a change you made helped?\nWhere this picks up #The last post was January. It\u0026rsquo;s August.\nSome of that gap is honest and some of it isn\u0026rsquo;t. I moved from Berlin to Dubai, and the tail of a move is longer than the move — residency, a driving licence, a one-year-old adjusting to a new flat, a stretch of health stuff I hadn\u0026rsquo;t planned for. I\u0026rsquo;d built the whole of v1 in about twenty days and drew exactly the wrong conclusion from that: that this was my speed. It wasn\u0026rsquo;t. It was a burst.\nWhat actually happened is that the building never stopped, it just slowed to a few hours on a good week — thirty-odd commits since February. The publishing stopped. Those are different failures and I\u0026rsquo;d rather be honest about which one I committed.\nThe project also got renamed. llm_lab was accurate when it was a lab; it stopped being accurate somewhere around the point it grew a regression gate. It\u0026rsquo;s ritam now.\nThe system, and how to read its numbers #The first post described a service that answers questions from a set of documents, and the shape hasn\u0026rsquo;t changed: documents get cut into chunks, each chunk becomes an embeddingA list of numbers representing a piece of text, produced by a model, positioned so that text with similar meaning lands nearby. Comparing embeddings is how the system finds relevant chunks without matching words literally. in a vector database (Qdrant, now, rather than a JSON file), and a question retrieves the nearest ones, which go into the promptThe full block of text sent to the model: the question, the retrieved chunks, and the instruction to answer only from those chunks. Everything in it is billed as input. for the model to answer from.\nWhat\u0026rsquo;s new since v1 is that I can now tell whether a change to that pipeline helped — a harder problem, it turns out, than building the pipeline was. It\u0026rsquo;s a 78-question eval set over a deliberately fictional corpusMade-up facts, on purpose: seven documents about a duck civil war. On real-world topics the model can answer from what it absorbed in pretraining, so a correct answer tells you nothing about whether your retrieval worked. Fiction forces every correct answer to come through the retrieval path., and the questions come in three groups that almost everything below depends on:\nfactual — the answer sits in one document. multi-hop — the answer needs two documents combined, and both have to come back or the question fails. out-of-scope — the corpus genuinely can\u0026rsquo;t answer it, and the right behaviour is to say so. Three numbers run through the rest of this: recall@3Of the questions that do have an answer in the corpus, how often the right document came back in the three chunks returned. Three is the top-k — return more and you have a better chance of catching the answer, and a proportionally larger prompt to pay for. Higher is better., abstentionOf the questions the corpus genuinely cannot answer, how often the system correctly returned nothing at all, rather than confidently retrieving something irrelevant and answering from it. Higher is better., and coverageHow often anything at all came back, across every question. Low coverage means the system has gone quiet — which flatters abstention and wrecks recall at the same time, so it's the number that catches a system for refusing too much..\nThe first two pull against each other. Loosen the filter and you answer more, including things you should have refused; tighten it and you refuse more, including things you could have answered. Most of this post is about that trade, and about how easy it is to think you\u0026rsquo;ve improved it when you haven\u0026rsquo;t. A regression gateAn automated check that re-runs the eval set on every code change and refuses the change if any metric dropped below the recorded baseline. A unit test for quality rather than correctness — with the caveat that it can only compare against a number you previously agreed to. runs the set in CI on every push and fails the build if any of the three drops.\nWhy I wanted hybrid search #In June I swept 33 configurations — two embedding models, three chunk sizes, a range of similarity thresholds. 2,574 queries, $0.84 of API spend, no errored rows.\nThe result that mattered wasn\u0026rsquo;t the winner. It was that there was no good operating point at all. Nothing in 33 cells reached recall above 0.8 and abstention above 0.3 together. The best compromise got recall 0.841 at abstention 0.133 — which reads fine until you translate it: for 87% of the questions my corpus genuinely couldn\u0026rsquo;t answer, the system confidently retrieved something anyway.\nThe reason is in the scores. I was using one cutoff on a similarity score to decide what counts as relevant, but relevant and irrelevant chunks share a band, roughly 0.74 to 0.88. One cutoff can\u0026rsquo;t separate two overlapping distributions. That\u0026rsquo;s architectural, not a tuning problem.\nThe standard answer is to stop treating the retrieval score as the relevance score: retrieve generously, then re-score the shortlist with a cross-encoderAlso called a reranker. Vector search compares two texts that were each summarised into an embedding separately and never seen side by side — fast, and lossy. A cross-encoder reads the question and the chunk together as one input and scores the match directly. Far more accurate, and far too slow to run across a whole index, so it goes on a shortlist. — a model that reads the question and the chunk together.\nSo: that, plus BM25A classic keyword-ranking algorithm. It scores a document by how often the query's words appear in it, weighted so that rare words count for much more than common ones. keyword matching alongside the vector search — the hybridRunning two different retrievers over the same corpus and combining their results: vector search, which matches on meaning, and keyword search, which matches on exact words. Each catches things the other misses — product codes and proper nouns for keyword search, paraphrases for vectors. part, meant to catch exact terms vectors are bad at — with the two lists merged by RRFReciprocal Rank Fusion — combines several ranked lists by scoring each result on its rank position in each list, so anything ranking well in both rises to the top.. Textbook.\nThe reranker came with a constraint attached #Before I ran a single eval, a review of the implementation turned up something I\u0026rsquo;d otherwise have found much later and much more expensively: the cross-encoder only reads about 512 tokensThe units models actually process — roughly a short word or a fragment of one. Every model has a hard ceiling on how many it can read at once, and text past that ceiling is simply cut off. — roughly 2,000 characters.\nThe model I\u0026rsquo;d picked, jina-reranker-v1-turbo-en, advertises an 8,000-token context window. The library I was calling it through pins the tokeniser\u0026rsquo;s truncation to 512. My chunks were 2,500, 5,000 and 10,000 characters. So every candidate on every query was being silently clipped before scoring, and the largest chunks were having roughly 80% of their content thrown away.\nI checked it directly, by putting the same answer at the end of chunks of increasing length and reading the score the reranker gave back:\nanswer at END of 1500 chars -\u0026gt; score -2.178 answer at END of 2500 chars -\u0026gt; score -3.323 answer at END of 5000 chars -\u0026gt; score -4.068 answer at END of 10000 chars -\u0026gt; score -4.068 \u0026lt;- identical: stopped reading Identical scores at 5,000 and 10,000 characters, because by then it\u0026rsquo;s reading the same first 512 tokens either way. Anything past the cut may as well not exist.\nWhich has a consequence I didn\u0026rsquo;t like: chunk size stops being a free parameter and becomes bounded by the reranker\u0026rsquo;s window. Shrink the chunks to fit, or score a different unit than you retrieve. I shrank them — 2500 down to 1500 — because it was a flag change rather than a project.\nAnd that is how a second variable walks into an experiment without ever feeling like one. I didn\u0026rsquo;t change chunk size absent-mindedly. I changed it because I was adding the reranker, so it felt like part of installing the thing rather than an independent intervention. Which is exactly why it took me weeks to see it as one.\nThe results looked good #They did, genuinely. Against the June baseline:\nConfig Recall@3 Factual Multi-hop Abstention $/query baseline (chunk 2500) 0.841 0.933 0.611 0.133 $0.000420 hybrid, chunk 1500, loose cutoff 0.730 0.956 0.167 0.400 $0.000204 hybrid, chunk 1500, mid cutoff 0.810 0.978 0.389 0.133 $0.000247 hybrid, chunk 1500, no filtering 0.841 1.000 0.444 0.000 $0.000304 Factual recall of 0.956 while correctly refusing 40% of the unanswerable questions — the region the June sweep said didn\u0026rsquo;t exist. Cost down 28% at identical recall. I wrote up four findings, and I was pleased with them.\nMulti-hop had collapsed, from 0.611 to somewhere between 0.167 and 0.444, and I explained that too: a cross-encoder scores each chunk on its own and never sees the result set as a whole, so whichever document matches the question best produces several strong chunks and takes every slot. The evidence was right there — on one run, 11 of 18 multi-hop questions came back with chunks from a single document, and only 3 of 18 got both documents the answer needed.\nneed = [duck_wars_extended_history.md, isoprene_planetary_survey.md] got = [duck_wars_extended_history.md, duck_wars_extended_history.md, duck_wars_extended_history.md] Clean diagnosis. Confident write-up. Three variables in the experiment.\nThe ablation #An ablation is the boring, essential experiment where you remove exactly one thing and re-run everything else unchanged. It\u0026rsquo;s the only way to know which part of a change was responsible for the result. If you alter three things at once and the numbers move, you\u0026rsquo;ve learned that something worked — which feels like knowledge and isn\u0026rsquo;t.\nI knew mine was confounded. I\u0026rsquo;d written \u0026ldquo;the outstanding ablation is dense-only at chunk 1500\u0026rdquo; into my own notes and then not run it, because the results were good and it cost an evening. That\u0026rsquo;s the whole failure mode, and it\u0026rsquo;s worth saying plainly: ablations don\u0026rsquo;t get skipped when the numbers look bad. They get skipped when the numbers look good.\nSo I checked out the code from before the hybrid work, set the chunk size to 1500, rebuilt the index from scratch, and ran the same 78 questions. Same corpus, same embedding model, same top-3. The only difference from the hybrid run is the pipeline itself.\nrecall factual multi-hop coverage plain vector search, chunk 1500 0.841 1.000 0.444 1.000 hybrid + reranker, chunk 1500 0.841 1.000 0.444 1.000 Not close. The same numbers. With filtering off, both pipelines put the same documents in the top three on all 63 answerable questions. And across the rest of the range, plain vector search was marginally ahead.\nEach point is one run at a different filter setting. The two pipelines start on the same point and the simpler one stays slightly above — it answers a little more at any given level of caution. Three of my four findings were wrong. Reranking hadn\u0026rsquo;t opened up that high-abstention region — halving the chunk size had. Reranking hadn\u0026rsquo;t broken multi-hop — plain vector search at the same chunk size breaks it identically. The cost saving wasn\u0026rsquo;t the pipeline either: plain vector search at chunk 1500 costs $0.000312 per query against the baseline\u0026rsquo;s $0.000420, a 26% reduction with no pipeline change whatsoever.\nMeanwhile the hybrid pipeline was adding about 160 ms of CPU-bound reranking to every query, and a 150 MB model to a service that scales to zero and therefore downloads it again on every cold start.\nWhat chunk size was actually doing #Truncation is half of it, and the probe above tells that story. The failure breakdown says the same thing from the other direction: of 11 factual misses in the 2500-character run, 10 returned zero chunks rather than wrong ones. Retrieval had found the right candidates. The reranker scored them below the cutoff because it never read that far into them.\nThe other half is the one I keep thinking about.\nSame eval set, same everything except how finely the documents were cut. The two buckets move in opposite directions by almost exactly the same amount. Smaller chunks fix the last three factual questions and break three multi-hop ones. Fifty-three of sixty-three answered either way — the identical total, because the two effects are the identical size. That\u0026rsquo;s why overall recall sits at 0.841 in both rows while the system\u0026rsquo;s behaviour changes underneath it.\nIf I\u0026rsquo;d been tracking aggregate recall, chunk size would have looked like a no-op. Splitting the questions into buckets is the only reason I can see the trade at all.\nIt\u0026rsquo;s also worth being honest about what three questions can and can\u0026rsquo;t tell you. On eighteen multi-hop rows, three is a 17-point swing, and I wouldn\u0026rsquo;t defend the precise size of that trade-off from this corpus. What I would defend is its direction and its mechanism, which the failure breakdown explains independently of the score. The numbers in this post are the smallest part of it; the method is the part I\u0026rsquo;d argue for.\nThe mechanism, once the buckets tell you where to look, is embarrassingly simple. Smaller chunks mean more chunks per document. I return the top 3 with nothing stopping all three from coming out of the same file — and the better a document matches, the more of its chunks crowd the shortlist. Multi-hop questions here always need two distinct documents. So it isn\u0026rsquo;t a retrieval-strategy problem at all. It\u0026rsquo;s a selection problem, and capping how many chunks any one document can contribute fixes it directly, at no cost per query.\nA note on the 26% #Every query in the eval set records what it cost, so that saving isn\u0026rsquo;t an estimate — and splitting the bill shows it had nothing to do with retrieval either. The model wrote the same amount of answer both times. It read a third less, because I\u0026rsquo;d halved the chunks going into the prompt.\nTwo things follow that are worth carrying out of here. Cost per query is trivially minimised by answering fewer questions — a system that refuses everything is nearly free — so the number is meaningless without recall beside it, which is why every table above carries both. And the reranker\u0026rsquo;s 160 ms of CPU and its 150 MB cold-start download appear nowhere in that dollar column at all. The component with the weakest case for existing was invisible to the metric I was judging everything else by.\nSo I\u0026rsquo;m keeping hybrid search #Which reads as sunk cost, so let me put the argument down properly.\nThe ablation is a null result on a test my corpus can\u0026rsquo;t run. Seven documents, 52 chunks, and a shortlist of 9 — about 17% of everything I have. A reranker earns its keep by reordering a shortlist drawn from a much larger pool; mine already holds nearly everything relevant, so there\u0026rsquo;s little left to fix. BM25\u0026rsquo;s entire signal is which words are rare, and across seven documents there\u0026rsquo;s no rarity to measure. So the claim is narrow — hybrid search didn\u0026rsquo;t pay for itself on this corpus at this scale, not \u0026ldquo;hybrid search doesn\u0026rsquo;t work.\u0026rdquo;\nBut read that back, because there\u0026rsquo;s something worse in it than a null result. If my corpus can\u0026rsquo;t exhibit the failure hybrid search exists to fix, then I built the fix before I had any test capable of judging it — a bigger error than the confounded experiment, and one no amount of careful measurement afterwards could have caught. The measurement was fine. The thing being measured was never in scope. Find a query your retrieval genuinely fails on, then go looking for what fixes it. I did it backwards and spent a quarter finding out.\nIt stays anyway, and not only out of stubbornness: the reranker\u0026rsquo;s 512-token window is what turned chunk size from a default I\u0026rsquo;d stopped seeing into a live parameter, so every gain I\u0026rsquo;m now crediting to chunking exists because it forced me there. But it stays labelled — an unproven bet with a stated test. Build a corpus where plain vector search genuinely fails, re-run this comparison, and if it still adds nothing, it comes out.\nThough \u0026ldquo;stays\u0026rdquo; flatters it. There\u0026rsquo;s no flag to turn it off: query prefetches both vectors, fuses them and calls the reranker on every request, with the threshold and model name as the only knobs I exposed. I wired it in as the only path rather than one of two, which is what building-before-testing looks like by the time it reaches the code.\nThe part I\u0026rsquo;d keep #The interesting output here isn\u0026rsquo;t a finding about chunk sizes. It\u0026rsquo;s that I ran an experiment with three variables in it, produced a confident and internally coherent explanation, and was wrong — and the only reason I know is that I\u0026rsquo;d built the instrument before I needed it.\nThe write-ups in the repo now carry their corrections inline, with the wrong conclusions left standing and annotated rather than quietly edited out. That\u0026rsquo;s deliberate. A results directory that only contains things that worked is marketing.\nMost RAG write-ups I read change three things and credit the interesting one. I did exactly that, in public, with a straight face, in my own notes. The eval set caught it. That is what the eval set is for.\nNext is the half of the bill I can\u0026rsquo;t honestly attribute per query at all — GPU-seconds, cold starts, what it costs to run a model rather than rent one. That\u0026rsquo;s the part I understand least, so that\u0026rsquo;s where I\u0026rsquo;m going.\nThe code, the eval harness, the full result write-ups and the corrections are all in ritam. If you\u0026rsquo;re running retrieval in production and your evals report a single aggregate number, split it into buckets — I\u0026rsquo;d like to hear what falls out.\n","date":"12 August 2026","permalink":"https://vbhargava.org/writing/hybrid-search-ablation/","section":"Blog","summary":"\u003cp\u003eI added hybrid search and a cross-encoder reranker to my RAG service, measured it, and wrote up four findings. Then I ran the one comparison I\u0026rsquo;d been putting off, and three of the four didn\u0026rsquo;t survive.\u003c/p\u003e","title":"Hybrid search didn't help. It still fixed my retrieval."},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/inference-cost/","section":"Tags","summary":"","title":"Inference-Cost"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/llm/","section":"Tags","summary":"","title":"LLM"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/rag/","section":"Tags","summary":"","title":"Rag"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/retrieval/","section":"Tags","summary":"","title":"Retrieval"},{"content":"","date":null,"permalink":"https://vbhargava.org/series/ritam/","section":"Series","summary":"","title":"Ritam"},{"content":"","date":null,"permalink":"https://vbhargava.org/series/","section":"Series","summary":"","title":"Series"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://vbhargava.org/categories/technology/","section":"Categories","summary":"","title":"Technology"},{"content":"Built a production-shaped RAG v1: offline indexing + FastAPI /query on Cloud Run (Terraform), with sources, logs, and basic monitoring.\nNote: This is a walkthrough of the smallest RAG system that still behaves like a real service: deployable, observable, and debuggable. I keep the RAG explanation beginner-friendly, but the focus is the engineering reality (latency, rate limits, errors, and trade-offs).\nWhy I built this #I’ve spent most of my career building and operating platform infrastructure — CI/CD, cloud services, and the systems that have to keep working when nobody is watching. Over the last year, AI and LLMsLarge Language Models, a type of artificial intelligence that can generate human-like text and other content. have moved from novelty to something teams expect to ship as product features. What I didn’t want was to stay at the level of “I use ChatGPT at work”. I wanted a hands-on understanding of what it takes to run an LLM-backed service in a way that’s reliable, observable, and cost-aware.\nI also wanted to get past the vague feeling of “this is magic” and replace it with a concrete mental model I could explain and debug. I learn best by building: something small enough to understand end-to-end, but real enough to behave like production.\nSo I built a small, production-shaped RAG service end-to-end: an indexing pipeline, a retrieval layer, and a FastAPI /query endpoint that returns an answer with sources. I deployed it to Cloud Run using Terraform and added basic logging and monitoring so I could see real latency, errors, and rate limits — not just a happy-path demo.\nThis post is a walkthrough of the smallest RAG system that still behaves like a production service: deployable, observable, and debuggable.\nProject links # Repo (code + Terraform): GitHub Code snapshot used for this post: tagged q1 RAG in 60 seconds (for people new to it) #LLMs are good at generating text, but they don’t “know” your private docs. RAG (Retrieval-Augmented Generation) is a practical pattern to fix that:\nIndex your documents by turning chunks of text into vectors (“embeddingsVector representations of words or phrases”). Retrieve the most relevant chunks for a question (similarity search). Generate an answer using the retrieved chunks as grounded contextInformation provided to an LLM from a reliable source to ensure its response is based on facts, not just its internal knowledge. — and return sources. With that mental model, here’s the smallest version I could build that still behaves like a real service when deployed.\nWhat I built #RAG v1 is deliberately small and production-shaped. It has two paths:\nOffline indexing (CLI, run locally before deploy) Read Markdown docs from assets/docs/ (in the repo) Chunk them (simple splitter for v1) Generate embeddings via an embedding API Write an index file: assets/indexed_chunks.json Query serving (FastAPI on Cloud Run) Load the precomputed index at startup On POST /query: embed the question, retrieve top-k chunks via cosine similarity, build a grounded promptThe input given to a large language model to guide its response., call a chat model Return JSON: answer + sources (doc + chunk references) A key v1 constraint: the index is baked into the container image. That keeps the system predictable and easy to deploy, at the cost of not supporting live ingestion. Dynamic ingestion, vector DBs, and hybrid search are explicitly v2 work.\nHere’s the full v1 architecture (two paths: offline indexing + query serving):\nOffline indexing builds a static index; Cloud Run serves grounded answers with sources. Demo (local) #Here’s what a minimal query looks like end-to-end.\nRequest\ncurl -s http://localhost:8080/query \\ -H \u0026#34;Content-Type: application/json\u0026#34; \\ -d \u0026#39;{ \u0026#34;query\u0026#34;: \u0026#34;Why is the embedding index baked into the container image in this project?\u0026#34;, \u0026#34;top_k\u0026#34;: 2 }\u0026#39; | jq . Response\n{ \u0026#34;answer\u0026#34;: \u0026#34;In RAG v1, the index is baked into the container image to keep deployments predictable and reproducible. The trade-off is you don’t get live ingestion: updating docs requires re-indexing and redeploying. This constraint is intentional in v1 so the focus stays on retrieval, API shape, deployment, and observability.\u0026#34;, \u0026#34;sources\u0026#34;: [ { \u0026#34;source\u0026#34;: \u0026#34;assets/docs/rag_intro.md\u0026#34;, \u0026#34;chunk_id\u0026#34;: 3 }, { \u0026#34;source\u0026#34;: \u0026#34;assets/docs/architecture_notes.md\u0026#34;, \u0026#34;chunk_id\u0026#34;: 7 } ] } Observed behaviour (v1) #These are early measurements from Cloud Run + a simple Monitoring dashboard (light traffic + a couple of burst tests):\np95 latency (/query): ~1.7–3.5s Failure mode under burst: provider rate limiting (HTTP 429) dominated; in one burst test the error rate spiked up to ~85% since I was using a free-tier API key with low limits Where time goes: generation latency dominates; retrieval is comparatively cheap in v1 due to the small, in-memory index Numbers are indicative, not benchmark-grade; v2 work adds caching/batching + evals + more granular tracing.\nEnd-to-end flow (10 steps) #Here is the full end-to-end flow from writing source docs to getting a grounded answer back:\nEnd-to-end flow (10 steps) Write source docs (local)\nMy “knowledge base” is a small set of Markdown notes in assets/docs/ (repo).\nRun the indexing CLI (offline)\nI run an index command locally with LLM_API_KEY set.\nLoad and chunk documents\nThe indexer reads all *.md files, validates they’re non-empty, and splits content into chunks (v1 uses a simple paragraph-based splitter).\nEmbed every chunk\nFor each chunk, the CLI calls an embedding model via the provider API.\nWrite the embedding index JSON\nThe CLI writes assets/indexed_chunks.json, containing metadata and an array of chunks:\n{chunk_id, source, text, embedding[]}.\nBake the index into the container image\nFor v1, the index is bundled into the Docker image so the runtime stays predictable.\nDeploy the API service (Cloud Run via Terraform)\nTerraform provisions Artifact Registry + a Cloud Run service (in europe-west3) and configures env vars (API key, model names).\nHandle a query request\nA client sends POST /query with { \u0026quot;query\u0026quot;: \u0026quot;...\u0026quot;, \u0026quot;top_k\u0026quot;: 3 }. FastAPI validates the payload with Pydantic.\nRetrieve + generate a grounded answer\nThe service embeds the incoming question, computes cosine similarityA measure of similarity between two non-zero vectors. It's used to find text chunks with meanings similar to the user's query. against chunk embeddings, selects the top-k chunks, builds a prompt that includes only those chunks (with source + chunk_id), and calls the chat model.\nObserve what actually happened\nA request logging middleware emits structured JSON logs, and Cloud Run metrics feed a small dashboard (request count, p95 latency, error rate).\nKey engineering decisions (and trade-offs) #A few choices in v1 are intentionally “boring” — not because I don’t know the fancier options, but because I wanted a tight loop where I can measure failures and iterate.\nKey engineering decisions (and trade-offs) Static index baked into the image (v1 constraint)\nWhy: predictable deploys, minimal moving parts, easy to reproduce.\nTrade-off: no live ingestion; updating docs requires rebuilding the index and redeploying.\nNext: ingestion v2 + datasets + idempotency (and later, vector DB).\nSimple chunking on purpose\nWhy: chunking is a major driver of retrieval quality. Starting simple makes failure modes obvious.\nTrade-off: uneven chunk sizes and missed structure.\nNext: configurable chunking strategies + richer metadata.\nCosine similarity over an in-memory index\nWhy: v1 is about correctness and system shape, not scale.\nTrade-off: doesn’t scale; scoring is linear over all chunks.\nNext: move retrieval to a vector store (e.g., Qdrant) + backup/restore.\nGrounded prompting + explicit sources\nWhy: returning sources makes answers auditable and debuggable.\nTrade-off: prompt formatting matters; answers can be more conservative.\nNext: eval-driven iteration on prompt and retrieval knobs.\nTyped service boundaries (core vs API vs provider client)\nWhy: keep HTTP concerns separate from retrieval/orchestration so the core can be reused and tested cleanly.\nPayoff: easier testing, clearer error handling, and safer refactors.\nObservability early (not after the demo works)\nWhy: LLM systems fail in ways you won’t see from CPU/memory graphs alone.\nNext: request IDs, per-stage latency (retrieve vs generate), and token/cost signals where available.\nWhat I learned (v1) #This project was a deliberate step outside my comfort zone. Coming from a career in platform infrastructure, I’m used to thinking in terms of deployments, observability, and system resilience. Writing the core logic for this RAG service was a dive into a different kind of complexity. It wasn’t just about making it run; it was about shaping the logic that produces the answer. It was a sharp reminder that the concerns of a software developer and a DevOps engineer are distinct, even when they work on the same system.\nBefore I started, the world of LLMs and RAG felt like a black box—something magical and slightly intimidating. But building this system end-to-end demystified it. I realized that RAG isn’t magic; it’s an engineering pipeline. You take documents, you chunk them, you embed them, you retrieve them, and you use them to generate a response. Once you see it as a series of steps, you can reason about it, debug it, and improve it. That was the biggest takeaway for me: the \u0026ldquo;magic\u0026rdquo; is just a well-structured pipeline.\nBeyond that, a few key lessons stood out:\nRetrieval quality is everything. The quality of the answer depends almost entirely on the quality of the retrieved context. Chunking strategy, top-k selection, and prompt construction are the most important levers you have. Operations are not an afterthought. Rate limits, API errors, and latency are not edge cases; they are the reality of running a service that depends on other services. Building in logging and monitoring from day one was crucial. Start simple. By using a static index and in-memory retrieval, I kept the system predictable and easy to debug. This allowed me to focus on the core logic of the RAG pipeline. Scaling comes next. What I’m improving next (v2) #v1 was about getting the system shape right: something correct, deployable, and observable.\nv2 is about making it realistic at scale — and easy to improve without guessing.\nMake ingestion real\nMulti-source inputs (dir + URL), richer metadata (dataset/doc_id/timestamps), and idempotent re-ingestion so updates don’t create duplicates.\nMake retrieval scalable\nMove from an in-memory JSON index to a vector database, and add hybrid search (BM25A ranking function used by search engines to estimate the relevance of documents to a given search query. It is based on the probabilistic retrieval model. + vectors) where it actually helps.\nMake it measurable\nAdd a small eval set, track retrieval quality over time, and break down latency + cost per stage (embed → retrieve → generate) so I know what to optimize.\nThe goal is to turn this from a production-shaped demo into a repeatable reference service for building and operating LLM-backed systems responsibly — and eventually a small framework other teams could build on.\n","date":"29 January 2026","permalink":"https://vbhargava.org/writing/llm-lab-rag-v1/","section":"Blog","summary":"\u003cp\u003eBuilt a production-shaped RAG v1: offline indexing + FastAPI \u003ccode\u003e/query\u003c/code\u003e on Cloud Run (Terraform), with sources, logs, and basic monitoring.\u003c/p\u003e","title":"Building a RAG service: architecture, trade-offs, and lessons"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/fastapi/","section":"Tags","summary":"","title":"Fastapi"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/gcp/","section":"Tags","summary":"","title":"GCP"},{"content":"","date":null,"permalink":"https://vbhargava.org/tags/observability/","section":"Tags","summary":"","title":"Observability"},{"content":"I\u0026rsquo;m Vishal Bhargava. I build and run systems that have to work in the real world — not just in clean diagrams.\nMost of my career has been platform and infrastructure work: delivery pipelines, internal tooling, and the foundations other engineers depend on. Twelve years of it. I tend to care about reliability, debugging, and cost more than novelty.\nThese days I apply that same platform mindset to LLM systems — not from a research angle, but an operational one. How they get deployed, observed, and paid for. The part I keep coming back to is inference cost: it\u0026rsquo;s the line item that surprises teams, it\u0026rsquo;s rarely measured properly, and it\u0026rsquo;s usually treated as a model problem when it\u0026rsquo;s an infrastructure problem. Measurement, routing, and serving efficiency move it far more than swapping models does.\nWhat I\u0026rsquo;m working on # ritam — an end-to-end RAG service (API, infra, evals, dashboards) I keep building in the open as a way to work through these problems properly rather than theoretically. Cost and latency per stage — embed → retrieve → generate, measured rather than guessed. Retrieval quality — chunking, hybrid search, and evals that tell you whether a change actually helped. Quick facts # Platform/infrastructure engineer, ~12 years across AWS, Kubernetes, and Terraform Tech Lead Manager, Platform @ Kittl — remote, for a Berlin team Based in Dubai since early 2026, after seven years in Berlin Outside of work, I care a lot about food. I cook often, experiment a lot, and probably overthink simple dishes more than necessary. It\u0026rsquo;s one of the few places where I enjoy slowing down and paying attention to small details.\nThis site is where I write things down — about work, projects, and change — when they\u0026rsquo;re worth thinking through properly.\nGet in touch #If you\u0026rsquo;re running LLM systems in production and wrestling with inference cost, serving, retrieval, or evals, I\u0026rsquo;d like to hear about it. Mail me at vishal@vbhargava.org — I read everything and reply to most of it.\nYou can also find me on:\nGitHub LinkedIn Instagram ","date":null,"permalink":"https://vbhargava.org/about/","section":"","summary":"","title":"About me"}]