# Building a RAG service: architecture, trade-offs, and lessons
> A production-shaped RAG v1: offline indexing + FastAPI /query on Cloud Run (Terraform), with sources, logs, and basic monitoring.
Published: 2026-01-29
Author: Vishal Bhargava
Canonical URL: https://vbhargava.org/writing/llm-lab-rag-v1/
Tags: rag, fastapi, GCP, observability, LLM
Series: ritam
---
Built a production-shaped RAG v1: offline indexing + FastAPI `/query` on Cloud Run (Terraform), with sources, logs, and basic monitoring.
> **Note:**
> This is a walkthrough of the smallest RAG system that still behaves like a real service: deployable, observable, and debuggable.
> I keep the RAG explanation beginner-friendly, but the focus is the engineering reality (latency, rate limits, errors, and trade-offs).
## Why I built this
I’ve spent most of my career building and operating platform infrastructure — CI/CD, cloud services, and the systems that have to keep working when nobody is watching. Over the last year, AI and LLMs (Large Language Models, a type of artificial intelligence that can generate human-like text and other content.) have moved from novelty to something teams expect to ship as product features. What I didn’t want was to stay at the level of “I use ChatGPT at work”. I wanted a hands-on understanding of what it takes to run an LLM-backed service in a way that’s reliable, observable, and cost-aware.
I also wanted to get past the vague feeling of “this is magic” and replace it with a concrete mental model I could explain and debug. I learn best by building: something small enough to understand end-to-end, but real enough to behave like production.
So I built a small, production-shaped RAG service end-to-end: an indexing pipeline, a retrieval layer, and a FastAPI `/query` endpoint that returns an answer *with sources*. I deployed it to Cloud Run using Terraform and added basic logging and monitoring so I could see real latency, errors, and rate limits — not just a happy-path demo.
This post is a walkthrough of the smallest RAG system that still behaves like a production service: deployable, observable, and debuggable.
## Project links
- Repo (code + Terraform): [GitHub](https://github.com/bhvishal9/llm_lab)
- Code snapshot used for this post: [tagged q1](https://github.com/bhvishal9/llm_lab/tree/q1)
## RAG in 60 seconds (for people new to it)
LLMs are good at generating text, but they don’t “know” your private docs. RAG (Retrieval-Augmented Generation) is a practical pattern to fix that:
1) **Index** your documents by turning chunks of text into vectors (“embeddings (Vector representations of words or phrases)”).
2) **Retrieve** the most relevant chunks for a question (similarity search).
3) **Generate** an answer using the retrieved chunks as grounded context (Information provided to an LLM from a reliable source to ensure its response is based on facts, not just its internal knowledge.) — and return sources.
With that mental model, here’s the smallest version I could build that still behaves like a real service when deployed.
## What I built
RAG v1 is deliberately small and production-shaped. It has two paths:
1) **Offline indexing (CLI, run locally before deploy)**
- Read Markdown docs from `assets/docs/` (in the repo)
- Chunk them (simple splitter for v1)
- Generate embeddings via an embedding API
- Write an index file: `assets/indexed_chunks.json`
2) **Query serving (FastAPI on Cloud Run)**
- Load the precomputed index at startup
- On `POST /query`: embed the question, retrieve top-k chunks via cosine similarity, build a grounded prompt (The input given to a large language model to guide its response.), call a chat model
- Return JSON: `answer` + `sources` (doc + chunk references)
A key v1 constraint: the index is **baked into the container image**. That keeps the system predictable and easy to deploy, at the cost of not supporting live ingestion. Dynamic ingestion, vector DBs, and hybrid search are explicitly v2 work.
Here’s the full v1 architecture (two paths: offline indexing + query serving):
## Demo (local)
Here’s what a minimal query looks like end-to-end.
**Request**
```bash
curl -s http://localhost:8080/query \
-H "Content-Type: application/json" \
-d '{
"query": "Why is the embedding index baked into the container image in this project?",
"top_k": 2
}' | jq .
```
**Response**
```json
{
"answer": "In RAG v1, the index is baked into the container image to keep deployments predictable and reproducible. The trade-off is you don’t get live ingestion: updating docs requires re-indexing and redeploying. This constraint is intentional in v1 so the focus stays on retrieval, API shape, deployment, and observability.",
"sources": [
{ "source": "assets/docs/rag_intro.md", "chunk_id": 3 },
{ "source": "assets/docs/architecture_notes.md", "chunk_id": 7 }
]
}
```
## Observed behaviour (v1)
These are early measurements from Cloud Run + a simple Monitoring dashboard (light traffic + a couple of burst tests):
- **p95 latency (/query):** ~1.7–3.5s
- **Failure mode under burst:** provider rate limiting (HTTP 429) dominated; in one burst test the **error rate spiked up to ~85%** since I was using a free-tier API key with low limits
- **Where time goes:** generation latency dominates; retrieval is comparatively cheap in v1 due to the small, in-memory index
Numbers are indicative, not benchmark-grade; v2 work adds caching/batching + evals + more granular tracing.
## End-to-end flow (10 steps)
Here is the full end-to-end flow from writing source docs to getting a grounded answer back:
End-to-end flow (10 steps)
1) **Write source docs (local)**
My “knowledge base” is a small set of Markdown notes in `assets/docs/` (repo).
2) **Run the indexing CLI (offline)**
I run an `index` command locally with `LLM_API_KEY` set.
3) **Load and chunk documents**
The indexer reads all `*.md` files, validates they’re non-empty, and splits content into chunks (v1 uses a simple paragraph-based splitter).
4) **Embed every chunk**
For each chunk, the CLI calls an embedding model via the provider API.
5) **Write the embedding index JSON**
The CLI writes `assets/indexed_chunks.json`, containing metadata and an array of chunks:
`{chunk_id, source, text, embedding[]}`.
6) **Bake the index into the container image**
For v1, the index is bundled into the Docker image so the runtime stays predictable.
7) **Deploy the API service (Cloud Run via Terraform)**
Terraform provisions Artifact Registry + a Cloud Run service (in `europe-west3`) and configures env vars (API key, model names).
8) **Handle a query request**
A client sends `POST /query` with `{ "query": "...", "top_k": 3 }`. FastAPI validates the payload with Pydantic.
9) **Retrieve + generate a grounded answer**
The service embeds the incoming question, computes cosine similarity (A measure of similarity between two non-zero vectors. It's used to find text chunks with meanings similar to the user's query.) against chunk embeddings, selects the top-k chunks, builds a prompt that includes *only* those chunks (with `source` + `chunk_id`), and calls the chat model.
10) **Observe what actually happened**
A request logging middleware emits structured JSON logs, and Cloud Run metrics feed a small dashboard (request count, p95 latency, error rate).
## Key engineering decisions (and trade-offs)
A few choices in v1 are intentionally “boring” — not because I don’t know the fancier options, but because I wanted a tight loop where I can measure failures and iterate.
Key engineering decisions (and trade-offs)
- **Static index baked into the image (v1 constraint)**
*Why:* predictable deploys, minimal moving parts, easy to reproduce.
*Trade-off:* no live ingestion; updating docs requires rebuilding the index and redeploying.
*Next:* ingestion v2 + datasets + idempotency (and later, vector DB).
- **Simple chunking on purpose**
*Why:* chunking is a major driver of retrieval quality. Starting simple makes failure modes obvious.
*Trade-off:* uneven chunk sizes and missed structure.
*Next:* configurable chunking strategies + richer metadata.
- **Cosine similarity over an in-memory index**
*Why:* v1 is about correctness and system shape, not scale.
*Trade-off:* doesn’t scale; scoring is linear over all chunks.
*Next:* move retrieval to a vector store (e.g., Qdrant) + backup/restore.
- **Grounded prompting + explicit sources**
*Why:* returning sources makes answers auditable and debuggable.
*Trade-off:* prompt formatting matters; answers can be more conservative.
*Next:* eval-driven iteration on prompt and retrieval knobs.
- **Typed service boundaries (core vs API vs provider client)**
*Why:* keep HTTP concerns separate from retrieval/orchestration so the core can be reused and tested cleanly.
*Payoff:* easier testing, clearer error handling, and safer refactors.
- **Observability early (not after the demo works)**
*Why:* LLM systems fail in ways you won’t see from CPU/memory graphs alone.
*Next:* request IDs, per-stage latency (retrieve vs generate), and token/cost signals where available.
## What I learned (v1)
This project was a deliberate step outside my comfort zone. Coming from a career in platform infrastructure, I’m used to thinking in terms of deployments, observability, and system resilience. Writing the core logic for this RAG service was a dive into a different kind of complexity. It wasn’t just about making it run; it was about shaping the logic that produces the *answer*. It was a sharp reminder that the concerns of a software developer and a DevOps engineer are distinct, even when they work on the same system.
Before I started, the world of LLMs and RAG felt like a black box—something magical and slightly intimidating. But building this system end-to-end demystified it. I realized that RAG isn’t magic; it’s an engineering pipeline. You take documents, you chunk them, you embed them, you retrieve them, and you use them to generate a response. Once you see it as a series of steps, you can reason about it, debug it, and improve it. That was the biggest takeaway for me: the "magic" is just a well-structured pipeline.
Beyond that, a few key lessons stood out:
- **Retrieval quality is everything.** The quality of the answer depends almost entirely on the quality of the retrieved context. Chunking strategy, top-k selection, and prompt construction are the most important levers you have.
- **Operations are not an afterthought.** Rate limits, API errors, and latency are not edge cases; they are the reality of running a service that depends on other services. Building in logging and monitoring from day one was crucial.
- **Start simple.** By using a static index and in-memory retrieval, I kept the system predictable and easy to debug. This allowed me to focus on the core logic of the RAG pipeline. Scaling comes next.
## What I’m improving next (v2)
v1 was about getting the *system shape* right: something correct, deployable, and observable.
v2 is about making it **realistic at scale** — and easy to improve without guessing.
1) **Make ingestion real**
Multi-source inputs (dir + URL), richer metadata (dataset/doc_id/timestamps), and idempotent re-ingestion so updates don’t create duplicates.
2) **Make retrieval scalable**
Move from an in-memory JSON index to a vector database, and add hybrid search (BM25 (A ranking function used by search engines to estimate the relevance of documents to a given search query. It is based on the probabilistic retrieval model.) + vectors) where it actually helps.
3) **Make it measurable**
Add a small eval set, track retrieval quality over time, and break down latency + cost per stage (embed → retrieve → generate) so I know what to optimize.
The goal is to turn this from a production-shaped demo into a **repeatable reference service** for building and operating LLM-backed systems responsibly — and eventually a small framework other teams could build on.