Building a RAG service: architecture, trade-offs, and lessons

Built a production-shaped RAG v1: offline indexing + FastAPI /query on Cloud Run (Terraform), with sources, logs, and basic monitoring.

Note: This is a walkthrough of the smallest RAG system that still behaves like a real service: deployable, observable, and debuggable. I keep the RAG explanation beginner-friendly, but the focus is the engineering reality (latency, rate limits, errors, and trade-offs).

Why I built this #

I’ve spent most of my career building and operating platform infrastructure — CI/CD, cloud services, and the systems that have to keep working when nobody is watching. Over the last year, AI and LLMsLarge Language Models, a type of artificial intelligence that can generate human-like text and other content. have moved from novelty to something teams expect to ship as product features. What I didn’t want was to stay at the level of “I use ChatGPT at work”. I wanted a hands-on understanding of what it takes to run an LLM-backed service in a way that’s reliable, observable, and cost-aware.

I also wanted to get past the vague feeling of “this is magic” and replace it with a concrete mental model I could explain and debug. I learn best by building: something small enough to understand end-to-end, but real enough to behave like production.

So I built a small, production-shaped RAG service end-to-end: an indexing pipeline, a retrieval layer, and a FastAPI /query endpoint that returns an answer with sources. I deployed it to Cloud Run using Terraform and added basic logging and monitoring so I could see real latency, errors, and rate limits — not just a happy-path demo.

This post is a walkthrough of the smallest RAG system that still behaves like a production service: deployable, observable, and debuggable.

Project links #

• Repo (code + Terraform): GitHub
• Code snapshot used for this post: tagged q1

RAG in 60 seconds (for people new to it) #

LLMs are good at generating text, but they don’t “know” your private docs. RAG (Retrieval-Augmented Generation) is a practical pattern to fix that:

1. Index your documents by turning chunks of text into vectors (“embeddingsVector representations of words or phrases”).
2. Retrieve the most relevant chunks for a question (similarity search).
3. Generate an answer using the retrieved chunks as grounded contextInformation provided to an LLM from a reliable source to ensure its response is based on facts, not just its internal knowledge. — and return sources.

With that mental model, here’s the smallest version I could build that still behaves like a real service when deployed.

What I built #

RAG v1 is deliberately small and production-shaped. It has two paths:

1. Offline indexing (CLI, run locally before deploy)

• Read Markdown docs from assets/docs/ (in the repo)
• Chunk them (simple splitter for v1)
• Generate embeddings via an embedding API
• Write an index file: assets/indexed_chunks.json

2. Query serving (FastAPI on Cloud Run)

• Load the precomputed index at startup
• On POST /query: embed the question, retrieve top-k chunks via cosine similarity, build a grounded promptThe input given to a large language model to guide its response., call a chat model
• Return JSON: answer + sources (doc + chunk references)

A key v1 constraint: the index is baked into the container image. That keeps the system predictable and easy to deploy, at the cost of not supporting live ingestion. Dynamic ingestion, vector DBs, and hybrid search are explicitly v2 work.

Here’s the full v1 architecture (two paths: offline indexing + query serving):

Demo (local) #

Here’s what a minimal query looks like end-to-end.

Request

curl -s http://localhost:8080/query \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Why is the embedding index baked into the container image in this project?",
    "top_k": 2
  }' | jq .

Response

{
  "answer": "In RAG v1, the index is baked into the container image to keep deployments predictable and reproducible. The trade-off is you don’t get live ingestion: updating docs requires re-indexing and redeploying. This constraint is intentional in v1 so the focus stays on retrieval, API shape, deployment, and observability.",
  "sources": [
    { "source": "assets/docs/rag_intro.md", "chunk_id": 3 },
    { "source": "assets/docs/architecture_notes.md", "chunk_id": 7 }
  ]
}

Observed behaviour (v1) #

These are early measurements from Cloud Run + a simple Monitoring dashboard (light traffic + a couple of burst tests):

• p95 latency (/query): ~1.7–3.5s
• Failure mode under burst: provider rate limiting (HTTP 429) dominated; in one burst test the error rate spiked up to ~85% since I was using a free-tier API key with low limits
• Where time goes: generation latency dominates; retrieval is comparatively cheap in v1 due to the small, in-memory index

Numbers are indicative, not benchmark-grade; v2 work adds caching/batching + evals + more granular tracing.

End-to-end flow (10 steps) #

Here is the full end-to-end flow from writing source docs to getting a grounded answer back:

Key engineering decisions (and trade-offs) #

A few choices in v1 are intentionally “boring” — not because I don’t know the fancier options, but because I wanted a tight loop where I can measure failures and iterate.

What I learned (v1) #

This project was a deliberate step outside my comfort zone. Coming from a career in platform infrastructure, I’m used to thinking in terms of deployments, observability, and system resilience. Writing the core logic for this RAG service was a dive into a different kind of complexity. It wasn’t just about making it run; it was about shaping the logic that produces the answer. It was a sharp reminder that the concerns of a software developer and a DevOps engineer are distinct, even when they work on the same system.

Before I started, the world of LLMs and RAG felt like a black box—something magical and slightly intimidating. But building this system end-to-end demystified it. I realized that RAG isn’t magic; it’s an engineering pipeline. You take documents, you chunk them, you embed them, you retrieve them, and you use them to generate a response. Once you see it as a series of steps, you can reason about it, debug it, and improve it. That was the biggest takeaway for me: the “magic” is just a well-structured pipeline.

Beyond that, a few key lessons stood out:

• Retrieval quality is everything. The quality of the answer depends almost entirely on the quality of the retrieved context. Chunking strategy, top-k selection, and prompt construction are the most important levers you have.
• Operations are not an afterthought. Rate limits, API errors, and latency are not edge cases; they are the reality of running a service that depends on other services. Building in logging and monitoring from day one was crucial.
• Start simple. By using a static index and in-memory retrieval, I kept the system predictable and easy to debug. This allowed me to focus on the core logic of the RAG pipeline. Scaling comes next.

What I’m improving next (v2) #

v1 was about getting the system shape right: something correct, deployable, and observable.
v2 is about making it realistic at scale — and easy to improve without guessing.

1. Make ingestion real
   Multi-source inputs (dir + URL), richer metadata (dataset/doc_id/timestamps), and idempotent re-ingestion so updates don’t create duplicates.

2. Make retrieval scalable
   Move from an in-memory JSON index to a vector database, and add hybrid search (BM25A ranking function used by search engines to estimate the relevance of documents to a given search query. It is based on the probabilistic retrieval model. + vectors) where it actually helps.

3. Make it measurable
   Add a small eval set, track retrieval quality over time, and break down latency + cost per stage (embed → retrieve → generate) so I know what to optimize.

The goal is to turn this from a production-shaped demo into a repeatable reference service for building and operating LLM-backed systems responsibly — and eventually a small framework other teams could build on.