RAG LLM Systems: Dhavan’s Personal Assistant

  1. Home
  2. Projects
  3. RAG Assistant (AWS + SSE)
RAG LLM Systems Project

Production-style RAG assistant: retrieval, streaming SSE responses, rate limiting, cost guardrails, and cloud deployment.

Project information

  • Category: LLM Systems Engineering · Retrieval Augmented Generation (RAG)
  • Frontend: GitHub Pages (home-page chat UI)
  • Backend: FastAPI on AWS Lambda (Web Adapter + RESPONSE_STREAM)
  • Edge: Cloudflare Worker + WAF in front of the Function URL
  • Cloud: Lambda, Function URL, API Gateway (health/metrics), S3, DynamoDB, CloudWatch, CDK
  • Core features: Hybrid BM25+vector+RRF retrieval, SSE streaming, rate limits, atomic budget reservation, evidence verification, CI/CD

Overview

This project is a production-style Retrieval Augmented Generation (RAG) system that answers questions about Dhavan Shah (portfolio, experience, Kaggle achievements, and projects). The focus is not just model output quality, but end-to-end AI systems engineering: retrieval, streaming responses, deployment architecture, cost controls, and operational maturity.

Dhavan's Assistant: you can test the grounded assistant at:

Try The Assistant

Back to Projects

Problem

A portfolio assistant can invent, stall, or become expensive without grounding, streaming, and abuse controls.

Mechanism

Document ingestion, embeddings, hybrid BM25/vector fusion (RRF), fail-closed commercial facts, LLM synthesis with sentence-level verification, SSE streaming, and edge/app abuse controls work as one pipeline.

What It Proves

RAG quality depends on production behavior too: latency, citations, rate limits, observability, and cost control.

Engagement Relevance

Useful for teams replacing demo chatbots with grounded assistants that need answer discipline and operating limits.

CTA

Use this pattern to review grounding, streaming, and cost controls in an assistant workflow.

Send a problem brief

Project Goal

Build a public, portfolio-grade AI assistant that demonstrates real-world engineering around LLM applications: LLM pipelines, vector retrieval, streaming SSE, cloud deployment, rate limiting, cost-aware guardrails, API design, and CI/CD + infrastructure-as-code.

High-Level Architecture

Request/response flow:

  • Frontend (GitHub Pages) → home-page chat UI posts to the public API
  • Cloudflare Worker → origin secret, trusted client IP, WAF/rate limits; kill switch
  • Lambda Function URL → Lambda Web Adapter + RESPONSE_STREAM for SSE
  • FastAPI query Lambda → rate limit → buyer-walk pipeline → budget reservation → retrieve/generate
  • Retrieval → hybrid BM25 + vector similarity fused with RRF (not a separate reranker service)
  • Index in S3 → schema-v2 bundle loaded into memory on cold start / refresh
  • Hugging Face–hosted model → streams provisional Thought tokens
  • SSE → searching → progress? → thinking? → token* → final or error

Backend API Design

  • POST /v1/query-stream: SSE streaming endpoint (public via edge only).
  • GET /v1/health: readiness (index loaded).
  • GET /v1/metrics: sanitized public metrics (no spend detail).
  • Bundle publication is a trusted publisher workflow — not an internet-facing reindex route.

RAG Pipeline (How Answers Are Grounded)

The system follows RAG with an explicit product contract: retrieve published evidence first, generate once, then verify protected claims before committing final.answer.

  • Ingestion: private Markdown with structured claims is compiled into a versioned S3 bundle.
  • Embedding: query embedding for vector search (paid path after budget reservation).
  • Index: vector_index.npy, documents.json, facts.json as one validated snapshot.
  • Retrieval: BM25 + cosine fusion via RRF → context assembly.
  • Generation: system prompt + evidence + visitor question → streamed draft (Thought).
  • Commit: only verified final.answer is authoritative; drafts are provisional.

Streaming Responses (SSE)

Responses use Server-Sent Events. Token frames are append-only drafts shown as provisional Thought text. Only final.answer is committed to the assistant bubble. Event names are searching, optional progress/thinking, token, then final or error — not generic chunk/done aliases.

Cost & Abuse Controls

  • Edge: Cloudflare Worker origin secret, WAF, IP rate limits, body size limits, kill switch.
  • App rate limiting: DynamoDB fixed windows (e.g. 5/minute, 20/hour) keyed by trusted client IP fingerprint.
  • Budget reservation: atomic worst-case reserve before embedding/inference; daily/monthly caps; fail closed if the store is unavailable.

Observability

Structured failure metrics (rate limit, budget, bundle, provider, etc.) and Lambda alarms publish to CloudWatch/SNS. Visitor prompt text is not written into metric dimensions; accepted Q&A may be retained ~30 days for debugging.

Deployment & CI/CD

  • Infrastructure-as-Code: AWS CDK (split query/publisher roles, reserved concurrency).
  • GitHub Actions: offline tests/security + deploy with smoke through https://api.dhavanshah.com.
  • Cold start: S3 index bundle loaded into memory; edge health warm reduces first-hit latency.

Operating Limits

  • App rate limits: production uses fixed windows such as 5/minute and 20/hour per trusted client fingerprint.
  • Spend caps: atomic reservation against daily/monthly budgets before paid embedding/inference.
  • Concurrency: query Lambda reserved concurrency is capped to bound blast radius under traffic spikes.
  • Latency: end-to-end times depend on cold starts and the provider; live smoke gates first SSE and final events rather than fixed marketing numbers.

Why This Project Is Valuable

This is an AI systems engineering project, not just a model demo. It demonstrates the practical building blocks needed for modern LLM products: grounded retrieval, streaming UX, cost controls, cloud-native deployment, and operational readiness.