Week 12 · Capstone

AI-Powered Data Platform — Architecture

The system that ties every prior week together: raw sources land in a data lake, get orchestrated and transformed into a warehouse, then split into two consumers — traditional analytics, and a RAG pipeline that turns the same warehouse data into something an LLM can reason over.

End-to-end flow

Rendered with Mermaid — each color marks one architectural layer, matching the legend below.

What: A single picture showing how every piece of the roadmap fits together into one system.
Where: The final capstone project of the 12-week plan.
When: Once you've learned the individual pieces — this shows how they connect.
How: Follow the arrows from raw sources on the left through to the AI Assistant at the bottom.

Scroll horizontally on small screens

Sources

API, CSV exports, PDFs — heterogeneous, arrive on their own schedules

S3 data lake

Raw landing zone, partitioned by source and date (Week 5)

Airflow

Schedules and sequences every downstream step (Week 7–8)

PySpark / dbt

Distributed transform + SQL modeling into warehouse shape (Week 9–10, 7–8)

Data warehouse

Star-schema fact/dim tables, the single source of truth (Week 4)

Analytics

BI dashboards, reporting — the traditional consumer

RAG → AI Assistant

Embeddings, vector search, and an LLM — your GenAI/LangChain differentiator

How to read it

What: A plain-English explanation of what each arrow and box in the diagram means.
Where: Directly below the diagram itself.
When: Read this after looking at the diagram once, so the boxes have context.
How: Read top to bottom — each bullet matches one stage of the pipeline.

Sources → S3 — every format lands untouched in the raw layer first; this is the ELT pattern from Week 4, deferring transformation until data is safely stored.
Airflow is the spine of the whole thing, not just one more box — it doesn't transform any data itself, it triggers and orders every step downstream, retries anything that fails, and gives you one place to watch the whole run happen.
PySpark / dbt is where raw data actually turns into something structured. PySpark handles anything too big for a single machine to process; dbt handles the SQL modeling that turns staging data into the warehouse's star schema.
The warehouse splits into two directions — this is the one idea worth remembering from this whole diagram: the same clean, modeled data feeds both a normal BI/dashboard stack and a RAG pipeline, instead of the AI assistant working off its own separate, untrustworthy copy of the data.
RAG path — warehouse data (or raw documents) gets turned into vectors ("embeddings"), stored in a vector database, and pulled back out based on similarity whenever someone asks a question. Those results get handed to an LLM so its answer is grounded in your real data — this whole flow is what "RAG" (retrieval-augmented generation) means.

▶ YouTube: Architecture diagrams with Mermaid.js ▶ YouTube: RAG architecture explained 📘 Mermaid.js Flowchart Syntax