Databricks

What Databricks actually is

A managed platform built by the original creators of Apache Spark, running on top of your existing AWS/Azure/GCP account.

• Not a replacement for Spark — it is Spark, managed: no cluster provisioning by hand, a notebook UI, built-in job scheduling, and a storage layer (Delta Lake) layered on top of plain Parquet.
• Marketed as a "lakehouse" — the idea is combining a data lake's cheap, flexible storage (like Week 5's S3) with a warehouse's reliability (like Week 1-2's Postgres), so you don't have to pick one or the other.
• One platform spans what used to be three separate tools in your roadmap: PySpark compute, Airflow-style job orchestration, and (via MLflow) ML/model tracking — plus governance via Unity Catalog.

When to reach for it (and when not to)

Here's the honest way to think about it: Databricks is basically PySpark with extra tools built in. If you already decided in Week 9-10 that you don't need Spark at all — your data fits comfortably in Postgres or Redshift — then you don't suddenly need Databricks either. It's asking you the exact same question ("do I actually need distributed compute?"), just with a nicer interface wrapped around it.

Architecture: control plane vs data plane

What

How Databricks splits responsibility — its own control plane, and your cloud account as the data plane.

Where

Behind the scenes of every Databricks workspace.

When

Worth knowing before a security/compliance conversation — your data never leaves your cloud account.

How

Databricks manages the UI/scheduler; your AWS/Azure account runs the actual compute and holds the data.

• This split matters for a very real reason: your raw data always stays inside your own cloud account's storage. Databricks' control plane just orchestrates compute — it's never actually where your data lives, which tends to be the first question any security review asks.
• A workspace is your team's environment — notebooks, jobs, and cluster configs live here, tied to one control-plane deployment per cloud region.
• This is the same idea you saw with Airflow — the thing that triggers work doesn't have to be where the data lives — and with IAM roles in Week 5, where a service borrows temporary access instead of storing its own credentials.

Notebooks

What

An interactive, shareable document mixing code, SQL, and notes.

Where

The main place you write and run code in Databricks.

When

For exploration, development, and even scheduled jobs.

How

Write cells in Python or SQL (switch with %sql), run them, and see results inline.

# Cmd 1 — a widget makes the notebook parameterizable, like an Airflow DAG's {{ ds }}
dbutils.widgets.text("run_date", "2026-08-18")
run_date = dbutils.widgets.get("run_date")

# Cmd 2 — switch language per cell with a magic command
%sql
SELECT * FROM analytics.orders WHERE order_date = '${run_date}'

# Cmd 3 — back to Python, using the SQL result
df = _sqldf   # Databricks exposes the last %sql cell's result as a DataFrame
df.groupBy("customer_id").sum("total_amount").display()

• One notebook can mix %python, %sql, %scala, and %md (markdown) cells — genuinely useful for the "explore in SQL, transform in Python" workflow instead of switching tools.
• Multiple people can co-edit a notebook live, with per-cell run history — a real difference from a local .py script or a Jupyter notebook on your laptop.
• Notebooks are version-controllable (Databricks Repos syncs a workspace folder to a real Git repository) — so the Week 6 Git workflow still applies, notebooks aren't a dead end for version control.

Clusters & compute

What

The actual compute — a group of machines — that runs your Spark code in Databricks.

Where

Attached to a notebook or a job.

When

All-purpose for interactive work; job clusters for scheduled production runs (cheaper).

How

Pick a cluster size, or let autoscaling adjust it, and set auto-termination to avoid paying for idle time.

# cluster config (JSON, via API or UI) — the same DataFrame code from Week 9-10
# runs unchanged regardless of which cluster type executes it
{
  "num_workers": 4,
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "i3.xlarge",
  "autoscale": {"min_workers": 2, "max_workers": 8},
  "autotermination_minutes": 30
}

• Autoscaling adds/removes workers within your min/max range based on load — the managed version of the partition/parallelism tuning you did by hand in Week 9-10.
• Auto-termination shuts an idle all-purpose cluster down after N minutes — the single most common cost-control setting people forget to set.
• Cluster policies let an admin restrict what configs a user can even choose (e.g. cap max workers, forbid certain instance types) — a governance lever, same spirit as the least-privilege IAM policies from Week 5.

Delta Lake Must know

Databricks' flagship contribution: ACID transactions, schema enforcement, and time travel layered on top of plain Parquet files sitting in S3/ADLS/GCS.

What

A storage format that adds reliable transactions and history on top of plain files in cloud storage.

Where

Any table you'd otherwise store as plain Parquet on S3.

When

Whenever you need safe concurrent writes, upserts, or the ability to look at past versions of a table.

How

Write with .format("delta"), then use MERGE INTO for upserts.

# write a Delta table — looks just like a Parquet write from Week 9-10, format is the only change
df.write.format("delta").mode("overwrite").save("s3://bucket/curated/orders")

# or register it as a table other people can query by name
df.write.format("delta").saveAsTable("analytics.orders")

# time travel — query the table exactly as it looked at a prior version or timestamp
spark.read.format("delta").option("versionAsOf", 5).load("s3://bucket/curated/orders")
spark.read.format("delta").option("timestampAsOf", "2026-08-01").table("analytics.orders")

# see the full change history of a table
spark.sql("DESCRIBE HISTORY analytics.orders").show()

Here's why this actually matters: plain Parquet files on S3 have no concept of a transaction. A job that crashes halfway through writing can leave a table half-finished, and two pipelines writing at the same time can corrupt each other's output. Delta Lake wraps every write in a real transaction log (_delta_log/), so a table is always either fully updated or completely untouched — even with several pipelines writing to it at once. It's solving the same "safe to retry" problem from Week 4, just at the storage level instead of in your own code.

# MERGE INTO — the Delta version of the upsert pattern from Week 4's idempotency section
from delta.tables import DeltaTable

target = DeltaTable.forName(spark, "analytics.orders")

target.alias("t").merge(
    updates_df.alias("s"),
    "t.order_id = s.order_id"
).whenMatchedUpdate(set={"total_amount": "s.total_amount", "updated_at": "s.updated_at"}) \
 .whenNotMatchedInsertAll() \
 .execute()

This is a built-in replacement for the hand-written INSERT ... ON CONFLICT DO UPDATE from Week 4, and even for dbt's snapshot SCD Type 2 logic from Week 7-8. MERGE INTO is the exact same idea — match on a key, update if found, insert if not — just running natively over Delta files instead of a SQL warehouse table.

Autoloader & the medallion architecture

What

A way to automatically pick up new files as they land in cloud storage, without re-scanning everything.

Where

The first step of ingesting raw files into Databricks.

When

When new files keep arriving in S3/ADLS and you want to process only the new ones.

How

spark.readStream.format("cloudFiles").load(path), with a checkpoint to remember what's been read.

# incrementally ingest new files landing in S3, without re-scanning the whole bucket each run
df = (
    spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "s3://bucket/checkpoints/orders_schema")
    .load("s3://bucket/raw/orders/")
)

(df.writeStream
   .format("delta")
   .option("checkpointLocation", "s3://bucket/checkpoints/orders")
   .trigger(availableNow=True)   # process everything currently sitting there, then stop — good for a scheduled batch-style run
   .table("bronze.orders"))

• Autoloader keeps track of which files it's already processed, using a checkpoint, so re-running only picks up new ones. It's the file-based version of the watermark pattern from Week 4 — without you having to manage a watermark table by hand.
• trigger(availableNow=True) makes a streaming job behave like a batch job — it runs once against everything currently available, then stops. Handy for scheduling from Airflow (Week 7-8) without leaving a stream running around the clock.
• Compare to Week 11's Kafka + Spark Structured Streaming: same streaming API, different source — cloudFiles reads incrementally from object storage instead of a Kafka topic.

Medallion architecture is Databricks' name for the same layered pattern from Week 4/5's raw → staging → curated:

Databricks SQL

A SQL-only experience on top of the same Delta tables — built for analysts, not just engineers writing PySpark.

What

SQL-only compute, built for analysts running queries and dashboards.

Where

The "Databricks SQL" part of the platform.

When

When the people using the data want to write plain SQL, not Python/Spark.

How

Point a SQL Warehouse at your Delta tables and query them like a normal database.

• SQL Warehouses are a separate type of compute from all-purpose or job clusters — built and billed specifically for running SQL queries, including a serverless option that starts up almost instantly, with no cluster spin-up wait.
• Every SQL skill from Week 1-2 transfers directly — the query editor, dashboards, and alerts all run standard SQL against Delta tables.
• This is Databricks competing directly with a traditional warehouse like Redshift or Snowflake for analysts — while still being the exact same lake your PySpark jobs already write to. There's no separate step to copy data into "the warehouse," because here, the lake is the warehouse.

Jobs & Workflows

Databricks' own orchestrator — a multi-task DAG of notebooks/scripts, roughly Airflow's DAG concept, but native to the platform.

What

Databricks' own way to schedule and chain tasks together.

Where

Inside a Databricks workspace.

When

For pipelines that live entirely within Databricks, or as one task triggered by a larger Airflow DAG.

How

Define tasks and their dependencies in the Workflows UI, or trigger the whole job from Airflow.

# Airflow DAG triggering a Databricks job instead of running Spark itself —
# the common real-world pattern: Airflow stays the org-wide orchestrator (Week 7-8),
# Databricks Workflows handles the internal task DAG of one specific job
from airflow.providers.databricks.operators.databricks import DatabricksRunNowOperator

run_orders_pipeline = DatabricksRunNowOperator(
    task_id="run_databricks_orders_job",
    databricks_conn_id="databricks_default",
    job_id=12345,
)

• Tasks in a Databricks Workflow can depend on each other, retry when they fail, and pass small values between each other — the same DAG/Task/Retry/XCom ideas from Week 7-8, just running inside Databricks' own scheduler instead of Airflow's.
• In practice, it's rarely "Airflow or Databricks Workflows" — it's often both. Airflow runs the big-picture pipeline across Postgres, S3, Databricks, Redshift, and whatever else, and triggers a Databricks job as just one task in that larger DAG. Databricks Workflows then handles the steps inside that one Spark job.

Unity Catalog

What

A central place to control who can access which tables, across every workspace.

Where

Sits above all your catalogs, schemas, and tables.

When

As soon as more than one team is sharing the same data.

How

Grant access with GRANT SELECT ON catalog.schema.table TO group.

-- three-level namespace: catalog.schema.table (one level deeper than plain Postgres schema.table)
SELECT * FROM production_catalog.analytics.fct_orders;

GRANT SELECT ON production_catalog.analytics.fct_orders TO `data-analysts`;

• A centralized governance layer across every workspace — one place to manage who can access which catalog/schema/table, instead of per-workspace permissions.
• Automatic data lineage keeps track of which notebooks and jobs read and wrote each table. It's Databricks' built-in version of what dbt's lineage graph gives you (Week 7-8) for SQL models — except it covers PySpark jobs too.
• This is the same access-control mindset as IAM (Week 5) and dbt's relationships tests (Week 7-8) — give people only the access they need, and keep everything traceable — just applied at the catalog level instead of a single cloud account or warehouse.

MLflow — your GenAI angle

What

A tool for tracking machine learning (and LLM/RAG) experiments — what you tried, and how well it worked.

Where

Any notebook or job doing ML or GenAI work.

When

Any time you're comparing different models, prompts, or settings.

How

Wrap your code in mlflow.start_run(), then log parameters and metrics as you go.

import mlflow

with mlflow.start_run():
    mlflow.log_param("chunk_size", 500)
    mlflow.log_param("embedding_model", "text-embedding-3-small")
    mlflow.log_metric("retrieval_precision", 0.87)
    mlflow.langchain.log_model(rag_chain, artifact_path="rag_chain")   # log a LangChain pipeline directly

• MLflow (built by Databricks, usable standalone too) tracks experiments, parameters, and metrics — and has first-class support for logging LangChain/LLM pipelines, not just traditional ML models.
• Databricks also includes Vector Search, a managed vector database built right into Delta tables. That means the embeddings/vector-DB step in your capstone's RAG pipeline can live on the same platform as the data pipeline feeding it, instead of needing a separate standalone vector database.
• This is the real, concrete version of "Data Engineer → AI Data Engineer" from your roadmap. Databricks is one of the few platforms where the data pipeline, the warehouse, and the RAG/LLM tooling are genuinely one system, instead of three separate tools you have to wire together yourself.

Practice project

Sign up for Databricks Community Edition or a free trial, and rebuild your Week 4/5 pipeline as a medallion architecture.

What

A full mini pipeline using the medallion architecture (Bronze/Silver/Gold).

Where

A free Databricks trial or Community Edition account.

When

After you're comfortable with Delta Lake and Autoloader.

How

Ingest raw data into Bronze, clean it into Silver, and aggregate it into Gold tables.