Don't learn all of AWS. Focus on the handful of services that show up in almost every real data pipeline: IAM, S3 (the most important one), Glue, Lambda, CloudWatch, and Redshift.
Controls who (or what service) can do what, on which AWS resources.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::my-de-bucket/raw/*"
}
]
}"Action": "*" in real pipelines.Let's read that policy line by line: "Effect": "Allow" says this rule grants access (the other option, "Deny", always wins if both apply to the same thing). "Action" lists exactly which operations are allowed — s3:GetObject means "download a file," s3:PutObject means "upload a file." Notice s3:DeleteObject isn't listed, so whoever has this policy can read and write, but never delete. "Resource" limits all of this to one folder in one bucket — not the whole account's S3.
How a role actually gets used — this is the part that trips people up coming from user/password thinking: a Glue job doesn't have a username and password for S3 at all. Instead:
# 1. create a role that Glue is allowed to "assume"
# 2. attach the policy above to that role
# 3. when you configure the Glue job, you specify the role's ARN, e.g.:
# arn:aws:iam::123456789012:role/GlueOrdersJobRole
# 4. Glue automatically gets temporary, auto-rotating credentials scoped to exactly
# that policy for the duration of the job run — nothing to store, rotate, or leakThis "assume a role" approach is why roles are almost always preferred over long-lived access keys for anything a service does on your behalf — there's no fixed secret sitting in a config file that could ever leak out.
Object storage — the landing zone for almost every data pipeline. This is the one AWS service you should know cold.
# AWS CLI basics
aws s3 mb s3://my-de-bucket
aws s3 cp orders.csv s3://my-de-bucket/raw/orders/orders.csv
aws s3 sync ./local_data/ s3://my-de-bucket/raw/
aws s3 ls s3://my-de-bucket/raw/ --recursive
# read directly from Python (boto3)
import boto3
s3 = boto3.client("s3")
s3.upload_file("orders.csv", "my-de-bucket", "raw/orders/orders.csv")
obj = s3.get_object(Bucket="my-de-bucket", Key="raw/orders/orders.csv")
data = obj["Body"].read()COPYs from it."There are no real folders" trips almost everyone up the first time. raw/orders/dt=2026-08-17/orders.json looks like a file path, but S3 actually stores it as one single object, with that whole string as its name (its "key"). There's no real raw folder underneath. The AWS Console just splits that key on / to draw folder-looking breadcrumbs for you. This is exactly why the prefix-based layout below works so well: tools can quickly list "everything starting with raw/orders/dt=2026-08-17/" without needing a real folder tree.
# list only objects under one prefix — cheap, doesn't scan the whole bucket
aws s3 ls s3://my-de-bucket/raw/orders/dt=2026-08-17/ --recursive
# generate a temporary, expiring URL to share one object without making the bucket public
aws s3 presign s3://my-de-bucket/curated/report.csv --expires-in 3600
Storage classes are a real way to save money, not just trivia. A file in S3 Standard costs roughly 5x more per month than the same file in S3 Glacier — but getting a Glacier file back can take minutes to hours instead of being instant. A common setup is a lifecycle rule that automatically moves files from Standard to cheaper storage as they get older — for example, raw data past 90 days old that you're only keeping around for compliance, and almost never actually query.
Organize S3 keys by layer and source so every tool downstream knows where to look.
dt=2026-08-17/) lets tools like Glue/PySpark/Athena scan only the relevant slice instead of the whole bucket — huge cost/speed win at scale.source/dt=YYYY-MM-DD/file) so every pipeline that reads/writes the bucket agrees on structure.Why write dt=2026-08-17 instead of just 2026-08-17? This key=value style (called Hive-style partitioning) isn't just a style choice — Glue, Athena, and Spark all recognize this exact pattern automatically, and turn it into a real, filterable column:
# with Hive-style partitioning, this Spark read...
df = spark.read.parquet("s3://my-de-bucket/raw/orders/")
# ...automatically exposes `dt` as a real column, and this filter
# is pushed down to only read the matching S3 prefixes — never touches other dates
df.filter(df.dt == "2026-08-17")Without the dt= naming, Spark would have to open every single file and check its contents just to find the date. With it, the date is right there in the path, so filtering by date just means listing fewer files — no need to open and scan them at all. This is called partition pruning, and on a large data lake it can easily be a 10-100x difference in cost.
AWS's managed ETL service — Spark under the hood, plus a metadata catalog.
A minimal Glue job — this is literally PySpark (Week 9-10) with a small wrapper for reading Glue job arguments and wiring up the Spark context Glue provides:
import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session
df = spark.read.parquet("s3://my-de-bucket/raw/orders/")
result = df.groupBy("customer_id").sum("total_amount")
result.write.mode("overwrite").parquet("s3://my-de-bucket/curated/orders_by_customer/")What Glue actually adds, compared to just running this script on your own server, happens around the code: AWS sets up the Spark cluster and tears it back down for you — you never manage a server yourself. And once a Crawler has scanned curated/orders_by_customer/, the Glue Data Catalog means that output is instantly queryable by name from Athena or Redshift — no extra step to register its schema anywhere.
import json, boto3
def lambda_handler(event, context):
# triggered automatically when a new file lands in S3
bucket = event["Records"][0]["s3"]["bucket"]["name"]
key = event["Records"][0]["s3"]["object"]["key"]
print(f"New file: s3://{bucket}/{key}")
# e.g. kick off validation, trigger a Glue job, notify downstream
return {"statusCode": 200}A realistic use in a DE pipeline: validate a file's structure the instant it lands, before an expensive downstream Glue job wastes compute on something malformed:
import json, boto3
s3 = boto3.client("s3")
REQUIRED_COLUMNS = {"order_id", "customer_id", "total_amount"}
def lambda_handler(event, context):
bucket = event["Records"][0]["s3"]["bucket"]["name"]
key = event["Records"][0]["s3"]["object"]["key"]
obj = s3.get_object(Bucket=bucket, Key=key)
first_record = json.loads(obj["Body"].readline())
if not REQUIRED_COLUMNS.issubset(first_record.keys()):
# move the bad file aside instead of letting it enter the pipeline
s3.copy_object(Bucket=bucket, CopySource=f"{bucket}/{key}", Key=key.replace("raw/", "quarantine/"))
s3.delete_object(Bucket=bucket, Key=key)
return {"statusCode": 400, "body": "Schema check failed, file quarantined"}
return {"statusCode": 200, "body": "Schema OK"}This is the same "quarantine bad data instead of crashing the whole pipeline" idea from Week 4's validation section — just running as an S3-triggered gatekeeper instead of a step inside your Python/dbt pipeline. It's cheap, fires in milliseconds, and stops obviously broken files before a 20-minute Glue job even gets a chance to start.
# tail a Lambda function's logs from the CLI while debugging
aws logs tail /aws/lambda/orders-schema-check --follow
# a CloudWatch alarm: page someone if a Lambda's error count exceeds 0 in a 5-minute window
aws cloudwatch put-metric-alarm \
--alarm-name orders-lambda-errors \
--metric-name Errors --namespace AWS/Lambda \
--dimensions Name=FunctionName,Value=orders-schema-check \
--statistic Sum --period 300 --threshold 0 \
--comparison-operator GreaterThanThreshold --evaluation-periods 1Every AWS service on this page writes here automatically: a Glue job's print() output, a Lambda's error messages, a slow-query log from Redshift. Keep this mental model: a log group is one per resource (one per Lambda function, say), and a log stream is one per individual run inside that group. So debugging one failed run means finding its stream inside the right group — not searching through everything at once.
COPY analytics.orders
FROM 's3://my-de-bucket/curated/orders/'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftLoadRole'
FORMAT AS PARQUET;COPY command is the standard bulk-load pattern: pull directly from S3 in parallel, far faster than row-by-row INSERT.Why COPY beats INSERT once data gets big: this is the same idea as psycopg2's copy_expert from Week 3, just spread across many machines. Redshift has multiple compute nodes working together, and COPY lets every one of them load a different slice of the S3 data at the same time. A loop of single-row INSERTs only ever uses one connection and one node, one row at a time — for a multi-GB load, that's the difference between minutes and hours.
Distribution and sort keys — the two warehouse-specific tuning levers that don't exist in plain Postgres:
CREATE TABLE analytics.fact_orders (
order_id BIGINT,
customer_key BIGINT,
order_date DATE,
amount NUMERIC(10,2)
)
DISTKEY(customer_key) -- rows with the same customer_key land on the same compute node
SORTKEY(order_date); -- data is physically stored sorted by date on each nodeDISTKEY decides which node each row lives on. Pick a column you join on often (like customer_key), so a join against dim_customers doesn't need to move data across nodes over the network — the same goal as Spark's broadcast joins in Week 9-10. SORTKEY lets Redshift skip whole chunks of data that fall outside a query's date range, similar to the S3 partition pruning above — a query filtering on order_date can skip most of the table instead of scanning all of it.