Docker + Linux + Git

Linux essentials

What

The command-line operating system almost every server, container, and cloud machine runs on.

Where

Any time you're working inside a terminal — locally, in Docker, or on a remote server.

When

Constantly — this is baseline literacy for any data engineering job.

How

Learn a handful of commands (ls, cd, grep, find, curl) and combine them with pipes (|).

ls -la                     # list files, including hidden, with details
cd /var/log                # change directory
grep -r "ERROR" ./logs/    # search recursively for text
find . -name "*.csv"       # find files by pattern
curl -s https://api.example.com/orders | jq .   # hit an API from the shell
chmod +x run_pipeline.sh   # make a script executable
ps aux | grep python       # list running processes, filter for python
tail -f pipeline.log       # follow a log file live
df -h                      # disk space usage
top                        # live process/resource monitor

• Every Docker container, cloud VM, and CI runner you touch is Linux underneath — these commands are non-negotiable baseline literacy.
• grep + find are your primary debugging tools when a pipeline fails on a remote machine and you only have shell access.
• curl lets you sanity-check an API endpoint before writing any Python against it.
• chmod/permissions matter constantly in Docker — a script that isn't executable, or a mounted volume with the wrong owner, is a classic "works on my machine" bug.

Chaining commands together is where the shell actually earns its keep in day-to-day debugging — piping the output of one command into the next instead of running them separately:

# find the 5 largest files under a directory
find data/ -type f -exec du -h {} \; | sort -rh | head -5

# count how many ERROR lines happened per hour in a log
grep "ERROR" pipeline.log | cut -d' ' -f1-2 | uniq -c

# watch memory/CPU on a specific process while a pipeline runs
ps aux | grep pipeline.py | grep -v grep

# check disk space isn't the reason a job failed to write output
df -h /data

The idea behind all of these is command1 | command2 | command3 — the output of one command feeds straight into the next one. That lets you combine small, simple tools instead of needing one giant command that does everything. It's the same "small pieces you can check individually" idea as building a pipeline: grep filters, cut/sort/uniq reshape — basically the extract/transform/load steps from Week 4, just on the command line.

Git essentials

What

A tool for tracking changes to code over time, and collaborating with others.

Where

Every real codebase — pipelines, dbt models, and DAGs all live in Git.

When

From the very first line of code in any real project.

How

git add, git commit, git push to save and share changes; branches to work without disturbing others.

git clone https://github.com/you/de-roadmap.git
git checkout -b feature/airflow-dag        # branch
git add pipelines/orders_dag.py
git commit -m "Add orders extraction DAG"
git push origin feature/airflow-dag
git pull origin main
git merge main                              # merge main into current branch
git rebase main                             # replay your commits on top of main

• Branch — isolated line of work. Commit — a saved snapshot with a message. Push/pull — sync with the remote (GitHub/GitLab).
• Merge vs rebase — merge preserves history with a merge commit; rebase replays your commits on top of the target branch for a linear history. Know both; most teams standardize on one.
• Data pipeline code (DAGs, dbt models, SQL) lives in Git just like application code — expect PR review workflows in any real DE job.
• Practical habit: small, frequent commits with clear messages beat one giant commit at the end of the day.

Merge vs rebase, side by side — say main gained 2 new commits while you were working on feature/airflow-dag with 3 commits of your own:

# merge: creates a new "merge commit" that ties both histories together
git checkout feature/airflow-dag
git merge main
# history now shows both branches' commits interleaved, plus one merge commit —
# nothing is rewritten, so this is always safe even on a branch others are using

# rebase: replays your 3 commits on top of main's latest commit, one at a time
git checkout feature/airflow-dag
git rebase main
# history is now perfectly linear — looks as if you started your branch today —
# but your commits' hashes have changed, which is DANGEROUS if anyone else
# already pulled the old versions of those commits

Simple rule of thumb: rebase freely on a branch that's only yours, to keep history clean before opening a PR. Never rebase a branch that someone else has already pulled or built work on top of — the commit hashes change, and that causes a confusing mess for everyone else.

Everyday recovery commands worth knowing before you need them under pressure:

git status                       # what's changed, what's staged
git diff                         # see unstaged changes
git stash                        # temporarily shelve changes without committing
git stash pop                    # bring them back
git log --oneline -10            # last 10 commits, one line each
git restore --staged file.py     # unstage a file (keeps your edits)
git revert <commit-hash>         # undo a commit by creating a new opposite commit (safe on shared branches)

git revert is worth calling out specifically. Unlike reset --hard (which rewrites history and can lose work), revert just adds a brand-new commit that undoes a previous one. That makes it safe to use on main or any shared branch — which is why it's the standard way to undo a bad deploy.

Docker concepts

What

A way to package code and everything it needs to run into one portable “container.”

Where

Any project where “works on my machine” needs to become “works everywhere.”

When

When you need your pipeline to run identically on your laptop, in CI, and in production.

How

Write a Dockerfile describing the environment, then build and run it as a container.

• Docker solves "works on my machine" — your pipeline runs in an identical, isolated environment everywhere (your laptop, CI, production).
• Without volumes, any data written inside a container disappears when the container is removed — critical to know before you "lose" a database.
• Containers on the same Docker network can reach each other by service name (e.g. your Python container connects to postgres:5432, not localhost:5432).

Image vs container, made concrete — one image, many containers:

docker build -t my-pipeline .        # builds ONE image from your Dockerfile

docker run --name run1 my-pipeline   # container 1: an isolated running instance
docker run --name run2 my-pipeline   # container 2: a completely separate instance,
                                      # same image, independent filesystem/memory/process

docker ps                            # list currently running containers
docker ps -a                         # include stopped containers too
docker logs run1                     # see what container 1 printed
docker exec -it run1 bash            # get an interactive shell INSIDE a running container

docker exec -it <container> bash is probably the single most useful debugging command in Docker. It drops you into a live shell inside a running container, as if you'd logged into it directly — so you can look at files, check environment variables, or test a connection from inside the exact environment your code is actually running in.

Dockerfile

What

The recipe file Docker uses to build an image.

Where

One per project (or per service), usually at the project root.

When

Whenever you're packaging an app or pipeline to run in Docker.

How

FROM a base image, COPY your code in, RUN to install dependencies, CMD to say what runs on start.

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "pipeline.py"]

• FROM — base image to build on. WORKDIR — sets the working directory inside the container. COPY — brings files from your machine into the image.
• RUN executes at build time (installing dependencies); CMD defines what runs when the container starts.
• Copy requirements.txt and install dependencies before copying the rest of your code — Docker caches layers, so code changes won't force a slow dependency reinstall.
• Build and run: docker build -t my-pipeline . then docker run my-pipeline.

Layer caching, concretely — every instruction in a Dockerfile creates a cached "layer." Docker only re-runs an instruction (and everything after it) if that instruction or anything it depends on changed:

# BAD: copies everything first, so ANY code change invalidates the pip install layer too
COPY . .
RUN pip install --no-cache-dir -r requirements.txt

# GOOD: requirements.txt copied and installed FIRST — this layer only
# re-runs when requirements.txt itself changes, not on every code edit
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

With the "GOOD" ordering, editing pipeline.py and rebuilding skips the slow pip install step entirely. Docker notices requirements.txt hasn't changed, reuses that cached layer, and only re-runs COPY . . and whatever comes after it. On a real project with 20+ dependencies, that's the difference between a 2-second rebuild and a 2-minute one.

# multi-stage build: keep the final image small by discarding build-only tools
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /deps /usr/local/lib/python3.11/site-packages
COPY . .
CMD ["python", "pipeline.py"]

Multi-stage builds start to matter once a Dockerfile needs compilers or build tools just to install its dependencies (fairly common with some pandas/psycopg2 versions). The final image only keeps what you explicitly copy over with COPY --from=builder — none of the build tools tag along, so the image you actually ship stays smaller and faster to download.

docker-compose

What

A way to describe and run multiple containers together as one stack.

Where

Local development, or any multi-service setup — a database + your app + a scheduler.

When

Whenever your project needs more than one container working together.

How

Describe each service in a docker-compose.yml file, then run docker compose up.

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: ecommerce
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

  pipeline:
    build: .
    depends_on:
      - postgres
    environment:
      DB_HOST: postgres        # reach Postgres by service name, not localhost

  airflow:
    image: apache/airflow:2.9.0
    depends_on:
      - postgres
    ports:
      - "8080:8080"
    volumes:
      - ./dags:/opt/airflow/dags

volumes:
  pgdata:

• One command, docker compose up -d, spins up Postgres + your Python pipeline + Airflow together, wired to talk to each other.
• depends_on controls start order (not full readiness — for real projects add a healthcheck/wait step so the pipeline doesn't start before Postgres is ready to accept connections).
• volumes: keeps your Postgres data across restarts and lets Airflow pick up DAG files you edit locally without rebuilding the image.
• docker compose down stops everything; add -v to also wipe volumes (careful — that deletes your database data).

Fixing the "starts before it's ready" problem mentioned above: depends_on on its own only waits for the Postgres container to start — not for Postgres itself to finish setting up and start accepting connections, which can take a few extra seconds. A healthcheck closes that gap:

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

  pipeline:
    build: .
    depends_on:
      postgres:
        condition: service_healthy   # waits for the healthcheck to pass, not just container start

Without a healthcheck, the classic symptom is a pipeline container that fails on its very first run with "connection refused," then works fine if you just try again. The healthcheck fixes this for good, by making pipeline wait until pg_isready actually succeeds inside the Postgres container.

# everyday compose commands
docker compose up -d              # start everything in the background
docker compose logs -f airflow    # stream one service's logs
docker compose exec postgres psql -U postgres -d ecommerce   # open psql inside the running container
docker compose down                # stop and remove containers (volumes survive)
docker compose down -v             # also wipe volumes — deletes your Postgres data

Practice project

Run PostgreSQL + Python + Airflow locally with a single docker-compose.yml.

What

Postgres, your Python pipeline, and Airflow, all running together locally.

Where

Your own machine, using docker-compose.

When

After learning Docker basics — this is the practical payoff.

How

One docker-compose.yml with three services, wired together by service name.