In this post we are going to cover every core Claude Code concept a data engineer needs, using the same sales project as part 1. We go through permission modes and rules, plan mode, the CLAUDE.md file, slash commands, context, checkpoints, models and effort, and the last step is headless runs with claude -p.
These concepts decide how Claude Code behaves: when it asks before it acts, what it remembers between sessions, how much it keeps in mind, and how you undo a change. Along the way we do one real task. A new orders export just arrived and it has bad rows, so we plan and build a data check for it and run it for real. Every command output in this post comes from really running the code.
Part 2 of the Claude for Data Engineers series on TechBrothersIT.
- Let's create some sample data
- Step 1: Permission modes
- Step 2: Permission rules
- Step 3: Plan mode for a real data check
- Step 4: CLAUDE.md and memory
- Step 5: Slash commands
- Step 6: Context and /compact
- Step 7: Checkpoints and /rewind
- Step 8: Models and effort
- Step 9: Headless runs with claude -p
- The full script
Let's create some sample data
We start where part 1 ended: the sales-pipeline project with its CLAUDE.md and revenue_by_country.py. One thing changed. The new data/orders.csv export has 18 orders instead of 16, and the two new rows are bad on purpose: order 5017 has an amount of -20.00, and order 5018 uses customer C09, who is not in customers.csv.
# sales-pipeline
Daily sales reporting for the online shop.
- data/ raw CSV exports from the shop
- sql/ reporting queriesduckdb==2.0.0order_id,order_date,customer_id,status,amount
5001,2026-10-01,C01,completed,120.00
5002,2026-10-01,C02,completed,75.50
5003,2026-10-02,C03,cancelled,200.00
5004,2026-10-02,C01,completed,42.00
5005,2026-10-03,C04,completed,310.00
5006,2026-10-03,C05,refunded,89.99
5007,2026-10-04,C02,completed,150.00
5008,2026-10-04,C06,completed,64.25
5009,2026-10-05,C03,completed,99.00
5010,2026-10-05,C04,cancelled,45.00
5011,2026-10-06,C05,completed,210.00
5012,2026-10-06,C06,completed,38.75
5013,2026-10-07,C01,completed,500.00
5014,2026-10-07,C02,refunded,60.00
5015,2026-10-08,C03,completed,85.50
5016,2026-10-08,C04,completed,130.00
5017,2026-10-09,C02,completed,-20.00
5018,2026-10-09,C09,completed,75.00customer_id,name,country
C01,Ava Patel,USA
C02,Liam Chen,Canada
C03,Sofia Rossi,USA
C04,Noah Weber,Germany
C05,Mia Santos,Canada
C06,Omar Haddad,Germany-- Orders and amount per status
SELECT status,
count(*) AS orders,
sum(amount) AS amount
FROM read_csv('data/orders.csv', header = true)
GROUP BY status
ORDER BY status;# CLAUDE.md
This file provides guidance to Claude Code
(claude.ai/code) when working with code in this
repository.
## Project
Daily sales reporting for an online shop. Raw CSV
exports in `data/` are queried with DuckDB.
## Layout
- `data/orders.csv`: one row per order (order_id,
order_date, customer_id, status, amount)
- `data/customers.csv`: customer_id, name, country
- `sql/`: reporting queries in DuckDB SQL
## Commands
- Install: `pip install -r requirements.txt`
- Run a query: `duckdb -f sql/orders_by_status.sql`
## Conventions
- Use DuckDB for queries (version pinned in
`requirements.txt`).
- Keep reporting SQL in `sql/`, one query per file.
## Business rules
- Revenue counts only orders with status
`completed`. Cancelled and refunded orders are
excluded.import duckdb
con = duckdb.connect()
con.execute("CREATE VIEW orders AS FROM 'data/orders.csv'")
con.execute("CREATE VIEW customers AS FROM 'data/customers.csv'")
def scalar(sql):
return con.execute(sql).fetchall()[0][0]
total = scalar("SELECT count(*) FROM orders")
done = scalar("SELECT count(*) FROM orders WHERE status = 'completed'")
print(f"Orders read: {total}, completed: {done}")
# Business rule (CLAUDE.md): revenue counts completed orders only
con.sql("""
SELECT c.country,
count(*) AS orders,
round(sum(o.amount), 2) AS revenue
FROM orders o
JOIN customers c USING (customer_id)
WHERE o.status = 'completed'
GROUP BY c.country
ORDER BY revenue DESC
""").show()Step 1: Permission modes
Permission modes decide when Claude Code asks you before it acts. The current mode is shown in the status line at the bottom, and Shift+Tab moves to the next one.
New sessions start in auto mode. Claude checks each action for risk, runs the low risk ones, and blocks the rest. It reduces prompts, but it does not guarantee safety.
Press Shift+Tab once and you are in manual mode. Claude asks before every file edit and every shell command that is not read only.
Press it again for accept edits. File edits are approved for you, and so are simple file commands like mkdir, touch, mv and cp. Other commands still ask.
Press it again for plan mode. Claude can read and explore, but it does not change your files until you approve a plan. One more press brings you back to auto mode.
Shift+Tab cycle
auto → manual → accept edits → plan → auto
On a production repo, start in manual mode until you trust the setup.
Step 2: Permission rules
Modes are the big switch. Permission rules are the fine control.
When you choose Yes, and don't ask again at a prompt, Claude Code saves an allow rule in the settings.local.json file, in the .claude folder of your repo. You can also write rules yourself. Here is an example for our project.
Allow and deny rules
{
"permissions": {
"allow": [
"Bash(python check_orders.py)",
"Bash(duckdb *)"
],
"deny": [
"Read(./.env)",
"Bash(rm *)"
]
}
}The data check and any DuckDB command are allowed, so Claude runs them without asking. Reading the .env file, or running any rm command, is denied. Deny rules are checked first, so an allow rule can never open up something you denied.
Put team rules in .claude/settings.json and commit it, so everyone on the repo gets the same rules. Keep personal ones in settings.local.json. The /permissions command shows all of them in one place.
Step 3: Plan mode for a real data check
Now let's use plan mode on a real task. The new orders export arrived, and we want a check before it goes into any report. We press Shift+Tab three times to get to plan mode, then describe the task.
New orders export arrived. Plan check_orders.py: fail on negative amounts or unknown customersIn plan mode Claude only reads. It reads data/orders.csv, data/customers.csv and the existing revenue_by_country.py. Then it shows its plan: create check_orders.py with DuckDB, one check for orders with a negative amount, one check for orders whose customer is not in customers.csv, print each problem, exit with code 1 if any check fails, and run it once on the new export.
Claude asks if we are ready to code. We choose Yes, manually approve edits. That approves the plan and puts us in manual mode. Next it asks to create the file, and we choose Yes. Then it asks to run the script, and we choose Yes again. Here is the check and the real run.
import sys
import duckdb
con = duckdb.connect()
con.execute("CREATE VIEW orders AS FROM 'data/orders.csv'")
con.execute("CREATE VIEW customers AS FROM 'data/customers.csv'")
CHECKS = {
"negative amount":
"SELECT order_id, customer_id, amount"
" FROM orders WHERE amount < 0",
"unknown customer":
"SELECT order_id, customer_id, amount"
" FROM orders ANTI JOIN customers"
" USING (customer_id)",
}
total = con.execute("SELECT count(*) FROM orders").fetchall()[0][0]
print(f"Checked {total} orders")
problems = 0
for name, sql in CHECKS.items():
for order_id, customer, amount in con.execute(sql).fetchall():
print(f"FAIL {name}: order {order_id}, {customer}, {amount}")
problems += 1
print(f"{problems} problem(s) found")
sys.exit(1 if problems else 0)python check_orders.pyChecked 18 orders
FAIL negative amount: order 5017, C02, -20.0
FAIL unknown customer: order 5018, C09, 75.0
2 problem(s) foundThe run fails on purpose. It checked 18 orders and found 2 problems: order 5017 for C02 has a negative amount of -20.0, and order 5018 points to customer C09 with 75.0, who is not in the customers file. The script exits with code 1, so a pipeline step that runs it would stop there.
For anything bigger than a one line fix, plan first. Reading a plan takes a minute. Undoing a wrong change in a pipeline takes much longer.
Step 4: CLAUDE.md and memory
The CLAUDE.md file is how Claude Code remembers your project between sessions. We want every future session to know about the new check, so we just tell Claude to add it.
Add to CLAUDE.md: the check_orders.py command, and never edit data/ files by handClaude edits CLAUDE.md. It adds the data check command under Commands, and a rule never to edit the raw files in data/ by hand under Conventions. Here is the file after the edit.
# CLAUDE.md
This file provides guidance to Claude Code
(claude.ai/code) when working with code in this
repository.
## Project
Daily sales reporting for an online shop. Raw CSV
exports in `data/` are queried with DuckDB.
## Layout
- `data/orders.csv`: one row per order (order_id,
order_date, customer_id, status, amount)
- `data/customers.csv`: customer_id, name, country
- `sql/`: reporting queries in DuckDB SQL
## Commands
- Install: `pip install -r requirements.txt`
- Run a query: `duckdb -f sql/orders_by_status.sql`
- Data checks: `python check_orders.py` (exit 1 means
a problem was found). Run it after every change.
## Conventions
- Use DuckDB for queries (version pinned in
`requirements.txt`).
- Keep reporting SQL in `sql/`, one query per file.
- Never edit files in `data/` by hand: they are raw
exports from the shop.
## Business rules
- Revenue counts only orders with status
`completed`. Cancelled and refunded orders are
excluded.Older tutorials show a # shortcut for adding memory. It was removed, so just ask Claude like this, or open the files with /memory.
Three levels of CLAUDE.md
~/.claude/CLAUDE.md: you, every project
./CLAUDE.md: the team (commit it)
CLAUDE.local.md: just you (gitignore it)
There are three levels you will use: one in your home folder for all your projects, one in the repo shared with your team, and CLAUDE.local.md just for you. All of them load when a session starts.
Keep each CLAUDE.md under about 200 lines, and put in only the rules Claude really needs.
Step 5: Slash commands
Type a slash at the prompt and the menu shows every command. These are the ones you will use most.
Eight commands to know
| Command | What it does |
|---|---|
/init | Create CLAUDE.md for this project |
/memory | Edit CLAUDE.md files and auto memory |
/permissions | View and edit allow and deny rules |
/model · /effort | Pick the model and the effort level |
/context | See what fills the context window |
/compact | Summarize the conversation to free context |
/clear | Start a new session with empty context |
/rewind | Go back to an earlier checkpoint |
/init creates the CLAUDE.md file, and /memory edits it. /permissions shows your rules. /model and /effort pick the model and how hard it thinks. /context shows what fills the context window, and /compact summarizes the conversation to free up space. /clear starts a fresh session, and /rewind goes back to an earlier point.
Two more are worth knowing. /resume picks up an old conversation, and /status shows your version, model and account.
Step 6: Context and /compact
Everything in a session lives in Claude's context window: your messages, the files it read, command output, and the CLAUDE.md file.
What fills the context
Your messages · files Claude read · command output · CLAUDE.md · auto memory · skills
When the window gets close to full, Claude Code compacts it for you. It summarizes the old parts and keeps going. You can also do it yourself. In our session we already have the plan, the check and the CLAUDE.md edit, so let's compact it.
/compactClaude summarizes the conversation so far and reports that the conversation was compacted. You can add a few words after the command to say what to keep.
Steer the summary/compact focus on the order checks
When you switch to a different task, use /clear instead. A fresh context gives better answers than a long, mixed one.
Step 7: Checkpoints and /rewind
Claude Code saves a checkpoint for every prompt you send. To go back, press Esc twice on an empty prompt, or type /rewind.
Go back to any prompt
- A checkpoint for every prompt you send
- Open with Esc Esc on an empty prompt, or
/rewind - Restore code and conversation, only one, or summarize
- Not tracked: files changed by Bash (
rm,mv,cp) - Not undone: databases, APIs, deployments. Use git.
You can restore the code and the conversation, only the conversation, or only the code. You can also summarize from a point to free up context.
Know the limits. File changes made by shell commands, like rm, mv or cp, are not tracked. And anything outside your files, like a table you dropped in a database, cannot be rewound.
Checkpoints are not a backup. Use git and a dev database for real safety.
Step 8: Models and effort
Opus 5.5 is the default model in Claude Code. Switch with /model, or start with claude --model sonnet. Alt+P (Option+P on a Mac) switches inside a session. opusplan uses Opus to plan and Sonnet to build.
Pick the model, then the effort
opus: Opus 5.5 (default)sonnet: Sonnet 5.5haiku: Haiku 5.5opusplan: Opus plans, Sonnet builds/model·--model· Alt+P
- low · medium (Opus 5.5 default)
- high · xhigh · max
/effort highfor a tricky bug- ultrathink in a prompt: one deeper turn
Effort is how hard the model thinks: low, medium, high, xhigh and max. Opus 5.5 defaults to medium. Raise it with /effort for a tricky bug, or put the word ultrathink in one prompt for deeper reasoning on just that turn.
Step 9: Headless runs with claude -p
claude -p runs one task without the interactive screen, prints the answer and exits. This makes Claude Code a command you can put in a script or a scheduled job.
One task, then exit
| What | Command |
|---|---|
| One task, print, exit | claude -p "explain what check_orders.py does" |
| Allow the tools it needs | claude -p "run the order checks" --allowedTools "Bash,Read" |
| A result a script can read | claude -p "list the files in data" --output-format json |
| Pipe a file in | cat check_errors.txt | claude -p 'explain the root cause' |
| CI: skip local setup (API key) | claude --bare -p "explain what check_orders.py does" |
Headless runs start in manual mode, and nobody is there to say yes. So allow the tools the task needs with --allowedTools. --output-format json gives you a result a script can read. And you can pipe a file in, like an error log, and ask what went wrong.
For CI pipelines, look at the --bare flag. It skips hooks, plugins, MCP servers and the CLAUDE.md file, and it needs an API key. We come back to it in the CI part of this series.
That is every core Claude Code concept: modes and rules, plan mode, CLAUDE.md, slash commands, context, checkpoints, models and headless runs. In the next part we use Claude to write, debug and speed up SQL against a real Postgres database.
Key points to remember
- Auto, manual, accept edits, plan
- Allow and deny rules
- Plan first, then build
- CLAUDE.md: tell Claude to edit it
- Slash commands to know
/compactand/clear- Checkpoints: Esc Esc
- Models, effort,
claude -p
The full script
This bash script writes every file Claude wrote, in the same order, and runs every command, so you can reproduce the results without Claude.
Show the full script (every file and every command, in order)
#!/usr/bin/env bash
# Claude for Data Engineers #2: Every Claude Code Concept Explained (for Data Engineers)
# Rebuilds the demo project without Claude: writes every file Claude wrote, in the same order,
# and runs every command from the video. Run it in an empty folder.
# Tools used in the video: Python 3.13, DuckDB 2.0.0, pandas 3.0.5, PySpark 4.3.0 (local mode, Java 21),
# dbt-core 1.12.6 with dbt-duckdb 1.11.0, PostgreSQL 16. Install the ones this project uses first.
mkdir -p sales-pipeline && cd sales-pipeline
# ---------- starting files ----------
cat > README.md <<'CLAUDE_DE_EOF'
# sales-pipeline
Daily sales reporting for the online shop.
- data/ raw CSV exports from the shop
- sql/ reporting queries
CLAUDE_DE_EOF
cat > requirements.txt <<'CLAUDE_DE_EOF'
duckdb==2.0.0
CLAUDE_DE_EOF
mkdir -p data
cat > data/orders.csv <<'CLAUDE_DE_EOF'
order_id,order_date,customer_id,status,amount
5001,2026-10-01,C01,completed,120.00
5002,2026-10-01,C02,completed,75.50
5003,2026-10-02,C03,cancelled,200.00
5004,2026-10-02,C01,completed,42.00
5005,2026-10-03,C04,completed,310.00
5006,2026-10-03,C05,refunded,89.99
5007,2026-10-04,C02,completed,150.00
5008,2026-10-04,C06,completed,64.25
5009,2026-10-05,C03,completed,99.00
5010,2026-10-05,C04,cancelled,45.00
5011,2026-10-06,C05,completed,210.00
5012,2026-10-06,C06,completed,38.75
5013,2026-10-07,C01,completed,500.00
5014,2026-10-07,C02,refunded,60.00
5015,2026-10-08,C03,completed,85.50
5016,2026-10-08,C04,completed,130.00
5017,2026-10-09,C02,completed,-20.00
5018,2026-10-09,C09,completed,75.00
CLAUDE_DE_EOF
mkdir -p data
cat > data/customers.csv <<'CLAUDE_DE_EOF'
customer_id,name,country
C01,Ava Patel,USA
C02,Liam Chen,Canada
C03,Sofia Rossi,USA
C04,Noah Weber,Germany
C05,Mia Santos,Canada
C06,Omar Haddad,Germany
CLAUDE_DE_EOF
mkdir -p sql
cat > sql/orders_by_status.sql <<'CLAUDE_DE_EOF'
-- Orders and amount per status
SELECT status,
count(*) AS orders,
sum(amount) AS amount
FROM read_csv('data/orders.csv', header = true)
GROUP BY status
ORDER BY status;
CLAUDE_DE_EOF
cat > CLAUDE.md <<'CLAUDE_DE_EOF'
# CLAUDE.md
This file provides guidance to Claude Code
(claude.ai/code) when working with code in this
repository.
## Project
Daily sales reporting for an online shop. Raw CSV
exports in `data/` are queried with DuckDB.
## Layout
- `data/orders.csv`: one row per order (order_id,
order_date, customer_id, status, amount)
- `data/customers.csv`: customer_id, name, country
- `sql/`: reporting queries in DuckDB SQL
## Commands
- Install: `pip install -r requirements.txt`
- Run a query: `duckdb -f sql/orders_by_status.sql`
## Conventions
- Use DuckDB for queries (version pinned in
`requirements.txt`).
- Keep reporting SQL in `sql/`, one query per file.
## Business rules
- Revenue counts only orders with status
`completed`. Cancelled and refunded orders are
excluded.
CLAUDE_DE_EOF
cat > revenue_by_country.py <<'CLAUDE_DE_EOF'
import duckdb
con = duckdb.connect()
con.execute("CREATE VIEW orders AS FROM 'data/orders.csv'")
con.execute("CREATE VIEW customers AS FROM 'data/customers.csv'")
def scalar(sql):
return con.execute(sql).fetchall()[0][0]
total = scalar("SELECT count(*) FROM orders")
done = scalar("SELECT count(*) FROM orders WHERE status = 'completed'")
print(f"Orders read: {total}, completed: {done}")
# Business rule (CLAUDE.md): revenue counts completed orders only
con.sql(""
SELECT c.country,
count(*) AS orders,
round(sum(o.amount), 2) AS revenue
FROM orders o
JOIN customers c USING (customer_id)
WHERE o.status = 'completed'
GROUP BY c.country
ORDER BY revenue DESC
"").show()
CLAUDE_DE_EOF
# ---------- step 1: Plan mode: a data check for the new export ----------
cat > check_orders.py <<'CLAUDE_DE_EOF'
import sys
import duckdb
con = duckdb.connect()
con.execute("CREATE VIEW orders AS FROM 'data/orders.csv'")
con.execute("CREATE VIEW customers AS FROM 'data/customers.csv'")
CHECKS = {
"negative amount":
"SELECT order_id, customer_id, amount"
" FROM orders WHERE amount < 0",
"unknown customer":
"SELECT order_id, customer_id, amount"
" FROM orders ANTI JOIN customers"
" USING (customer_id)",
}
total = con.execute("SELECT count(*) FROM orders").fetchall()[0][0]
print(f"Checked {total} orders")
problems = 0
for name, sql in CHECKS.items():
for order_id, customer, amount in con.execute(sql).fetchall():
print(f"FAIL {name}: order {order_id}, {customer}, {amount}")
problems += 1
print(f"{problems} problem(s) found")
sys.exit(1 if problems else 0)
CLAUDE_DE_EOF
python check_orders.py # expected to fail
# ---------- step 2: Tell Claude to update CLAUDE.md ----------
cat > CLAUDE.md <<'CLAUDE_DE_EOF'
# CLAUDE.md
This file provides guidance to Claude Code
(claude.ai/code) when working with code in this
repository.
## Project
Daily sales reporting for an online shop. Raw CSV
exports in `data/` are queried with DuckDB.
## Layout
- `data/orders.csv`: one row per order (order_id,
order_date, customer_id, status, amount)
- `data/customers.csv`: customer_id, name, country
- `sql/`: reporting queries in DuckDB SQL
## Commands
- Install: `pip install -r requirements.txt`
- Run a query: `duckdb -f sql/orders_by_status.sql`
- Data checks: `python check_orders.py` (exit 1 means
a problem was found). Run it after every change.
## Conventions
- Use DuckDB for queries (version pinned in
`requirements.txt`).
- Keep reporting SQL in `sql/`, one query per file.
- Never edit files in `data/` by hand: they are raw
exports from the shop.
## Business rules
- Revenue counts only orders with status
`completed`. Cancelled and refunded orders are
excluded.
CLAUDE_DE_EOF
Part 3: Write, Debug and Speed Up SQL with Claude
The project and its data are invented for teaching. Every file on this page is the real file from the demo project, and every output is the real output of the command above it, captured when the video was made (each step ran twice in a fresh folder and the outputs were compared). The Claude screens in the video are a simplified recreation for teaching. Always review what Claude changes and runs. Claude plans, features and limits are current as of November 2026; check the official Claude and Claude Code documentation before you rely on them. Part of the Claude for Data Engineers series on TechBrothersIT.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.