Claude for Data Engineers · Part 10 of 10
In this post we are going to cover a complete data project with Claude Code, from a raw CSV export to a dashboard. We plan it first, profile the raw data, load it, clean it and model it in layers, test it with real data checks, build an HTML dashboard, and the last step is one command that runs the whole pipeline.
This is the last part of the series, so we use the things we learned along the way: plan mode, the CLAUDE.md file, real runs, and checks that stop bad data. An end to end project is where these come together. Claude Code reads your files, writes the SQL and Python, runs every stage in your terminal, and you check each result before moving on. Every command output in this post comes from really running the code.
Part 10 of the Claude for Data Engineers series on TechBrothersIT.
On this page
Let's create some sample data
Our project is shop-analytics, a new folder with two raw exports in data/raw/: orders.csv with order rows from an online shop, and products.csv with 8 products in 4 categories. There is also a short CLAUDE.md with the goal, the tools, the layer names and the revenue rule. The orders file has a few problems on purpose, and we let the profile find them.
order_id,order_ts,customer_email,product_id,qty,status
1001,2026-10-01 11:36:00,jack@example.com,P08,2,completed
1002,2026-10-01 19:33:00,liam@example.com,P04,1,completed
1003,2026-10-01 11:45:00,zoe@example.com,P04,3,completed
1004,2026-10-02 21:38:00,lucas@example.com,P01,2,completed
1005,2026-10-02 10:13:00,emma@example.com,P08,2,completed
1006,2026-10-02 19:20:00,emma@example.com,P02,2,cancelled
1007,2026-10-02 09:42:00,lucas@example.com,P03,2,completed
1008,2026-10-02 09:23:00,ethan@example.com,P03,3,completed
1009,2026-10-03 20:09:00,noah@example.com,P07,2,Completed
1010,2026-10-03 08:03:00,,P04,3,completed
1011,2026-10-03 13:02:00,omar@example.com,P05,1,completed
1012,2026-10-03 12:36:00,noah@example.com,P04,3,completed
1013,2026-10-03 18:12:00,mia@example.com,P07,2,completed
1013,2026-10-03 18:12:00,mia@example.com,P07,2,completed
1014,2026-10-03 11:05:00,lucas@example.com,P04,2,completed
1015,2026-10-04 08:17:00,ruby@example.com,P07,2,completed
1016,2026-10-04 13:28:00,mia@example.com,P01,2,completed
1017,2026-10-04 08:52:00,noah@example.com,P04,2,completed
1018,2026-10-04 17:22:00,leo@example.com,P99,2,completed
1019,2026-10-05 12:25:00,ethan@example.com,P01,1,completed
1020,2026-10-05 13:34:00,mia@example.com,P02,2,completed
1021,2026-10-05 08:30:00,nora@example.com,P02,3,cancelled
1022,2026-10-05 13:09:00,ruby@example.com,P06,1,Completed
1023,2026-10-05 13:04:00,jack@example.com,P02,2,cancelled
1024,2026-10-05 20:21:00,ruby@example.com,P05,1,completed
1025,2026-10-06 21:34:00,liam@example.com,P02,3,cancelled
1026,2026-10-06 16:20:00,nora@example.com,P04,1,completed
1027,2026-10-06 13:23:00,mia@example.com,P02,1,completed
1028,2026-10-07 17:12:00,omar@example.com,P03,1,COMPLETED
1029,2026-10-07 08:01:00,ethan@example.com,P08,2,COMPLETED
1030,2026-10-07 14:07:00,noah@example.com,P02,1,completed
1031,2026-10-08 11:07:00,test@shop.example,P08,0,cancelled
1032,2026-10-08 10:10:00,ruby@example.com,P08,2,completed
1033,2026-10-08 13:06:00,nora@example.com,P02,2,Completed
1034,2026-10-09 19:01:00,lucas@example.com,P02,3,cancelled
1035,2026-10-09 09:47:00,zoe@example.com,P04,3,completed
1036,2026-10-09 16:28:00,emma@example.com,P08,2,completed
1037,2026-10-09 14:43:00,emma@example.com,P08,1,completed
1038,2026-10-10 09:46:00,nora@example.com,P06,1,COMPLETED
1039,2026-10-10 20:11:00,omar@example.com,P07,1,completed
1039,2026-10-10 20:11:00,omar@example.com,P07,1,completed
1040,2026-10-10 18:02:00,mia@example.com,P08,1,completed
1041,2026-10-10 15:18:00,ruby@example.com,P04,1,completed
1042,2026-10-11 15:23:00,noah@example.com,P08,3,completed
1043,2026-10-11 15:47:00,jack@example.com,P04,3,completed
1044,2026-10-11 09:51:00,mia@example.com,P08,3,completed
1045,2026-10-11 17:32:00,ethan@example.com,P01,3,completed
1046,2026-10-12 10:03:00,leo@example.com,P07,3,completed
1047,2026-10-12 18:18:00,ruby@example.com,P01,3,completed
1048,2026-10-12 14:43:00,ava@example.com,P02,2,Completed
1049,2026-10-13 11:30:00,ruby@example.com,P05,2,completed
1050,2026-10-13 21:54:00,lucas@example.com,P05,1,completed
1051,2026-10-13 11:37:00,ruby@example.com,P04,2,completed
1052,2026-10-13 08:37:00,ethan@example.com,P03,3,COMPLETED
1053,2026-10-13 15:38:00,sofia@example.com,P04,1,COMPLETED
1054,2026-10-13 16:01:00,isla@example.com,P08,2,cancelled
1055,2026-10-14 09:31:00,liam@example.com,P02,1,completed
1056,2026-10-14 15:14:00,ruby@example.com,P02,1,completed
1057,2026-10-14 20:10:00,omar@example.com,P07,1,cancelled
1058,2026-10-14 13:34:00,ruby@example.com,P07,1,cancelled
1059,2026-10-14 08:03:00,nora@example.com,P05,1,cancelledproduct_id,name,category,unit_price
P01,Trail Running Shoes,Footwear,89.00
P02,Everyday Sneakers,Footwear,64.00
P03,Rain Jacket,Apparel,120.00
P04,Merino T-Shirt,Apparel,38.00
P05,Daypack 20L,Bags,55.00
P06,Travel Duffel,Bags,95.00
P07,Water Bottle,Accessories,18.00
P08,Running Cap,Accessories,22.00# shop-analytics
Raw order exports land in `data/raw/` (`orders.csv`, `products.csv`). Goal: a daily sales dashboard.
- Python 3.13 + DuckDB. Warehouse file: `warehouse.duckdb` (do not commit it).
- Layers: `raw_*` tables (as loaded), `stg_*` models (clean), marts (`daily_sales`, `category_sales`).
- Revenue = qty * unit_price, completed orders only.Step 1: Plan the project in plan mode
We start Claude Code in the project folder and trust it, because this is our own project. Then we press Shift+Tab three times to get to plan mode. In plan mode Claude reads and plans, but it does not change any files. We describe the whole job in one prompt: raw files in, clean tables, data checks, and a dashboard out.
Build an end-to-end pipeline: data/raw CSVs -> clean DuckDB tables -> data checks -> dashboard.html. Plan it first.Claude Code loaded CLAUDE.md at startup, so Claude only reads the two raw files, products.csv and orders.csv. Then it comes back with a plan in six parts: profile orders.csv with DuckDB, a raw layer in pipeline/ingest.py that loads both files into warehouse.duckdb, staging and marts in models/, data checks in checks/quality.py that exit with code 1 on failure, a dashboard.py that writes a self-contained dashboard.html, and a run_pipeline.sh that runs it all, documented in CLAUDE.md.
We choose Yes, and use auto mode. Claude turns the plan into a to-do list and starts with the first item, profiling the raw data.
Tip
On a real project, read the plan carefully. It is much cheaper to fix a plan than to fix a pipeline.
Step 2: Profile the raw CSV
Before writing any pipeline code, Claude looks at the raw data. It writes a small profile query file with two queries. The first counts rows, order IDs, duplicates, missing emails and zero quantities. The second shows the status values exactly as they arrive. Then it runs the file with the DuckDB command line.
-- 1. Size, duplicates and gaps
SELECT count(*) AS row_count,
count(DISTINCT order_id) AS order_ids,
count(*) - count(DISTINCT order_id) AS duplicates,
count(*) - count(customer_email) AS no_email,
count(*) FILTER (WHERE qty <= 0) AS qty_le_0
FROM 'data/raw/orders.csv';
SELECT status, -- 2. Status values as they arrive
count(*) AS row_count
FROM 'data/raw/orders.csv'
GROUP BY status
ORDER BY row_count DESC;duckdb -f profile.sql┌───────────┬───────────┬────────────┬──────────┬──────────┐
│ row_count │ order_ids │ duplicates │ no_email │ qty_le_0 │
│ int64 │ int64 │ int64 │ int64 │ int64 │
├───────────┼───────────┼────────────┼──────────┼──────────┤
│ 61 │ 59 │ 2 │ 1 │ 1 │
└───────────┴───────────┴────────────┴──────────┴──────────┘
┌───────────┬───────────┐
│ status │ row_count │
│ varchar │ int64 │
├───────────┼───────────┤
│ completed │ 42 │
│ cancelled │ 10 │
│ COMPLETED │ 5 │
│ Completed │ 4 │
└───────────┴───────────┘There are 61 rows but only 59 order IDs, so 2 rows are duplicates. One order has no email, and one has a quantity of zero, which looks like a test order. And the status column has completed in three spellings: completed with 42 rows, COMPLETED with 5 and Completed with 4, next to 10 cancelled. So the clean layer must trim and lowercase the status.
Tip
Always profile first. These four problems would have quietly changed every number on the dashboard.
Step 3: Ingest and model in raw, staging and marts
Now the pipeline itself, in three layers, like bronze, silver and gold in a lakehouse. The raw layer loads each CSV file as it is into a raw_ table. There is no cleaning here, so we can always go back to the original data.
"""Raw layer: load the CSV exports as they are, one raw_* table per file."""
import duckdb
con = duckdb.connect("warehouse.duckdb")
for name in ("orders", "products"):
con.execute(f"CREATE OR REPLACE TABLE raw_{name} AS SELECT * FROM read_csv('data/raw/{name}.csv', header = true)")
rows = con.execute(f"SELECT count(*) FROM raw_{name}").fetchone()[0]
print(f"raw_{name:<9} {rows:>3} rows <- data/raw/{name}.csv")python pipeline/ingest.pyraw_orders 61 rows <- data/raw/orders.csv
raw_products 8 rows <- data/raw/products.csv61 order rows and 8 products, exactly what the profile showed. Next Claude writes the staging model, two marts and a small runner that builds the three models in order.
-- staging: one clean row per order
with deduped as (
select distinct * from raw_orders
)
select
o.order_id,
cast(o.order_ts as date) as order_date,
lower(trim(o.customer_email)) as email,
o.product_id,
p.name as product_name,
p.category,
o.qty,
o.qty * p.unit_price as revenue,
lower(trim(o.status)) as status
from deduped o
left join raw_products p
on p.product_id = o.product_id
where o.qty > 0-- mart: completed sales per day
select
order_date,
count(*) as orders,
sum(revenue) as revenue,
count(distinct email) as customers
from stg_orders
where status = 'completed'
group by order_date
order by order_date-- mart: completed sales per product category
select
category,
count(*) as orders,
sum(qty) as units,
sum(revenue) as revenue
from stg_orders
where status = 'completed'
group by category
order by revenue desc"""Staging and marts: run the SQL models in order, each one becomes a table."""
import pathlib
import duckdb
MODELS = ["stg_orders", "daily_sales", "category_sales"]
con = duckdb.connect("warehouse.duckdb")
for name in MODELS:
sql = pathlib.Path(f"models/{name}.sql").read_text()
con.execute(f"CREATE OR REPLACE TABLE {name} AS {sql}")
rows = con.execute(f"SELECT count(*) FROM {name}").fetchone()[0]
print(f"{name:<15} {rows:>3} rows")python pipeline/transform.pystg_orders 58 rows
daily_sales 14 rows
category_sales 5 rowsThe staging model is where the cleaning happens. select distinct removes the duplicate rows, and lower(trim(...)) fixes the status values. It joins the products for the category and the price, and the where o.qty > 0 line drops the test order. The two marts, daily_sales and category_sales, use completed orders only.
The run shows stg_orders with 58 rows and daily_sales with 14 days. But category_sales has 5 rows, and our products only have 4 categories. Keep that in mind.
Step 4: Data-quality checks catch a bad row
The pipeline runs, but does it produce the right numbers? That is the job of data checks. Claude writes six checks: duplicates, unknown products, bad quantities, unexpected status values, and two checks on the daily sales mart. Here is the first run.
"""Data-quality checks on the staging model and the marts. Exit code 1 when any check fails."""
import sys
import duckdb
con = duckdb.connect("warehouse.duckdb", read_only=True)
CHECKS = {
"stg_orders: duplicate order_id": "select order_id from stg_orders group by order_id having count(*) > 1",
"stg_orders: unknown product": "select order_id || ' (' || product_id || ')' from stg_orders where category is null",
"stg_orders: qty <= 0": "select order_id from stg_orders where qty <= 0",
"stg_orders: unexpected status": "select distinct status from stg_orders where status not in ('completed', 'cancelled')",
"daily_sales: matches staging": """select round(d, 2) || ' vs ' || round(s, 2) from (
select (select sum(revenue) from daily_sales) d,
(select sum(revenue) from stg_orders where status = 'completed') s) where d is distinct from s""",
"daily_sales: no missing days": """select cast(day as date) from generate_series(
(select min(order_date) from daily_sales), (select max(order_date) from daily_sales), interval 1 day) t(day)
where cast(day as date) not in (select order_date from daily_sales)""",
}
failed = 0
for name, sql in CHECKS.items():
bad = [str(r[0]) for r in con.execute(sql).fetchall()]
failed += bool(bad)
print(f"{'FAIL' if bad else 'pass'} {name:<32} {', '.join(bad)}".rstrip())
print(f"{len(CHECKS) - failed} of {len(CHECKS)} checks passed")
sys.exit(1 if failed else 0)python checks/quality.pypass stg_orders: duplicate order_id
FAIL stg_orders: unknown product 1018 (P99)
pass stg_orders: qty <= 0
pass stg_orders: unexpected status
pass daily_sales: matches staging
pass daily_sales: no missing days
5 of 6 checks passed5 of 6 checks pass, one fails, and the script exits with code 1. Order 1018 points to product P99, which is not in the products file. That is our fifth category.
Claude replies
stg_orders kept order 1018 with product P99, which is not in products.csv. The left join gives it no category and no revenue.
The left join kept that order with no category and no price. The profile did not look for it, but the check did. Claude changes the left join into an inner join, so staging only keeps orders with a known product. Then it rebuilds the models and runs the checks again.
-- staging: one clean row per order
with deduped as (
select distinct * from raw_orders
)
select
o.order_id,
cast(o.order_ts as date) as order_date,
lower(trim(o.customer_email)) as email,
o.product_id,
p.name as product_name,
p.category,
o.qty,
o.qty * p.unit_price as revenue,
lower(trim(o.status)) as status
from deduped o
join raw_products p
on p.product_id = o.product_id
where o.qty > 0python pipeline/transform.py && python checks/quality.pystg_orders 57 rows
daily_sales 14 rows
category_sales 4 rows
pass stg_orders: duplicate order_id
pass stg_orders: unknown product
pass stg_orders: qty <= 0
pass stg_orders: unexpected status
pass daily_sales: matches staging
pass daily_sales: no missing days
6 of 6 checks passedStaging now has 57 rows, category_sales is back to 4 rows, and all 6 checks pass.
Tip
In a real project, do not just drop bad rows. Write them to a quarantine table and count them, so someone can fix the source.
Step 5: Build the HTML dashboard
Now the part everyone sees, the dashboard. We ask for a single HTML file with no server and no libraries, so we can open it anywhere or send it to someone.
Build dashboard.html from the marts: KPI tiles, revenue per day, revenue by category, top 5 products. One file, no libraries.Claude writes a Python script that reads the marts and draws the charts as plain SVG. Then it runs the script.
"""Build dashboard.html from the marts: one self-contained page, no server, no libraries."""
import html
import duckdb
con = duckdb.connect("warehouse.duckdb", read_only=True)
days = con.execute("select order_date, orders, revenue from daily_sales order by order_date").fetchall()
cats = con.execute("select category, revenue from category_sales order by revenue desc").fetchall()
top = con.execute("""select product_name, sum(qty) as units, sum(revenue) as revenue from stg_orders
where status = 'completed' group by product_name order by revenue desc limit 5""").fetchall()
revenue, orders = sum(d[2] for d in days), sum(d[1] for d in days)
customers = con.execute("select count(distinct email) from stg_orders where status = 'completed'").fetchone()[0]
money = lambda v: f"${v:,.0f}"
def tile(label, value):
return f'<div class="tile"><div class="lb">{label}</div><div class="v">{value}</div></div>'
def day_bars(w=700, h=190):
top_v = max(d[2] for d in days) * 1.15
bw = w / len(days)
out = [f'<line x1="0" y1="{h}" x2="{w}" y2="{h}" class="axis"/>']
for g in (0.5, 1.0):
y = h - h * g / 1.15
out.append(f'<line x1="0" y1="{y:.0f}" x2="{w}" y2="{y:.0f}" class="grid"/>'
f'<text x="0" y="{y - 4:.0f}" class="tick">{money(top_v * g / 1.15)}</text>')
best = max(days, key=lambda d: d[2])
for i, (day, n, rev) in enumerate(days):
bh = h * rev / top_v
x = i * bw + 6
out.append(f'<rect x="{x:.0f}" y="{h - bh:.0f}" width="{bw - 12:.0f}" height="{bh:.0f}" rx="4" class="bar">'
f'<title>{day:%b %d}: {money(rev)}, {n} orders</title></rect>')
if i % 2 == 0:
out.append(f'<text x="{x + (bw - 12) / 2:.0f}" y="{h + 18}" class="tick" text-anchor="middle">{day:%b %d}</text>')
if (day, n, rev) == best:
out.append(f'<text x="{x + (bw - 12) / 2:.0f}" y="{h - bh - 6:.0f}" class="val" text-anchor="middle">{money(rev)}</text>')
return f'<svg viewBox="0 -10 {w} {h + 30}" width="100%">{"".join(out)}</svg>'
def cat_bars(w=330):
peak = cats[0][1]
rows = []
for i, (cat, rev) in enumerate(cats):
y = i * 38
rows.append(f'<text x="0" y="{y + 17}" class="lbl">{html.escape(cat)}</text>'
f'<rect x="100" y="{y + 3}" width="{(w - 170) * rev / peak:.0f}" height="20" rx="4" class="bar"><title>{cat}: {money(rev)}</title></rect>'
f'<text x="{106 + (w - 170) * rev / peak:.0f}" y="{y + 18}" class="val">{money(rev)}</text>')
return f'<svg viewBox="0 0 {w} {len(cats) * 38}" width="100%">{"".join(rows)}</svg>'
rows = "".join(f"<tr><td>{html.escape(p)}</td><td class=n>{u}</td><td class=n>{money(r)}</td></tr>" for p, u, r in top)
page = f"""<!doctype html><html><head><meta charset="utf-8"><title>Shop sales dashboard</title><style>
body {{ margin:0; padding:22px 26px; background:#fcfcfb; color:#0b0b0b; font:15px/1.4 Inter, Arial, sans-serif; }}
h1 {{ font-size:22px; margin:0; }} .sub {{ color:#52514e; font-size:13px; margin:2px 0 16px; }}
.tiles {{ display:grid; grid-template-columns:repeat(4, 1fr); gap:12px; margin-bottom:16px; }}
.tile, .card {{ background:#fff; border:1px solid #e4e2dc; border-radius:10px; padding:12px 14px; }}
.lb {{ color:#52514e; font-size:12px; text-transform:uppercase; letter-spacing:.5px; }} .v {{ font-size:26px; font-weight:700; }}
h2 {{ font-size:14px; margin:0 0 8px; }} .row {{ display:grid; grid-template-columns:1fr 1fr; gap:12px; margin-top:12px; }}
.bar {{ fill:#2a78d6; }} .grid {{ stroke:#ecebe7; }} .axis {{ stroke:#b9b7b0; }}
.tick {{ fill:#6d6b66; font-size:11px; }} .val {{ fill:#0b0b0b; font-size:12px; font-weight:600; }} .lbl {{ fill:#52514e; font-size:13px; }}
table {{ width:100%; border-collapse:collapse; font-size:13px; }} td {{ padding:5px 4px; border-top:1px solid #ecebe7; }}
th {{ text-align:left; color:#52514e; font-weight:600; font-size:12px; padding:0 4px 4px; }} .n {{ text-align:right; }}
</style></head><body>
<h1>Shop sales dashboard</h1>
<div class="sub">Completed orders, {days[0][0]:%b %d} to {days[-1][0]:%b %d, %Y} · built from data/raw by run_pipeline.sh</div>
<div class="tiles">{tile("Revenue", money(revenue))}{tile("Orders", orders)}{tile("Avg order", money(revenue / orders))}{tile("Customers", customers)}</div>
<div class="card"><h2>Revenue per day</h2>{day_bars()}</div>
<div class="row"><div class="card"><h2>Revenue by category</h2>{cat_bars()}</div>
<div class="card"><h2>Top 5 products</h2><table><tr><th>Product</th><th class=n>Units</th><th class=n>Revenue</th></tr>{rows}</table></div></div>
</body></html>"""
open("dashboard.html", "w").write(page)
print(f"dashboard.html written: {money(revenue)} revenue, {orders} orders, {len(days)} days, {len(cats)} categories")python dashboard.pydashboard.html written: $4,690 revenue, 48 orders, 14 days, 4 categoriesThe script prints $4,690 revenue, 48 orders, 14 days, 4 categories. Open dashboard.html in the browser and you get KPI tiles at the top for revenue, orders, average order and customers. Below them are revenue per day with the best day labeled, revenue by category, and the top 5 products. Every number on the page comes from the marts we just tested. Each bar also has a tooltip with the day, the revenue and the orders.
Step 6: One command for the whole pipeline
A pipeline you can only run by asking Claude is not finished. It needs one command. We ask Claude for a single script that runs every stage, and to write the command into CLAUDE.md.
Add run_pipeline.sh that runs every stage and stops on the first failure, then add a Run section to CLAUDE.mdThe script runs ingest, transform, checks and the dashboard, and set -euo pipefail stops it at the first failure. So bad data never reaches the dashboard. Claude also adds a Run section to CLAUDE.md, so the next session, or the next person, knows how to run it. Then it deletes the warehouse and the dashboard and runs everything from scratch.
#!/usr/bin/env bash
# Raw CSV to dashboard in one command. Stops at the first failing stage.
set -euo pipefail
echo "== 1/4 ingest"; python pipeline/ingest.py
echo "== 2/4 transform"; python pipeline/transform.py
echo "== 3/4 checks"; python checks/quality.py
echo "== 4/4 dashboard"; python dashboard.py# shop-analytics
Raw order exports land in `data/raw/` (`orders.csv`, `products.csv`). Goal: a daily sales dashboard.
- Python 3.13 + DuckDB. Warehouse file: `warehouse.duckdb` (do not commit it).
- Layers: `raw_*` tables (as loaded), `stg_*` models (clean), marts (`daily_sales`, `category_sales`).
- Revenue = qty * unit_price, completed orders only.
## Run
- `bash run_pipeline.sh` runs ingest, transform, checks and the dashboard, and stops at the first failure.
- Data checks: `python checks/quality.py` (exit code 1 when a check fails).rm -f warehouse.duckdb dashboard.html && bash run_pipeline.sh== 1/4 ingest
raw_orders 61 rows <- data/raw/orders.csv
raw_products 8 rows <- data/raw/products.csv
== 2/4 transform
stg_orders 57 rows
daily_sales 14 rows
category_sales 4 rows
== 3/4 checks
pass stg_orders: duplicate order_id
pass stg_orders: unknown product
pass stg_orders: qty <= 0
pass stg_orders: unexpected status
pass daily_sales: matches staging
pass daily_sales: no missing days
6 of 6 checks passed
== 4/4 dashboard
dashboard.html written: $4,690 revenue, 48 orders, 14 days, 4 categoriesAll four stages pass: 61 raw orders and 8 products, 57 staging rows, 6 of 6 checks passed, and the dashboard is written again with the same $4,690 revenue, 48 orders.
Tip
Run the pipeline from a clean start like this before you share it. It proves nothing depends on files left over from earlier runs.
The whole project
Here is the whole project, start to finish.
RAW CSV TO DASHBOARD
Six steps, all real
- Plan in plan mode, then profile the raw CSV
- Raw tables as loaded, staging cleans, marts answer questions
- Checks fail loudly (exit code 1), then fix and re-run
- Dashboard + run_pipeline.sh + a Run section in CLAUDE.md
Plan first, and profile the raw data before you write code. Build in layers: raw as it arrives, clean staging, and small marts for the questions people ask. Test the data, not only the code, and finish with one command that anyone can run.
That is a complete data project with Claude Code, from a raw CSV export to tested tables to a dashboard, with one command to run it all. And that is the end of the Claude for Data Engineers series.
Key points to remember
- Plan mode first
- Profile the raw CSV with DuckDB
- Raw, staging and marts
- Data checks: fail, fix, pass
- A self-contained dashboard.html
- run_pipeline.sh + CLAUDE.md
The full script
This bash script writes every file Claude wrote, in the same order, and runs every command, so you can reproduce the results without Claude.
Show the full script (every file and every command, in order)
#!/usr/bin/env bash
# Claude for Data Engineers #10: End-to-End Data Project with Claude Code: Raw CSV to Dashboard
# Rebuilds the demo project without Claude: writes every file Claude wrote, in the same order,
# and runs every command from the video. Run it in an empty folder.
# Tools used in the video: Python 3.13, DuckDB 2.0.0, pandas 3.0.5, PySpark 4.3.0 (local mode, Java 21),
# dbt-core 1.12.6 with dbt-duckdb 1.11.0, PostgreSQL 16. Install the ones this project uses first.
mkdir -p shop-analytics && cd shop-analytics
# ---------- starting files ----------
mkdir -p data/raw
cat > data/raw/orders.csv <<'CLAUDE_DE_EOF'
order_id,order_ts,customer_email,product_id,qty,status
1001,2026-10-01 11:36:00,jack@example.com,P08,2,completed
1002,2026-10-01 19:33:00,liam@example.com,P04,1,completed
1003,2026-10-01 11:45:00,zoe@example.com,P04,3,completed
1004,2026-10-02 21:38:00,lucas@example.com,P01,2,completed
1005,2026-10-02 10:13:00,emma@example.com,P08,2,completed
1006,2026-10-02 19:20:00,emma@example.com,P02,2,cancelled
1007,2026-10-02 09:42:00,lucas@example.com,P03,2,completed
1008,2026-10-02 09:23:00,ethan@example.com,P03,3,completed
1009,2026-10-03 20:09:00,noah@example.com,P07,2,Completed
1010,2026-10-03 08:03:00,,P04,3,completed
1011,2026-10-03 13:02:00,omar@example.com,P05,1,completed
1012,2026-10-03 12:36:00,noah@example.com,P04,3,completed
1013,2026-10-03 18:12:00,mia@example.com,P07,2,completed
1013,2026-10-03 18:12:00,mia@example.com,P07,2,completed
1014,2026-10-03 11:05:00,lucas@example.com,P04,2,completed
1015,2026-10-04 08:17:00,ruby@example.com,P07,2,completed
1016,2026-10-04 13:28:00,mia@example.com,P01,2,completed
1017,2026-10-04 08:52:00,noah@example.com,P04,2,completed
1018,2026-10-04 17:22:00,leo@example.com,P99,2,completed
1019,2026-10-05 12:25:00,ethan@example.com,P01,1,completed
1020,2026-10-05 13:34:00,mia@example.com,P02,2,completed
1021,2026-10-05 08:30:00,nora@example.com,P02,3,cancelled
1022,2026-10-05 13:09:00,ruby@example.com,P06,1,Completed
1023,2026-10-05 13:04:00,jack@example.com,P02,2,cancelled
1024,2026-10-05 20:21:00,ruby@example.com,P05,1,completed
1025,2026-10-06 21:34:00,liam@example.com,P02,3,cancelled
1026,2026-10-06 16:20:00,nora@example.com,P04,1,completed
1027,2026-10-06 13:23:00,mia@example.com,P02,1,completed
1028,2026-10-07 17:12:00,omar@example.com,P03,1,COMPLETED
1029,2026-10-07 08:01:00,ethan@example.com,P08,2,COMPLETED
1030,2026-10-07 14:07:00,noah@example.com,P02,1,completed
1031,2026-10-08 11:07:00,test@shop.example,P08,0,cancelled
1032,2026-10-08 10:10:00,ruby@example.com,P08,2,completed
1033,2026-10-08 13:06:00,nora@example.com,P02,2,Completed
1034,2026-10-09 19:01:00,lucas@example.com,P02,3,cancelled
1035,2026-10-09 09:47:00,zoe@example.com,P04,3,completed
1036,2026-10-09 16:28:00,emma@example.com,P08,2,completed
1037,2026-10-09 14:43:00,emma@example.com,P08,1,completed
1038,2026-10-10 09:46:00,nora@example.com,P06,1,COMPLETED
1039,2026-10-10 20:11:00,omar@example.com,P07,1,completed
1039,2026-10-10 20:11:00,omar@example.com,P07,1,completed
1040,2026-10-10 18:02:00,mia@example.com,P08,1,completed
1041,2026-10-10 15:18:00,ruby@example.com,P04,1,completed
1042,2026-10-11 15:23:00,noah@example.com,P08,3,completed
1043,2026-10-11 15:47:00,jack@example.com,P04,3,completed
1044,2026-10-11 09:51:00,mia@example.com,P08,3,completed
1045,2026-10-11 17:32:00,ethan@example.com,P01,3,completed
1046,2026-10-12 10:03:00,leo@example.com,P07,3,completed
1047,2026-10-12 18:18:00,ruby@example.com,P01,3,completed
1048,2026-10-12 14:43:00,ava@example.com,P02,2,Completed
1049,2026-10-13 11:30:00,ruby@example.com,P05,2,completed
1050,2026-10-13 21:54:00,lucas@example.com,P05,1,completed
1051,2026-10-13 11:37:00,ruby@example.com,P04,2,completed
1052,2026-10-13 08:37:00,ethan@example.com,P03,3,COMPLETED
1053,2026-10-13 15:38:00,sofia@example.com,P04,1,COMPLETED
1054,2026-10-13 16:01:00,isla@example.com,P08,2,cancelled
1055,2026-10-14 09:31:00,liam@example.com,P02,1,completed
1056,2026-10-14 15:14:00,ruby@example.com,P02,1,completed
1057,2026-10-14 20:10:00,omar@example.com,P07,1,cancelled
1058,2026-10-14 13:34:00,ruby@example.com,P07,1,cancelled
1059,2026-10-14 08:03:00,nora@example.com,P05,1,cancelled
CLAUDE_DE_EOF
mkdir -p data/raw
cat > data/raw/products.csv <<'CLAUDE_DE_EOF'
product_id,name,category,unit_price
P01,Trail Running Shoes,Footwear,89.00
P02,Everyday Sneakers,Footwear,64.00
P03,Rain Jacket,Apparel,120.00
P04,Merino T-Shirt,Apparel,38.00
P05,Daypack 20L,Bags,55.00
P06,Travel Duffel,Bags,95.00
P07,Water Bottle,Accessories,18.00
P08,Running Cap,Accessories,22.00
CLAUDE_DE_EOF
cat > CLAUDE.md <<'CLAUDE_DE_EOF'
# shop-analytics
Raw order exports land in `data/raw/` (`orders.csv`, `products.csv`). Goal: a daily sales dashboard.
- Python 3.13 + DuckDB. Warehouse file: `warehouse.duckdb` (do not commit it).
- Layers: `raw_*` tables (as loaded), `stg_*` models (clean), marts (`daily_sales`, `category_sales`).
- Revenue = qty * unit_price, completed orders only.
CLAUDE_DE_EOF
# ---------- step 1: Profile the raw orders file with DuckDB ----------
cat > profile.sql <<'CLAUDE_DE_EOF'
-- 1. Size, duplicates and gaps
SELECT count(*) AS row_count,
count(DISTINCT order_id) AS order_ids,
count(*) - count(DISTINCT order_id) AS duplicates,
count(*) - count(customer_email) AS no_email,
count(*) FILTER (WHERE qty <= 0) AS qty_le_0
FROM 'data/raw/orders.csv';
SELECT status, -- 2. Status values as they arrive
count(*) AS row_count
FROM 'data/raw/orders.csv'
GROUP BY status
ORDER BY row_count DESC;
CLAUDE_DE_EOF
duckdb -f profile.sql
# ---------- step 2: Raw layer: load the CSV files as raw tables ----------
mkdir -p pipeline
cat > pipeline/ingest.py <<'CLAUDE_DE_EOF'
"""Raw layer: load the CSV exports as they are, one raw_* table per file."""
import duckdb
con = duckdb.connect("warehouse.duckdb")
for name in ("orders", "products"):
con.execute(f"CREATE OR REPLACE TABLE raw_{name} AS SELECT * FROM read_csv('data/raw/{name}.csv', header = true)")
rows = con.execute(f"SELECT count(*) FROM raw_{name}").fetchone()[0]
print(f"raw_{name:<9} {rows:>3} rows <- data/raw/{name}.csv")
CLAUDE_DE_EOF
python pipeline/ingest.py
# ---------- step 3: Staging and marts: clean one row per order, then daily and category sales ----------
mkdir -p models
cat > models/stg_orders.sql <<'CLAUDE_DE_EOF'
-- staging: one clean row per order
with deduped as (
select distinct * from raw_orders
)
select
o.order_id,
cast(o.order_ts as date) as order_date,
lower(trim(o.customer_email)) as email,
o.product_id,
p.name as product_name,
p.category,
o.qty,
o.qty * p.unit_price as revenue,
lower(trim(o.status)) as status
from deduped o
left join raw_products p
on p.product_id = o.product_id
where o.qty > 0
CLAUDE_DE_EOF
mkdir -p models
cat > models/daily_sales.sql <<'CLAUDE_DE_EOF'
-- mart: completed sales per day
select
order_date,
count(*) as orders,
sum(revenue) as revenue,
count(distinct email) as customers
from stg_orders
where status = 'completed'
group by order_date
order by order_date
CLAUDE_DE_EOF
mkdir -p models
cat > models/category_sales.sql <<'CLAUDE_DE_EOF'
-- mart: completed sales per product category
select
category,
count(*) as orders,
sum(qty) as units,
sum(revenue) as revenue
from stg_orders
where status = 'completed'
group by category
order by revenue desc
CLAUDE_DE_EOF
mkdir -p pipeline
cat > pipeline/transform.py <<'CLAUDE_DE_EOF'
"""Staging and marts: run the SQL models in order, each one becomes a table."""
import pathlib
import duckdb
MODELS = ["stg_orders", "daily_sales", "category_sales"]
con = duckdb.connect("warehouse.duckdb")
for name in MODELS:
sql = pathlib.Path(f"models/{name}.sql").read_text()
con.execute(f"CREATE OR REPLACE TABLE {name} AS {sql}")
rows = con.execute(f"SELECT count(*) FROM {name}").fetchone()[0]
print(f"{name:<15} {rows:>3} rows")
CLAUDE_DE_EOF
python pipeline/transform.py
# ---------- step 4: Data-quality checks (first run finds a bad row) ----------
mkdir -p checks
cat > checks/quality.py <<'CLAUDE_DE_EOF'
"""Data-quality checks on the staging model and the marts. Exit code 1 when any check fails."""
import sys
import duckdb
con = duckdb.connect("warehouse.duckdb", read_only=True)
CHECKS = {
"stg_orders: duplicate order_id": "select order_id from stg_orders group by order_id having count(*) > 1",
"stg_orders: unknown product": "select order_id || ' (' || product_id || ')' from stg_orders where category is null",
"stg_orders: qty <= 0": "select order_id from stg_orders where qty <= 0",
"stg_orders: unexpected status": "select distinct status from stg_orders where status not in ('completed', 'cancelled')",
"daily_sales: matches staging": ""select round(d, 2) || ' vs ' || round(s, 2) from (
select (select sum(revenue) from daily_sales) d,
(select sum(revenue) from stg_orders where status = 'completed') s) where d is distinct from s"",
"daily_sales: no missing days": ""select cast(day as date) from generate_series(
(select min(order_date) from daily_sales), (select max(order_date) from daily_sales), interval 1 day) t(day)
where cast(day as date) not in (select order_date from daily_sales)"",
}
failed = 0
for name, sql in CHECKS.items():
bad = [str(r[0]) for r in con.execute(sql).fetchall()]
failed += bool(bad)
print(f"{'FAIL' if bad else 'pass'} {name:<32} {', '.join(bad)}".rstrip())
print(f"{len(CHECKS) - failed} of {len(CHECKS)} checks passed")
sys.exit(1 if failed else 0)
CLAUDE_DE_EOF
python checks/quality.py # expected to fail
# ---------- step 5: Fix: keep only orders with a known product, rebuild, re-run the checks ----------
mkdir -p models
cat > models/stg_orders.sql <<'CLAUDE_DE_EOF'
-- staging: one clean row per order
with deduped as (
select distinct * from raw_orders
)
select
o.order_id,
cast(o.order_ts as date) as order_date,
lower(trim(o.customer_email)) as email,
o.product_id,
p.name as product_name,
p.category,
o.qty,
o.qty * p.unit_price as revenue,
lower(trim(o.status)) as status
from deduped o
join raw_products p
on p.product_id = o.product_id
where o.qty > 0
CLAUDE_DE_EOF
python pipeline/transform.py && python checks/quality.py
# ---------- step 6: Build dashboard.html from the marts ----------
cat > dashboard.py <<'CLAUDE_DE_EOF'
"""Build dashboard.html from the marts: one self-contained page, no server, no libraries."""
import html
import duckdb
con = duckdb.connect("warehouse.duckdb", read_only=True)
days = con.execute("select order_date, orders, revenue from daily_sales order by order_date").fetchall()
cats = con.execute("select category, revenue from category_sales order by revenue desc").fetchall()
top = con.execute(""select product_name, sum(qty) as units, sum(revenue) as revenue from stg_orders
where status = 'completed' group by product_name order by revenue desc limit 5"").fetchall()
revenue, orders = sum(d[2] for d in days), sum(d[1] for d in days)
customers = con.execute("select count(distinct email) from stg_orders where status = 'completed'").fetchone()[0]
money = lambda v: f"${v:,.0f}"
def tile(label, value):
return f'<div class="tile"><div class="lb">{label}</div><div class="v">{value}</div></div>'
def day_bars(w=700, h=190):
top_v = max(d[2] for d in days) * 1.15
bw = w / len(days)
out = [f'<line x1="0" y1="{h}" x2="{w}" y2="{h}" class="axis"/>']
for g in (0.5, 1.0):
y = h - h * g / 1.15
out.append(f'<line x1="0" y1="{y:.0f}" x2="{w}" y2="{y:.0f}" class="grid"/>'
f'<text x="0" y="{y - 4:.0f}" class="tick">{money(top_v * g / 1.15)}</text>')
best = max(days, key=lambda d: d[2])
for i, (day, n, rev) in enumerate(days):
bh = h * rev / top_v
x = i * bw + 6
out.append(f'<rect x="{x:.0f}" y="{h - bh:.0f}" width="{bw - 12:.0f}" height="{bh:.0f}" rx="4" class="bar">'
f'<title>{day:%b %d}: {money(rev)}, {n} orders</title></rect>')
if i % 2 == 0:
out.append(f'<text x="{x + (bw - 12) / 2:.0f}" y="{h + 18}" class="tick" text-anchor="middle">{day:%b %d}</text>')
if (day, n, rev) == best:
out.append(f'<text x="{x + (bw - 12) / 2:.0f}" y="{h - bh - 6:.0f}" class="val" text-anchor="middle">{money(rev)}</text>')
return f'<svg viewBox="0 -10 {w} {h + 30}" width="100%">{"".join(out)}</svg>'
def cat_bars(w=330):
peak = cats[0][1]
rows = []
for i, (cat, rev) in enumerate(cats):
y = i * 38
rows.append(f'<text x="0" y="{y + 17}" class="lbl">{html.escape(cat)}</text>'
f'<rect x="100" y="{y + 3}" width="{(w - 170) * rev / peak:.0f}" height="20" rx="4" class="bar"><title>{cat}: {money(rev)}</title></rect>'
f'<text x="{106 + (w - 170) * rev / peak:.0f}" y="{y + 18}" class="val">{money(rev)}</text>')
return f'<svg viewBox="0 0 {w} {len(cats) * 38}" width="100%">{"".join(rows)}</svg>'
rows = "".join(f"<tr><td>{html.escape(p)}</td><td class=n>{u}</td><td class=n>{money(r)}</td></tr>" for p, u, r in top)
page = f"""<!doctype html><html><head><meta charset="utf-8><title>Shop sales dashboard</title><style>
body {{ margin:0; padding:22px 26px; background:#fcfcfb; color:#0b0b0b; font:15px/1.4 Inter, Arial, sans-serif; }}
h1 {{ font-size:22px; margin:0; }} .sub {{ color:#52514e; font-size:13px; margin:2px 0 16px; }}
.tiles {{ display:grid; grid-template-columns:repeat(4, 1fr); gap:12px; margin-bottom:16px; }}
.tile, .card {{ background:#fff; border:1px solid #e4e2dc; border-radius:10px; padding:12px 14px; }}
.lb {{ color:#52514e; font-size:12px; text-transform:uppercase; letter-spacing:.5px; }} .v {{ font-size:26px; font-weight:700; }}
h2 {{ font-size:14px; margin:0 0 8px; }} .row {{ display:grid; grid-template-columns:1fr 1fr; gap:12px; margin-top:12px; }}
.bar {{ fill:#2a78d6; }} .grid {{ stroke:#ecebe7; }} .axis {{ stroke:#b9b7b0; }}
.tick {{ fill:#6d6b66; font-size:11px; }} .val {{ fill:#0b0b0b; font-size:12px; font-weight:600; }} .lbl {{ fill:#52514e; font-size:13px; }}
table {{ width:100%; border-collapse:collapse; font-size:13px; }} td {{ padding:5px 4px; border-top:1px solid #ecebe7; }}
th {{ text-align:left; color:#52514e; font-weight:600; font-size:12px; padding:0 4px 4px; }} .n {{ text-align:right; }}
</style></head><body>
<h1>Shop sales dashboard</h1>
<div class="sub">Completed orders, {days[0][0]:%b %d} to {days[-1][0]:%b %d, %Y} · built from data/raw by run_pipeline.sh</div>
<div class="tiles">{tile("Revenue", money(revenue))}{tile("Orders", orders)}{tile("Avg order", money(revenue / orders))}{tile("Customers", customers)}</div>
<div class="card"><h2>Revenue per day</h2>{day_bars()}</div>
<div class="row"><div class="card"><h2>Revenue by category</h2>{cat_bars()}</div>
<div class="card"><h2>Top 5 products</h2><table><tr><th>Product</th><th class=n>Units</th><th class=n>Revenue</th></tr>{rows}</table></div></div>
</body></html>""
open("dashboard.html", "w").write(page)
print(f"dashboard.html written: {money(revenue)} revenue, {orders} orders, {len(days)} days, {len(cats)} categories")
CLAUDE_DE_EOF
python dashboard.py
# ---------- step 7: One command for the whole pipeline, documented in CLAUDE.md ----------
cat > run_pipeline.sh <<'CLAUDE_DE_EOF'
#!/usr/bin/env bash
# Raw CSV to dashboard in one command. Stops at the first failing stage.
set -euo pipefail
echo "== 1/4 ingest"; python pipeline/ingest.py
echo "== 2/4 transform"; python pipeline/transform.py
echo "== 3/4 checks"; python checks/quality.py
echo "== 4/4 dashboard"; python dashboard.py
CLAUDE_DE_EOF
cat > CLAUDE.md <<'CLAUDE_DE_EOF'
# shop-analytics
Raw order exports land in `data/raw/` (`orders.csv`, `products.csv`). Goal: a daily sales dashboard.
- Python 3.13 + DuckDB. Warehouse file: `warehouse.duckdb` (do not commit it).
- Layers: `raw_*` tables (as loaded), `stg_*` models (clean), marts (`daily_sales`, `category_sales`).
- Revenue = qty * unit_price, completed orders only.
## Run
- `bash run_pipeline.sh` runs ingest, transform, checks and the dashboard, and stops at the first failure.
- Data checks: `python checks/quality.py` (exit code 1 when a check fails).
CLAUDE_DE_EOF
rm -f warehouse.duckdb dashboard.html && bash run_pipeline.sh
That is the whole series
All 10 parts are listed on the Claude for Data Engineers: Every Tutorial page on this blog.
The project and its data are invented for teaching. Every file on this page is the real file from the demo project, and every output is the real output of the command above it, captured when the video was made (each step ran twice in a fresh folder and the outputs were compared). The Claude screens in the video are a simplified recreation for teaching. Always review what Claude changes and runs. Claude plans, features and limits are current as of November 2026; check the official Claude and Claude Code documentation before you rely on them. Part of the Claude for Data Engineers series on TechBrothersIT.