In this post we are going to cover the core Python that trips up data engineers: mutable defaults, shallow and deep copies, lists vs sets vs dictionaries, generators for big files, float vs Decimal, and is vs ==. These are 7 real Python interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario where you review a junior's script that works on small files but fails on big ones.
None of this is advanced Python. These are the small language rules that decide whether a loader leaks data between files, runs out of memory, or is off by a cent. Interviewers ask about them because every one of them shows up in real pipelines, and the bugs usually pass a quick test and fail in production.
Every Python step below runs on the sample data, and the output under each step is the real result.
Part 1 of the Real Time Python for Data Engineers Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Let's create some sample data
- Question 1: A function that builds a list of records keeps returning rows from the previous call. Why, and how do you fix it?
- Question 2: You copied a list of dictionaries, changed the copy, and the original changed too. What is the difference between a shallow and a deep copy?
- Question 3: A lookup over a million IDs inside a loop takes minutes. When do you use a list, a set or a dictionary?
- Question 4: A script that reads a 20 GB file runs out of memory. How do generators help, and when do you use one?
- Question 5: The finance totals are off by a cent. Why does float give the wrong answer, and when do you use Decimal?
- Question 6: A teammate compares values with is instead of ==, and a check fails in production. What is the difference?
- Scenario 7: You review a junior's script that loads customer files, and it works on small files but fails on big ones. What do you look for?
- The full script
The short answer
Defaults are evaluated once, so never use a mutable default. Copies are shallow unless you deep copy. Use a set or dict for lookups, generators to stream big files, Decimal for money, and is only for None and other singletons.
Keep that in mind. Every question below is one piece of it.
Let's create some sample data
The script writes two small CSV files: orders.csv with 1,800 orders, where every third one is cancelled, and customers.csv with 100,000 customer rows but only 60,000 unique customer IDs, so the rest are repeats. Run it in an empty folder.
import csv
# Q4: 1,800 orders, every third one cancelled
with open("orders.csv", "w", encoding="utf-8", newline="") as f:
w = csv.writer(f)
w.writerow(["order_id", "status", "amount"])
for i in range(1_800):
w.writerow([50_000 + i, "CANCELLED" if i % 3 == 0 else "PAID", f"{(i % 90) + 10}.25"])
# Q7: 100,000 customer rows, 60,000 unique customer IDs (the rest are repeats)
with open("customers.csv", "w", encoding="utf-8", newline="") as f:
w = csv.writer(f)
w.writerow(["cust_id", "name", "amount"])
for i in range(100_000):
w.writerow([f"C{i % 60_000:06d}", f"Customer {i % 60_000}", f"{(i % 500) + 1}.50"])Question 1: A function that builds a list of records keeps returning rows from the previous call. Why, and how do you fix it?
The classic cause is a mutable default argument. Python evaluates a default value once, when the def statement runs, not on every call.
Let's see it. This function takes a row, and a rows list that defaults to an empty list. We call it twice, with two different rows.
def add_row(row, rows=[]):
rows.append(row)
return rows
print(add_row({"id": 1}))
print(add_row({"id": 2}))
print(add_row.__defaults__)[{'id': 1}]
[{'id': 1}, {'id': 2}]
([{'id': 1}, {'id': 2}],)The first call returns [{'id': 1}], but the second call returns both rows, [{'id': 1}, {'id': 2}]. Every call shares one list object, and the last line shows it: the function's __defaults__ now hold both records.
The fix is to default to None and create a new list inside the function.
def add_row(row, rows=None):
if rows is None:
rows = []
rows.append(row)
return rows
print(add_row({"id": 1}))
print(add_row({"id": 2}))[{'id': 1}]
[{'id': 2}]Now each call gets its own list: [{'id': 1}], then [{'id': 2}].
The same rule applies to dicts and sets. A dataclass refuses a list default with a ValueError, so there you use field(default_factory=list).
This bug hides in helpers that collect rows or errors per file. It passes a unit test that calls it once, and leaks data across files in a long-running job.
Question 2: You copied a list of dictionaries, changed the copy, and the original changed too. What is the difference between a shallow and a deep copy?
A shallow copy creates a new outer container that still points to the same inner objects. A deep copy copies the container and everything inside it, recursively.
Let's try it. Here are two records, each with a list of tags. We make a shallow copy with the list copy() method, a deep copy with copy.deepcopy(), and then append vip to the first record's tags in the shallow copy.
import copy
rows = [{"id": 1, "tags": ["new"]}, {"id": 2, "tags": []}]
shallow = rows.copy()
deep = copy.deepcopy(rows)
shallow[0]["tags"].append("vip")
print(rows[0])
print(deep[0])
print(shallow is rows, shallow[0] is rows[0]){'id': 1, 'tags': ['new', 'vip']}
{'id': 1, 'tags': ['new']}
False TrueThe original now has ['new', 'vip'], but the deep copy still has only ['new']. The last line explains why: shallow is rows is False, so the copy is a different list, but shallow[0] is rows[0] is True, so its first element is the very same dict.
Be careful here: copy.copy(x), list(x), slicing with x[:], dict(d) and the copy() method are all shallow. Use deepcopy when you will change nested data.
You tweak a template record or a config per file, and the change leaks into the next file. Build a fresh record per row, or deep copy the template.
Question 3: A lookup over a million IDs inside a loop takes minutes. When do you use a list, a set or a dictionary?
A membership check on a list scans the items one by one: O(n). A set or a dictionary hashes the key and jumps straight to it: O(1) on average. Inside a loop, the list version becomes O(n times m). So build the set once, before the loop.
| Membership test | Use it for | |
|---|---|---|
| LIST | scan: O(n) | ordered rows, duplicates |
| SET | hash: O(1) average | is this ID there? |
| DICT | hash: O(1) average | ID to its record |
Let's prove it. This Key class counts every equality check. We build a million keys and look for the last one, first in the list and then in a set.
class Key(int):
checks = 0 # counts every == comparison
__hash__ = int.__hash__
def __eq__(self, other):
Key.checks += 1
return super().__eq__(other)
ids = [Key(i) for i in range(1_000_000)]
for name, box in [("list", ids), ("set", set(ids))]:
Key.checks = 0
found = Key(999_999) in box
print(f"{name}: found={found}, checks={Key.checks:,}")list: found=True, checks=1,000,000
set: found=True, checks=1Both find it, but the list needed 1,000,000 checks, and a missing ID costs the same full scan. The set matched the hash, then needed 1 check.
So use a list for ordered rows, a set to ask whether an ID is there, and a dictionary when you need the record behind a key, like enriching orders with customers.
Sets also make reconciliation easy. Let's put the source keys and the loaded keys in two sets and subtract.
source_ids = {"C01", "C02", "C07", "C09"}
loaded_ids = {"C01", "C02"}
print(sorted(source_ids - loaded_ids))['C07', 'C09']The result is the keys that were never loaded: C07 and C09.
Say the complexity out loud: list scan O(n), set and dict O(1) on average, and the set is built once, outside the loop.
Question 4: A script that reads a 20 GB file runs out of memory. How do generators help, and when do you use one?
A list holds every item in memory at once. A generator produces one item at a time, only when the loop asks for it.
Let's compare a million numbers in a list and in a generator.
import sys
ids_list = [i for i in range(1_000_000)]
ids_gen = (i for i in range(1_000_000))
print(f"list: {sys.getsizeof(ids_list):,} bytes")
print(f"generator: {sys.getsizeof(ids_gen):,} bytes")
print(sum(ids_gen))
print(sum(ids_gen))list: 8,448,728 bytes
generator: 200 bytes
499999500000
0The list takes 8,448,728 bytes, and getsizeof counts only the list itself, its pointers, not the numbers in it. The generator takes 200 bytes. But a generator is single pass: the first sum gives 499999500000, and the second gives 0.
Now the big file. A generator function opens the file with an explicit encoding, and yield from csv.DictReader(f) hands back one row at a time. The filter for paid orders is a generator expression, so it is lazy too. Then itertools.batched, new in Python 3.12, groups the stream into batches of 500 for the load.
import csv
from itertools import batched
def read_rows(path):
with open(path, encoding="utf-8", newline="") as f:
yield from csv.DictReader(f)
rows = read_rows("orders.csv")
paid = (r for r in rows if r["status"] == "PAID")
for n, batch in enumerate(batched(paid, 500), 1):
print(f"batch {n}: {len(batch)} rows") # load herebatch 1: 500 rows
batch 2: 500 rows
batch 3: 200 rowsThree batches: 500, 500 and 200 rows, which is the 1,200 paid orders. Memory holds one batch, whatever the file size.
When do you use which?
- Generator: read once, top to bottom, like files, API pages and database cursors.
- List: you need to index, sort, call
len(), or loop over the data twice.
With a generator and batches, memory is one batch, not the whole file.
Question 5: The finance totals are off by a cent. Why does float give the wrong answer, and when do you use Decimal?
Float is binary. Most decimal fractions, like 0.1, have no exact binary form, so the nearest value is stored. Decimal stores base ten digits exactly.
Let's look at the float side first.
print(0.1 + 0.2)
print(0.1 + 0.2 == 0.3)
print(round(2.675, 2))
print(round(2.5), round(3.5))0.30000000000000004
False
2.67
2 40.1 + 0.2 gives 0.30000000000000004, a bit more than 0.3, so the equality check is False. Rounding shows the cent problem: round(2.675, 2) gives 2.67, because the stored float is just below the half. And round() uses banker's rounding, half to even: round(2.5) gives 2 and round(3.5) gives 4.
Now Decimal. Build it from a string, never from a float.
import json
from decimal import ROUND_HALF_UP, Decimal
print(Decimal(0.1))
print(Decimal("0.1") + Decimal("0.2") == Decimal("0.3"))
cent = Decimal("0.01")
money = Decimal("2.675")
print(money.quantize(cent, rounding=ROUND_HALF_UP))
print(json.loads('{"amount": 19.99}', parse_float=Decimal))0.1000000000000000055511151231257827021181583404541015625
True
2.68
{'amount': Decimal('19.99')}Decimal(0.1) shows the full binary error, 0.1000000000000000055511151231257827021181583404541015625. From strings, the sum equals exactly 0.3: True. For money, quantize to the cent with the rule finance agrees on, here ROUND_HALF_UP, so 2.675 becomes 2.68. And parse JSON with parse_float=Decimal, so amounts never pass through float: the result is Decimal('19.99').
Store money as a DECIMAL(p, s) column, or as integer cents. Keep float for measurements, and compare floats with math.isclose().
Question 6: A teammate compares values with is instead of ==, and a check fails in production. What is the difference?
== compares values. is compares identity: are these two names the very same object? Two equal values can still be two different objects.
| Compares | Use it for | |
|---|---|---|
| == | values | data: codes, amounts, text |
| is | identity: same object | None and sentinel objects |
Let's parse 100 twice and 5000 twice, the way a file load would, then compare a status string built at run time, and finish with None and NaN.
a, b = int("100"), int("100")
c, d = int("5000"), int("5000")
print(a == b, a is b)
print(c == d, c is d)
status, expected = "".join(["PA", "ID"]), "PAID"
print(status == expected, status is expected)
value = None
print(value is None)
nan = float("nan")
print(nan == nan)True True
True False
True False
True
False100 passes both checks: True True. 5000 is equal, but is gives False. That is why it works in testing and fails in production. CPython caches some small integers and some strings, so is happens to work on small test values, but that is an implementation detail. The status string built at run time equals "PAID", but the is check gives False again.
Use is only for None and your own sentinel objects, so value is None is the right check, and it prints True. Don't write flag is True: a value from a pandas boolean column is a NumPy bool, so that check gives False. Just write if flag. For data values, use ==.
NaN is not equal to itself, so the last line prints False, and == can't find missing values either. In pandas use pd.isna(), and in plain Python use math.isnan().
Scenario 7: You review a junior's script that loads customer files, and it works on small files but fails on big ones. What do you look for?
Here is the script.
def load(path, seen=[]):
lines = open(path).readlines()
out = pd.DataFrame()
for line in lines[1:]:
cust_id, name, amount = line.strip().split(",")
if cust_id in seen:
continue
seen.append(cust_id)
row = pd.DataFrame([{"cust_id": cust_id,
"amount": float(amount)}])
out = pd.concat([out, row], ignore_index=True)
return outLet's go through it in order.
First, anything that holds the whole file in memory. readlines() pulls every line at once. The file is also opened without an encoding, and without a with block to close it.
Second, work that grows faster than the data. The seen list makes every lookup a scan, so the loop is quadratic. And pd.concat inside the loop copies the whole DataFrame on every row.
Third, the traps from this post. seen=[] is a mutable default, so the next file starts with old IDs. Splitting on commas by hand breaks on quoted values. And the amount is a float.
So the checklist is:
- Whole file in memory:
readlines(),read() - List lookups in a loop: use a set
- concat in a loop: copies every time
- Mutable default:
seen=[]leaks - Hand-split commas: use the csv module
- Float for money: use Decimal
Now the fix, which streams it. It reuses read_rows from Question 4, which reads rows with the csv module. A set of IDs, not rows, skips repeated customers, and batched sends 20,000 rows at a time.
def unique_by(rows, key):
seen = set()
for row in rows:
if row[key] not in seen:
seen.add(row[key])
yield row
rows = unique_by(read_rows("customers.csv"), "cust_id")
for n, batch in enumerate(batched(rows, 20_000), 1):
print(f"batch {n}: {len(batch):,} rows") # insert herebatch 1: 20,000 rows
batch 2: 20,000 rows
batch 3: 20,000 rowsThree batches of 20,000 rows: the 60,000 unique customers out of 100,000 rows.
Then test it with a production-sized file, and check memory use, not only run time.
Memory should depend on the batch size, never on the file size. If even the set of IDs is too big, dedupe in the database.
Key points to remember
- Defaults are evaluated once: use None, not a mutable default
- Copies are shallow unless you deepcopy
- set and dict lookups are O(1) on average; a list scans
- Generators stream big files; batched loads them in batches
- Decimal from strings for money; is only for None and singletons
The full script
Here is everything in one script: the sample data and every step from this post, in order. Run it top to bottom in an empty folder.
Show the full Python script (sample data and every step)
# ========================================================================
# Python Interview Q&A #1 - Core Python that trips up data engineers
# TechBrothersIT. Sample data is invented for teaching.
# Every step below was run with Python 3.14.6 and pandas 3.1.0; the outputs in the video and the blog
# post are the real outputs. Run it in an empty folder: it writes small sample files.
# ========================================================================
# Step 0: sample data
import csv
# Q4: 1,800 orders, every third one cancelled
with open("orders.csv", "w", encoding="utf-8", newline="") as f:
w = csv.writer(f)
w.writerow(["order_id", "status", "amount"])
for i in range(1_800):
w.writerow([50_000 + i, "CANCELLED" if i % 3 == 0 else "PAID", f"{(i % 90) + 10}.25"])
# Q7: 100,000 customer rows, 60,000 unique customer IDs (the rest are repeats)
with open("customers.csv", "w", encoding="utf-8", newline="") as f:
w = csv.writer(f)
w.writerow(["cust_id", "name", "amount"])
for i in range(100_000):
w.writerow([f"C{i % 60_000:06d}", f"Customer {i % 60_000}", f"{(i % 500) + 1}.50"])
# Step 1: The bug: a mutable default argument
def add_row(row, rows=[]):
rows.append(row)
return rows
print(add_row({"id": 1}))
print(add_row({"id": 2}))
print(add_row.__defaults__)
# Step 2: The fix: default to None, create the list inside
def add_row(row, rows=None):
if rows is None:
rows = []
rows.append(row)
return rows
print(add_row({"id": 1}))
print(add_row({"id": 2}))
# Step 3: Shallow copy vs deep copy of a list of dicts
import copy
rows = [{"id": 1, "tags": ["new"]}, {"id": 2, "tags": []}]
shallow = rows.copy()
deep = copy.deepcopy(rows)
shallow[0]["tags"].append("vip")
print(rows[0])
print(deep[0])
print(shallow is rows, shallow[0] is rows[0])
# Step 4: Count the equality checks: list vs set membership
class Key(int):
checks = 0 # counts every == comparison
__hash__ = int.__hash__
def __eq__(self, other):
Key.checks += 1
return super().__eq__(other)
ids = [Key(i) for i in range(1_000_000)]
for name, box in [("list", ids), ("set", set(ids))]:
Key.checks = 0
found = Key(999_999) in box
print(f"{name}: found={found}, checks={Key.checks:,}")
# Step 5: Reconcile keys with set difference
source_ids = {"C01", "C02", "C07", "C09"}
loaded_ids = {"C01", "C02"}
print(sorted(source_ids - loaded_ids))
# Step 6: A list holds everything, a generator holds one item
import sys
ids_list = [i for i in range(1_000_000)]
ids_gen = (i for i in range(1_000_000))
print(f"list: {sys.getsizeof(ids_list):,} bytes")
print(f"generator: {sys.getsizeof(ids_gen):,} bytes")
print(sum(ids_gen))
print(sum(ids_gen))
# Step 7: Stream a big file with a generator and load it in batches
import csv
from itertools import batched
def read_rows(path):
with open(path, encoding="utf-8", newline="") as f:
yield from csv.DictReader(f)
rows = read_rows("orders.csv")
paid = (r for r in rows if r["status"] == "PAID")
for n, batch in enumerate(batched(paid, 500), 1):
print(f"batch {n}: {len(batch)} rows") # load here
# Step 8: Why float is off: binary fractions and rounding
print(0.1 + 0.2)
print(0.1 + 0.2 == 0.3)
print(round(2.675, 2))
print(round(2.5), round(3.5))
# Step 9: Decimal for money: from strings, quantize to the cent
import json
from decimal import ROUND_HALF_UP, Decimal
print(Decimal(0.1))
print(Decimal("0.1") + Decimal("0.2") == Decimal("0.3"))
cent = Decimal("0.01")
money = Decimal("2.675")
print(money.quantize(cent, rounding=ROUND_HALF_UP))
print(json.loads('{"amount": 19.99}', parse_float=Decimal))
# Step 10: is compares identity, == compares values
a, b = int("100"), int("100")
c, d = int("5000"), int("5000")
print(a == b, a is b)
print(c == d, c is d)
status, expected = "".join(["PA", "ID"]), "PAID"
print(status == expected, status is expected)
value = None
print(value is None)
nan = float("nan")
print(nan == nan)
# Step 11: Scenario: stream, skip duplicates with a set, load in batches
def unique_by(rows, key):
seen = set()
for row in rows:
if row[key] not in seen:
seen.add(row[key])
yield row
rows = unique_by(read_rows("customers.csv"), "cust_id")
for n, batch in enumerate(batched(rows, 20_000), 1):
print(f"batch {n}: {len(batch):,} rows") # insert here
Part 2: Reading and Writing CSV, JSON and Parquet
The sample data on this page is invented for teaching. Every output is the real output of the code above it, run with Python 3.14.6 and pandas 3.1.0. Part of the Real Time Python for Data Engineers Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Python and library documentation for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.