In this post we are going to cover monitoring, inventory and cost in Azure Data Lake Storage Gen2: blob inventory reports, the metrics to watch, throttling and retries, diagnostic logs, transaction costs and cost by team. These are 7 real ADLS Gen2 and Blob Storage interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: the CFO asks why the data lake costs keep rising and what you will do about it.
Running a lake in production means knowing what is stored, who touches it, and what it costs. Azure gives us inventory reports, metrics and logs for that, and each answers a different question. Interviewers ask about this because a lake nobody watches gets slow and expensive, and because "who deleted this file?" always comes up on the day the logs were never turned on.
Part 9 of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Let's set up the project
- Question 1: Your manager asks how to see what is stored in the lake and how big it is. How do blob inventory reports help?
- Question 2: Someone asks which metrics to watch on a storage account. What do you monitor?
- Question 3: A pipeline fails with throttling errors from storage. What are the limits, and how do you handle them?
- Question 4: A teammate asks how to find out who deleted or read a file. How do diagnostic logs help?
- Question 5: Storage transaction costs jumped this month. Where do you look?
- Question 6: Someone asks how to tag and track data cost by team. What do you set up?
- Scenario 7: The CFO asks why the data lake costs keep rising and what you will do about it. What is your plan?
- The full script
The short answer
Know what you store with blob inventory and the capacity metrics, watch Transactions split by ResponseType, ApiName and Authentication, and find callers in StorageBlobLogs. Throttling is 503 ServerBusy: retry with backoff and send fewer, larger requests. Cut cost with tiering, compaction and showback by team.
Let's set up the project
We run Azurite, Microsoft's open-source Azure Storage emulator, locally, and small Python scripts with the azure-storage-blob client and pandas, so every output below is real. Features the emulator does not support, like inventory, metrics, logs, lifecycle and throttling, are shown as code. common.py creates the client, and its container() helper creates a container, with optional metadata, if it is missing.
import os
from azure.core.exceptions import ResourceExistsError
from azure.storage.blob import BlobServiceClient
CONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
svc = BlobServiceClient.from_connection_string(CONN)
def container(name, **meta):
try:
return svc.create_container(name, metadata=meta or None)
except ResourceExistsError:
return svc.get_container_client(name)Question 1: Your manager asks how to see what is stored in the lake and how big it is. How do blob inventory reports help?
Blob inventory is a scheduled report of every blob, or every container, in the account. Azure writes a CSV or Parquet file to a container you choose, daily or weekly, with the fields you pick, like name, size and access tier.
- Storage account - the lake
- Inventory rule - daily or weekly
- inventory container - report + manifest
- pandas / Spark - query the report
Azurite has no inventory, so let's build the same kind of table. seed_lake.py loads a small lake, and inventory.py lists it and summarizes it with pandas: files and kilobytes per container and area, then by tier. We run python seed_lake.py, then python inventory.py.
from common import container
lake = { # container: [(path, size in KB)]
"raw": [("sales/orders/2026/10/29/orders.csv", 96),
("sales/orders/2026/10/30/orders.csv", 104),
("clickstream/2026/10/30/h09/events.json", 380),
("clickstream/2026/10/30/h10/events.json", 420),
("finance/gl/2026/10/gl_extract.csv", 64)],
"curated": [("sales/orders/part-0.parquet", 48),
("finance/gl/part-0.parquet", 22)],
"serving": [("sales/daily_revenue.parquet", 6)],
}
for name, files in lake.items():
c = container(name)
for path, kb in files:
c.upload_blob(path, b"x" * kb * 1024, overwrite=True)
print("lake loaded:", sum(len(f) for f in lake.values()), "files")import pandas as pd
from common import svc
rows = [{"container": c.name, "name": b.name, "tier": b.blob_tier,
"size": b.size}
for c in svc.list_containers()
for b in svc.get_container_client(c.name).list_blobs()]
df = pd.DataFrame(rows) # one row per blob
df["area"] = df["name"].str.split("/").str[0]
report = (df.groupby(["container", "area"])
.agg(files=("name", "count"), kb=("size", "sum")))
report["kb"] //= 1024
print(report.sort_values("kb", ascending=False).to_string())
print("total:", len(df), "files,", df["size"].sum() // 1024, "KB")
print("KB by tier:", (df.groupby("tier")["size"].sum() // 1024)
.to_dict())$ python seed_lake.py
lake loaded: 8 files
$ python inventory.py
files kb
container area
raw clickstream 2 800
sales 2 200
finance 1 64
curated sales 1 48
finance 1 22
serving sales 1 6
total: 8 files, 1140 KB
KB by tier: {'Hot': 1140}Clickstream in raw is the biggest area, 800 KB in 2 files. In total we have 8 files and 1140 KB, and every file sits in Hot. On Azure, you read the inventory file the same way, instead of listing millions of blobs yourself.
The policy is JSON: a destination container that must exist, the format, the schedule, and filters. Each run writes a manifest and a checksum file, which marks the run as complete.
{
"enabled": true,
"rules": [{
"enabled": true,
"name": "lake-daily",
"destination": "inventory",
"definition": {
"format": "parquet",
"schedule": "daily",
"objectType": "blob",
"schemaFields": ["Name", "Content-Length",
"AccessTier", "Last-Modified"],
"filters": {"blobTypes": ["blockBlob"],
"prefixMatch": ["raw/", "curated/"]}
}
}]
}
az storage account blob-inventory-policy create \
--account-name stlakeprod -g rg-lake --policy @policy.jsonInventory is billed per million objects scanned, on every run, so use weekly where daily isn't needed. And for a quick total, the BlobCapacity and BlobCount metrics already split by tier.
Question 2: Someone asks which metrics to watch on a storage account. What do you monitor?
Four groups: capacity, traffic, health and errors. And the Transactions metric, split by its dimensions, answers most of the questions.
| Group | Metric | Split by |
|---|---|---|
| Capacity | UsedCapacity, BlobCapacity, BlobCount | Tier, BlobType |
| Traffic | Transactions, Ingress, Egress | ApiName: which operations you pay for |
| Health | Availability, SuccessE2ELatency, SuccessServerLatency | ApiName |
| Errors | Transactions | ResponseType: ServerBusyError, AuthorizationError |
| Who | Transactions | Authentication: AccountKey, SAS, OAuth, Anonymous |
Let's go through it. Capacity is used capacity, plus blob capacity and blob count split by tier. Capacity values refresh several times a day, not on a fixed schedule, so watch the trend, not the minute. Traffic is transactions, ingress and egress. Health is availability and latency.
Transactions is the most useful metric because of its dimensions. Split by ResponseType to see throttling and authorization errors, by ApiName to see which operations you pay for, and by Authentication to see who still uses account keys.
Then alert on what needs action:
- Availability - drops below the target
- ServerBusyError - transactions above zero
- Capacity - grows faster than the plan
# metric alerts on the lake account (Azure, not run here)
LAKE=$(az storage account show -n stlakeprod -g rg-lake \
--query id -o tsv)
T="total Transactions > 0 where ResponseType includes"
az monitor metrics alert create -n lake-busy -g rg-lake \
--scopes $LAKE --window-size 5m --evaluation-frequency 1m \
--condition "$T ServerBusyError"
az monitor metrics alert create -n lake-availability \
-g rg-lake --scopes $LAKE --condition "avg Availability < 99"
# which operations, hour by hour, over the last day
az monitor metrics list --resource $LAKE --metrics Transactions \
--dimension ApiName --aggregation Total --interval 1h --offset 1dMetrics tell you what happened, not who did it. That is the logs, in question 4.
I keep these alerts in code next to the storage account, so every new account gets them on day one.
Question 3: A pipeline fails with throttling errors from storage. What are the limits, and how do you handle them?
A standard account has scale targets for requests, ingress and egress. Past them, storage answers 503 ServerBusy, or 500 OperationTimedOut.
| Standard account scale target | Value |
|---|---|
| Requests per second | 20,000 (40,000 in the large regions) |
| Ingress | 25 Gbps (60 Gbps in the large regions) |
| Egress | 50 Gbps (200 Gbps in the large regions) |
| One block blob | up to 3,000 requests per second |
The message names the limit: "Ingress / Egress / Operations per second is over the account limit." And OperationTimedOut may or may not have succeeded, so check the state before you retry.
The Python SDK retries these by itself, with exponential backoff. The emulator never throttles, so the limits and messages come from the Microsoft docs, but the retry numbers come from the real SDK. retry_policy.py prints the default policy, the planned waits, and the waits of a shorter nightly policy with six retries. We run python retry_policy.py.
from azure.storage.blob import BlobServiceClient, ExponentialRetry
from common import CONN
d = ExponentialRetry() # every client's default
print("policy:", type(d).__name__)
print("retries:", d.total_retries, "| initial_backoff:",
d.initial_backoff, "s | increment_base:", d.increment_base,
"| jitter: +/-", d.random_jitter_range, "s")
plans = {"default": ExponentialRetry(random_jitter_range=0),
"nightly": ExponentialRetry(initial_backoff=5,
increment_base=2, retry_total=6,
random_jitter_range=0)}
for name, p in plans.items(): # the SDK counts from 1
waits = [round(p.get_backoff_time({"count": n}))
for n in range(1, p.total_retries + 1)]
print(f"{name:7} waits (s): {waits} | total {sum(waits)}")
nightly = ExponentialRetry(initial_backoff=5, increment_base=2,
retry_total=6)
svc = BlobServiceClient.from_connection_string(CONN,
retry_policy=nightly)$ python retry_policy.py
policy: ExponentialRetry
retries: 3 | initial_backoff: 15 s | increment_base: 3 | jitter: +/- 3 s
default waits (s): [18, 24, 42] | total 84
nightly waits (s): [7, 9, 13, 21, 37, 69] | total 156The default is ExponentialRetry with 3 retries, an initial backoff of 15 s and an increment base of 3. That gives waits of about 18, 24 and 42 seconds, 84 in total, each plus or minus 3 seconds of jitter. The nightly policy starts shorter: 7, 9, 13, 21, 37 and 69 seconds, 156 in total.
Retries alone don't fix it. Send fewer, larger requests:
- Blocks over 256 KiB - fewer, larger requests
- Files 256 MB to 100 GB - not thousands of tiny ones
- Spread the load - or ask support for higher limits
I start shorter and allow six retries, so a busy minute slows the job down instead of failing it.
Question 4: A teammate asks how to find out who deleted or read a file. How do diagnostic logs help?
Create a diagnostic setting on the blob service with the StorageRead, StorageWrite and StorageDelete categories, and send them to a Log Analytics workspace. Every request then lands in the StorageBlobLogs table.
- Blob service - read, write, delete
- Diagnostic setting - StorageRead, Write, Delete
- Log Analytics - table StorageBlobLogs
- KQL query - who, when, from where
First, the setting:
# send blob logs to Log Analytics (Azure, not run here)
az monitor diagnostic-settings create -n lake-blob-logs \
--resource "$LAKE/blobServices/default" --workspace $LAW \
--logs '[{"category": "StorageRead", "enabled": true},
{"category": "StorageWrite", "enabled": true},
{"category": "StorageDelete", "enabled": true}]'Each row has the operation name, the status, the caller's IP address and the authentication type. When the caller used Microsoft Entra ID, it also has the requester's object ID, and for a user, the UPN.
| Field | Tells you |
|---|---|
| OperationName | what: DeleteBlob, GetBlob |
| StatusCode | did it work |
| AuthenticationType | AccountKey, SAS, OAuth |
| RequesterObjectId | who: Entra ID (UPN for users) |
| CallerIpAddress | from where |
Then this KQL query lists the deletes under one path over the last seven days:
// who deleted files under raw/sales/orders?
StorageBlobLogs
| where TimeGenerated > ago(7d)
| where OperationName startswith "Delete"
| where Uri has "/raw/sales/orders/"
| project TimeGenerated, OperationName, StatusCode,
AuthenticationType, RequesterUpn, RequesterObjectId,
CallerIpAddress, Uri
| order by TimeGenerated descWith an account key or a SAS, the log shows the key type and the IP address, not a person. That is one more reason to move pipelines and people to Entra ID.
Logs start when you enable them, and nothing before that is recorded. And Log Analytics costs money, so set a sensible retention and don't send every account at full volume forever.
Question 5: Storage transaction costs jumped this month. Where do you look?
First the bill, then the Transactions metric by ApiName, then the logs. The usual cause is small files.
- The bill - which meter grew: write, read, list, other
- Transactions by ApiName - PutBlob, GetBlob, ListBlobs, SetBlobTier
- StorageBlobLogs - which job or caller sends them
Operations are billed per 10,000 and metered in 4 MiB steps, so a tiny file costs a full operation. Let's count. small_files.py hooks into the SDK and counts every HTTP request it sends, by method. It writes and reads 500 tiny event files, then the same events as one compacted file. We run python small_files.py.
import json
from collections import Counter
from azure.storage.blob import BlobServiceClient
from common import CONN, container
calls = Counter()
def count(resp): # every HTTP request sent
calls[resp.http_request.method] += 1
def report(label):
print(label, dict(calls)); calls.clear()
container("clicks")
svc = BlobServiceClient.from_connection_string(CONN,
raw_response_hook=count)
c = svc.get_container_client("clicks")
events = [{"user": f"u{i % 40}", "page": "/cart", "n": i}
for i in range(500)]
for e in events: # one tiny file per event
c.upload_blob(f"tiny/ev-{e['n']:04}.json", json.dumps(e))
report("500 tiny files, write:")
for b in c.list_blobs(name_starts_with="tiny/"):
c.download_blob(b.name).readall()
report("500 tiny files, read :")
c.upload_blob("compact/events.jsonl",
"\n".join(json.dumps(e) for e in events))
report("1 compacted file, write:")
for b in c.list_blobs(name_starts_with="compact/"):
c.download_blob(b.name).readall()
report("1 compacted file, read :")$ python small_files.py
500 tiny files, write: {'PUT': 500}
500 tiny files, read : {'GET': 501}
1 compacted file, write: {'PUT': 1}
1 compacted file, read : {'GET': 2}The 500 tiny files cost 500 PUTs to write, and 501 GETs to read: one list, plus one per file. The same events in one compacted file cost 1 PUT, and 2 GETs.
Other suspects to check:
- Full listings - every few minutes
- HNS list and rename - their own meters
- Bulk tier changes
- Last access tracking - updates
Not in storage: batch small writes, compact files, and list only the partition you need.
Question 6: Someone asks how to tag and track data cost by team. What do you set up?
In my experience, the cost reports see the storage account as one resource, and containers don't get their own cost line. So if a team must pay its exact bill, it gets its own account, with Azure tags like team and cost center.
# one account per team: tags flow into the cost reports (not run)
az storage account update -n stlakefinance -g rg-lake \
--tags team=finance costcenter=cc-200Inside a shared account, use showback. Give each container metadata, team and cost center, and measure the bytes per team. showback_seed.py creates four team containers with that metadata, and showback.py lists every container with its metadata and sums the blob sizes. We run python showback_seed.py, then python showback.py.
from common import container
teams = {"sales-raw": ("sales", "cc-100", [300, 220, 180]),
"sales-curated": ("sales", "cc-100", [90, 60]),
"finance-raw": ("finance", "cc-200", [64, 64, 48, 40]),
"marketing-raw": ("marketing", "cc-300", [900, 700])}
for name, (team, cc, sizes) in teams.items():
c = container(name, team=team, costcenter=cc)
for i, kb in enumerate(sizes):
c.upload_blob(f"2026/10/file-{i}.csv", b"x" * kb * 1024,
overwrite=True)
print("team containers:", len(teams))from collections import defaultdict
from common import svc
usage = defaultdict(lambda: [0, 0, 0]) # containers, files, bytes
for c in svc.list_containers(include_metadata=True):
meta = c.metadata or {}
key = (meta.get("team", "(untagged)"), meta.get("costcenter", "-"))
usage[key][0] += 1
for b in svc.get_container_client(c.name).list_blobs():
usage[key][1] += 1
usage[key][2] += b.size
total = sum(u[2] for u in usage.values())
for (team, cc), (n, files, size) in sorted(usage.items()):
print(f"{team:10} {cc:6} containers {n} files {files:>3}"
f" {size // 1024:>5} KB {size / total:4.0%}")$ python showback_seed.py
team containers: 4
$ python showback.py
(untagged) - containers 4 files 509 1181 KB 31%
finance cc-200 containers 1 files 4 216 KB 6%
marketing cc-300 containers 1 files 2 1600 KB 42%
sales cc-100 containers 2 files 5 850 KB 22%Marketing holds 42 percent of the bytes, sales 22 and finance 6. And the first line is the untagged data: 4 containers, 509 files, 31 percent that nobody owns. On a real lake you get these numbers from the inventory report, not from listing.
Blob index tags can label single files, but on hierarchical namespace accounts they are in preview, and they don't drive billing.
Check for missing team tags in the deployment pipeline, so untagged storage never gets created.
Scenario 7: The CFO asks why the data lake costs keep rising and what you will do about it. What is your plan?
I answer with data, in four steps:
- Measure - inventory, capacity by tier
- Find the drivers - transactions by ApiName, small files
- Act on the biggest line - tiering, deletes, compaction
- Report every month - bytes by tier, cost per team
Let's run it. cost_seed.py writes one 40 KB telemetry file per day since April, all in Hot. cost_report.py groups the kilobytes by age and tier as of October 30, and --apply moves old files in batch calls: over 90 days to Cold, 31 to 90 to Cool. We run python cost_seed.py, python cost_report.py --apply, then python cost_report.py.
from datetime import date, timedelta
from common import container
c = container("telemetry")
day = date(2026, 4, 1)
while day <= date(2026, 10, 30): # one file per day
c.upload_blob(f"raw/devices/{day:%Y/%m/%d}/readings.json",
b"x" * 40 * 1024, overwrite=True)
day += timedelta(days=1)
print("telemetry: one 40 KB file per day, 2026-04-01 to 2026-10-30")import sys
from collections import Counter
from datetime import date
from common import svc
AS_OF = date(2026, 10, 30) # the report date
c = svc.get_container_client("telemetry")
age, tier, moves = Counter(), Counter(), {"Cool": [], "Cold": []}
for b in c.list_blobs(name_starts_with="raw/"):
y, m, d = map(int, b.name.split("/")[2:5]) # date in the path
days = (AS_OF - date(y, m, d)).days
age["0-30" if days <= 30 else "31-90" if days <= 90
else "over 90"] += b.size // 1024
tier[b.blob_tier] += b.size // 1024
if days > 30:
moves["Cold" if days > 90 else "Cool"].append(b.name)
print("KB by age :", dict(age))
print("KB by tier:", dict(tier))
if "--apply" in sys.argv:
for t, names in moves.items():
for i in range(0, len(names), 256): # up to 256 per batch
c.set_standard_blob_tier_blobs(t, *names[i:i + 256])
print("moved to", t + ":", len(names), "files")$ python cost_seed.py
telemetry: one 40 KB file per day, 2026-04-01 to 2026-10-30
$ python cost_report.py --apply
KB by age : {'over 90': 4880, '31-90': 2400, '0-30': 1240}
KB by tier: {'Hot': 8520}
moved to Cool: 60 files
moved to Cold: 122 files
$ python cost_report.py
KB by age : {'over 90': 4880, '31-90': 2400, '0-30': 1240}
KB by tier: {'Cold': 4880, 'Cool': 2400, 'Hot': 1240}The report finds 4880 KB older than 90 days, 2400 KB between 31 and 90, and 1240 KB newer, all in Hot. The batches move 60 files to Cool and 122 to Cold, and the second report shows Cold 4880, Cool 2400, Hot 1240.
In production, a lifecycle policy does this daily: cool after 30 days, cold after 90, delete raw data after a year.
{"rules": [{
"name": "telemetry-tiering",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {"blobTypes": ["blockBlob"],
"prefixMatch": ["telemetry/raw/"]},
"actions": {"baseBlob": {
"tierToCool": {"daysAfterModificationGreaterThan": 30},
"tierToCold": {"daysAfterModificationGreaterThan": 90},
"delete": {"daysAfterModificationGreaterThan": 365}}}
}
}]}
az storage account management-policy create \
--account-name stlakeprod -g rg-lake --policy @policy.jsonMind the minimums: 30 days for cool, 90 for cold, 180 for archive, or you pay early deletion. Then the other levers:
- Smart tier - GA April 2026, ZRS family, no archive
- Compact - small files
- Reserved capacity - 1 or 3 years
Every month: bytes by tier, transactions by meter, and cost per team. The CFO wants the trend, not one cleanup.
Key points to remember
- Blob inventory: a daily or weekly CSV or Parquet report of every blob
- Watch Transactions by ResponseType, ApiName and Authentication
- Throttling = 503 ServerBusy: backoff retries, fewer and larger requests
- StorageBlobLogs say who did what; turn them on before the incident
- Cut cost: tiering, compaction, showback by team, a monthly report
The full script
This is a Python script that writes every file and runs every command above in order against Azurite: run pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas first.
Show the full script that writes the files and runs every step
# ========================================================================
# ADLS Gen2 Interview Q&A #9 - Monitoring, inventory and cost
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run on Azurite 3.37.0 (Microsoft's open-source Azure Storage emulator, Blob service)
# with azure-storage-blob 12.31.1, pyarrow and pandas; the outputs in the video and the blog post are the real outputs.
# To run it yourself:
# 1. npm install -g azurite then azurite-blob --skipApiVersionCheck --location ./azurite-data
# (or the Docker image mcr.microsoft.com/azure-storage/azurite), and leave it running on port 10000.
# --skipApiVersionCheck lets a newer azure-storage-blob talk to it.
# 2. pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas requests
# 3. Set AZURE_STORAGE_CONNECTION_STRING to "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
# then run python ADLS_Interview_QA_09_Code.py.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================
import os, subprocess, time
def write(path, text):
if text is None:
if os.path.exists(path): os.remove(path)
return
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
f.write(text.strip("\n") + "\n")
def run(cmd, expect_error=False, allow_error=False, env=None):
print("\n$ " + cmd, flush=True)
rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
if rc != 0 and not (expect_error or allow_error):
raise SystemExit("stopped: this command was expected to succeed")
# Step 0: the starting files
write('common.py', """\
import os
from azure.core.exceptions import ResourceExistsError
from azure.storage.blob import BlobServiceClient
CONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
svc = BlobServiceClient.from_connection_string(CONN)
def container(name, **meta):
try:
return svc.create_container(name, metadata=meta or None)
except ResourceExistsError:
return svc.get_container_client(name)
""")
# Step 1: A small lake, then an inventory-style report: files and KB per container, area and tier
write('seed_lake.py', """\
from common import container
lake = { # container: [(path, size in KB)]
"raw": [("sales/orders/2026/10/29/orders.csv", 96),
("sales/orders/2026/10/30/orders.csv", 104),
("clickstream/2026/10/30/h09/events.json", 380),
("clickstream/2026/10/30/h10/events.json", 420),
("finance/gl/2026/10/gl_extract.csv", 64)],
"curated": [("sales/orders/part-0.parquet", 48),
("finance/gl/part-0.parquet", 22)],
"serving": [("sales/daily_revenue.parquet", 6)],
}
for name, files in lake.items():
c = container(name)
for path, kb in files:
c.upload_blob(path, b"x" * kb * 1024, overwrite=True)
print("lake loaded:", sum(len(f) for f in lake.values()), "files")
""")
write('inventory.py', """\
import pandas as pd
from common import svc
rows = [{"container": c.name, "name": b.name, "tier": b.blob_tier,
"size": b.size}
for c in svc.list_containers()
for b in svc.get_container_client(c.name).list_blobs()]
df = pd.DataFrame(rows) # one row per blob
df["area"] = df["name"].str.split("/").str[0]
report = (df.groupby(["container", "area"])
.agg(files=("name", "count"), kb=("size", "sum")))
report["kb"] //= 1024
print(report.sort_values("kb", ascending=False).to_string())
print("total:", len(df), "files,", df["size"].sum() // 1024, "KB")
print("KB by tier:", (df.groupby("tier")["size"].sum() // 1024)
.to_dict())
""")
run('python seed_lake.py')
run('python inventory.py')
# Step 2: The SDK retry policy: the default and a nightly policy, with the planned waits
write('retry_policy.py', """\
from azure.storage.blob import BlobServiceClient, ExponentialRetry
from common import CONN
d = ExponentialRetry() # every client's default
print("policy:", type(d).__name__)
print("retries:", d.total_retries, "| initial_backoff:",
d.initial_backoff, "s | increment_base:", d.increment_base,
"| jitter: +/-", d.random_jitter_range, "s")
plans = {"default": ExponentialRetry(random_jitter_range=0),
"nightly": ExponentialRetry(initial_backoff=5,
increment_base=2, retry_total=6,
random_jitter_range=0)}
for name, p in plans.items(): # the SDK counts from 1
waits = [round(p.get_backoff_time({"count": n}))
for n in range(1, p.total_retries + 1)]
print(f"{name:7} waits (s): {waits} | total {sum(waits)}")
nightly = ExponentialRetry(initial_backoff=5, increment_base=2,
retry_total=6)
svc = BlobServiceClient.from_connection_string(CONN,
retry_policy=nightly)
""")
run('python retry_policy.py')
# Step 3: Count the HTTP requests: 500 tiny files against one compacted file
write('small_files.py', 'import json\nfrom collections import Counter\nfrom azure.storage.blob import BlobServiceClient\nfrom common import CONN, container\n\ncalls = Counter()\ndef count(resp): # every HTTP request sent\n calls[resp.http_request.method] += 1\ndef report(label):\n print(label, dict(calls)); calls.clear()\n\ncontainer("clicks")\nsvc = BlobServiceClient.from_connection_string(CONN,\n raw_response_hook=count)\nc = svc.get_container_client("clicks")\nevents = [{"user": f"u{i % 40}", "page": "/cart", "n": i}\n for i in range(500)]\nfor e in events: # one tiny file per event\n c.upload_blob(f"tiny/ev-{e[\'n\']:04}.json", json.dumps(e))\nreport("500 tiny files, write:")\nfor b in c.list_blobs(name_starts_with="tiny/"):\n c.download_blob(b.name).readall()\nreport("500 tiny files, read :")\nc.upload_blob("compact/events.jsonl",\n "\\n".join(json.dumps(e) for e in events))\nreport("1 compacted file, write:")\nfor b in c.list_blobs(name_starts_with="compact/"):\n c.download_blob(b.name).readall()\nreport("1 compacted file, read :")\n')
run('python small_files.py')
# Step 4: Showback: containers with team metadata, bytes per team
write('showback_seed.py', """\
from common import container
teams = {"sales-raw": ("sales", "cc-100", [300, 220, 180]),
"sales-curated": ("sales", "cc-100", [90, 60]),
"finance-raw": ("finance", "cc-200", [64, 64, 48, 40]),
"marketing-raw": ("marketing", "cc-300", [900, 700])}
for name, (team, cc, sizes) in teams.items():
c = container(name, team=team, costcenter=cc)
for i, kb in enumerate(sizes):
c.upload_blob(f"2026/10/file-{i}.csv", b"x" * kb * 1024,
overwrite=True)
print("team containers:", len(teams))
""")
write('showback.py', """\
from collections import defaultdict
from common import svc
usage = defaultdict(lambda: [0, 0, 0]) # containers, files, bytes
for c in svc.list_containers(include_metadata=True):
meta = c.metadata or {}
key = (meta.get("team", "(untagged)"), meta.get("costcenter", "-"))
usage[key][0] += 1
for b in svc.get_container_client(c.name).list_blobs():
usage[key][1] += 1
usage[key][2] += b.size
total = sum(u[2] for u in usage.values())
for (team, cc), (n, files, size) in sorted(usage.items()):
print(f"{team:10} {cc:6} containers {n} files {files:>3}"
f" {size // 1024:>5} KB {size / total:4.0%}")
""")
run('python showback_seed.py')
run('python showback.py')
# Step 5: Cost drivers: KB by age and tier, then move old files to Cool and Cold in batches
write('cost_seed.py', """\
from datetime import date, timedelta
from common import container
c = container("telemetry")
day = date(2026, 4, 1)
while day <= date(2026, 10, 30): # one file per day
c.upload_blob(f"raw/devices/{day:%Y/%m/%d}/readings.json",
b"x" * 40 * 1024, overwrite=True)
day += timedelta(days=1)
print("telemetry: one 40 KB file per day, 2026-04-01 to 2026-10-30")
""")
write('cost_report.py', """\
import sys
from collections import Counter
from datetime import date
from common import svc
AS_OF = date(2026, 10, 30) # the report date
c = svc.get_container_client("telemetry")
age, tier, moves = Counter(), Counter(), {"Cool": [], "Cold": []}
for b in c.list_blobs(name_starts_with="raw/"):
y, m, d = map(int, b.name.split("/")[2:5]) # date in the path
days = (AS_OF - date(y, m, d)).days
age["0-30" if days <= 30 else "31-90" if days <= 90
else "over 90"] += b.size // 1024
tier[b.blob_tier] += b.size // 1024
if days > 30:
moves["Cold" if days > 90 else "Cool"].append(b.name)
print("KB by age :", dict(age))
print("KB by tier:", dict(tier))
if "--apply" in sys.argv:
for t, names in moves.items():
for i in range(0, len(names), 256): # up to 256 per batch
c.set_standard_blob_tier_blobs(t, *names[i:i + 256])
print("moved to", t + ":", len(names), "files")
""")
run('python cost_seed.py')
run('python cost_report.py --apply')
run('python cost_report.py')
Part 10: Real-World Manager Scenarios
The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run on Azurite 3.37.0, Microsoft's open-source Azure Storage emulator, with azure-storage-blob 12.31.1, pyarrow and pandas. Features the emulator does not support are shown as code. Part of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Azure Storage documentation (learn.microsoft.com/azure/storage) for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.