In this post we are going to cover access tiers and lifecycle management. These are 7 real ADLS Gen2 and Blob Storage interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: storage costs keep growing, most data is never read after ninety days, and you have to cut the bill.
Access tiers are price points for the same data: hot, cool, cold and archive. Lifecycle management is the set of rules that moves blobs between those tiers, or deletes them, by age or by last access. Interviewers ask about this because storage cost is one of the first things a manager notices, and because a wrong rule can either waste money or delete data you still need.
Part 3 of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Let's set up the project
- Question 1: Your manager asks what hot, cool, cold and archive tiers are. How do you explain them?
- Question 2: A teammate wants to read a file that is in the archive tier. What happens, and how long does it take?
- Question 3: Someone asks how to move old data to cheaper tiers automatically. How do lifecycle management policies work?
- Question 4: Data is moved to cool and then read again a week later. What extra costs appear?
- Question 5: A teammate asks about default account tiers versus setting the tier per blob. What is the difference?
- Question 6: Someone asks how to delete raw files automatically after a year. How do you set it up safely?
- Scenario 7: Storage costs keep growing and most data is never read after ninety days. How do you cut the bill?
- The full script
The short answer
Hot, cool, cold and archive trade storage price against access price, with minimum stays of 30, 90 and 180 days. Archive is offline: rehydration takes hours. Lifecycle policies tier and delete by age or last access once a day; measure first, dry-run prefixes and turn on soft delete before any delete rule.
Let's set up the project
We run everything against Azurite, Microsoft's open-source Azure Storage emulator, running locally, with small Python scripts that use the azure-storage-blob client and pyarrow. Features the emulator does not support, such as the hierarchical namespace, lifecycle policies and rehydration times, are shown as code. Every script imports common.py, which connects to the account, creates a container on first use and prints errors in a short form.
import os
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
def container(name):
"""Get a container client, creating the container on first use."""
c = svc.get_container_client(name)
if not c.exists():
c.create_container()
return c
def show_err(e):
code = getattr(e.error_code, "value", e.error_code)
print(" ", type(e).__name__, e.status_code, code)
print(" ", str(e).split("\n")[0])Question 1: Your manager asks what hot, cool, cold and archive tiers are. How do you explain them?
They are price points for the same block blob. Hot costs the most to store and the least to read. Cool, then cold, store more cheaply but charge more to read. Archive is the cheapest to store, but it is offline.
| Tier | Storage cost | Access cost | Minimum stay | Read latency |
|---|---|---|---|---|
| Hot | highest | lowest | none | milliseconds |
| Cool | lower | higher | 30 days | milliseconds |
| Cold | lower still | higher still | 90 days | milliseconds |
| Archive | lowest | highest, plus rehydration | 180 days | hours: offline |
Each cooler tier has a minimum stay: 30 days for cool, 90 for cold and 180 for archive. If you delete, overwrite or change the tier earlier, you pay an early deletion charge for the remaining days, prorated.
Let's put four monthly order exports into the raw container, each with its own tier at upload: the account default, cool, cold and archive. We run python tiers.py, which uploads them and lists the tier of each blob.
from azure.storage.blob import StandardBlobTier as Tier
from common import container
raw = container("raw")
exports = {"sales/2026/10/orders.csv": None, # account default
"sales/2026/09/orders.csv": Tier.COOL,
"sales/2026/07/orders.csv": Tier.COLD,
"sales/2025/10/orders.csv": Tier.ARCHIVE}
for name, tier in exports.items():
raw.upload_blob(name, b"order_id,amount\n1001,25.50\n",
standard_blob_tier=tier, overwrite=True)
for b in raw.list_blobs(name_starts_with="sales/"):
print(f"{b.name} {b.size} bytes tier {b.blob_tier}")$ python tiers.py
sales/2025/10/orders.csv 27 bytes tier Archive
sales/2026/07/orders.csv 27 bytes tier Cold
sales/2026/09/orders.csv 27 bytes tier Cool
sales/2026/10/orders.csv 27 bytes tier HotEach file is 27 bytes. sales/2026/10/orders.csv follows the account default, tier Hot. September is Cool, July is Cold, and last October is Archive.
Hot, cool and cold all read in milliseconds; archive takes hours. And tiers apply to block blobs only, not to append or page blobs.
I pick the tier from how often data is read, not from its age. Old data that a monthly report reads belongs in cool, not archive.
Question 2: A teammate wants to read a file that is in the archive tier. What happens, and how long does it take?
The read fails. An archived blob is offline: you can read its properties, but not its data, until you rehydrate it to an online tier, and that takes hours.
Let's try it on last October's export. We run python read_archive.py, which reads the blob properties first and then tries to download the data.
from azure.core.exceptions import HttpResponseError
from common import container, show_err
blob = container("raw").get_blob_client("sales/2025/10/orders.csv")
p = blob.get_blob_properties() # properties: allowed
print("tier:", p.blob_tier, "| size:", p.size, "bytes")
try:
blob.download_blob().readall() # the data: offline
except HttpResponseError as e:
show_err(e)$ python read_archive.py
tier: Archive | size: 27 bytes
ResourceExistsError 409 BlobArchived
This operation is not permitted on an archived blob.The properties work and show tier: Archive | size: 27 bytes. The download fails with ResourceExistsError 409 BlobArchived: "This operation is not permitted on an archived blob."
There are two ways back. First, set the tier to hot, cool or cold, and the blob stays archived until rehydration finishes. Or copy it to a new blob in an online tier, which has no early deletion charge. The emulator rehydrates instantly, so the Azure behavior is shown as code.
from azure.storage.blob import RehydratePriority, StandardBlobTier
# 1. rehydrate in place: the blob stays Archive until it is done
blob.set_standard_blob_tier("Hot", rehydrate_priority="Standard")
print(blob.get_blob_properties().archive_status)
# Azure: rehydrate-pending-to-hot, for hours
# 2. copy to a new blob in an online tier: no early deletion charge
copy = container("raw").get_blob_client("restore/2025-10-orders.csv")
copy.start_copy_from_url(blob.url,
standard_blob_tier=StandardBlobTier.HOT,
rehydrate_priority=RehydratePriority.HIGH)How long it takes depends on the priority you choose:
| Priority | Objects under 10 GB |
|---|---|
| Standard | might take up to 15 hours |
| High | might finish in under 1 hour |
I don't poll in a loop. I react to the BlobTierChanged event, and I tell the business: archive is for data you can wait a day for.
Question 3: Someone asks how to move old data to cheaper tiers automatically. How do lifecycle management policies work?
A lifecycle policy is a set of rules on the storage account. Filters pick the blobs, by blob type and prefix. Actions tier or delete them after a number of days.
Let's look at a policy for our sales exports. It moves them to cool after 30 days, cold after 90, and archive after 180.
{
"rules": [
{
"name": "sales-exports-tiering",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": ["blockBlob"],
"prefixMatch": ["raw/sales/"]
},
"actions": {
"baseBlob": {
"tierToCool": {"daysAfterModificationGreaterThan": 30},
"tierToCold": {"daysAfterModificationGreaterThan": 90},
"tierToArchive": {
"daysAfterModificationGreaterThan": 180,
"daysAfterLastTierChangeGreaterThan": 7
}
}
}
}
}
]
}Notice the prefix, raw/sales/. It starts with the container name, it is case sensitive, and it has no wildcards. I also add daysAfterLastTierChangeGreaterThan to the archive action, so a freshly rehydrated blob isn't sent straight back.
Then we deploy it with Azure CLI. The emulator has no lifecycle engine, so this is code only.
az storage account management-policy create \
--account-name stshoplake --resource-group rg-lake \
--policy @policy.jsonA few things to know about how it runs:
- Runs once a day.
- First run: up to 24 hours after a change.
- The policy is free: tier changes are billed as operations.
A policy is written in full, so read it, add your rule to the existing JSON and write it back, never replace it. And archive isn't supported on ZRS, GZRS or RA-GZRS accounts.
Question 4: Data is moved to cool and then read again a week later. What extra costs appear?
Three extras: cool read prices, a retrieval charge for every GB read, and an early deletion charge if you move it back to hot within 30 days.
Let's see the storage side first. September's export is in cool. We run python cool_read.py, which reads it and then moves it back to hot.
from common import container
blob = container("raw").get_blob_client("sales/2026/09/orders.csv")
print("tier:", blob.get_blob_properties().blob_tier)
data = blob.download_blob().readall() # cool is online
print("read:", len(data), "bytes, first line", data.split(b"\n")[0])
blob.set_standard_blob_tier("Hot") # back after a week
print("tier now:", blob.get_blob_properties().blob_tier)$ python cool_read.py
tier: Cool
read: 27 bytes, first line b'order_id,amount'
tier now: HotIt starts as tier: Cool, the read works right away and returns 27 bytes, because cool is an online tier. Then tier now: Hot.
The emulator bills nothing, so the costs come from the Azure pricing model. On Azure, that week looks like this:
| That week on Azure | Billed as |
|---|---|
| Move Hot to Cool | a write operation, at the cool rate |
| Read it in Cool | read operations at cool prices, plus retrieval per GB |
| Move Cool back to Hot | a read operation at the cool rate, plus retrieval per GB |
| Back after 7 days | early deletion for the other 23 of 30 days (prorated) |
Copying it to a new hot blob avoids early deletion, but you pay for two blobs.
So the rule should count from last access time instead of modification time. Auto tier back to hot helps, but within 30 days it still pays early deletion.
"actions": {
"baseBlob": {
"enableAutoTierToHotFromCool": true,
"tierToCool": {"daysAfterLastAccessTimeGreaterThan": 30}
}
}Last access tracking has to be turned on first, and it is billed at most once a day per object.
az storage account blob-service-properties update \
-n stshoplake -g rg-lake --enable-last-access-tracking trueTier on last access time, so data that is still read stays hot, and turn on tracking before you write the rule.
Question 5: A teammate asks about default account tiers versus setting the tier per blob. What is the difference?
The default access tier is an account setting, and blobs without their own tier follow it. A per-blob tier is set on one blob, at upload or with Set Blob Tier, and lifecycle rules set tiers the same way.
| Account default tier | Tier set per blob | |
|---|---|---|
| Set on | the storage account | one block blob: at upload or Set Blob Tier |
| Values | Hot (new GPv2 default), Cool, Cold, Smart (zone-redundant); not Archive | Hot, Cool, Cold, Archive |
| Applies to | blobs without their own tier: "inferred" | that blob only |
| Change it | every inferred blob moves and is billed | one operation per blob |
A new general purpose v2 account starts at hot, and archive can't be the default. Blobs that follow the default show their tier as inferred.
Let's see the difference. We run python default_tier.py. It uploads one click file with no tier and one as cool, then sets the first one to hot explicitly.
from azure.storage.blob import StandardBlobTier
from common import container
clicks = container("landing")
clicks.upload_blob("clicks/2026-10-30.json", b"{}", overwrite=True)
clicks.upload_blob("clicks/2026-10-01.json", b"{}", overwrite=True,
standard_blob_tier=StandardBlobTier.COOL)
def show(when):
for b in clicks.list_blobs(name_starts_with="clicks/"):
p = clicks.get_blob_client(b.name).get_blob_properties()
print(when, b.name, p.blob_tier,
"| inferred:", p.blob_tier_inferred)
show("before:")
today = clicks.get_blob_client("clicks/2026-10-30.json")
today.set_standard_blob_tier("Hot") # now an explicit tier
show("after: ")$ python default_tier.py
before: clicks/2026-10-01.json Cool | inferred: False
before: clicks/2026-10-30.json Hot | inferred: True
after: clicks/2026-10-01.json Cool | inferred: False
after: clicks/2026-10-30.json Hot | inferred: FalseBefore, clicks/2026-10-30.json shows Hot | inferred: True: it follows the default. The cool file shows inferred: False. After we set it to hot explicitly, the first file also shows inferred: False, so it no longer follows the default.
Why it matters: change the account default, and every inferred blob moves with it. A cooler default bills write operations for all of them; a warmer one bills reads and retrieval. The emulator can't change the default, so that is code.
az storage account update -n stshoplake -g rg-lake \
--access-tier CoolI keep the default at hot and let lifecycle rules set explicit tiers. A surprise default change on a big account is a surprise bill.
Question 6: Someone asks how to delete raw files automatically after a year. How do you set it up safely?
A lifecycle rule that deletes blobs 365 days after modification, on the raw prefix. Safe means soft delete, a prefix dry run, the right date, and a policy review.
- Soft delete first: a wrong rule can be undone.
- Dry run the prefix: container name first, case sensitive, no wildcards.
- Check the date: modification time, not the folder name.
- Review the whole policy: it is read and written in full.
Here is the new rule:
{
"name": "clickstream-delete-after-1-year",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": ["blockBlob"],
"prefixMatch": ["raw/clickstream/"]
},
"actions": {
"baseBlob": {
"delete": {"daysAfterModificationGreaterThan": 365}
}
}
}
}Let's dry run it. We run python seed_clicks.py to write 26 months of clickstream files, then python dry_run.py to see what the prefix matches and how old the blobs really are.
from common import container
raw = container("raw")
months = [f"{y}/{m:02}" for y in (2024, 2025, 2026)
for m in range(1, 13)]
months = [ym for ym in months if "2024/09" <= ym <= "2026/10"]
for i, ym in enumerate(months, 1):
line = f'{{"month": "{ym}", "page": "/cart"}}\n'.encode()
raw.upload_blob(f"clickstream/{ym}/events.json", line * (20 * i),
overwrite=True)
print("clickstream files:", len(months), "from", months[0],
"to", months[-1])from datetime import datetime, timezone
from common import container
raw = container("raw")
# rule prefixMatch "raw/clickstream/" = container raw + "clickstream/"
for prefix in ["clickstream/", "ClickStream/", "clickstream/*/"]:
n = len(list(raw.list_blob_names(name_starts_with=prefix)))
print(f"prefix {prefix:15} -> {n} blobs")
blobs = list(raw.list_blobs(name_starts_with="clickstream/"))
month = lambda b: "/".join(b.name.split("/")[1:3])
print("named for a month before 2025/10:",
sum(month(b) < "2025/10" for b in blobs))
now = datetime.now(timezone.utc)
print("last modified more than 365 days ago:",
sum((now - b.last_modified).days > 365 for b in blobs))$ python seed_clicks.py
clickstream files: 26 from 2024/09 to 2026/10
$ python dry_run.py
prefix clickstream/ -> 26 blobs
prefix ClickStream/ -> 0 blobs
prefix clickstream/*/ -> 0 blobs
named for a month before 2025/10: 13
last modified more than 365 days ago: 0clickstream/ matches 26 blobs. With a capital S, or with a star, it matches 0 blobs. Then the date: 13 files are named for months before 2025/10, but last modified more than 365 days ago: 0. All were written today, and the rule counts from the write, not the folder name. So after a backfill or a migration, the clock starts again.
Now turn on blob soft delete first, for example 14 days. The emulator has no lifecycle engine, so the rule and soft delete are code.
az storage account blob-service-properties update \
-n stshoplake -g rg-lake \
--enable-delete-retention true --delete-retention-days 14I also check the matches with a blob inventory report first. And the delete action doesn't act on blobs in immutable containers.
Scenario 7: Storage costs keep growing and most data is never read after ninety days. How do you cut the bill?
First measure, then automate. Find where the bytes are, in which tier and how old, then let rules or smart tier move them.
At scale, a blob inventory report answers that, as CSV or Parquet. Here a small listing does it for the raw container. We run python measure.py.
from collections import defaultdict
from common import container
REPORT = (2026, 10) # month of this report
tiers = defaultdict(lambda: [0, 0]) # tier -> [blobs, bytes]
hot = old_hot = 0
for b in container("raw").list_blobs():
tiers[b.blob_tier][0] += 1
tiers[b.blob_tier][1] += b.size
y, m = (int(x) for x in b.name.split("/")[1:3])
months_old = (REPORT[0] - y) * 12 + REPORT[1] - m
if b.blob_tier == "Hot":
hot += b.size
if months_old >= 3:
old_hot += b.size
for tier, (n, size) in sorted(tiers.items()):
print(f"{tier:8} {n:>3} blobs {size:>7} bytes")
print(f"hot bytes 3 or more months old: {old_hot / hot:.0%}")$ python measure.py
Archive 1 blobs 27 bytes
Cold 1 blobs 27 bytes
Hot 28 blobs 266814 bytes
hot bytes 3 or more months old: 79%The output shows Hot 28 blobs 266814 bytes, against one archive and one cold blob, and hot bytes 3 or more months old: 79%. That is the bill: data nobody reads, at the highest storage price.
Then automate. Let's compare the options:
| Option | How it moves data | Watch out |
|---|---|---|
| Lifecycle: days after modification | Cool at 30 days, Cold at 90, delete after retention | tier changes billed; early deletion |
| Lifecycle: days after last access | tiers only what nobody reads | needs last access tracking |
| Smart tier (GA, April 2026) | Cool after 30 days, Cold after 90 without access; Hot again on access | ZRS, GZRS, RA-GZRS only; no archive; monitoring fee |
Lifecycle rules move data to cool at 30 days and cold at 90, by age or by last access, with a delete at the end of retention. Smart tier, generally available since April 2026 on zone-redundant accounts, tiers by access with no early deletion or retrieval charges, for a monitoring fee.
Then avoid the traps:
- Archive only what can wait hours: and not on ZRS, GZRS, RA-GZRS.
- Rules longer than the minimum stay: or pay early deletion.
- Plan backfills that read cold data: retrieval per GB.
I report bytes per tier every month from the BlobCapacity metric split by tier, so everyone sees the curve bend.
Key points to remember
- Hot, cool, cold, archive: storage price vs access price; minimum stays 30 / 90 / 180 days
- Archive is offline: 409 BlobArchived; rehydrate (Standard up to 15 h, High under 1 h) or copy
- Lifecycle rules: filters + actions, once a day, written in full, prefix starts with the container
- Cool data read again: retrieval, cool read prices and early deletion
- Delete safely: soft delete first, dry-run the prefix, rules count from the last write
The full script
Here is a Python script that writes every file and runs every command in order against Azurite; run pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas first.
Show the full script that writes the files and runs every step
# ========================================================================
# ADLS Gen2 Interview Q&A #3 - Access tiers and lifecycle management
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run on Azurite 3.37.0 (Microsoft's open-source Azure Storage emulator, Blob service)
# with azure-storage-blob 12.31.1, pyarrow and pandas; the outputs in the video and the blog post are the real outputs.
# To run it yourself:
# 1. npm install -g azurite then azurite-blob --skipApiVersionCheck --location ./azurite-data
# (or the Docker image mcr.microsoft.com/azure-storage/azurite), and leave it running on port 10000.
# --skipApiVersionCheck lets a newer azure-storage-blob talk to it.
# 2. pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas requests
# 3. Set AZURE_STORAGE_CONNECTION_STRING to "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
# then run python ADLS_Interview_QA_03_Code.py.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================
import os, subprocess, time
def write(path, text):
if text is None:
if os.path.exists(path): os.remove(path)
return
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
f.write(text.strip("\n") + "\n")
def run(cmd, expect_error=False, allow_error=False, env=None):
print("\n$ " + cmd, flush=True)
rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
if rc != 0 and not (expect_error or allow_error):
raise SystemExit("stopped: this command was expected to succeed")
# Step 0: the starting files
write('common.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\n\nsvc = BlobServiceClient.from_connection_string(\n os.environ["AZURE_STORAGE_CONNECTION_STRING"])\n\n\ndef container(name):\n """Get a container client, creating the container on first use."""\n c = svc.get_container_client(name)\n if not c.exists():\n c.create_container()\n return c\n\n\ndef show_err(e):\n code = getattr(e.error_code, "value", e.error_code)\n print(" ", type(e).__name__, e.status_code, code)\n print(" ", str(e).split("\\n")[0])\n')
# Step 1: Four monthly order exports uploaded with a tier per blob: default, Cool, Cold, Archive
write('tiers.py', 'from azure.storage.blob import StandardBlobTier as Tier\nfrom common import container\n\nraw = container("raw")\nexports = {"sales/2026/10/orders.csv": None, # account default\n "sales/2026/09/orders.csv": Tier.COOL,\n "sales/2026/07/orders.csv": Tier.COLD,\n "sales/2025/10/orders.csv": Tier.ARCHIVE}\nfor name, tier in exports.items():\n raw.upload_blob(name, b"order_id,amount\\n1001,25.50\\n",\n standard_blob_tier=tier, overwrite=True)\nfor b in raw.list_blobs(name_starts_with="sales/"):\n print(f"{b.name} {b.size} bytes tier {b.blob_tier}")\n')
run('python tiers.py')
# Step 2: Reading an archived blob: properties work, the download fails with 409 BlobArchived
write('read_archive.py', """\
from azure.core.exceptions import HttpResponseError
from common import container, show_err
blob = container("raw").get_blob_client("sales/2025/10/orders.csv")
p = blob.get_blob_properties() # properties: allowed
print("tier:", p.blob_tier, "| size:", p.size, "bytes")
try:
blob.download_blob().readall() # the data: offline
except HttpResponseError as e:
show_err(e)
""")
run('python read_archive.py')
# Step 3: A Cool blob is online: read it, then move it back to Hot
write('cool_read.py', 'from common import container\n\nblob = container("raw").get_blob_client("sales/2026/09/orders.csv")\nprint("tier:", blob.get_blob_properties().blob_tier)\ndata = blob.download_blob().readall() # cool is online\nprint("read:", len(data), "bytes, first line", data.split(b"\\n")[0])\nblob.set_standard_blob_tier("Hot") # back after a week\nprint("tier now:", blob.get_blob_properties().blob_tier)\n')
run('python cool_read.py')
# Step 4: Inferred (account default) tier vs an explicit tier per blob
write('default_tier.py', """\
from azure.storage.blob import StandardBlobTier
from common import container
clicks = container("landing")
clicks.upload_blob("clicks/2026-10-30.json", b"{}", overwrite=True)
clicks.upload_blob("clicks/2026-10-01.json", b"{}", overwrite=True,
standard_blob_tier=StandardBlobTier.COOL)
def show(when):
for b in clicks.list_blobs(name_starts_with="clicks/"):
p = clicks.get_blob_client(b.name).get_blob_properties()
print(when, b.name, p.blob_tier,
"| inferred:", p.blob_tier_inferred)
show("before:")
today = clicks.get_blob_client("clicks/2026-10-30.json")
today.set_standard_blob_tier("Hot") # now an explicit tier
show("after: ")
""")
run('python default_tier.py')
# Step 5: Dry run of a delete rule: what the prefix matches, and the modification date vs the folder date
write('seed_clicks.py', 'from common import container\n\nraw = container("raw")\nmonths = [f"{y}/{m:02}" for y in (2024, 2025, 2026)\n for m in range(1, 13)]\nmonths = [ym for ym in months if "2024/09" <= ym <= "2026/10"]\nfor i, ym in enumerate(months, 1):\n line = f\'{{"month": "{ym}", "page": "/cart"}}\\n\'.encode()\n raw.upload_blob(f"clickstream/{ym}/events.json", line * (20 * i),\n overwrite=True)\nprint("clickstream files:", len(months), "from", months[0],\n "to", months[-1])\n')
write('dry_run.py', """\
from datetime import datetime, timezone
from common import container
raw = container("raw")
# rule prefixMatch "raw/clickstream/" = container raw + "clickstream/"
for prefix in ["clickstream/", "ClickStream/", "clickstream/*/"]:
n = len(list(raw.list_blob_names(name_starts_with=prefix)))
print(f"prefix {prefix:15} -> {n} blobs")
blobs = list(raw.list_blobs(name_starts_with="clickstream/"))
month = lambda b: "/".join(b.name.split("/")[1:3])
print("named for a month before 2025/10:",
sum(month(b) < "2025/10" for b in blobs))
now = datetime.now(timezone.utc)
print("last modified more than 365 days ago:",
sum((now - b.last_modified).days > 365 for b in blobs))
""")
run('python seed_clicks.py')
run('python dry_run.py')
# Step 6: Measure first: blobs and bytes per tier, and how much hot data is old
write('measure.py', """\
from collections import defaultdict
from common import container
REPORT = (2026, 10) # month of this report
tiers = defaultdict(lambda: [0, 0]) # tier -> [blobs, bytes]
hot = old_hot = 0
for b in container("raw").list_blobs():
tiers[b.blob_tier][0] += 1
tiers[b.blob_tier][1] += b.size
y, m = (int(x) for x in b.name.split("/")[1:3])
months_old = (REPORT[0] - y) * 12 + REPORT[1] - m
if b.blob_tier == "Hot":
hot += b.size
if months_old >= 3:
old_hot += b.size
for tier, (n, size) in sorted(tiers.items()):
print(f"{tier:8} {n:>3} blobs {size:>7} bytes")
print(f"hot bytes 3 or more months old: {old_hot / hot:.0%}")
""")
run('python measure.py')
Part 4: Security and Access Control
The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run on Azurite 3.37.0, Microsoft's open-source Azure Storage emulator, with azure-storage-blob 12.31.1, pyarrow and pandas. Features the emulator does not support are shown as code. Part of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Azure Storage documentation (learn.microsoft.com/azure/storage) for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.