In this post we are going to cover ADLS Gen2 and Blob Storage fundamentals. These are 7 real ADLS Gen2 and Blob Storage interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: the company is building its first data lake for sales, finance and clickstream data, and you have to set up the storage.
Azure Data Lake Storage Gen2 is where most Azure data platforms keep their files. It is not a separate service. It is Blob Storage with one extra switch, the hierarchical namespace. Interviewers ask about the basics because every later decision, such as layout, security, cost and recovery, depends on them. If you mix up containers and directories, or the blob and dfs endpoints, it shows quickly.
Part 1 of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Let's set up the project
- Question 1: Your manager asks what Azure Data Lake Storage Gen2 is and how it differs from plain Blob Storage. What do you say?
- Question 2: A new teammate asks how storage accounts, containers, directories and blobs fit together. How do you explain it?
- Question 3: Someone asks what the hierarchical namespace actually changes. How do you explain it?
- Question 4: A teammate asks about block blobs, append blobs and page blobs. When do you use each?
- Question 5: Someone asks the difference between the blob endpoint and the dfs endpoint. When does it matter?
- Question 6: Your manager asks which redundancy option to pick: LRS, ZRS, GRS or GZRS. How do you decide?
- Scenario 7: The company is building its first data lake for sales, finance and clickstream data. How do you set up the storage?
- The full script
The short answer
ADLS Gen2 is Blob Storage with the hierarchical namespace: real directories, POSIX-like ACLs and directory operations on the dfs endpoint, and it can't be turned off. A lake stores block blobs; Microsoft recommends ZRS in the primary region, and GZRS covers a region outage.
Let's set up the project
We run everything against Azurite, Microsoft's open-source Azure Storage emulator, running locally, with small Python scripts that use the azure-storage-blob client and pyarrow. Features the emulator does not support, such as the hierarchical namespace and ACLs, are shown as code. Here are the two helper files every script uses: common.py connects to the account, and lakeview.py prints a folder tree.
import os
from azure.storage.blob import BlobServiceClient
CONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
svc = BlobServiceClient.from_connection_string(CONN)
def show_err(e):
code = getattr(e.error_code, "value", e.error_code)
print(" ", type(e).__name__, e.status_code, code)
print(" ", str(e).split("\n")[0])from azure.storage.blob import BlobPrefix
def tree(container, prefix="", depth=1):
"""Print virtual folders and blobs, one level at a time."""
for item in container.walk_blobs(name_starts_with=prefix,
delimiter="/"):
label = " " * depth + item.name[len(prefix):]
if isinstance(item, BlobPrefix): # a virtual folder
print(label)
tree(container, item.name, depth + 1)
else: # a blob
print(label, item.size, "bytes")Question 1: Your manager asks what Azure Data Lake Storage Gen2 is and how it differs from plain Blob Storage. What do you say?
ADLS Gen2 is not a separate service. It is Blob Storage with the hierarchical namespace turned on, usually in a general purpose v2 account, with the same blobs, tiers and redundancy options.
Let's compare the two side by side.
| Blob Storage (flat) | ADLS Gen2 (hierarchical namespace) | |
|---|---|---|
| Folders | a prefix in the blob name | real directories |
| Rename a folder | copy + delete every blob | one operation |
| Permissions | Azure RBAC, SAS | + POSIX-like ACLs on files and folders |
| dfs endpoint | there, but no real directories or ACLs | directories, rename, ACLs (abfss:// for Spark) |
| Switch | can be upgraded to HNS | can't go back to flat |
The namespace adds what analytics needs. First, real directories, so a folder rename is one operation. Then POSIX-like access control lists on files and folders. And directory operations on the dfs endpoint, which Spark reaches through the ABFS driver.
Now let's see what a plain account looks like. Our emulator is a flat Blob Storage account. We run python account.py, which prints the account kind, uploads three CSV files into a sales container and lists them.
from common import svc
info = svc.get_account_information()
print("account kind:", info["account_kind"])
print("hierarchical namespace:", info["is_hns_enabled"])
sales = svc.create_container("sales")
csv = b"order_id,amount\n1001,25.50\n1002,14.00\n"
sales.upload_blob("orders/2026/10/29/orders.csv", csv)
sales.upload_blob("orders/2026/10/30/orders.csv", csv)
sales.upload_blob("returns/2026/10/30/returns.csv", csv[:27])
print("container sales, one flat list of names:")
for b in sales.list_blobs():
print(" ", b.name, b.size, "bytes")$ python account.py
account kind: StorageV2
hierarchical namespace: False
container sales, one flat list of names:
orders/2026/10/29/orders.csv 38 bytes
orders/2026/10/30/orders.csv 38 bytes
returns/2026/10/30/returns.csv 27 bytesThe output shows account kind: StorageV2 and hierarchical namespace: False. The three files come back as one flat list of names, such as orders/2026/10/29/orders.csv 38 bytes. The slashes are just characters in the name.
Microsoft recommends the namespace for analytics workloads. For backups or image storage, where the folder structure lives somewhere else, a flat account is fine.
Once the namespace is on, you can't turn it off, and some blob features, like versioning, change feed and object replication, aren't supported yet on these accounts. Decide before you build.
Question 2: A new teammate asks how storage accounts, containers, directories and blobs fit together. How do you explain it?
There are four levels: the storage account, containers, directories, and blobs, which are your files.
Let's go from the top down:
- Storage account: a name unique in Azure, 3 to 24 lowercase letters and numbers, with its own endpoints, keys, network rules and redundancy.
- Container: ADLS Gen2 calls it a file system, but it is the same thing.
- Directories: name prefixes on a flat account, real directories with their own permissions when the namespace is on.
- Blobs: your files.
To see this, we run python tree.py. It adds a marketing container with one file, then walks every container with a / delimiter. Prefixes come back as folders, blobs as files with their sizes.
from common import svc
from lakeview import tree
marketing = svc.create_container("marketing")
marketing.upload_blob("campaigns/budget_2026.csv",
b"campaign,budget\nautumn,5000\n")
print("storage account devstoreaccount1")
for c in svc.list_containers():
print(" container:", c.name)
tree(svc.get_container_client(c.name), depth=2)
blob = svc.get_blob_client("sales", "orders/2026/10/30/orders.csv")
print("URL:", blob.url)$ python tree.py
storage account devstoreaccount1
container: marketing
campaigns/
budget_2026.csv 28 bytes
container: sales
orders/
2026/
10/
29/
orders.csv 38 bytes
30/
orders.csv 38 bytes
returns/
2026/
10/
30/
returns.csv 27 bytes
URL: http://127.0.0.1:10000/devstoreaccount1/sales/orders/2026/10/30/orders.csvUnder marketing we get campaigns/budget_2026.csv 28 bytes. Under sales we see orders and returns, down to the day: orders/2026/10/29/ and orders/2026/10/30/, each with a 38-byte orders.csv, and returns/2026/10/30/returns.csv at 27 bytes. The last line is the blob URL on the emulator. On Azure the address is the account endpoint, then the container, then the path, for example https://shoplake.blob.core.windows.net/sales/orders/2026/10/30/orders.csv.
I keep few containers, one per zone or security boundary, because a container is the narrowest scope for a role assignment. Folders do the rest.
Question 3: Someone asks what the hierarchical namespace actually changes. How do you explain it?
It changes what a folder is. Without it, a folder is only a prefix in blob names. With it, a directory is a real object: you rename or delete it in one operation, and it has its own ACLs.
Let's see the flat side first. A job wrote 48 clickstream files into staging, and they belong under events. Without the namespace, a rename means copying each blob and then deleting the original. We run python rename_folder.py, which does exactly that, counts the operations, checks both folders halfway through, and then tries to delete the folder name itself.
from azure.core.exceptions import ResourceNotFoundError
from common import svc, show_err
clicks = svc.create_container("clickstream")
for h in range(24): # 48 files in the wrong folder
for part in (1, 2):
clicks.upload_blob(f"staging/h{h:02}-{part}.json", b"{}")
def count(prefix):
return len(list(clicks.list_blob_names(name_starts_with=prefix)))
names = list(clicks.list_blob_names(name_starts_with="staging/"))
ops = 0
for i, old in enumerate(names, 1): # "rename" staging/ -> events/
new = clicks.get_blob_client(old.replace("staging/", "events/"))
copy = new.start_copy_from_url(clicks.get_blob_client(old).url)
assert copy["copy_status"] == "success" # Azure: may be pending
clicks.delete_blob(old)
ops += 2 # one copy + one delete
if i == 24:
print("halfway, a reader sees: staging", count("staging/"),
"| events", count("events/"))
print("files moved:", len(names), "| operations:", ops)
try:
clicks.delete_blob("staging") # is the folder an object?
except ResourceNotFoundError as e:
show_err(e)$ python rename_folder.py
halfway, a reader sees: staging 24 | events 24
files moved: 48 | operations: 96
ResourceNotFoundError 404 BlobNotFound
The specified blob does not exist.The output shows files moved: 48 | operations: 96. Halfway, a reader sees staging 24 | events 24, half the data in each place. Deleting staging gives ResourceNotFoundError 404 BlobNotFound, because the folder does not exist as an object.
Now the same move with the namespace. On Azure it is one rename_directory call on the dfs endpoint. Azurite has no dfs endpoint, so this part is code only.
# Azure, account with HNS (not run: Azurite has no dfs)
from azure.identity import DefaultAzureCredential
from azure.storage.filedatalake import DataLakeServiceClient
lake = DataLakeServiceClient(
"https://shoplake.dfs.core.windows.net",
credential=DefaultAzureCredential())
staging = lake.get_file_system_client(
"clickstream").get_directory_client("staging")
staging.rename_directory("clickstream/events") # one callMicrosoft warns that without the namespace, this kind of rename can often take longer than the analytics process itself.
It is one way. You can upgrade an existing account after validation, but never go back, and features like soft delete must be turned off and page blobs removed first.
Question 4: A teammate asks about block blobs, append blobs and page blobs. When do you use each?
Block blobs are the default and what a lake stores: Parquet, CSV, images. Append blobs only grow at the end, which fits logs. Page blobs are 512-byte pages for random writes, like virtual machine disks.
| Block blob | Append blob | Page blob | |
|---|---|---|---|
| Built from | blocks, committed as a list | blocks added at the end | 512-byte pages |
| Writes | upload or replace | append only | random, in place |
| Use for | lake files: Parquet, CSV, images | logs | virtual machine disks |
| Access tiers | hot, cool, cold, archive | no | no |
| Max size | about 190.7 TiB | about 195 GiB | 8 TiB |
Let's try all three. We run python block_blob.py to build a block blob from staged blocks, then python append_page.py for an append blob and a page blob.
from azure.storage.blob import BlobBlock
from common import svc
types = svc.create_container("blobtypes")
orders = types.get_blob_client("orders/orders_big.csv")
parts = [b"order_id,amount\n", b"1001,25.50\n", b"1002,14.00\n"]
blocks = []
for i, data in enumerate(parts):
block_id = f"block-{i:04}" # same length for every id
orders.stage_block(block_id, data)
blocks.append(BlobBlock(block_id))
print("staged 3 blocks, blob exists:", orders.exists())
orders.commit_block_list(blocks) # now it is one blob
committed, _ = orders.get_block_list("committed")
p = orders.get_blob_properties()
print("committed:", [b.id for b in committed])
print(p.blob_type.value, p.size, "bytes, tier", p.blob_tier)from azure.core.exceptions import HttpResponseError
from common import svc, show_err
types = svc.get_container_client("blobtypes")
log = types.get_blob_client("logs/pipeline.log")
log.create_append_blob()
for line in ["start\n", "loaded 2 rows\n", "done\n"]:
r = log.append_block(line) # always at the end
print("append at offset", r["blob_append_offset"])
print(log.download_blob().readall())
try:
log.set_standard_blob_tier("Cool") # tiers: block blobs only
except HttpResponseError as e:
show_err(e)
disk = types.get_blob_client("disks/vm-disk.vhd")
disk.create_page_blob(size=1024) # two 512-byte pages
disk.upload_page(b"A" * 512, offset=512, length=512)
ranges, _ = disk.get_page_ranges()
p = disk.get_blob_properties()
print(p.blob_type.value, p.size, "bytes, written:", ranges)$ python block_blob.py
staged 3 blocks, blob exists: False
committed: ['block-0000', 'block-0001', 'block-0002']
BlockBlob 38 bytes, tier Hot
$ python append_page.py
append at offset 0
append at offset 6
append at offset 20
b'start\nloaded 2 rows\ndone\n'
HttpResponseError 400 AccessTierNotSupportedForBlobType
The access tier is not supported for this blob type.
PageBlob 1024 bytes, written: [{'start': 512, 'end': 1023}]First, we stage three blocks and the blob does not exist yet: blob exists: False. Then we commit the block list, and it is one blob: BlockBlob 38 bytes, tier Hot. Above 64 MiB, the Python SDK does exactly this for you.
The append blob takes three log lines, at offsets 0, 6 and 20. Setting it to cool fails with 400 AccessTierNotSupportedForBlobType, because tiers apply to block blobs only. The page blob is 1024 bytes, and we wrote one page at offset 512: [{'start': 512, 'end': 1023}].
A few limits worth knowing: a block blob holds up to 50,000 blocks of up to 4,000 MiB, about 190.7 TiB. An append block is at most 4 MiB.
Page blobs aren't supported on accounts with the hierarchical namespace, and they block the upgrade. In a lake, it is block blobs.
Question 5: Someone asks the difference between the blob endpoint and the dfs endpoint. When does it matter?
Same account, same data, two doors. The blob endpoint speaks the Blob REST API. The dfs endpoint speaks the Data Lake Storage API: directories, rename and ACLs. Spark reaches it with abfss:// paths.
| Endpoint | Address | Speaks |
|---|---|---|
| blob | https://shoplake.blob.core.windows.net/sales/... | Blob REST API: blobs, tiers, SAS |
| dfs | https://shoplake.dfs.core.windows.net/sales/... | Data Lake Storage API: directories, rename, ACLs |
| abfss | abfss://sales@shoplake.dfs.core.windows.net/orders/ | ABFS driver (Spark) to dfs, always TLS |
Let's see the difference on the emulator. We run python dfs.py, which uses the DataLake client to list file systems, read a file, and then create a directory.
import logging
from azure.core.exceptions import HttpResponseError
from azure.storage.filedatalake import DataLakeServiceClient
from common import CONN
logging.getLogger("azure").setLevel(logging.ERROR) # quiet SDK logs
lake = DataLakeServiceClient.from_connection_string(CONN)
print("file systems:", [fs.name for fs in lake.list_file_systems()])
sales = lake.get_file_system_client("sales")
f = sales.get_file_client("orders/2026/10/30/orders.csv")
print("read through the DataLake client:",
f.download_file().readall()[:15])
try:
sales.create_directory("orders/2026/10/31") # a dfs operation
except HttpResponseError as e:
print("create_directory:", type(e).__name__, e.status_code)$ python dfs.py
file systems: ['blobtypes', 'clickstream', 'marketing', 'sales']
read through the DataLake client: b'order_id,amount'
create_directory: HttpResponseError 400Listing file systems works and returns ['blobtypes', 'clickstream', 'marketing', 'sales'], and the read returns b'order_id,amount', because those calls map to blob calls. create_directory is a true dfs call, and the emulator has no hierarchical namespace and no dfs endpoint, so it fails with HttpResponseError 400.
It matters in three places:
- Engines: Databricks Unity Catalog, Fabric shortcuts and the Data Factory connector all take the dfs URL.
- Private endpoints: one for blob and one for dfs. With only blob, managing ACLs or creating directories fails, and the Python SDK reads files through the blob endpoint.
- Events: on a namespace account, a Data Lake API write fires
BlobCreatedfor the create file call and again for the final flush.
Here is how we create both private endpoints:
# Azure CLI (not run: Azurite has no networking)
for sub in blob dfs; do
az network private-endpoint create -g rg-lake \
-n pe-shoplake-$sub --connection-name pe-shoplake-$sub \
--vnet-name vnet-data --subnet snet-storage \
--private-connection-resource-id "$LAKE_ID" \
--group-id $sub
done # + DNS zones privatelink.blob and privatelink.dfsAnd here is an Event Grid subscription that only fires on the final flush:
# Azure CLI (not run: Azurite has no events)
az eventgrid event-subscription create -n lake-new-files \
--source-resource-id "$LAKE_ID" \
--endpoint-type storagequeue --endpoint "$QUEUE_ID" \
--included-event-types Microsoft.Storage.BlobCreated \
--advanced-filter data.api StringIn FlushWithCloseFilter Event Grid on data.api equal to FlushWithClose, or your pipeline starts on an empty file.
Question 6: Your manager asks which redundancy option to pick: LRS, ZRS, GRS or GZRS. How do you decide?
By the failure you must survive. Microsoft recommends ZRS in the primary region for Data Lake Storage, so my default is ZRS, and GZRS when the lake must survive a region outage.
Let's go through the four options:
| Option | Where the copies are | Durability | Survives |
|---|---|---|---|
| LRS | one datacenter in the primary region | 11 nines | a node failure |
| ZRS | availability zones in the primary region, sync | 12 nines | a zone outage |
| GRS | LRS + async copy in a secondary region | 16 nines | a region outage, after failover |
| GZRS | ZRS + async copy in a secondary region | 16 nines | a zone outage; a region after failover |
LRS keeps copies in one datacenter, 11 nines, but a fire or flood there can take them all. ZRS writes synchronously across availability zones, 12 nines, and survives a zone outage. GRS adds an asynchronous copy in a secondary region hundreds of miles away, and GZRS combines zone redundancy with that geo copy: 16 nines. The read access versions also let you read the secondary.
The catch with geo is that the copy is asynchronous, so check the last sync time. An unplanned failover can lose recent writes and leaves the account LRS. A planned failover, generally available since November 2025, keeps geo-redundancy. Here is how we create a GZRS account, check the last sync time and run a planned failover. Azurite has no redundancy, so this is code only.
# Azure CLI (not run: Azurite has no redundancy)
az storage account create -n shoplake -g rg-lake -l eastus2 \
--kind StorageV2 --hns true --sku Standard_GZRS
# writes after the last sync time can be lost in a failover
az storage account show -n shoplake -g rg-lake \
--expand geoReplicationStats \
--query geoReplicationStats.lastSyncTime
# planned failover, e.g. a DR drill (default type: Unplanned)
az storage account failover -n shoplake -g rg-lake \
--failover-type Planned --yesThe archive tier isn't available on ZRS, GZRS or RA-GZRS accounts. If you need archive, plan for it in the account design.
Scenario 7: The company is building its first data lake for sales, finance and clickstream data. How do you set up the storage?
One general purpose v2 account with the hierarchical namespace on from creation, on ZRS, or GZRS if the lake must survive a region outage, and locked down from day one.
Here is the checklist I start with:
- HNS on: at creation.
- ZRS: GZRS for a region outage.
- No anonymous access.
- Shared Key off: Microsoft Entra ID only.
- Private endpoints: blob and dfs.
- Soft delete: blobs and containers.
In Azure CLI that looks like this:
# Azure CLI (not run: Azurite has no account settings)
az storage account create -n shoplake -g rg-lake -l eastus2 \
--kind StorageV2 --hns true --sku Standard_ZRS \
--min-tls-version TLS1_2 --allow-blob-public-access false \
--allow-shared-key-access false --public-network-access Disabled
az storage account blob-service-properties update \
-n shoplake -g rg-lake \
--enable-delete-retention true --delete-retention-days 14 \
--enable-container-delete-retention true \
--container-delete-retention-days 14Then the layout I use: containers by zone, raw and curated, with a folder per domain. Finance gets its own container, since a container is the narrowest scope for a role assignment. Let's build that layout on the emulator. We run python build_lake.py, which creates three containers with owner and zone metadata and one sample file per domain.
from common import svc
from lakeview import tree
layout = {"raw": ["sales/orders_20261030.csv",
"clickstream/events_20261030.json"],
"curated": ["sales/daily_sales.csv",
"clickstream/sessions.csv"],
"finance": ["raw/gl_20261030.csv",
"curated/gl_balances.csv"]}
for name, files in layout.items():
owner = "finance" if name == "finance" else "data-platform"
zone = "restricted" if name == "finance" else name
lake = svc.create_container(
name, metadata={"owner": owner, "zone": zone})
for path in files:
lake.upload_blob(path, b"sample\n")
for name in layout:
lake = svc.get_container_client(name)
meta = lake.get_container_properties().metadata
print("container:", name, dict(sorted(meta.items())))
tree(lake)$ python build_lake.py
container: raw {'owner': 'data-platform', 'zone': 'raw'}
clickstream/
events_20261030.json 7 bytes
sales/
orders_20261030.csv 7 bytes
container: curated {'owner': 'data-platform', 'zone': 'curated'}
clickstream/
sessions.csv 7 bytes
sales/
daily_sales.csv 7 bytes
container: finance {'owner': 'finance', 'zone': 'restricted'}
curated/
gl_balances.csv 7 bytes
raw/
gl_20261030.csv 7 bytesThe output shows raw and curated owned by data-platform, each with clickstream/ and sales/, and finance with {'owner': 'finance', 'zone': 'restricted'} holding its own raw/ and curated/ folders.
Access goes through Microsoft Entra groups, and ACLs handle anything finer:
| Who (Entra group) | Role, scope |
|---|---|
| data platform team | Storage Blob Data Contributor, account |
| analysts | Storage Blob Data Reader, curated |
| finance analysts | Storage Blob Data Reader, finance |
| finer folder needs | ACLs on directories |
A container per source dumped at the root, no zones, and account keys in notebooks. Fixing that later means moving every file.
Key points to remember
- ADLS Gen2 = Blob Storage + hierarchical namespace; it can't be turned off
- Account, container (file system), directories, blobs
- HNS makes folders real: rename is one call, not copy + delete per blob
- Lakes store block blobs; page blobs aren't supported with HNS
- Engines use the dfs endpoint; private endpoints for blob and dfs; ZRS by default
The full script
Here is a Python script that writes every file and runs every command in order against Azurite; run pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas first.
Show the full script that writes the files and runs every step
# ========================================================================
# ADLS Gen2 Interview Q&A #1 - ADLS Gen2 and Blob Storage fundamentals
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run on Azurite 3.37.0 (Microsoft's open-source Azure Storage emulator, Blob service)
# with azure-storage-blob 12.31.1, pyarrow and pandas; the outputs in the video and the blog post are the real outputs.
# To run it yourself:
# 1. npm install -g azurite then azurite-blob --skipApiVersionCheck --location ./azurite-data
# (or the Docker image mcr.microsoft.com/azure-storage/azurite), and leave it running on port 10000.
# --skipApiVersionCheck lets a newer azure-storage-blob talk to it.
# 2. pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas requests
# 3. Set AZURE_STORAGE_CONNECTION_STRING to "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
# then run python ADLS_Interview_QA_01_Code.py.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================
import os, subprocess, time
def write(path, text):
if text is None:
if os.path.exists(path): os.remove(path)
return
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
f.write(text.strip("\n") + "\n")
def run(cmd, expect_error=False, allow_error=False, env=None):
print("\n$ " + cmd, flush=True)
rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
if rc != 0 and not (expect_error or allow_error):
raise SystemExit("stopped: this command was expected to succeed")
# Step 0: the starting files
write('common.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\n\nCONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]\nsvc = BlobServiceClient.from_connection_string(CONN)\n\n\ndef show_err(e):\n code = getattr(e.error_code, "value", e.error_code)\n print(" ", type(e).__name__, e.status_code, code)\n print(" ", str(e).split("\\n")[0])\n')
write('lakeview.py', 'from azure.storage.blob import BlobPrefix\n\n\ndef tree(container, prefix="", depth=1):\n """Print virtual folders and blobs, one level at a time."""\n for item in container.walk_blobs(name_starts_with=prefix,\n delimiter="/"):\n label = " " * depth + item.name[len(prefix):]\n if isinstance(item, BlobPrefix): # a virtual folder\n print(label)\n tree(container, item.name, depth + 1)\n else: # a blob\n print(label, item.size, "bytes")\n')
# Step 1: A plain Blob Storage account (the emulator): no hierarchical namespace, one flat list of names
write('account.py', 'from common import svc\n\ninfo = svc.get_account_information()\nprint("account kind:", info["account_kind"])\nprint("hierarchical namespace:", info["is_hns_enabled"])\n\nsales = svc.create_container("sales")\ncsv = b"order_id,amount\\n1001,25.50\\n1002,14.00\\n"\nsales.upload_blob("orders/2026/10/29/orders.csv", csv)\nsales.upload_blob("orders/2026/10/30/orders.csv", csv)\nsales.upload_blob("returns/2026/10/30/returns.csv", csv[:27])\nprint("container sales, one flat list of names:")\nfor b in sales.list_blobs():\n print(" ", b.name, b.size, "bytes")\n')
run('python account.py')
# Step 2: Account, containers, virtual folders and blobs: walk the account with a / delimiter
write('tree.py', 'from common import svc\nfrom lakeview import tree\n\nmarketing = svc.create_container("marketing")\nmarketing.upload_blob("campaigns/budget_2026.csv",\n b"campaign,budget\\nautumn,5000\\n")\nprint("storage account devstoreaccount1")\nfor c in svc.list_containers():\n print(" container:", c.name)\n tree(svc.get_container_client(c.name), depth=2)\nblob = svc.get_blob_client("sales", "orders/2026/10/30/orders.csv")\nprint("URL:", blob.url)\n')
run('python tree.py')
# Step 3: Flat namespace: "rename" a folder of 48 blobs = copy + delete each blob
write('rename_folder.py', """\
from azure.core.exceptions import ResourceNotFoundError
from common import svc, show_err
clicks = svc.create_container("clickstream")
for h in range(24): # 48 files in the wrong folder
for part in (1, 2):
clicks.upload_blob(f"staging/h{h:02}-{part}.json", b"{}")
def count(prefix):
return len(list(clicks.list_blob_names(name_starts_with=prefix)))
names = list(clicks.list_blob_names(name_starts_with="staging/"))
ops = 0
for i, old in enumerate(names, 1): # "rename" staging/ -> events/
new = clicks.get_blob_client(old.replace("staging/", "events/"))
copy = new.start_copy_from_url(clicks.get_blob_client(old).url)
assert copy["copy_status"] == "success" # Azure: may be pending
clicks.delete_blob(old)
ops += 2 # one copy + one delete
if i == 24:
print("halfway, a reader sees: staging", count("staging/"),
"| events", count("events/"))
print("files moved:", len(names), "| operations:", ops)
try:
clicks.delete_blob("staging") # is the folder an object?
except ResourceNotFoundError as e:
show_err(e)
""")
run('python rename_folder.py')
# Step 4: Block blob from staged blocks, an append blob, a page blob
write('block_blob.py', 'from azure.storage.blob import BlobBlock\nfrom common import svc\n\ntypes = svc.create_container("blobtypes")\norders = types.get_blob_client("orders/orders_big.csv")\nparts = [b"order_id,amount\\n", b"1001,25.50\\n", b"1002,14.00\\n"]\nblocks = []\nfor i, data in enumerate(parts):\n block_id = f"block-{i:04}" # same length for every id\n orders.stage_block(block_id, data)\n blocks.append(BlobBlock(block_id))\nprint("staged 3 blocks, blob exists:", orders.exists())\norders.commit_block_list(blocks) # now it is one blob\ncommitted, _ = orders.get_block_list("committed")\np = orders.get_blob_properties()\nprint("committed:", [b.id for b in committed])\nprint(p.blob_type.value, p.size, "bytes, tier", p.blob_tier)\n')
write('append_page.py', 'from azure.core.exceptions import HttpResponseError\nfrom common import svc, show_err\n\ntypes = svc.get_container_client("blobtypes")\nlog = types.get_blob_client("logs/pipeline.log")\nlog.create_append_blob()\nfor line in ["start\\n", "loaded 2 rows\\n", "done\\n"]:\n r = log.append_block(line) # always at the end\n print("append at offset", r["blob_append_offset"])\nprint(log.download_blob().readall())\ntry:\n log.set_standard_blob_tier("Cool") # tiers: block blobs only\nexcept HttpResponseError as e:\n show_err(e)\n\ndisk = types.get_blob_client("disks/vm-disk.vhd")\ndisk.create_page_blob(size=1024) # two 512-byte pages\ndisk.upload_page(b"A" * 512, offset=512, length=512)\nranges, _ = disk.get_page_ranges()\np = disk.get_blob_properties()\nprint(p.blob_type.value, p.size, "bytes, written:", ranges)\n')
run('python block_blob.py')
run('python append_page.py')
# Step 5: The DataLake client on a blob-only emulator: blob-backed calls work, a dfs call fails
write('dfs.py', """\
import logging
from azure.core.exceptions import HttpResponseError
from azure.storage.filedatalake import DataLakeServiceClient
from common import CONN
logging.getLogger("azure").setLevel(logging.ERROR) # quiet SDK logs
lake = DataLakeServiceClient.from_connection_string(CONN)
print("file systems:", [fs.name for fs in lake.list_file_systems()])
sales = lake.get_file_system_client("sales")
f = sales.get_file_client("orders/2026/10/30/orders.csv")
print("read through the DataLake client:",
f.download_file().readall()[:15])
try:
sales.create_directory("orders/2026/10/31") # a dfs operation
except HttpResponseError as e:
print("create_directory:", type(e).__name__, e.status_code)
""")
run('python dfs.py')
# Step 6: A first lake layout: raw, curated and finance containers with domain folders
write('build_lake.py', 'from common import svc\nfrom lakeview import tree\n\nlayout = {"raw": ["sales/orders_20261030.csv",\n "clickstream/events_20261030.json"],\n "curated": ["sales/daily_sales.csv",\n "clickstream/sessions.csv"],\n "finance": ["raw/gl_20261030.csv",\n "curated/gl_balances.csv"]}\nfor name, files in layout.items():\n owner = "finance" if name == "finance" else "data-platform"\n zone = "restricted" if name == "finance" else name\n lake = svc.create_container(\n name, metadata={"owner": owner, "zone": zone})\n for path in files:\n lake.upload_blob(path, b"sample\\n")\nfor name in layout:\n lake = svc.get_container_client(name)\n meta = lake.get_container_properties().metadata\n print("container:", name, dict(sorted(meta.items())))\n tree(lake)\n')
run('python build_lake.py')
Part 2: Lake Layout, Naming and Partitioning
The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run on Azurite 3.37.0, Microsoft's open-source Azure Storage emulator, with azure-storage-blob 12.31.1, pyarrow and pandas. Features the emulator does not support are shown as code. Part of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Azure Storage documentation (learn.microsoft.com/azure/storage) for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.