Azure Data Lake Storage Interview Questions and Answers Part 10 - Real-World Manager Scenarios

ADLS Gen2 Interview Q&A · Part 10 of 10

In this post we are going to cover real-world manager scenarios for Azure Data Lake Storage Gen2: a "not authorized" error at night, duplicate rows after a rerun, a migration, a container that went public, a slow listing and testing storage code. These are 7 real ADLS Gen2 and Blob Storage interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: in the final round, the hiring manager asks you to design the storage layer of a lakehouse on Azure.

This is the last part of the series, and it pulls the earlier parts together. Every question here starts with something going wrong, or a decision a manager needs from you today. Interviewers ask these because they show whether you have run a lake in production: how you debug, how you avoid repeating a mistake, and how you explain a design in a few minutes.

VIDEO COMING SOON

Part 10 of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT.

On this page
  1. The short answer
  2. Let's set up the project
  3. Question 1: A nightly load fails with "This request is not authorized to perform this operation". How do you investigate?
  4. Question 2: Analysts see duplicate rows after a pipeline rerun. How do you make file loads idempotent?
  5. Question 3: A team needs to migrate from Gen1 or another cloud's object store to ADLS Gen2. How do you plan it?
  6. Question 4: Someone accidentally made a container public. What do you do?
  7. Question 5: A job lists millions of files and takes an hour before it starts. How do you fix it?
  8. Question 6: A teammate asks how to test storage code locally and in CI. What do you set up?
  9. Scenario 7: In the final round, the hiring manager asks you to design the storage layer of a lakehouse on Azure. What is your design?
  10. The full script

The short answer

Say this first

Debug a 403 by its error code, not its message. Make loads idempotent with names that depend on the data. Reconcile every migration. Close a public container at the account level. List one partition, not the lake. Test on Azurite in CI, and design the lake with HNS, ZRS, Entra ID and private endpoints.

Let's set up the project

We run Azurite, Microsoft's open-source Azure Storage emulator, locally, and small Python scripts with the azure-storage-blob client and pandas, so every output below is real. Features the emulator does not support, like the hierarchical namespace, logs, role checks and account settings, are shown as code. common.py creates the client, and has a container() helper and an attempt() helper that runs a call and prints the error type, status, code and first line of the message.

common.py
import os
from azure.core.exceptions import HttpResponseError, ResourceExistsError
from azure.storage.blob import BlobServiceClient

CONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
svc = BlobServiceClient.from_connection_string(CONN)


def container(name, **kw):
    try:
        return svc.create_container(name, **kw)
    except ResourceExistsError:
        return svc.get_container_client(name)


def attempt(label, call):
    print(label)
    try:
        call()
        print("  ok")
    except HttpResponseError as e:
        print(" ", type(e).__name__, e.status_code, e.error_code)
        print(" ", str(e).split("\n")[0])

Question 1: A nightly load fails with "This request is not authorized to perform this operation". How do you investigate?

Don't stop at the message. Read the error code, because several 403 errors start with the same words. Then ask what changed.

Let's prove it. auth403.py runs three broken jobs against the emulator: a read-only SAS that writes, an https-only SAS sent over http, and an object-only account SAS that lists. We run python auth403.py.

auth403.py
from datetime import datetime, timezone
from azure.storage.blob import (AccountSasPermissions, BlobClient,
    BlobSasPermissions, ContainerClient, ResourceTypes,
    generate_account_sas, generate_blob_sas)
from common import attempt, container, svc

c = container("nightly")
c.upload_blob("orders/day.csv", "id,amount\n1,20.5\n", overwrite=True)
acct, key = svc.account_name, svc.credential.account_key
END = datetime(2030, 1, 1, tzinfo=timezone.utc)
def blob(**perm):                       # a client with a blob SAS
    sas = generate_blob_sas(acct, "nightly", "orders/day.csv",
                            account_key=key, expiry=END, **perm)
    return BlobClient(svc.url, "nightly", "orders/day.csv",
                      credential=sas)

ro = blob(permission=BlobSasPermissions(read=True))
attempt("1. read-only SAS, the job writes:",
        lambda: ro.upload_blob("id,amount\n", overwrite=True))
tls = blob(permission=BlobSasPermissions(read=True), protocol="https")
attempt("2. https-only SAS, sent over http:", tls.download_blob)
objs = generate_account_sas(acct, key, ResourceTypes(object=True),
    AccountSasPermissions(read=True, list=True), expiry=END)
lister = ContainerClient(svc.url, "nightly", credential=objs)
attempt("3. object-only account SAS, the job lists:",
        lambda: list(lister.list_blobs()))
Terminal
$ python auth403.py
1. read-only SAS, the job writes:
  HttpResponseError 403 AuthorizationPermissionMismatch
  This request is not authorized to perform this operation using this permission.
2. https-only SAS, sent over http:
  HttpResponseError 403 AuthorizationProtocolMismatch
  This request is not authorized to perform this operation using this protocol.
3. object-only account SAS, the job lists:
  HttpResponseError 403 AuthorizationResourceTypeMismatch
  This request is not authorized to perform this operation using this resource type.

All three messages start with "This request is not authorized to perform this operation". The codes are different: AuthorizationPermissionMismatch, AuthorizationProtocolMismatch and AuthorizationResourceTypeMismatch.

On Azure, map the code to a cause:

Error code (Azure)Usual cause
AuthorizationPermissionMismatchmissing data role, ACL or SAS permission
AuthorizationFailureusually the network: firewall, private endpoints
KeyBasedAuthenticationNotPermittedshared key disabled: use Microsoft Entra ID
AuthenticationFailedbad signature, expired SAS, clock skew

Then check the facts, one by one:

  • The identity - which principal, which secret
  • Its data role or ACLs - on this account, container, path
  • The network - firewall, private endpoints
  • The key or SAS - rotated, expired, wrong permissions

The StorageBlobLogs table shows the failed calls and the caller's IP, and two CLI calls show the roles and the network rules:

triage (KQL and Azure CLI, not run)
// 1. the failed calls (Log Analytics, not run here)
StorageBlobLogs
| where TimeGenerated > ago(1d) and StatusCode == "403"
| summarize count() by StatusText, AuthenticationType,
    CallerIpAddress, RequesterObjectId

# 2. the pipeline identity's roles on the lake (Azure CLI)
az role assignment list --assignee $PIPELINE_ID \
  --scope $LAKE --include-inherited -o table
# 3. the network rules
az storage account show -n stlakeprod -g rg-lake \
  --query "[networkRuleSet.defaultAction, publicNetworkAccess]"
Two classic causes

Owner or Contributor gives no access to the data; only a data role does. And a new role assignment can take up to 10 minutes, so a rerun right after the fix can still fail.

Question 2: Analysts see duplicate rows after a pipeline rerun. How do you make file loads idempotent?

Make the output path depend on the data, not on the run. Same input, same file name, so a rerun replaces the file instead of adding a second one.

  • run-0105.csv + run-0412.csv - a rerun adds a file: duplicates
  • part-0.csv - a rerun replaces it: idempotent

Let's see the bug, then the fix. load_bad.py names its file after the run time. load_good.py writes part-0.csv for the load date with overwrite, writes a done marker only after success, and checks that marker first. count_rows.py counts files and rows in a partition. We run the bad load twice, as runs 0105 and 0412, and count. Then the good load twice, and count.

load_bad.py
import sys
from common import container

run = sys.argv[1]                       # e.g. the run's start time
rows = "order_id,amount\n101,20.5\n102,7.25\n103,12.0\n"
lake = container("sales")
lake.upload_blob(f"orders/load_date=2026-10-29/run-{run}.csv", rows)
print("run", run, "wrote", f"run-{run}.csv")
load_good.py
from common import container

DAY = "2026-10-29"
rows = "order_id,amount\n101,20.5\n102,7.25\n103,12.0\n"
lake = container("sales")
done = lake.get_blob_client(f"_loads/orders/{DAY}.done")
if done.exists():                          # processed-files log
    print("already loaded", DAY, "-> skip")
    raise SystemExit
path = f"orders_v2/load_date={DAY}/part-0.csv"   # same name every run
lake.upload_blob(path, rows, overwrite=True)     # a rerun replaces it
done.upload_blob(b"", overwrite=True)            # mark after success
print("loaded", path)
count_rows.py
import io, sys
import pandas as pd
from common import svc

lake = svc.get_container_client("sales")
prefix = sys.argv[1] + "/load_date=2026-10-29/"
files = [b.name for b in lake.list_blobs(name_starts_with=prefix)]
df = pd.concat(pd.read_csv(io.BytesIO(
    lake.download_blob(f).readall())) for f in files)
print(sys.argv[1], "| files:", len(files), "| rows:", len(df),
      "| distinct order_id:", df["order_id"].nunique())
Terminal
$ python load_bad.py 0105
run 0105 wrote run-0105.csv
$ python load_bad.py 0412
run 0412 wrote run-0412.csv
$ python count_rows.py orders
orders | files: 2 | rows: 6 | distinct order_id: 3
$ python load_good.py
loaded orders_v2/load_date=2026-10-29/part-0.csv
$ python load_good.py
already loaded 2026-10-29 -> skip
$ python count_rows.py orders_v2
orders_v2 | files: 1 | rows: 3 | distinct order_id: 3

The bad load leaves 2 files in the same partition: 6 rows, but only 3 distinct orders. The good load writes orders_v2/load_date=2026-10-29/part-0.csv, and the second run says already loaded 2026-10-29 -> skip. The result: 1 file, 3 rows.

For writes that must never replace anything, use if_none_match="*": a second write then fails with 409 BlobAlreadyExists.

On real projects

The processed-files log stops reprocessing, and the fixed name makes a rerun harmless. For Delta tables, I merge on the business key, or replace the partition.

Question 3: A team needs to migrate from Gen1 or another cloud's object store to ADLS Gen2. How do you plan it?

First, a fact to get right: Gen1 was retired on February 29, 2024, and its data is no longer accessible. So today the sources are other clouds, on-premises systems, and flat Blob Storage accounts.

Then pick the tool by size and network:

SourcePlan
ADLS Gen1retired 2024-02-29: no longer accessible
Amazon S3, Google Cloud StorageAzCopy v10, service to service
More than 40 TB, limited networkData Box
Flat Blob Storage accountupgrade in place to ADLS Gen2: one way

AzCopy copies from Amazon S3 and Google Cloud Storage, service to service:

s3_to_lake.sh (AzCopy, not run)
# Amazon S3 to the lake, service to service (AzCopy v10, not run)
# S3 access keys in the environment, Entra ID for the lake
azcopy login
azcopy copy "https://s3.amazonaws.com/shop-sales" \
  "https://stlakeprod.blob.core.windows.net/raw/sales" \
  --recursive=true --dry-run

A flat blob account can be upgraded in place, but only one way:

upgrade.sh (Azure CLI, not run)
# flat Blob account to ADLS Gen2, one way (Azure CLI, not run)
az storage account hns-migration start --type validation \
  -n stlegacy -g rg-lake
az storage account hns-migration start --type upgrade \
  -n stlegacy -g rg-lake

Whatever copies the data, reconcile it. Here a second local account plays the old store. We run old_store.py, migrate.py, then late_write.py, where the old system writes after the copy, then reconcile.py, which compares names, sizes and MD5 hashes. Then we run the copy and the reconcile once more.

old_store.py
import os
from azure.storage.blob import BlobServiceClient

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])   # the old store
src = old.create_container("legacy-sales")
for m in ["07", "08", "09", "10"]:
    src.upload_blob(f"orders/2026-{m}.csv", f"month,{m}\n" * 900)
src.upload_blob("customers/customers.csv", "id,name\n" * 400)
print("old store: 5 files in legacy-sales")
migrate.py
import os, time
from datetime import datetime, timezone
from azure.storage.blob import (BlobServiceClient,
    ContainerSasPermissions, generate_container_sas)
from common import container

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])
src = old.get_container_client("legacy-sales")
sas = generate_container_sas(old.account_name, "legacy-sales",
    account_key=old.credential.account_key,
    permission=ContainerSasPermissions(read=True, list=True),
    expiry=datetime(2030, 1, 1, tzinfo=timezone.utc))
dst = container("landing")
names = [b.name for b in src.list_blobs()]
for name in names:
    target = dst.get_blob_client("legacy-sales/" + name)
    target.start_copy_from_url(f"{src.url}/{name}?{sas}")
    while target.get_blob_properties().copy.status == "pending":
        time.sleep(10)              # Azure: a copy can be pending
print("copied", len(names), "files")
late_write.py
import os
from azure.storage.blob import BlobServiceClient

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])
src = old.get_container_client("legacy-sales")
src.upload_blob("orders/2026-10.csv", "month,10\n" * 950,
                overwrite=True)     # the old system is still writing
print("old store: orders/2026-10.csv changed after the copy")
reconcile.py
import os
from azure.storage.blob import BlobServiceClient
from common import svc

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])
def manifest(c, prefix=""):            # name -> (size, MD5 if set)
    return {b.name[len(prefix):]:
            (b.size, bytes(b.content_settings.content_md5 or b""))
            for b in c.list_blobs(name_starts_with=prefix)}
src = manifest(old.get_container_client("legacy-sales"))
dst = manifest(svc.get_container_client("landing"), "legacy-sales/")
print("files old:", len(src), "new:", len(dst), "| bytes old:",
      sum(s for s, _ in src.values()), "new:",
      sum(s for s, _ in dst.values()))
bad = sorted(n for n in src if dst.get(n) != src[n])
print("size or MD5 mismatch:", bad or "none")
Terminal
$ python old_store.py
old store: 5 files in legacy-sales
$ python migrate.py
copied 5 files
$ python late_write.py
old store: orders/2026-10.csv changed after the copy
$ python reconcile.py
files old: 5 new: 5 | bytes old: 36050 new: 35600
size or MD5 mismatch: ['orders/2026-10.csv']
$ python migrate.py
copied 5 files
$ python reconcile.py
files old: 5 new: 5 | bytes old: 36050 new: 36050
size or MD5 mismatch: none

After the first copy, both sides have 5 files, but the bytes differ, 36050 against 35600, and orders/2026-10.csv is a mismatch. After the second copy, both sides have 36050 bytes and the reconcile says none. The emulator finishes copies at once. On Azure a copy can stay pending, so migrate.py polls the status.

The cutover

End the plan with a final delta pass: freeze the writers, copy again, reconcile, then switch the pipelines.

Question 4: Someone accidentally made a container public. What do you do?

First, stop the exposure. Set the container back to private. And if no container in the account should ever be public, turn off AllowBlobPublicAccess on the account, which blocks it for every container at once.

close_public.sh (Azure CLI, not run)
# no container in this account may be public (Azure CLI, not run)
az storage account update -n stlakeprod -g rg-lake \
  --allow-blob-public-access false
# or just this container
az storage container set-permission --name exports \
  --public-access off --account-name stlakeprod

Let's try it. public_fix.py creates an exports container with container-level access, puts a payroll file in it, and makes anonymous list and read calls, before and after setting it back to private. We run python public_fix.py.

public_fix.py
import requests
from azure.storage.blob import PublicAccess
from common import container

c = container("exports", public_access=PublicAccess.CONTAINER)
c.upload_blob("finance/payroll_2026-10.csv", "emp,amount\n7,4100\n",
              overwrite=True)
anon_list = c.url + "?restype=container&comp=list"
anon_read = c.url + "/finance/payroll_2026-10.csv"
def check(when):                        # no credential: anonymous
    level = c.get_container_properties().public_access
    print(f"{when:6} public_access={level}",
          "| anonymous list:", requests.get(anon_list).status_code,
          "| anonymous read:", requests.get(anon_read).status_code)
check("before")
c.set_container_access_policy(signed_identifiers={},
                              public_access=None)    # back to private
check("after")
Terminal
$ python public_fix.py
before public_access=container | anonymous list: 200 | anonymous read: 200
after  public_access=None | anonymous list: 403 | anonymous read: 403

Before the fix, with public_access=container, anyone can list the container and read the payroll file: 200 and 200. After, both anonymous calls are refused. The emulator answers 403. Azure refuses them too, with 404, 409, 401 or 403, depending on the account setting and the request's version.

Then find out what leaked. Check the Transactions metric split by Authentication for Anonymous, and StorageBlobLogs for anonymous reads, by IP and path:

anonymous_reads.kql (Log Analytics, not run)
// what was read anonymously? (Log Analytics, not run here)
StorageBlobLogs
| where AuthenticationType == "Anonymous"
| where Uri has "/exports/"
| summarize reads = count() by CallerIpAddress, Uri
| order by reads desc

Then clean up:

  • Rotate - keys or SAS found in the files
  • Public data - in a separate account
Tell security

If personal data was read, tell security right away.

Question 5: A job lists millions of files and takes an hour before it starts. How do you fix it?

A listing returns at most 5,000 blobs per call, and each next page needs the token from the last one, so the calls run one after another.

  • List call 1 - 5,000 blobs + a token
  • List call 2 - needs the token from call 1
  • 10 million files - 2,000 calls in a row

Let's measure. listing_seed.py writes 6,000 event files in 20 daily partitions. listing.py counts the list calls three ways: the whole container, one day's prefix, and folders only with a delimiter. We run python listing_seed.py, then python listing.py.

listing_seed.py
from concurrent.futures import ThreadPoolExecutor
from common import container

c = container("events")
names = [f"clicks/date=2026-10-{d:02}/part-{n:04}.json"
         for d in range(11, 31) for n in range(300)]
with ThreadPoolExecutor(16) as pool:
    list(pool.map(lambda n: c.upload_blob(n, b"{}", overwrite=True),
                  names))
print("events: 20 days x 300 files =", len(names))
listing.py
from collections import Counter
from azure.storage.blob import BlobServiceClient
from common import CONN

calls = Counter()                        # every HTTP request sent
svc = BlobServiceClient.from_connection_string(
    CONN, raw_response_hook=lambda r: calls.update(["list"]))
c = svc.get_container_client("events")

pages = [len(list(p)) for p in c.list_blobs().by_page()]
print("whole container :", pages, "|", calls["list"], "calls")
calls.clear()
day = c.list_blobs(name_starts_with="clicks/date=2026-10-30/")
pages = [len(list(p)) for p in day.by_page()]
print("one day, prefix :", pages, "|", calls["list"], "call")
calls.clear()
days = [p.name for p in c.walk_blobs(name_starts_with="clicks/",
                                     delimiter="/")]
print("folders only    :", len(days), "folders, last", days[-1],
      "|", calls["list"], "call")
Terminal
$ python listing_seed.py
events: 20 days x 300 files = 6000
$ python listing.py
whole container : [5000, 1000] | 2 calls
one day, prefix : [300] | 1 call
folders only    : 20 folders, last clicks/date=2026-10-30/ | 1 call

The whole container takes 2 calls, with pages of 5000 and 1000. One day's prefix takes 1 call, for 300 files. And with a delimiter, 1 call returns just the 20 date folders, the last one clicks/date=2026-10-30/.

So the fix starts with the layout, and then we stop listing what we don't need:

  • Partition by date - list one prefix
  • Inventory report - instead of a full listing
  • BlobCreated events - + a processed-files log
Mind the meter

On the hierarchical namespace, list calls have their own meter too, so a job that lists the whole lake every run also shows up on the bill.

Question 6: A teammate asks how to test storage code locally and in CI. What do you set up?

Azurite, Microsoft's open-source storage emulator. Point the connection string at it, and the same SDK code runs without a cloud account.

  • Azurite - npm, Docker image, VS Code
  • Connection string - from the environment
  • Same SDK code - no cloud account

Let's write a test. loader.py writes one day of orders to a fixed name. test_loader.py creates a fresh container, runs the loader twice, asserts there is still one file with two rows, and deletes its container at the end, so tests don't step on each other. We run python test_loader.py.

loader.py
def load_orders(container, day, csv_text):
    """Write one day of orders; a rerun replaces the same file."""
    path = f"orders/load_date={day}/part-0.csv"
    container.upload_blob(path, csv_text, overwrite=True)
    return path
test_loader.py
import io, uuid
import pandas as pd
from common import svc
from loader import load_orders

info = svc.get_account_information()       # what do we test on?
print("hierarchical namespace:", info["is_hns_enabled"])
CSV = "order_id,amount\n101,20.5\n102,7.25\n"


def test_rerun_keeps_one_file():
    c = svc.create_container(f"test-{uuid.uuid4().hex[:8]}")
    try:
        path = load_orders(c, "2026-10-29", CSV)
        load_orders(c, "2026-10-29", CSV)              # the rerun
        assert [b.name for b in c.list_blobs()] == [path]
        df = pd.read_csv(io.BytesIO(c.download_blob(path).readall()))
        assert len(df) == 2 and df["amount"].sum() == 27.75
    finally:
        c.delete_container()                           # clean up


test_rerun_keeps_one_file()
print("PASS test_rerun_keeps_one_file")
Terminal
$ python test_loader.py
hierarchical namespace: False
PASS test_rerun_keeps_one_file

The test prints hierarchical namespace: False, then PASS test_rerun_keeps_one_file. That first line is the reminder of what the emulator can't do:

  • No HNS - directories, ACLs, rename: a real account
  • No role checks - and no throttling
  • Newer SDK? - --skipApiVersionCheck

In CI, start the Docker image with in-memory persistence before the tests:

storage-tests.yml (GitHub Actions, not run here)
# .github/workflows/storage-tests.yml
on: [push]
jobs:
  storage-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.13"
      - name: Start Azurite (Blob only, in memory)
        run: >
          docker run -d -p 10000:10000
          mcr.microsoft.com/azure-storage/azurite azurite-blob
          --blobHost 0.0.0.0 --inMemoryPersistence
          --skipApiVersionCheck
      - run: pip install azure-storage-blob pandas
      - run: python test_loader.py
        env:
          AZURE_STORAGE_CONNECTION_STRING: UseDevelopmentStorage=true
Two layers of tests

Keep fast tests on Azurite, and run a small set against a real ADLS Gen2 account before release.

Scenario 7: In the final round, the hiring manager asks you to design the storage layer of a lakehouse on Azure. What is your design?

One general-purpose v2 account per environment, with the hierarchical namespace, and zone-redundant storage in the primary region, as Microsoft recommends for Data Lake Storage. Then I walk through it, area by area.

  • Sources - ERP, apps, events
  • raw - as delivered
  • curated - Delta, by date
  • serving - Delta, models
  • Engines - Databricks, Fabric

The zones are containers, partitioned by date, with Delta tables in curated and serving.

Security: Microsoft Entra ID, with shared key disabled. Data roles for the platform team, ACLs through Entra groups on sensitive paths like finance, and private endpoints for both blob and dfs, with public access off.

Protection: blob and container soft delete, a resource lock, and copies to a second account, because versioning and point-in-time restore aren't available with the hierarchical namespace.

Engines: Databricks through Unity Catalog external locations with an Access Connector managed identity, and Fabric through OneLake shortcuts, both on the dfs endpoint. Cost: lifecycle tiering, inventory, and a monthly report.

AreaDecision
AccountGPv2 + hierarchical namespace, ZRS; one account per environment
SecurityEntra ID, shared key off; data roles + ACLs via Entra groups
Networkprivate endpoints for blob and dfs; public access off
Protectionblob + container soft delete, resource lock, copies
EnginesUC external location (access connector), OneLake shortcut: dfs
Costlifecycle tiering, inventory, monthly report
Confirm first

Say what you would confirm before building: the data volumes, the access patterns and the recovery targets, because they decide the tiers and the redundancy.

Key points to remember

  • 403: read the error code, then check identity, role, network, key
  • Idempotent loads: data-based names + overwrite, a processed-files log
  • Migrations: AzCopy, Data Box or an upgrade, then reconcile and a delta pass
  • Public container: back to private, AllowBlobPublicAccess false, check the logs
  • List one partition; test on Azurite; design with HNS, ZRS, Entra ID

The full script

This is a Python script that writes every file and runs every command above in order against Azurite: run pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas first.

Show the full script that writes the files and runs every step
Full script - writes the files and runs every step against Azurite (pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas), run it with Python
# ========================================================================
# ADLS Gen2 Interview Q&A #10 - Real-world manager scenarios
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run on Azurite 3.37.0 (Microsoft's open-source Azure Storage emulator, Blob service)
# with azure-storage-blob 12.31.1, pyarrow and pandas; the outputs in the video and the blog post are the real outputs.
# To run it yourself:
#   1. npm install -g azurite   then   azurite-blob --skipApiVersionCheck --location ./azurite-data
#      (or the Docker image mcr.microsoft.com/azure-storage/azurite), and leave it running on port 10000.
#      --skipApiVersionCheck lets a newer azure-storage-blob talk to it.
#      This video also uses a second local account: start Azurite with the environment variable
#      AZURITE_ACCOUNTS="devstoreaccount1:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;partnerlake:cGFydG5lcmxha2Vsb2NhbGtleWZvcmF6dXJpdGVkZW1vMDEyMzQ1Njc4OTAxMjM0NTY3OA=="
#   2. pip install azure-storage-blob azure-storage-file-datalake pyarrow pandas requests
#   3. Set AZURE_STORAGE_CONNECTION_STRING to "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"
#      and PARTNER_STORAGE_CONNECTION_STRING to "DefaultEndpointsProtocol=http;AccountName=partnerlake;AccountKey=cGFydG5lcmxha2Vsb2NhbGtleWZvcmF6dXJpdGVkZW1vMDEyMzQ1Njc4OTAxMjM0NTY3OA==;BlobEndpoint=http://127.0.0.1:10000/partnerlake;"
#   then run python ADLS_Interview_QA_10_Code.py.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================

import os, subprocess, time

def write(path, text):
    if text is None:
        if os.path.exists(path): os.remove(path)
        return
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        f.write(text.strip("\n") + "\n")

def run(cmd, expect_error=False, allow_error=False, env=None):
    print("\n$ " + cmd, flush=True)
    rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
    if rc != 0 and not (expect_error or allow_error):
        raise SystemExit("stopped: this command was expected to succeed")

# Step 0: the starting files
write('common.py', 'import os\nfrom azure.core.exceptions import HttpResponseError, ResourceExistsError\nfrom azure.storage.blob import BlobServiceClient\n\nCONN = os.environ["AZURE_STORAGE_CONNECTION_STRING"]\nsvc = BlobServiceClient.from_connection_string(CONN)\n\n\ndef container(name, **kw):\n    try:\n        return svc.create_container(name, **kw)\n    except ResourceExistsError:\n        return svc.get_container_client(name)\n\n\ndef attempt(label, call):\n    print(label)\n    try:\n        call()\n        print("  ok")\n    except HttpResponseError as e:\n        print(" ", type(e).__name__, e.status_code, e.error_code)\n        print(" ", str(e).split("\\n")[0])\n')

# Step 1: Three broken nightly jobs: the same 403 message, three different error codes
write('auth403.py', 'from datetime import datetime, timezone\nfrom azure.storage.blob import (AccountSasPermissions, BlobClient,\n    BlobSasPermissions, ContainerClient, ResourceTypes,\n    generate_account_sas, generate_blob_sas)\nfrom common import attempt, container, svc\n\nc = container("nightly")\nc.upload_blob("orders/day.csv", "id,amount\\n1,20.5\\n", overwrite=True)\nacct, key = svc.account_name, svc.credential.account_key\nEND = datetime(2030, 1, 1, tzinfo=timezone.utc)\ndef blob(**perm):                       # a client with a blob SAS\n    sas = generate_blob_sas(acct, "nightly", "orders/day.csv",\n                            account_key=key, expiry=END, **perm)\n    return BlobClient(svc.url, "nightly", "orders/day.csv",\n                      credential=sas)\n\nro = blob(permission=BlobSasPermissions(read=True))\nattempt("1. read-only SAS, the job writes:",\n        lambda: ro.upload_blob("id,amount\\n", overwrite=True))\ntls = blob(permission=BlobSasPermissions(read=True), protocol="https")\nattempt("2. https-only SAS, sent over http:", tls.download_blob)\nobjs = generate_account_sas(acct, key, ResourceTypes(object=True),\n    AccountSasPermissions(read=True, list=True), expiry=END)\nlister = ContainerClient(svc.url, "nightly", credential=objs)\nattempt("3. object-only account SAS, the job lists:",\n        lambda: list(lister.list_blobs()))\n')
run('python auth403.py')

# Step 2: A rerun with run-named files duplicates rows; a fixed name plus a done marker does not
write('load_bad.py', 'import sys\nfrom common import container\n\nrun = sys.argv[1]                       # e.g. the run\'s start time\nrows = "order_id,amount\\n101,20.5\\n102,7.25\\n103,12.0\\n"\nlake = container("sales")\nlake.upload_blob(f"orders/load_date=2026-10-29/run-{run}.csv", rows)\nprint("run", run, "wrote", f"run-{run}.csv")\n')
write('load_good.py', 'from common import container\n\nDAY = "2026-10-29"\nrows = "order_id,amount\\n101,20.5\\n102,7.25\\n103,12.0\\n"\nlake = container("sales")\ndone = lake.get_blob_client(f"_loads/orders/{DAY}.done")\nif done.exists():                          # processed-files log\n    print("already loaded", DAY, "-> skip")\n    raise SystemExit\npath = f"orders_v2/load_date={DAY}/part-0.csv"   # same name every run\nlake.upload_blob(path, rows, overwrite=True)     # a rerun replaces it\ndone.upload_blob(b"", overwrite=True)            # mark after success\nprint("loaded", path)\n')
write('count_rows.py', """\
import io, sys
import pandas as pd
from common import svc

lake = svc.get_container_client("sales")
prefix = sys.argv[1] + "/load_date=2026-10-29/"
files = [b.name for b in lake.list_blobs(name_starts_with=prefix)]
df = pd.concat(pd.read_csv(io.BytesIO(
    lake.download_blob(f).readall())) for f in files)
print(sys.argv[1], "| files:", len(files), "| rows:", len(df),
      "| distinct order_id:", df["order_id"].nunique())
""")
run('python load_bad.py 0105')
run('python load_bad.py 0412')
run('python count_rows.py orders')
run('python load_good.py')
run('python load_good.py')
run('python count_rows.py orders_v2')

# Step 3: Migrate from an old store, then reconcile names, sizes and MD5; a late write needs a delta pass
write('old_store.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\n\nold = BlobServiceClient.from_connection_string(\n    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])   # the old store\nsrc = old.create_container("legacy-sales")\nfor m in ["07", "08", "09", "10"]:\n    src.upload_blob(f"orders/2026-{m}.csv", f"month,{m}\\n" * 900)\nsrc.upload_blob("customers/customers.csv", "id,name\\n" * 400)\nprint("old store: 5 files in legacy-sales")\n')
write('migrate.py', """\
import os, time
from datetime import datetime, timezone
from azure.storage.blob import (BlobServiceClient,
    ContainerSasPermissions, generate_container_sas)
from common import container

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])
src = old.get_container_client("legacy-sales")
sas = generate_container_sas(old.account_name, "legacy-sales",
    account_key=old.credential.account_key,
    permission=ContainerSasPermissions(read=True, list=True),
    expiry=datetime(2030, 1, 1, tzinfo=timezone.utc))
dst = container("landing")
names = [b.name for b in src.list_blobs()]
for name in names:
    target = dst.get_blob_client("legacy-sales/" + name)
    target.start_copy_from_url(f"{src.url}/{name}?{sas}")
    while target.get_blob_properties().copy.status == "pending":
        time.sleep(10)              # Azure: a copy can be pending
print("copied", len(names), "files")
""")
write('late_write.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\n\nold = BlobServiceClient.from_connection_string(\n    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])\nsrc = old.get_container_client("legacy-sales")\nsrc.upload_blob("orders/2026-10.csv", "month,10\\n" * 950,\n                overwrite=True)     # the old system is still writing\nprint("old store: orders/2026-10.csv changed after the copy")\n')
write('reconcile.py', """\
import os
from azure.storage.blob import BlobServiceClient
from common import svc

old = BlobServiceClient.from_connection_string(
    os.environ["PARTNER_STORAGE_CONNECTION_STRING"])
def manifest(c, prefix=""):            # name -> (size, MD5 if set)
    return {b.name[len(prefix):]:
            (b.size, bytes(b.content_settings.content_md5 or b""))
            for b in c.list_blobs(name_starts_with=prefix)}
src = manifest(old.get_container_client("legacy-sales"))
dst = manifest(svc.get_container_client("landing"), "legacy-sales/")
print("files old:", len(src), "new:", len(dst), "| bytes old:",
      sum(s for s, _ in src.values()), "new:",
      sum(s for s, _ in dst.values()))
bad = sorted(n for n in src if dst.get(n) != src[n])
print("size or MD5 mismatch:", bad or "none")
""")
run('python old_store.py')
run('python migrate.py')
run('python late_write.py')
run('python reconcile.py')
run('python migrate.py')
run('python reconcile.py')

# Step 4: A public container: anonymous list and read, then back to private
write('public_fix.py', 'import requests\nfrom azure.storage.blob import PublicAccess\nfrom common import container\n\nc = container("exports", public_access=PublicAccess.CONTAINER)\nc.upload_blob("finance/payroll_2026-10.csv", "emp,amount\\n7,4100\\n",\n              overwrite=True)\nanon_list = c.url + "?restype=container&comp=list"\nanon_read = c.url + "/finance/payroll_2026-10.csv"\ndef check(when):                        # no credential: anonymous\n    level = c.get_container_properties().public_access\n    print(f"{when:6} public_access={level}",\n          "| anonymous list:", requests.get(anon_list).status_code,\n          "| anonymous read:", requests.get(anon_read).status_code)\ncheck("before")\nc.set_container_access_policy(signed_identifiers={},\n                              public_access=None)    # back to private\ncheck("after")\n')
run('python public_fix.py')

# Step 5: Listing 6,000 files: the whole container, one partition prefix, folders only
write('listing_seed.py', """\
from concurrent.futures import ThreadPoolExecutor
from common import container

c = container("events")
names = [f"clicks/date=2026-10-{d:02}/part-{n:04}.json"
         for d in range(11, 31) for n in range(300)]
with ThreadPoolExecutor(16) as pool:
    list(pool.map(lambda n: c.upload_blob(n, b"{}", overwrite=True),
                  names))
print("events: 20 days x 300 files =", len(names))
""")
write('listing.py', """\
from collections import Counter
from azure.storage.blob import BlobServiceClient
from common import CONN

calls = Counter()                        # every HTTP request sent
svc = BlobServiceClient.from_connection_string(
    CONN, raw_response_hook=lambda r: calls.update(["list"]))
c = svc.get_container_client("events")

pages = [len(list(p)) for p in c.list_blobs().by_page()]
print("whole container :", pages, "|", calls["list"], "calls")
calls.clear()
day = c.list_blobs(name_starts_with="clicks/date=2026-10-30/")
pages = [len(list(p)) for p in day.by_page()]
print("one day, prefix :", pages, "|", calls["list"], "call")
calls.clear()
days = [p.name for p in c.walk_blobs(name_starts_with="clicks/",
                                     delimiter="/")]
print("folders only    :", len(days), "folders, last", days[-1],
      "|", calls["list"], "call")
""")
run('python listing_seed.py')
run('python listing.py')

# Step 6: A storage test that runs on Azurite: the loader, a rerun, assertions, clean-up
write('loader.py', 'def load_orders(container, day, csv_text):\n    """Write one day of orders; a rerun replaces the same file."""\n    path = f"orders/load_date={day}/part-0.csv"\n    container.upload_blob(path, csv_text, overwrite=True)\n    return path\n')
write('test_loader.py', 'import io, uuid\nimport pandas as pd\nfrom common import svc\nfrom loader import load_orders\n\ninfo = svc.get_account_information()       # what do we test on?\nprint("hierarchical namespace:", info["is_hns_enabled"])\nCSV = "order_id,amount\\n101,20.5\\n102,7.25\\n"\n\n\ndef test_rerun_keeps_one_file():\n    c = svc.create_container(f"test-{uuid.uuid4().hex[:8]}")\n    try:\n        path = load_orders(c, "2026-10-29", CSV)\n        load_orders(c, "2026-10-29", CSV)              # the rerun\n        assert [b.name for b in c.list_blobs()] == [path]\n        df = pd.read_csv(io.BytesIO(c.download_blob(path).readall()))\n        assert len(df) == 2 and df["amount"].sum() == 27.75\n    finally:\n        c.delete_container()                           # clean up\n\n\ntest_rerun_keeps_one_file()\nprint("PASS test_rerun_keeps_one_file")\n')
run('python test_loader.py')
That is the whole series

All 10 parts are listed on the Real Time Azure Data Lake Storage Interview Questions and Answers page on this blog.

The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run on Azurite 3.37.0, Microsoft's open-source Azure Storage emulator, with azure-storage-blob 12.31.1, pyarrow and pandas. Features the emulator does not support are shown as code. Part of the Real Time Azure Data Lake Storage Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Azure Storage documentation (learn.microsoft.com/azure/storage) for anything you quote in an interview.

No comments:
Subscribe to: Posts (Atom)