In this post we are going to cover blob and file processing. These are 7 real Azure Functions interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: partners drop CSV files all day, and each must be validated, converted to Parquet and loaded.
Files landing in a container are one of the most common reasons a data team uses Azure Functions. A file arrives, a function picks it up, checks it and moves it on. Interviewers like this topic because the problems are very real: the same file loads twice, a big file runs out of memory, or one bad file keeps failing and nobody notices.
Part 3 of the Real Time Azure Functions Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Let's set up the project
- Question 1: Files land in a container and must be processed as soon as they arrive. Which trigger do you use?
- Question 2: Someone asks the difference between the polling blob trigger and the Event Grid based blob trigger. Which is better?
- Question 3: A teammate's blob function processes the same file twice. Why does that happen, and how do you stop it?
- Question 4: A 5 GB file must be processed by a function. What do you watch out for?
- Question 5: Someone asks what happens when a blob function keeps failing on one file. How do poison blobs work?
- Question 6: A teammate asks how to read and write blobs inside a function without loading everything into memory. What do you do?
- Scenario 7: Partners drop CSV files all day and each must be validated, converted to Parquet and loaded. How do you build it?
- The full script
The short answer
Use the blob trigger with the Event Grid source (the only source on Flex Consumption). Expect duplicates: receipts are per ETag, so make loads idempotent. A failing file goes to webjobs-blobtrigger-poison after five tries, and big files are streamed in chunks, because the default Python InputStream holds the whole blob.
Let's set up the project
Everything below runs on the real Azure Functions host running locally with Core Tools, with Azurite, Microsoft's open-source storage emulator, for blobs, queues and tables. fhost runs func start in the background and prints the cleaned host log. What only Azure can show, such as the Event Grid subscription and plan sizes, is shown as code. Here are the starting files: the app's config in files_app/, and small client scripts. containers.py creates the containers once, upload.py uploads a file (a | is a new line), and grid.py plays Event Grid, because Azurite raises no storage events.
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[4.*, 5.0.0)"
},
"logging": {
"logLevel": {
"default": "Warning",
"Function": "Information",
"Azure.Core": "Error",
"Host.Function.Console": "Information"
}
}
}{
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "python",
"AzureWebJobsStorage": "UseDevelopmentStorage=true"
}
}azure-functions
azure-storage-blob
azure-data-tables
pyarrowimport os
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for name in ["landing", "drops", "orders", "invoices", "quarantine",
"raw", "curated", "incoming"]:
if not svc.get_container_client(name).exists():
svc.create_container(name)
print("containers ready")import os, sys
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
container, name, text = sys.argv[1:4] # "|" = new line
svc.get_blob_client(container, name).upload_blob(
text.replace("|", "\n") + "\n", overwrite=True)
print("uploaded", f"{container}/{name}")import sys, requests
# Azurite raises no storage events, so this script plays Event Grid:
# it posts a BlobCreated event to the host's blob webhook.
container, blob = sys.argv[1].split("/", 1)
event = [{"id": "evt-" + blob, "eventType": "Microsoft.Storage.BlobCreated",
"subject": f"/blobServices/default/containers/{container}"
f"/blobs/{blob}",
"eventTime": "2026-11-01T09:00:00Z", "dataVersion": "",
"data": {"api": "PutBlob", "blobType": "BlockBlob",
"url": "http://127.0.0.1:10000/devstoreaccount1/"
f"{container}/{blob}"}}]
r = requests.post("http://localhost:7071/runtime/webhooks/blobs",
params={"functionName": "Host.Functions." + sys.argv[2]},
headers={"aeg-event-type": "Notification"}, json=event)
print("Event Grid delivery:", r.status_code)import os, sys
from azure.storage.blob import BlobServiceClient
hosts = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("azure-webjobs-hosts")
names = [b.name for b in hosts.list_blobs(name_starts_with="blobreceipts/")
if f"/Host.Functions.{sys.argv[1]}/" in b.name]
for n in names:
print(n) # .../<function>/<etag>/<blob>
print("receipts:", len(names), "| different ETags:",
len({n.split("/")[3] for n in names}))import os, sys
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for c in sys.argv[1:]:
names = sorted(svc.get_container_client(c).list_blob_names())
print(f"{c}:", names)import re, sys
log = open("files_app/.fhost.log").read() # the host console
runs = re.findall(f"Executing 'Functions.{sys.argv[1]}'", log)
print(f"{sys.argv[1]} runs so far:", len(runs))import os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64EncodePolicy
conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
pages = ["home", "product", "cart", "checkout"]
rows = [f"{i},{pages[i % 4]}" for i in range(1, 1_000_001)]
data = ("click_id,page\n" + "\n".join(rows) + "\n").encode()
BlobServiceClient.from_connection_string(conn).get_blob_client(
"raw", "clicks_1101.csv").upload_blob(data, overwrite=True)
print("raw/clicks_1101.csv:", len(data), "bytes,", len(rows), "clicks")
q = QueueClient.from_connection_string(
conn, "big-files", message_encode_policy=TextBase64EncodePolicy())
q.create_queue()
q.send_message("clicks_1101.csv") # just the name
print("queued: clicks_1101.csv")import subprocess, sys
H = "order_id,sku,qty,amount"
files = {"acme/orders_1101.csv": f"{H}|5001,tee-m,2,19.98|5002,mug,1,8.50",
"globex/orders_1101.csv": "order_id;sku;qty;amount|7001;cap;1;12",
"initech/orders_1101.csv": f"{H}|9001,hoodie,two,45.00"}
for name, text in files.items():
subprocess.run([sys.executable, "upload.py", "incoming", name, text])
for name in files: # Event Grid: one event each
subprocess.run([sys.executable, "grid.py", "incoming/" + name,
"validate_file"])import json, os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64DecodePolicy
conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
q = QueueClient.from_connection_string(
conn, "to-parquet", message_decode_policy=TextBase64DecodePolicy())
todo = sorted(json.loads(m.content)["blob"]
for m in q.peek_messages(max_messages=32))
print("to-parquet queue:", todo)
box = BlobServiceClient.from_connection_string(
conn).get_container_client("quarantine")
for b in sorted(box.list_blobs(include=["metadata"]), key=lambda b: b.name):
print("quarantine:", b.name, "| reason:", b.metadata["reason"])import io, os
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
box = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("curated")
for name in sorted(box.list_blob_names(name_starts_with="orders/")):
table = pq.read_table(io.BytesIO(box.download_blob(name).readall()))
print(name, "|", table.num_rows, "rows")
for row in table.to_pylist():
print(" ", row)Question 1: Files land in a container and must be processed as soon as they arrive. Which trigger do you use?
The blob trigger. It runs the function for each new or updated blob that matches a path. In production, we give it the Event Grid source: lower latency, and on Flex Consumption it is the only source supported.
The path is landing/{name}.csv, so only CSV files fire it, and {name} becomes a binding value we can reuse in other bindings. The function gets an InputStream with the blob name and length.
Let's start the host and upload an orders file with python upload.py landing orders_0901.csv.
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
rows = blob.read().decode().strip().splitlines()[1:]
logging.info("%s: %d rows, %d bytes", blob.name, len(rows),
blob.length)$ cd files_app && fhost start
host started, functions:
on_landing: blobTrigger
$ python upload.py landing orders_0901.csv "order_id,amount|1001,25.50|1002,14.00"
uploaded landing/orders_0901.csv
$ cd files_app && fhost wait "Executed 'Functions.on_landing'" 60 && fhost log
Executing 'Functions.on_landing' (Reason='New blob detected(LogsAndContainerScan):
landing/orders_0901.csv')
landing/orders_0901.csv: 2 rows, 38 bytes
Executed 'Functions.on_landing' (Succeeded)It runs once: landing/orders_0901.csv: 2 rows, 38 bytes. The reason says New blob detected(LogsAndContainerScan). That is the default polling source.
The blob trigger is not the only option. Here is when we pick each one:
| Option | When |
|---|---|
| Blob trigger | most file loads |
| Event Grid trigger | you need the raw event |
| Event Grid to a queue | several consumers, buffering |
For the raw event, or for several consumers, we use an Event Grid trigger, or Event Grid into a storage queue.
A function writes its output into the container it watches, and triggers itself again. Write outputs to another container, and filter the path, like {name}.csv.
Question 2: Someone asks the difference between the polling blob trigger and the Event Grid based blob trigger. Which is better?
The Event Grid source is better for new work. Polling scans the container and reads storage logs, best effort. The Event Grid source is pushed to the function.
Let's compare them:
| Polling (default) | Event Grid source | |
|---|---|---|
| how | scans the container, 10,000 blobs a batch | BlobCreated event pushed to a webhook |
| storage logs | best effort, no guarantee | not used |
| idle Consumption app | up to a 10-minute delay | lower latency |
| ADLS Gen2 (HNS) | not officially supported | supported |
| Flex Consumption | not supported | the only option |
| needs | nothing extra | GPv2 account, event subscription |
So the default source scans the container in batches of 10,000 blobs. An idle Consumption app can take up to 10 minutes, and polling isn't officially supported on ADLS Gen2. With the Event Grid source, a BlobCreated event calls the function's webhook.
Now let's try it locally. on_drop watches drops/ with source=func.BlobSource.EVENT_GRID. We upload a file, count the runs with python runs.py on_drop, then post a BlobCreated event to the local webhook with python grid.py.
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
logging.info("polling source: %s", blob.name)
@app.blob_trigger(arg_name="blob", path="drops/{name}.csv",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
def on_drop(blob: func.InputStream) -> None:
logging.info("event grid source: %s, %d bytes", blob.name,
blob.length)$ cd files_app && fhost start
host started, functions:
on_drop: blobTrigger
on_landing: blobTrigger
$ python upload.py drops orders_0902.csv "order_id,amount|1003,9.50"
uploaded drops/orders_0902.csv
$ sleep 5 && python runs.py on_drop
on_drop runs so far: 0
$ python grid.py drops/orders_0902.csv on_drop
Event Grid delivery: 202
$ cd files_app && fhost wait "Executed 'Functions.on_drop'" 60 && fhost log
Executing 'Functions.on_drop' (Reason='New blob detected(EventGrid): drops/orders_0902.csv')
event grid source: drops/orders_0902.csv, 26 bytes
Executed 'Functions.on_drop' (Succeeded)The upload alone gives on_drop runs so far: 0, because Azurite raises no events. After the event, delivery returns 202, and the reason says New blob detected(EventGrid): drops/orders_0902.csv. In Azure, we create a system topic on the storage account and a webhook subscription with the blobs extension key, filtered to BlobCreated under one container:
# 1. a system topic for the storage account's events
az eventgrid system-topic create -g rg-data -n st-files-events \
--topic-type microsoft.storage.storageaccounts \
--source "$STORAGE_ID" --location eastus2
# 2. the blobs_extension system key of the function app
KEY=$(az functionapp keys list -g rg-data -n fn-files \
--query systemKeys.blobs_extension -o tsv)
HOOK="https://fn-files.azurewebsites.net/runtime/webhooks/blobs"
HOOK="$HOOK?functionName=Host.Functions.on_drop&code=$KEY"
# 3. BlobCreated events under drops/ go to the webhook
az eventgrid system-topic event-subscription create \
-g rg-data --system-topic-name st-files-events -n on-drop \
--endpoint-type webhook --endpoint "$HOOK" \
--included-event-types Microsoft.Storage.BlobCreated \
--subject-begins-with /blobServices/default/containers/drops/It needs a general-purpose v2 account, push can't reach a private endpoint, and delivery is at least once, so dedupe is your job.
Question 3: A teammate's blob function processes the same file twice. Why does that happen, and how do you stop it?
The polling trigger keeps a blob receipt per function, blob and ETag, in the azure-webjobs-hosts container. Anything that changes the ETag or loses the receipt runs the file again, so the load must be idempotent.
Let's see it. A partner uploads daily_1031.csv, then uploads the exact same content again. We check the receipts with python receipts.py on_order_file.
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
logging.info("load %s, %d bytes", blob.name, blob.length)
# ... append the rows to the orders table ...$ cd files_app && fhost start
host started, functions:
on_order_file: blobTrigger
$ python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"
uploaded orders/daily_1031.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60
$ python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"
uploaded orders/daily_1031.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 2 && fhost log
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
orders/daily_1031.csv')
load orders/daily_1031.csv, 27 bytes
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
orders/daily_1031.csv')
load orders/daily_1031.csv, 27 bytes
Executed 'Functions.on_order_file' (Succeeded)
$ python receipts.py on_order_file
blobreceipts/tbit-fn-demo/Host.Functions.on_order_file/<etag>/orders/daily_1031.csv
blobreceipts/tbit-fn-demo/Host.Functions.on_order_file/<etag>/orders/daily_1031.csv
receipts: 2 | different ETags: 2Same bytes, but a new ETag, so the function loads it twice, 27 bytes each time, and we get receipts: 2 | different ETags: 2.
There are other causes too:
- Renamed function: no receipts yet, every file again
- New AzureWebJobsStorage: the receipts stay behind
- Retry after a failure: part of the work ran already
- Two apps, one container: each has its own receipts
- Event Grid: at-least-once delivery
The fix is an idempotent load. This version hashes the content and keeps a processedfiles table. We upload daily_1101.csv, the same file again, then a corrected file with a new amount.
import hashlib, logging, os
import azure.functions as func
from azure.core.exceptions import ResourceNotFoundError
from azure.data.tables import TableServiceClient
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
done = TableServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"]
).create_table_if_not_exists("processedfiles")
@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
key = blob.name.split("/", 1)[1].replace("/", "|") # no / in keys
digest = hashlib.sha256(blob.read()).hexdigest()
try:
seen = done.get_entity("orders", key)["sha256"]
except ResourceNotFoundError:
seen = None # never loaded
if seen == digest:
logging.info("skip %s: same content", blob.name)
return
# ... load, overwriting this file's own rows ...
done.upsert_entity({"PartitionKey": "orders", "RowKey": key,
"sha256": digest})
logging.info("loaded %s", blob.name)$ cd files_app && fhost start
host started, functions:
on_order_file: blobTrigger
$ python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60
$ python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 2
$ python upload.py orders daily_1101.csv "order_id,amount|2001,32.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 3 && fhost log
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
orders/daily_1101.csv')
loaded orders/daily_1101.csv
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
orders/daily_1101.csv')
skip orders/daily_1101.csv: same content
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
orders/daily_1101.csv')
loaded orders/daily_1101.csv
Executed 'Functions.on_order_file' (Succeeded)The first copy logs loaded, the resend logs skip orders/daily_1101.csv: same content, and the corrected file loads.
Make each file overwrite its own output, with a fixed name or a MERGE on the business key. Then even a real duplicate run replaces rows instead of appending them.
Question 4: A 5 GB file must be processed by a function. What do you watch out for?
Watch memory, the timeout, temp storage and parallelism. Then stream it in chunks, or hand the heavy work to Databricks.
Memory comes first. In Python, the default InputStream binding hands you the whole content in memory. A Flex Consumption instance has 512 MB, 2,048 MB by default, or 4,096 MB. Five gigabytes doesn't fit. Let's look at the plans:
| Plan | Memory per instance | Timeout default / max | Temp storage |
|---|---|---|---|
| Flex Consumption | 512 MB, 2,048 MB (default), 4,096 MB | 30 min / unbounded | 0.8 GB |
| Premium | 3.5 to 14 GB (EP1 to EP3) | 30 min / unbounded | 11 to 61 GB |
| Consumption (legacy) | 1.5 GB | 5 min / 10 min | 0.5 GB (Linux 1.5 GB) |
Then time and disk. Flex Consumption and Premium default to 30 minutes and can run unbounded, legacy Consumption stops at 10, and an HTTP trigger must respond within 230 seconds on any plan. Flex has about 0.8 GB of temp storage, so a full download to disk fails too.
And parallelism. By default, the blob extension runs up to 8 times the core count at once, across all the app's blob functions, and each needs memory. So we lower maxDegreeOfParallelism:
{
"version": "2.0",
"functionTimeout": "01:00:00",
"extensions": {
"blobs": { "maxDegreeOfParallelism": 2 }
}
}Even with the largest Flex instance size, 4,096 MB, we still stream:
# Flex Consumption with the largest instance size: 4,096 MB
az functionapp create -g rg-data -n fn-bigfiles \
--storage-account stfnbigfiles \
--flexconsumption-location eastus2 \
--runtime python --runtime-version 3.13 --instance-memory 4096What we do instead:
- Stream: storage SDK, chunks in and blocks out (question 6)
- Split: process the pieces as Durable activities
- Hand off: heavy transforms and joins go to Databricks or Data Factory
A small test file works, and the real one runs out of memory. A 5 MB sample hides it, so test with a real size.
Question 5: Someone asks what happens when a blob function keeps failing on one file. How do poison blobs work?
The blob trigger tries a failing file five times by default. Then it writes a message to the webjobs-blobtrigger-poison queue with the function, container, blob name and ETag. The file itself stays where it is: you handle it.
Here on_invoice checks the header. We upload a good invoice, then one with semicolons.
import logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="invoices/{name}.csv",
connection="AzureWebJobsStorage")
def on_invoice(blob: func.InputStream) -> None:
header = blob.read().decode().splitlines()[0]
if header != "invoice_id,amount":
raise ValueError(f"bad header {header!r} in {blob.name}")
logging.info("loaded %s", blob.name)
@app.queue_trigger(arg_name="msg", connection="AzureWebJobsStorage",
queue_name="webjobs-blobtrigger-poison")
def park_poison(msg: func.QueueMessage) -> None:
info = msg.get_json() # FunctionId, ContainerName, BlobName
box, name = info["ContainerName"], info["BlobName"]
src = blobs.get_blob_client(box, name)
dst = blobs.get_blob_client("quarantine", f"{box}/{name}")
dst.start_copy_from_url(src.url, metadata={"reason": "poison"})
if dst.get_blob_properties().copy.status == "success":
src.delete_blob() # same account
logging.info("parked %s/%s in quarantine", box, name)$ cd files_app && fhost start
host started, functions:
on_invoice: blobTrigger
park_poison: queueTrigger
$ python upload.py invoices inv_0901.csv "invoice_id,amount|1,50.00"
uploaded invoices/inv_0901.csv
$ cd files_app && fhost wait "Executed 'Functions.on_invoice'" 60
$ python upload.py invoices inv_0902.csv "id;amount|2;75.00"
uploaded invoices/inv_0902.csv
$ cd files_app && fhost wait "Executed 'Functions.park_poison'" 90 && fhost log
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0901.csv')
loaded invoices/inv_0901.csv
Executed 'Functions.on_invoice' (Succeeded)
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Message has reached MaxDequeueCount of 5. Moving message to queue 'webjobs-blobtrigger-poison'.
Executing 'Functions.park_poison' (Reason='New queue message detected on 'webjobs-blobtrigger-poison'.')
parked invoices/inv_0902.csv in quarantine
Executed 'Functions.park_poison' (Succeeded)
$ python where.py invoices quarantine
invoices: ['inv_0901.csv']
quarantine: ['invoices/inv_0902.csv']inv_0901.csv loads. inv_0902.csv fails five times with ValueError: bad header 'id;amount', then the host logs Moving message to queue 'webjobs-blobtrigger-poison'.
Nothing reads that queue unless we add a function. Every blob function in the app shares it, so park_poison is generic: it copies the blob to quarantine and deletes the original. At the end, invoices holds only inv_0901.csv, and quarantine has invoices/inv_0902.csv.
In a real project, we also alert on the poison queue length and tell the partner why. A fixed upload is a new version, so it runs.
With identity-based connections, the trigger's identity needs Storage Queue Data Contributor, or it can't write poison messages.
Question 6: A teammate asks how to read and write blobs inside a function without loading everything into memory. What do you do?
Don't bind the file as bytes or as the default InputStream: in Python, both hold the whole blob. Bind an SDK type like BlobClient, which is generally available for the Python v2 model, or trigger on a small message and stream with the storage SDK.
Here is the SDK type binding:
# requirements.txt: azurefunctions-extensions-bindings-blob
import azure.functions as func
import azurefunctions.extensions.bindings.blob as blob
app = func.FunctionApp()
@app.blob_trigger(arg_name="client", path="raw/{name}",
connection="AzureWebJobsStorage")
def stream_file(client: blob.BlobClient): # not the bytes
for chunk in client.download_blob().chunks():
... # one chunk at a timeOur demo takes the second way. filter_clicks gets just the file name from a queue, and the blob client downloads 4 MiB chunks. A helper, whole_lines, turns each chunk into whole lines and carries the partial last line over. For the output, it stages one block per chunk and commits the block list at the end, so readers never see a half-written file.
We run python make_big.py, which uploads a file with a million clicks and queues its name.
import logging, os
import azure.functions as func
from azure.storage.blob import BlobBlock, BlobServiceClient
from chunks import whole_lines
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
MiB = 1024 * 1024
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"],
max_single_get_size=4 * MiB, max_chunk_get_size=4 * MiB)
@app.queue_trigger(arg_name="msg", queue_name="big-files",
connection="AzureWebJobsStorage")
def filter_clicks(msg: func.QueueMessage) -> None:
name = msg.get_body().decode() # clicks_1101.csv
src = blobs.get_blob_client("raw", name).download_blob()
dst = blobs.get_blob_client("curated", "checkout_" + name)
blocks, kept = [], 0
for lines in whole_lines(src): # chunk by chunk
keep = [l + b"\n" for l in lines if l.endswith(b",checkout")]
kept += len(keep)
blocks.append(BlobBlock(f"{len(blocks):05d}"))
dst.stage_block(blocks[-1].id, b"".join(keep)) # one block
dst.commit_block_list(blocks) # the file appears only now
logging.info("%s: %d bytes in %d chunks, kept %d, %d blocks",
name, src.size, len(blocks), kept, len(blocks))def whole_lines(download):
"""Yield the complete lines of each downloaded chunk."""
rest = b""
for chunk in download.chunks(): # one chunk in memory
*lines, rest = (rest + chunk).split(b"\n")
yield lines # rest: a partial line
if rest:
yield [rest] # no newline at the end$ cd files_app && fhost start
host started, functions:
filter_clicks: queueTrigger
$ python make_big.py
raw/clicks_1101.csv: 13638910 bytes, 1000000 clicks
queued: clicks_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.filter_clicks'" 120 && fhost log
Executing 'Functions.filter_clicks' (Reason='New queue message detected on 'big-files'.')
clicks_1101.csv: 13638910 bytes in 4 chunks, kept 250000, 4 blocks
Executed 'Functions.filter_clicks' (Succeeded)The file is 13638910 bytes and 1000000 clicks. It came in 4 chunks, and the function kept 250000 checkout clicks in 4 blocks. It only ever held about one chunk in memory.
Keep clients at module level so invocations reuse them, and set the azure logger to WARNING, or the SDK logs every request.
Scenario 7: Partners drop CSV files all day and each must be validated, converted to Parquet and loaded. How do you build it?
Event-driven, one file at a time. Partners write to their own folders, an Event Grid blob trigger validates each file, and a queue hands good files to a convert function that writes Parquet for the load.
Here is the flow:
- incoming/: the landing container, a folder per partner
- validate_file: Event Grid blob trigger
- to-parquet: storage queue for good files
- quarantine/: bad files, the reason in metadata
- convert_to_parquet: queue trigger, writes
curated/orders/
Validation lives in a plain module, checks.py, with the partner contract: the header, at least one row, and numeric quantity and amount. We run python partners_drop.py, which uploads three files and posts one event each, then python report.py.
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="incoming/{name}",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
name = blob.name.split("/", 1)[1] # acme/orders_1101.csv
why = problem(blob.read().decode())
if why is None:
convert.set(json.dumps({"blob": name})) # to the converter
logging.info("valid %s -> to-parquet queue", name)
return
bad = blobs.get_blob_client("quarantine", name)
bad.start_copy_from_url(blob.uri, metadata={"reason": why})
logging.info("rejected %s: %s -> quarantine", name, why)import pyarrow as pa
HEADER = "order_id,sku,qty,amount"
TYPES = {"order_id": pa.int64(), "sku": pa.string(), # for Parquet
"qty": pa.int64(), "amount": pa.float64()}
def problem(text: str):
"""None when the partner file is valid, else the reason."""
head, *rows = text.strip().splitlines() or [""]
if head != HEADER:
return "bad header"
if not rows:
return "no rows"
try:
[(int(r.split(",")[2]), float(r.split(",")[3])) for r in rows]
except (IndexError, ValueError):
return "bad qty or amount"
return None$ cd files_app && fhost start
host started, functions:
validate_file: blobTrigger
$ python partners_drop.py
uploaded incoming/acme/orders_1101.csv
uploaded incoming/globex/orders_1101.csv
uploaded incoming/initech/orders_1101.csv
Event Grid delivery: 202
Event Grid delivery: 202
Event Grid delivery: 202
$ cd files_app && fhost wait "Executed 'Functions.validate_file'" 60 3
$ cd files_app && fhost log | grep -E '^(valid|rejected)' | sort
rejected globex/orders_1101.csv: bad header -> quarantine
rejected initech/orders_1101.csv: bad qty or amount -> quarantine
valid acme/orders_1101.csv -> to-parquet queue
$ python report.py
to-parquet queue: ['acme/orders_1101.csv']
quarantine: globex/orders_1101.csv | reason: bad header
quarantine: initech/orders_1101.csv | reason: bad qty or amount
quarantine: invoices/inv_0902.csv | reason: poison
$ cd files_app && fhost stop
host stoppedAcme's file is valid and goes to the to-parquet queue. Globex used semicolons, bad header, and Initech has bad qty or amount, so both go to quarantine with the reason. The invoices/inv_0902.csv line is left over from question 5.
Now we add the convert blueprint and start the host again. It reads the CSV with pyarrow using the contract's types, not guessed ones, and writes Parquet with a fixed name, so a rerun overwrites.
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
from convert import bp
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
app.register_functions(bp)
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="incoming/{name}",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
name = blob.name.split("/", 1)[1] # acme/orders_1101.csv
why = problem(blob.read().decode())
if why is None:
convert.set(json.dumps({"blob": name})) # to the converter
logging.info("valid %s -> to-parquet queue", name)
return
bad = blobs.get_blob_client("quarantine", name)
bad.start_copy_from_url(blob.uri, metadata={"reason": why})
logging.info("rejected %s: %s -> quarantine", name, why)import io, logging, os
import azure.functions as func
import pyarrow.csv as pv
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
from checks import TYPES # the same partner contract
bp = func.Blueprint() # registered in function_app.py
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
OPTS = pv.ConvertOptions(column_types=TYPES)
@bp.queue_trigger(arg_name="msg", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def convert_to_parquet(msg: func.QueueMessage) -> None:
name = msg.get_json()["blob"] # acme/orders_1101.csv
src = blobs.get_blob_client("incoming", name).download_blob()
table = pv.read_csv(io.BytesIO(src.readall()),
convert_options=OPTS) # no type guessing
out = io.BytesIO()
pq.write_table(table, out)
target = "orders/" + name.replace(".csv", ".parquet")
blobs.get_blob_client("curated", target).upload_blob(
out.getvalue(), overwrite=True) # rerun: same file
logging.info("%s -> curated/%s: %d rows", name, target,
table.num_rows)
logging.info("schema: %s", ", ".join(
f"{f.name} {f.type}" for f in table.schema))$ cd files_app && fhost start
host started, functions:
convert_to_parquet: queueTrigger
validate_file: blobTrigger
$ cd files_app && fhost wait "Executed 'Functions.convert_to_parquet'" 60 && fhost log
Executing 'Functions.convert_to_parquet' (Reason='New queue message detected on 'to-parquet'.')
acme/orders_1101.csv -> curated/orders/acme/orders_1101.parquet: 2 rows
schema: order_id int64, sku string, qty int64, amount double
Executed 'Functions.convert_to_parquet' (Succeeded)
$ python read_parquet.py
orders/acme/orders_1101.parquet | 2 rows
{'order_id': 5001, 'sku': 'tee-m', 'qty': 2, 'amount': 19.98}
{'order_id': 5002, 'sku': 'mug', 'qty': 1, 'amount': 8.5}
$ cd files_app && fhost stop
host stoppedWe get curated/orders/acme/orders_1101.parquet: 2 rows, with order_id int64, sku string, qty int64, amount double, typed as agreed.
A processed-files table keyed by name and content hash, alerts on the poison queues, and a note to the partner for each quarantined file. Very large files go to Databricks instead.
Key points to remember
- Blob trigger with the Event Grid source: lower latency, ADLS Gen2, the only option on Flex Consumption
- Polling scans the container and logs: best effort, up to 10 minutes on an idle Consumption app
- Receipts are per function, blob and ETag: an overwrite runs again, so loads must be idempotent
- Five tries, then webjobs-blobtrigger-poison: add a handler that quarantines and alerts
- Big files: the InputStream holds the whole blob; stream with chunks in and blocks out
The full script
Here is a Python script that writes every file and runs every command in order with Core Tools and Azurite; run pip install azure-functions azure-functions-durable azure-storage-blob azure-storage-queue azure-data-tables first.
Show the full script that writes the files and runs every step
# ========================================================================
# Azure Functions Interview Q&A #3 - Blob and file processing
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run with Azure Functions Core Tools 4.16.0 (Functions host 4.x, Python 3.13 worker,
# programming model v2, extension bundle 4.x) and Azurite 3.37.0; the outputs in the video and the blog post are the real outputs.
# To run it yourself (Linux or macOS shell, or WSL on Windows):
# 1. Install Azure Functions Core Tools 4 and Azurite (npm install -g azurite), then start Azurite:
# azurite --skipApiVersionCheck --location ./azurite-data (blob 10000, queue 10001, table 10002)
# 2. pip install azure-functions azure-functions-durable azure-storage-blob azure-storage-queue azure-data-tables
# 3. Set AZURE_STORAGE_CONNECTION_STRING to "UseDevelopmentStorage=true", then run python Functions_Interview_QA_03_Code.py.
# "fhost" in the commands is a small helper written by Step 0 (fhost.py): it runs `func start` in the background,
# waits until the functions are loaded, and prints the host log lines without timestamps and invocation ids.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================
import os, subprocess, time
def write(path, text):
if text is None:
if os.path.exists(path): os.remove(path)
return
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
f.write(text.strip("\n") + "\n")
def run(cmd, expect_error=False, allow_error=False, env=None):
print("\n$ " + cmd, flush=True)
rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
if rc != 0 and not (expect_error or allow_error):
raise SystemExit("stopped: this command was expected to succeed")
# Step 0: the fhost helper and the starting files
write("fhost.py", '"""fhost: run the Azure Functions host (func start) in the background of the current folder, for scripted demos.\n fhost start [port] [-- extra func args] start `func start`, wait until the functions are indexed, print them\n (and any indexing errors)\n fhost wait PATTERN [secs] [count] wait until `count` (default 1) NEW host log lines (not yet shown by\n `fhost log`) match the regex PATTERN (default timeout 60 s)\n fhost log [--all] [--details] [--since-start]\n print the host log lines written since the last `fhost log` (function logs,\n Executing / Executed lines, exceptions); --details also keeps the\n "Trigger Details" lines, --all keeps every line, --since-start starts at the\n first line after the function list. Timestamps, invocation ids, durations and\n other per-run values are removed.\n fhost stop stop the host\nThe log is kept in .fhost.log in the current folder (one host per folder)."""\nimport os, re, signal, subprocess, sys, time\nLOG, PID, OFF, START = \'.fhost.log\', \'.fhost.pid\', \'.fhost.off\', \'.fhost.start\'\nTS = re.compile(r\'\\[\\d{4}-\\d\\d-\\d\\dT[\\d:.]+Z\\] \')\n# host messages to hide (anchored at the start of the cleaned line, so user log lines that merely contain these words stay)\nNOISE = re.compile(r\'^(Worker process started|Host lock lease|Found Python version|Azure Functions Core Tools|Core Tools Version|\'\n r\'Function Runtime Version|For detailed output|Using cached version|When you have network connectivity|\'\n r\'Warning: Unable to download extension|Host started|Job host started|Host initialized|Loading functions metadata|\'\n r\'Reading functions metadata|Starting Host|Initializing Warmup|Generating \\d+ job function|Functions:$|\'\n r\'Debugger listening|Host Status|Starting worker process|Worker failed to load|Retrieving function|\'\n r\'\\d+ functions? (found|loaded)|Successfully (found|loaded|registered)|Application Insights|WARNING: )\')\nDETAILS = re.compile(r\'^Trigger Details:\')\n\ndef clean(l):\n l = TS.sub(\'\', l.rstrip(\'\\n\'))\n l = re.sub(r\', Id=[0-9a-f-]{36}\', \'\', l)\n l = re.sub(r\'Id=[0-9a-f-]{36}\', \'\', l)\n l = re.sub(r\', Duration=\\d+ms\', \'\', l)\n l = re.sub(r\'\\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\\b\', \'<id>\', l)\n l = re.sub(r\'\\b[0-9a-f]{32}\\b\', \'<instance-id>\', l)\n l = re.sub(r\'\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(Z|[+-]\\d\\d:\\d\\d)?\', \'<time>\', l)\n l = re.sub(r\'"?0x[0-9A-F]{15,16}"?\', \'<etag>\', l)\n l = re.sub(r\'([?&]code=)[^&\\s"]+\', r\'\\1<key>\', l)\n l = re.sub(r\'Threadpool id: \\d+\', \'Threadpool id: <n>\', l)\n l = re.sub(r\'(processUptime\\W+)\\d+\', r\'\\1<n>\', l)\n return l.replace(\' ()\', \'\').rstrip()\n\ndef read_from(off):\n """complete lines from byte offset `off`; returns (lines, new_offset). A glued line ([ts] a [ts] b) is split."""\n if not os.path.exists(LOG): return [], off\n with open(LOG, \'rb\') as f:\n f.seek(off); data = f.read()\n end = data.rfind(b\'\\n\')\n if end < 0: return [], off\n chunk = data[:end + 1].decode(\'utf-8\', \'replace\')\n out = []\n for l in chunk.split(\'\\n\')[:-1]:\n parts = [p for p in re.split(r\'(?=\\[\\d{4}-\\d\\d-\\d\\dT[\\d:.]+Z\\] )\', l) if p.strip()] or [\'\']\n out += parts\n return out, off + end + 1\n\ndef get(f, d=0):\n return int(open(f).read()) if os.path.exists(f) else d\n\ndef filtered(lines, all_=False, details=False):\n out, in_stack, last_exc = [], False, None\n for l in lines:\n raw = TS.sub(\'\', l)\n if in_stack:\n if raw.startswith((\' \', \'\\t\')): continue\n in_stack = False\n c = clean(l)\n if not c: continue\n if c.startswith(\'Stack:\') and not all_: in_stack = True; continue\n if not all_ and (c in (\'Type:\', \'.\') or c == last_exc): continue\n if c.startswith(\'Exception: \'): last_exc = c[len(\'Exception: \'):]\n if all_ or (details and DETAILS.search(c)): out.append(c); continue\n if NOISE.search(c) or DETAILS.search(c): continue\n out.append(c)\n return out\n\ndef start(*args):\n port, extra = \'7071\', []\n if args and args[0] != \'--\': port = args[0]; args = args[1:]\n if args and args[0] == \'--\': extra = list(args[1:])\n if os.path.exists(PID): stop(quiet=True)\n for f in (LOG, OFF, START):\n if os.path.exists(f): os.remove(f)\n out = open(LOG, \'w\')\n p = subprocess.Popen([\'func\', \'start\', \'--port\', port] + extra, stdout=out, stderr=subprocess.STDOUT,\n stdin=subprocess.DEVNULL, start_new_session=True)\n open(PID, \'w\').write(str(p.pid))\n t0 = time.time()\n while time.time() - t0 < 120:\n txt = open(LOG, encoding=\'utf-8\', errors=\'replace\').read()\n if p.poll() is not None:\n print(\'\\n\'.join(filtered(txt.split(\'\\n\'), all_=True)[-30:])); sys.exit(\'host exited\')\n m = re.search(r\'\\nFunctions:\\n(.*?)\\n(For detailed output[^\\n]*\\n)\', txt, re.S)\n if m and \'Worker process started and initialized\' in txt:\n errs = [clean(l) for l in txt[:m.start()].split(\'\\n\')\n if re.search(r\'in error|failed indexing|Error indexing|will be disabled\', l)]\n names = [clean(l).strip() for l in m.group(1).split(\'\\n\') if l.strip()]\n boundary = len(txt[:m.end()].encode(\'utf-8\'))\n open(START, \'w\').write(str(boundary)); open(OFF, \'w\').write(str(boundary))\n time.sleep(1.5) # let triggers start listening\n for e in errs: print(e)\n print(\'host started, functions:\')\n for n in names: print(\' \' + re.sub(r\'http://localhost:\\d+\', \'http://localhost:\' + port, n))\n return\n if not m and re.search(r\'\\n\\[[^\\]]*\\] (Job host started|Host started)\', txt) and \'Worker process started and initialized\' in txt:\n time.sleep(1.5)\n boundary = len(txt.encode(\'utf-8\')); open(START, \'w\').write(str(boundary)); open(OFF, \'w\').write(str(boundary))\n for l in txt.split(\'\\n\'):\n if re.search(r\'in error|failed indexing|is disabled\', l): print(clean(l))\n print(\'host started, no enabled functions listed\'); return\n if re.search(r\'No job functions found|A host error has occurred\', txt):\n time.sleep(1); print(\'\\n\'.join(filtered(txt.split(\'\\n\'), all_=True)[-30:])); stop(quiet=True); sys.exit(\'host error\')\n time.sleep(0.3)\n print(\'\\n\'.join(filtered(open(LOG).read().split(\'\\n\'), all_=True)[-30:])); stop(quiet=True)\n sys.exit(\'host did not start in 120 s\')\n\ndef wait(pat, secs=\'60\', count=\'1\'):\n rx, t0, need = re.compile(pat), time.time(), int(count)\n while time.time() - t0 < float(secs):\n lines, _ = read_from(get(OFF))\n if sum(1 for l in lines if rx.search(clean(l))) >= need: return\n time.sleep(0.2)\n lines, _ = read_from(get(OFF))\n print(\'\\n\'.join(filtered(lines)[-20:])); sys.exit(\'timed out waiting for %r\' % pat)\n\ndef log(all_=False, details=False, since_start=False):\n off = get(START) if since_start else get(OFF)\n lines, new = read_from(off)\n if not since_start or new > get(OFF): open(OFF, \'w\').write(str(new))\n for c in filtered(lines, all_, details): print(c)\n\ndef stop(quiet=False):\n if not os.path.exists(PID): return\n pid = int(open(PID).read())\n for sig in (signal.SIGTERM, signal.SIGKILL):\n try: os.killpg(pid, sig)\n except Exception: break\n for _ in range(50):\n try: os.killpg(pid, 0); time.sleep(0.1)\n except Exception: break\n os.remove(PID)\n if not quiet: print(\'host stopped\')\n\nif __name__ == \'__main__\':\n a = sys.argv[1:] or [\'help\']\n if a[0] == \'start\': start(*a[1:])\n elif a[0] == \'wait\': wait(*a[1:])\n elif a[0] == \'log\': log(\'--all\' in a, \'--details\' in a, \'--since-start\' in a)\n elif a[0] == \'stop\': stop()\n else: print(__doc__)\n')
os.environ["FHOST"] = os.path.abspath("fhost.py") # commands below run it as: python "$FHOST" ...
write('files_app/host.json', """\
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[4.*, 5.0.0)"
},
"logging": {
"logLevel": {
"default": "Warning",
"Function": "Information",
"Azure.Core": "Error",
"Host.Function.Console": "Information"
}
}
}
""")
write('files_app/local.settings.json', """\
{
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "python",
"AzureWebJobsStorage": "UseDevelopmentStorage=true"
}
}
""")
write('files_app/requirements.txt', """\
azure-functions
azure-storage-blob
azure-data-tables
pyarrow
""")
write('containers.py', """\
import os
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for name in ["landing", "drops", "orders", "invoices", "quarantine",
"raw", "curated", "incoming"]:
if not svc.get_container_client(name).exists():
svc.create_container(name)
print("containers ready")
""")
write('upload.py', 'import os, sys\nfrom azure.storage.blob import BlobServiceClient\n\nsvc = BlobServiceClient.from_connection_string(\n os.environ["AZURE_STORAGE_CONNECTION_STRING"])\ncontainer, name, text = sys.argv[1:4] # "|" = new line\nsvc.get_blob_client(container, name).upload_blob(\n text.replace("|", "\\n") + "\\n", overwrite=True)\nprint("uploaded", f"{container}/{name}")\n')
write('grid.py', """\
import sys, requests
# Azurite raises no storage events, so this script plays Event Grid:
# it posts a BlobCreated event to the host's blob webhook.
container, blob = sys.argv[1].split("/", 1)
event = [{"id": "evt-" + blob, "eventType": "Microsoft.Storage.BlobCreated",
"subject": f"/blobServices/default/containers/{container}"
f"/blobs/{blob}",
"eventTime": "2026-11-01T09:00:00Z", "dataVersion": "",
"data": {"api": "PutBlob", "blobType": "BlockBlob",
"url": "http://127.0.0.1:10000/devstoreaccount1/"
f"{container}/{blob}"}}]
r = requests.post("http://localhost:7071/runtime/webhooks/blobs",
params={"functionName": "Host.Functions." + sys.argv[2]},
headers={"aeg-event-type": "Notification"}, json=event)
print("Event Grid delivery:", r.status_code)
""")
write('receipts.py', """\
import os, sys
from azure.storage.blob import BlobServiceClient
hosts = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("azure-webjobs-hosts")
names = [b.name for b in hosts.list_blobs(name_starts_with="blobreceipts/")
if f"/Host.Functions.{sys.argv[1]}/" in b.name]
for n in names:
print(n) # .../<function>/<etag>/<blob>
print("receipts:", len(names), "| different ETags:",
len({n.split("/")[3] for n in names}))
""")
write('where.py', """\
import os, sys
from azure.storage.blob import BlobServiceClient
svc = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for c in sys.argv[1:]:
names = sorted(svc.get_container_client(c).list_blob_names())
print(f"{c}:", names)
""")
write('runs.py', """\
import re, sys
log = open("files_app/.fhost.log").read() # the host console
runs = re.findall(f"Executing 'Functions.{sys.argv[1]}'", log)
print(f"{sys.argv[1]} runs so far:", len(runs))
""")
write('make_big.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\nfrom azure.storage.queue import QueueClient, TextBase64EncodePolicy\n\nconn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]\npages = ["home", "product", "cart", "checkout"]\nrows = [f"{i},{pages[i % 4]}" for i in range(1, 1_000_001)]\ndata = ("click_id,page\\n" + "\\n".join(rows) + "\\n").encode()\nBlobServiceClient.from_connection_string(conn).get_blob_client(\n "raw", "clicks_1101.csv").upload_blob(data, overwrite=True)\nprint("raw/clicks_1101.csv:", len(data), "bytes,", len(rows), "clicks")\nq = QueueClient.from_connection_string(\n conn, "big-files", message_encode_policy=TextBase64EncodePolicy())\nq.create_queue()\nq.send_message("clicks_1101.csv") # just the name\nprint("queued: clicks_1101.csv")\n')
write('partners_drop.py', """\
import subprocess, sys
H = "order_id,sku,qty,amount"
files = {"acme/orders_1101.csv": f"{H}|5001,tee-m,2,19.98|5002,mug,1,8.50",
"globex/orders_1101.csv": "order_id;sku;qty;amount|7001;cap;1;12",
"initech/orders_1101.csv": f"{H}|9001,hoodie,two,45.00"}
for name, text in files.items():
subprocess.run([sys.executable, "upload.py", "incoming", name, text])
for name in files: # Event Grid: one event each
subprocess.run([sys.executable, "grid.py", "incoming/" + name,
"validate_file"])
""")
write('report.py', """\
import json, os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64DecodePolicy
conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
q = QueueClient.from_connection_string(
conn, "to-parquet", message_decode_policy=TextBase64DecodePolicy())
todo = sorted(json.loads(m.content)["blob"]
for m in q.peek_messages(max_messages=32))
print("to-parquet queue:", todo)
box = BlobServiceClient.from_connection_string(
conn).get_container_client("quarantine")
for b in sorted(box.list_blobs(include=["metadata"]), key=lambda b: b.name):
print("quarantine:", b.name, "| reason:", b.metadata["reason"])
""")
write('read_parquet.py', """\
import io, os
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
box = BlobServiceClient.from_connection_string(
os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("curated")
for name in sorted(box.list_blob_names(name_starts_with="orders/")):
table = pq.read_table(io.BytesIO(box.download_blob(name).readall()))
print(name, "|", table.num_rows, "rows")
for row in table.to_pylist():
print(" ", row)
""")
run('python containers.py')
# Step 1: A blob trigger on landing/{name}.csv: upload an orders file and the function runs
write('files_app/function_app.py', """\
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
rows = blob.read().decode().strip().splitlines()[1:]
logging.info("%s: %d rows, %d bytes", blob.name, len(rows),
blob.length)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py landing orders_0901.csv "order_id,amount|1001,25.50|1002,14.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_landing\'" 60 && python "$FHOST" log')
# Step 2: The Event Grid source: an upload alone runs nothing; a BlobCreated event posted to the webhook runs it
write('files_app/function_app.py', """\
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
logging.info("polling source: %s", blob.name)
@app.blob_trigger(arg_name="blob", path="drops/{name}.csv",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
def on_drop(blob: func.InputStream) -> None:
logging.info("event grid source: %s, %d bytes", blob.name,
blob.length)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py drops orders_0902.csv "order_id,amount|1003,9.50"')
run('sleep 5 && python runs.py on_drop')
run('python grid.py drops/orders_0902.csv on_drop')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_drop\'" 60 && python "$FHOST" log')
# Step 3: The same file uploaded twice with the same content: two runs, two receipts
write('files_app/function_app.py', """\
import logging
import azure.functions as func
app = func.FunctionApp()
@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
logging.info("load %s, %d bytes", blob.name, blob.length)
# ... append the rows to the orders table ...
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60')
run('python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 2 && python "$FHOST" log')
run('python receipts.py on_order_file')
# Step 4: Idempotent: a content hash in a processed-files table; resend skipped, corrected file loaded
write('files_app/function_app.py', """\
import hashlib, logging, os
import azure.functions as func
from azure.core.exceptions import ResourceNotFoundError
from azure.data.tables import TableServiceClient
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
done = TableServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"]
).create_table_if_not_exists("processedfiles")
@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
key = blob.name.split("/", 1)[1].replace("/", "|") # no / in keys
digest = hashlib.sha256(blob.read()).hexdigest()
try:
seen = done.get_entity("orders", key)["sha256"]
except ResourceNotFoundError:
seen = None # never loaded
if seen == digest:
logging.info("skip %s: same content", blob.name)
return
# ... load, overwriting this file's own rows ...
done.upsert_entity({"PartitionKey": "orders", "RowKey": key,
"sha256": digest})
logging.info("loaded %s", blob.name)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 2')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,32.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 3 && python "$FHOST" log')
# Step 5: A bad invoice fails five times, goes to webjobs-blobtrigger-poison, and a handler parks it in quarantine
write('files_app/function_app.py', """\
import logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="invoices/{name}.csv",
connection="AzureWebJobsStorage")
def on_invoice(blob: func.InputStream) -> None:
header = blob.read().decode().splitlines()[0]
if header != "invoice_id,amount":
raise ValueError(f"bad header {header!r} in {blob.name}")
logging.info("loaded %s", blob.name)
@app.queue_trigger(arg_name="msg", connection="AzureWebJobsStorage",
queue_name="webjobs-blobtrigger-poison")
def park_poison(msg: func.QueueMessage) -> None:
info = msg.get_json() # FunctionId, ContainerName, BlobName
box, name = info["ContainerName"], info["BlobName"]
src = blobs.get_blob_client(box, name)
dst = blobs.get_blob_client("quarantine", f"{box}/{name}")
dst.start_copy_from_url(src.url, metadata={"reason": "poison"})
if dst.get_blob_properties().copy.status == "success":
src.delete_blob() # same account
logging.info("parked %s/%s in quarantine", box, name)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py invoices inv_0901.csv "invoice_id,amount|1,50.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_invoice\'" 60')
run('python upload.py invoices inv_0902.csv "id;amount|2;75.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.park_poison\'" 90 && python "$FHOST" log')
run('python where.py invoices quarantine')
# Step 6: Stream a 13 MB file: 4 MiB chunks in, one staged block per chunk out
write('files_app/function_app.py', 'import logging, os\nimport azure.functions as func\nfrom azure.storage.blob import BlobBlock, BlobServiceClient\nfrom chunks import whole_lines\n\nlogging.getLogger("azure").setLevel(logging.WARNING)\napp = func.FunctionApp()\nMiB = 1024 * 1024\nblobs = BlobServiceClient.from_connection_string(\n os.environ["AzureWebJobsStorage"],\n max_single_get_size=4 * MiB, max_chunk_get_size=4 * MiB)\n\n\n@app.queue_trigger(arg_name="msg", queue_name="big-files",\n connection="AzureWebJobsStorage")\ndef filter_clicks(msg: func.QueueMessage) -> None:\n name = msg.get_body().decode() # clicks_1101.csv\n src = blobs.get_blob_client("raw", name).download_blob()\n dst = blobs.get_blob_client("curated", "checkout_" + name)\n blocks, kept = [], 0\n for lines in whole_lines(src): # chunk by chunk\n keep = [l + b"\\n" for l in lines if l.endswith(b",checkout")]\n kept += len(keep)\n blocks.append(BlobBlock(f"{len(blocks):05d}"))\n dst.stage_block(blocks[-1].id, b"".join(keep)) # one block\n dst.commit_block_list(blocks) # the file appears only now\n logging.info("%s: %d bytes in %d chunks, kept %d, %d blocks",\n name, src.size, len(blocks), kept, len(blocks))\n')
write('files_app/chunks.py', 'def whole_lines(download):\n """Yield the complete lines of each downloaded chunk."""\n rest = b""\n for chunk in download.chunks(): # one chunk in memory\n *lines, rest = (rest + chunk).split(b"\\n")\n yield lines # rest: a partial line\n if rest:\n yield [rest] # no newline at the end\n')
run('cd files_app && python "$FHOST" start')
run('python make_big.py')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.filter_clicks\'" 120 && python "$FHOST" log')
# Step 7: Partner files: validate each one, good files to the to-parquet queue, bad files to quarantine
write('files_app/function_app.py', """\
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="incoming/{name}",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
name = blob.name.split("/", 1)[1] # acme/orders_1101.csv
why = problem(blob.read().decode())
if why is None:
convert.set(json.dumps({"blob": name})) # to the converter
logging.info("valid %s -> to-parquet queue", name)
return
bad = blobs.get_blob_client("quarantine", name)
bad.start_copy_from_url(blob.uri, metadata={"reason": why})
logging.info("rejected %s: %s -> quarantine", name, why)
""")
write('files_app/checks.py', 'import pyarrow as pa\n\nHEADER = "order_id,sku,qty,amount"\nTYPES = {"order_id": pa.int64(), "sku": pa.string(), # for Parquet\n "qty": pa.int64(), "amount": pa.float64()}\n\n\ndef problem(text: str):\n """None when the partner file is valid, else the reason."""\n head, *rows = text.strip().splitlines() or [""]\n if head != HEADER:\n return "bad header"\n if not rows:\n return "no rows"\n try:\n [(int(r.split(",")[2]), float(r.split(",")[3])) for r in rows]\n except (IndexError, ValueError):\n return "bad qty or amount"\n return None\n')
run('cd files_app && python "$FHOST" start')
run('python partners_drop.py')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.validate_file\'" 60 3')
run('cd files_app && python "$FHOST" log | grep -E \'^(valid|rejected)\' | sort')
run('python report.py')
run('cd files_app && python "$FHOST" stop')
# Step 8: The convert function (a blueprint): the queued file becomes Parquet with pyarrow; read it back
write('files_app/function_app.py', """\
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
from convert import bp
logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
app.register_functions(bp)
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
@app.blob_trigger(arg_name="blob", path="incoming/{name}",
connection="AzureWebJobsStorage",
source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
name = blob.name.split("/", 1)[1] # acme/orders_1101.csv
why = problem(blob.read().decode())
if why is None:
convert.set(json.dumps({"blob": name})) # to the converter
logging.info("valid %s -> to-parquet queue", name)
return
bad = blobs.get_blob_client("quarantine", name)
bad.start_copy_from_url(blob.uri, metadata={"reason": why})
logging.info("rejected %s: %s -> quarantine", name, why)
""")
write('files_app/convert.py', """\
import io, logging, os
import azure.functions as func
import pyarrow.csv as pv
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
from checks import TYPES # the same partner contract
bp = func.Blueprint() # registered in function_app.py
blobs = BlobServiceClient.from_connection_string(
os.environ["AzureWebJobsStorage"])
OPTS = pv.ConvertOptions(column_types=TYPES)
@bp.queue_trigger(arg_name="msg", queue_name="to-parquet",
connection="AzureWebJobsStorage")
def convert_to_parquet(msg: func.QueueMessage) -> None:
name = msg.get_json()["blob"] # acme/orders_1101.csv
src = blobs.get_blob_client("incoming", name).download_blob()
table = pv.read_csv(io.BytesIO(src.readall()),
convert_options=OPTS) # no type guessing
out = io.BytesIO()
pq.write_table(table, out)
target = "orders/" + name.replace(".csv", ".parquet")
blobs.get_blob_client("curated", target).upload_blob(
out.getvalue(), overwrite=True) # rerun: same file
logging.info("%s -> curated/%s: %d rows", name, target,
table.num_rows)
logging.info("schema: %s", ", ".join(
f"{f.name} {f.type}" for f in table.schema))
""")
run('cd files_app && python "$FHOST" start')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.convert_to_parquet\'" 60 && python "$FHOST" log')
run('python read_parquet.py')
run('cd files_app && python "$FHOST" stop')
Part 4: Queues, Event Hubs and Service Bus Triggers
The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run with Azure Functions Core Tools 4.16.0 (the real Functions host, Python 3.13 worker) and Azurite 3.37.0 for storage. Features the local host cannot show truthfully are shown as code. Part of the Real Time Azure Functions Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of November 2026; check the official Azure Functions documentation (learn.microsoft.com/azure/azure-functions) for anything you quote in an interview.



No comments:
Post a Comment
Note: Only a member of this blog may post a comment.