Azure Functions Interview Questions and Answers Part 3 - Blob and File Processing

Azure Functions Interview Q&A · Part 3 of 10

In this post we are going to cover blob and file processing. These are 7 real Azure Functions interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: partners drop CSV files all day, and each must be validated, converted to Parquet and loaded.

Files landing in a container are one of the most common reasons a data team uses Azure Functions. A file arrives, a function picks it up, checks it and moves it on. Interviewers like this topic because the problems are very real: the same file loads twice, a big file runs out of memory, or one bad file keeps failing and nobody notices.

Part 3 of the Real Time Azure Functions Interview Questions and Answers series on TechBrothersIT.

The short answer

Say this first

Use the blob trigger with the Event Grid source (the only source on Flex Consumption). Expect duplicates: receipts are per ETag, so make loads idempotent. A failing file goes to webjobs-blobtrigger-poison after five tries, and big files are streamed in chunks, because the default Python InputStream holds the whole blob.

Let's set up the project

Everything below runs on the real Azure Functions host running locally with Core Tools, with Azurite, Microsoft's open-source storage emulator, for blobs, queues and tables. fhost runs func start in the background and prints the cleaned host log. What only Azure can show, such as the Event Grid subscription and plan sizes, is shown as code. Here are the starting files: the app's config in files_app/, and small client scripts. containers.py creates the containers once, upload.py uploads a file (a | is a new line), and grid.py plays Event Grid, because Azurite raises no storage events.

files_app/host.json
{
  "version": "2.0",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  },
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Function": "Information",
      "Azure.Core": "Error",
      "Host.Function.Console": "Information"
    }
  }
}
files_app/local.settings.json
{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "AzureWebJobsStorage": "UseDevelopmentStorage=true"
  }
}
files_app/requirements.txt
azure-functions
azure-storage-blob
azure-data-tables
pyarrow
containers.py
import os
from azure.storage.blob import BlobServiceClient

svc = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for name in ["landing", "drops", "orders", "invoices", "quarantine",
             "raw", "curated", "incoming"]:
    if not svc.get_container_client(name).exists():
        svc.create_container(name)
print("containers ready")
upload.py
import os, sys
from azure.storage.blob import BlobServiceClient

svc = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"])
container, name, text = sys.argv[1:4]          # "|" = new line
svc.get_blob_client(container, name).upload_blob(
    text.replace("|", "\n") + "\n", overwrite=True)
print("uploaded", f"{container}/{name}")
grid.py
import sys, requests

# Azurite raises no storage events, so this script plays Event Grid:
# it posts a BlobCreated event to the host's blob webhook.
container, blob = sys.argv[1].split("/", 1)
event = [{"id": "evt-" + blob, "eventType": "Microsoft.Storage.BlobCreated",
          "subject": f"/blobServices/default/containers/{container}"
                     f"/blobs/{blob}",
          "eventTime": "2026-11-01T09:00:00Z", "dataVersion": "",
          "data": {"api": "PutBlob", "blobType": "BlockBlob",
                   "url": "http://127.0.0.1:10000/devstoreaccount1/"
                          f"{container}/{blob}"}}]
r = requests.post("http://localhost:7071/runtime/webhooks/blobs",
                  params={"functionName": "Host.Functions." + sys.argv[2]},
                  headers={"aeg-event-type": "Notification"}, json=event)
print("Event Grid delivery:", r.status_code)
receipts.py
import os, sys
from azure.storage.blob import BlobServiceClient

hosts = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("azure-webjobs-hosts")
names = [b.name for b in hosts.list_blobs(name_starts_with="blobreceipts/")
         if f"/Host.Functions.{sys.argv[1]}/" in b.name]
for n in names:
    print(n)                           # .../<function>/<etag>/<blob>
print("receipts:", len(names), "| different ETags:",
      len({n.split("/")[3] for n in names}))
where.py
import os, sys
from azure.storage.blob import BlobServiceClient

svc = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for c in sys.argv[1:]:
    names = sorted(svc.get_container_client(c).list_blob_names())
    print(f"{c}:", names)
runs.py
import re, sys

log = open("files_app/.fhost.log").read()          # the host console
runs = re.findall(f"Executing 'Functions.{sys.argv[1]}'", log)
print(f"{sys.argv[1]} runs so far:", len(runs))
make_big.py
import os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64EncodePolicy

conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
pages = ["home", "product", "cart", "checkout"]
rows = [f"{i},{pages[i % 4]}" for i in range(1, 1_000_001)]
data = ("click_id,page\n" + "\n".join(rows) + "\n").encode()
BlobServiceClient.from_connection_string(conn).get_blob_client(
    "raw", "clicks_1101.csv").upload_blob(data, overwrite=True)
print("raw/clicks_1101.csv:", len(data), "bytes,", len(rows), "clicks")
q = QueueClient.from_connection_string(
    conn, "big-files", message_encode_policy=TextBase64EncodePolicy())
q.create_queue()
q.send_message("clicks_1101.csv")             # just the name
print("queued: clicks_1101.csv")
partners_drop.py
import subprocess, sys

H = "order_id,sku,qty,amount"
files = {"acme/orders_1101.csv": f"{H}|5001,tee-m,2,19.98|5002,mug,1,8.50",
         "globex/orders_1101.csv": "order_id;sku;qty;amount|7001;cap;1;12",
         "initech/orders_1101.csv": f"{H}|9001,hoodie,two,45.00"}
for name, text in files.items():
    subprocess.run([sys.executable, "upload.py", "incoming", name, text])
for name in files:                         # Event Grid: one event each
    subprocess.run([sys.executable, "grid.py", "incoming/" + name,
                    "validate_file"])
report.py
import json, os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64DecodePolicy

conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
q = QueueClient.from_connection_string(
    conn, "to-parquet", message_decode_policy=TextBase64DecodePolicy())
todo = sorted(json.loads(m.content)["blob"]
              for m in q.peek_messages(max_messages=32))
print("to-parquet queue:", todo)
box = BlobServiceClient.from_connection_string(
    conn).get_container_client("quarantine")
for b in sorted(box.list_blobs(include=["metadata"]), key=lambda b: b.name):
    print("quarantine:", b.name, "| reason:", b.metadata["reason"])
read_parquet.py
import io, os
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient

box = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("curated")
for name in sorted(box.list_blob_names(name_starts_with="orders/")):
    table = pq.read_table(io.BytesIO(box.download_blob(name).readall()))
    print(name, "|", table.num_rows, "rows")
    for row in table.to_pylist():
        print("  ", row)

Question 1: Files land in a container and must be processed as soon as they arrive. Which trigger do you use?

The blob trigger. It runs the function for each new or updated blob that matches a path. In production, we give it the Event Grid source: lower latency, and on Flex Consumption it is the only source supported.

The path is landing/{name}.csv, so only CSV files fire it, and {name} becomes a binding value we can reuse in other bindings. The function gets an InputStream with the blob name and length.

Let's start the host and upload an orders file with python upload.py landing orders_0901.csv.

files_app/function_app.py
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
    rows = blob.read().decode().strip().splitlines()[1:]
    logging.info("%s: %d rows, %d bytes", blob.name, len(rows),
                 blob.length)
Terminal
$ cd files_app && fhost start
host started, functions:
  on_landing: blobTrigger
$ python upload.py landing orders_0901.csv "order_id,amount|1001,25.50|1002,14.00"
uploaded landing/orders_0901.csv
$ cd files_app && fhost wait "Executed 'Functions.on_landing'" 60 && fhost log
Executing 'Functions.on_landing' (Reason='New blob detected(LogsAndContainerScan):
  landing/orders_0901.csv')
landing/orders_0901.csv: 2 rows, 38 bytes
Executed 'Functions.on_landing' (Succeeded)

It runs once: landing/orders_0901.csv: 2 rows, 38 bytes. The reason says New blob detected(LogsAndContainerScan). That is the default polling source.

The blob trigger is not the only option. Here is when we pick each one:

OptionWhen
Blob triggermost file loads
Event Grid triggeryou need the raw event
Event Grid to a queueseveral consumers, buffering

For the raw event, or for several consumers, we use an Event Grid trigger, or Event Grid into a storage queue.

Pitfall

A function writes its output into the container it watches, and triggers itself again. Write outputs to another container, and filter the path, like {name}.csv.

Question 2: Someone asks the difference between the polling blob trigger and the Event Grid based blob trigger. Which is better?

The Event Grid source is better for new work. Polling scans the container and reads storage logs, best effort. The Event Grid source is pushed to the function.

Let's compare them:

Polling (default)Event Grid source
howscans the container, 10,000 blobs a batchBlobCreated event pushed to a webhook
storage logsbest effort, no guaranteenot used
idle Consumption appup to a 10-minute delaylower latency
ADLS Gen2 (HNS)not officially supportedsupported
Flex Consumptionnot supportedthe only option
needsnothing extraGPv2 account, event subscription

So the default source scans the container in batches of 10,000 blobs. An idle Consumption app can take up to 10 minutes, and polling isn't officially supported on ADLS Gen2. With the Event Grid source, a BlobCreated event calls the function's webhook.

Now let's try it locally. on_drop watches drops/ with source=func.BlobSource.EVENT_GRID. We upload a file, count the runs with python runs.py on_drop, then post a BlobCreated event to the local webhook with python grid.py.

files_app/function_app.py
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
    logging.info("polling source: %s", blob.name)


@app.blob_trigger(arg_name="blob", path="drops/{name}.csv",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
def on_drop(blob: func.InputStream) -> None:
    logging.info("event grid source: %s, %d bytes", blob.name,
                 blob.length)
Terminal
$ cd files_app && fhost start
host started, functions:
  on_drop: blobTrigger
  on_landing: blobTrigger
$ python upload.py drops orders_0902.csv "order_id,amount|1003,9.50"
uploaded drops/orders_0902.csv
$ sleep 5 && python runs.py on_drop
on_drop runs so far: 0
$ python grid.py drops/orders_0902.csv on_drop
Event Grid delivery: 202
$ cd files_app && fhost wait "Executed 'Functions.on_drop'" 60 && fhost log
Executing 'Functions.on_drop' (Reason='New blob detected(EventGrid): drops/orders_0902.csv')
event grid source: drops/orders_0902.csv, 26 bytes
Executed 'Functions.on_drop' (Succeeded)

The upload alone gives on_drop runs so far: 0, because Azurite raises no events. After the event, delivery returns 202, and the reason says New blob detected(EventGrid): drops/orders_0902.csv. In Azure, we create a system topic on the storage account and a webhook subscription with the blobs extension key, filtered to BlobCreated under one container:

event_subscription.sh (Azure, not run)
# 1. a system topic for the storage account's events
az eventgrid system-topic create -g rg-data -n st-files-events \
  --topic-type microsoft.storage.storageaccounts \
  --source "$STORAGE_ID" --location eastus2
# 2. the blobs_extension system key of the function app
KEY=$(az functionapp keys list -g rg-data -n fn-files \
  --query systemKeys.blobs_extension -o tsv)
HOOK="https://fn-files.azurewebsites.net/runtime/webhooks/blobs"
HOOK="$HOOK?functionName=Host.Functions.on_drop&code=$KEY"
# 3. BlobCreated events under drops/ go to the webhook
az eventgrid system-topic event-subscription create \
  -g rg-data --system-topic-name st-files-events -n on-drop \
  --endpoint-type webhook --endpoint "$HOOK" \
  --included-event-types Microsoft.Storage.BlobCreated \
  --subject-begins-with /blobServices/default/containers/drops/
Catches

It needs a general-purpose v2 account, push can't reach a private endpoint, and delivery is at least once, so dedupe is your job.

Question 3: A teammate's blob function processes the same file twice. Why does that happen, and how do you stop it?

The polling trigger keeps a blob receipt per function, blob and ETag, in the azure-webjobs-hosts container. Anything that changes the ETag or loses the receipt runs the file again, so the load must be idempotent.

Let's see it. A partner uploads daily_1031.csv, then uploads the exact same content again. We check the receipts with python receipts.py on_order_file.

files_app/function_app.py
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
    logging.info("load %s, %d bytes", blob.name, blob.length)
    # ... append the rows to the orders table ...
Terminal
$ cd files_app && fhost start
host started, functions:
  on_order_file: blobTrigger
$ python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"
uploaded orders/daily_1031.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60
$ python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"
uploaded orders/daily_1031.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 2 && fhost log
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
  orders/daily_1031.csv')
load orders/daily_1031.csv, 27 bytes
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
  orders/daily_1031.csv')
load orders/daily_1031.csv, 27 bytes
Executed 'Functions.on_order_file' (Succeeded)
$ python receipts.py on_order_file
blobreceipts/tbit-fn-demo/Host.Functions.on_order_file/<etag>/orders/daily_1031.csv
blobreceipts/tbit-fn-demo/Host.Functions.on_order_file/<etag>/orders/daily_1031.csv
receipts: 2 | different ETags: 2

Same bytes, but a new ETag, so the function loads it twice, 27 bytes each time, and we get receipts: 2 | different ETags: 2.

There are other causes too:

  • Renamed function: no receipts yet, every file again
  • New AzureWebJobsStorage: the receipts stay behind
  • Retry after a failure: part of the work ran already
  • Two apps, one container: each has its own receipts
  • Event Grid: at-least-once delivery

The fix is an idempotent load. This version hashes the content and keeps a processedfiles table. We upload daily_1101.csv, the same file again, then a corrected file with a new amount.

files_app/function_app.py
import hashlib, logging, os
import azure.functions as func
from azure.core.exceptions import ResourceNotFoundError
from azure.data.tables import TableServiceClient

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
done = TableServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"]
).create_table_if_not_exists("processedfiles")


@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
    key = blob.name.split("/", 1)[1].replace("/", "|")  # no / in keys
    digest = hashlib.sha256(blob.read()).hexdigest()
    try:
        seen = done.get_entity("orders", key)["sha256"]
    except ResourceNotFoundError:
        seen = None                                 # never loaded
    if seen == digest:
        logging.info("skip %s: same content", blob.name)
        return
    # ... load, overwriting this file's own rows ...
    done.upsert_entity({"PartitionKey": "orders", "RowKey": key,
                        "sha256": digest})
    logging.info("loaded %s", blob.name)
Terminal
$ cd files_app && fhost start
host started, functions:
  on_order_file: blobTrigger
$ python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60
$ python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 2
$ python upload.py orders daily_1101.csv "order_id,amount|2001,32.00"
uploaded orders/daily_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.on_order_file'" 60 3 && fhost log
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
  orders/daily_1101.csv')
loaded orders/daily_1101.csv
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
  orders/daily_1101.csv')
skip orders/daily_1101.csv: same content
Executed 'Functions.on_order_file' (Succeeded)
Executing 'Functions.on_order_file' (Reason='New blob detected(LogsAndContainerScan):
  orders/daily_1101.csv')
loaded orders/daily_1101.csv
Executed 'Functions.on_order_file' (Succeeded)

The first copy logs loaded, the resend logs skip orders/daily_1101.csv: same content, and the corrected file loads.

Real project

Make each file overwrite its own output, with a fixed name or a MERGE on the business key. Then even a real duplicate run replaces rows instead of appending them.

Question 4: A 5 GB file must be processed by a function. What do you watch out for?

Watch memory, the timeout, temp storage and parallelism. Then stream it in chunks, or hand the heavy work to Databricks.

Memory comes first. In Python, the default InputStream binding hands you the whole content in memory. A Flex Consumption instance has 512 MB, 2,048 MB by default, or 4,096 MB. Five gigabytes doesn't fit. Let's look at the plans:

PlanMemory per instanceTimeout default / maxTemp storage
Flex Consumption512 MB, 2,048 MB (default), 4,096 MB30 min / unbounded0.8 GB
Premium3.5 to 14 GB (EP1 to EP3)30 min / unbounded11 to 61 GB
Consumption (legacy)1.5 GB5 min / 10 min0.5 GB (Linux 1.5 GB)

Then time and disk. Flex Consumption and Premium default to 30 minutes and can run unbounded, legacy Consumption stops at 10, and an HTTP trigger must respond within 230 seconds on any plan. Flex has about 0.8 GB of temp storage, so a full download to disk fails too.

And parallelism. By default, the blob extension runs up to 8 times the core count at once, across all the app's blob functions, and each needs memory. So we lower maxDegreeOfParallelism:

host.json excerpt (big files)
{
  "version": "2.0",
  "functionTimeout": "01:00:00",
  "extensions": {
    "blobs": { "maxDegreeOfParallelism": 2 }
  }
}

Even with the largest Flex instance size, 4,096 MB, we still stream:

create_app.sh (Azure, not run)
# Flex Consumption with the largest instance size: 4,096 MB
az functionapp create -g rg-data -n fn-bigfiles \
  --storage-account stfnbigfiles \
  --flexconsumption-location eastus2 \
  --runtime python --runtime-version 3.13 --instance-memory 4096

What we do instead:

  • Stream: storage SDK, chunks in and blocks out (question 6)
  • Split: process the pieces as Durable activities
  • Hand off: heavy transforms and joins go to Databricks or Data Factory
Pitfall

A small test file works, and the real one runs out of memory. A 5 MB sample hides it, so test with a real size.

Question 5: Someone asks what happens when a blob function keeps failing on one file. How do poison blobs work?

The blob trigger tries a failing file five times by default. Then it writes a message to the webjobs-blobtrigger-poison queue with the function, container, blob name and ETag. The file itself stays where it is: you handle it.

Here on_invoice checks the header. We upload a good invoice, then one with semicolons.

files_app/function_app.py
import logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="invoices/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_invoice(blob: func.InputStream) -> None:
    header = blob.read().decode().splitlines()[0]
    if header != "invoice_id,amount":
        raise ValueError(f"bad header {header!r} in {blob.name}")
    logging.info("loaded %s", blob.name)


@app.queue_trigger(arg_name="msg", connection="AzureWebJobsStorage",
                   queue_name="webjobs-blobtrigger-poison")
def park_poison(msg: func.QueueMessage) -> None:
    info = msg.get_json()        # FunctionId, ContainerName, BlobName
    box, name = info["ContainerName"], info["BlobName"]
    src = blobs.get_blob_client(box, name)
    dst = blobs.get_blob_client("quarantine", f"{box}/{name}")
    dst.start_copy_from_url(src.url, metadata={"reason": "poison"})
    if dst.get_blob_properties().copy.status == "success":
        src.delete_blob()                           # same account
    logging.info("parked %s/%s in quarantine", box, name)
Terminal
$ cd files_app && fhost start
host started, functions:
  on_invoice: blobTrigger
  park_poison: queueTrigger
$ python upload.py invoices inv_0901.csv "invoice_id,amount|1,50.00"
uploaded invoices/inv_0901.csv
$ cd files_app && fhost wait "Executed 'Functions.on_invoice'" 60
$ python upload.py invoices inv_0902.csv "id;amount|2;75.00"
uploaded invoices/inv_0902.csv
$ cd files_app && fhost wait "Executed 'Functions.park_poison'" 90 && fhost log
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0901.csv')
loaded invoices/inv_0901.csv
Executed 'Functions.on_invoice' (Succeeded)
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
  System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
  System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
  System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
  System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Executing 'Functions.on_invoice' (Reason='New blob detected(LogsAndContainerScan):
  invoices/inv_0902.csv')
Executed 'Functions.on_invoice' (Failed)
System.Private.CoreLib: Exception while executing function: Functions.on_invoice.
  System.Private.CoreLib: Result: Failure
Exception: ValueError: bad header 'id;amount' in invoices/inv_0902.csv
Message has reached MaxDequeueCount of 5. Moving message to queue 'webjobs-blobtrigger-poison'.
Executing 'Functions.park_poison' (Reason='New queue message detected on 'webjobs-blobtrigger-poison'.')
parked invoices/inv_0902.csv in quarantine
Executed 'Functions.park_poison' (Succeeded)
$ python where.py invoices quarantine
invoices: ['inv_0901.csv']
quarantine: ['invoices/inv_0902.csv']

inv_0901.csv loads. inv_0902.csv fails five times with ValueError: bad header 'id;amount', then the host logs Moving message to queue 'webjobs-blobtrigger-poison'.

Nothing reads that queue unless we add a function. Every blob function in the app shares it, so park_poison is generic: it copies the blob to quarantine and deletes the original. At the end, invoices holds only inv_0901.csv, and quarantine has invoices/inv_0902.csv.

In a real project, we also alert on the poison queue length and tell the partner why. A fixed upload is a new version, so it runs.

Catch

With identity-based connections, the trigger's identity needs Storage Queue Data Contributor, or it can't write poison messages.

Question 6: A teammate asks how to read and write blobs inside a function without loading everything into memory. What do you do?

Don't bind the file as bytes or as the default InputStream: in Python, both hold the whole blob. Bind an SDK type like BlobClient, which is generally available for the Python v2 model, or trigger on a small message and stream with the storage SDK.

Here is the SDK type binding:

function_app.py with SDK type bindings (GA, Python v2 only, sync types)
# requirements.txt: azurefunctions-extensions-bindings-blob
import azure.functions as func
import azurefunctions.extensions.bindings.blob as blob

app = func.FunctionApp()

@app.blob_trigger(arg_name="client", path="raw/{name}",
                  connection="AzureWebJobsStorage")
def stream_file(client: blob.BlobClient):   # not the bytes
    for chunk in client.download_blob().chunks():
        ...                                  # one chunk at a time

Our demo takes the second way. filter_clicks gets just the file name from a queue, and the blob client downloads 4 MiB chunks. A helper, whole_lines, turns each chunk into whole lines and carries the partial last line over. For the output, it stages one block per chunk and commits the block list at the end, so readers never see a half-written file.

We run python make_big.py, which uploads a file with a million clicks and queues its name.

files_app/function_app.py
import logging, os
import azure.functions as func
from azure.storage.blob import BlobBlock, BlobServiceClient
from chunks import whole_lines

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
MiB = 1024 * 1024
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"],
    max_single_get_size=4 * MiB, max_chunk_get_size=4 * MiB)


@app.queue_trigger(arg_name="msg", queue_name="big-files",
                   connection="AzureWebJobsStorage")
def filter_clicks(msg: func.QueueMessage) -> None:
    name = msg.get_body().decode()               # clicks_1101.csv
    src = blobs.get_blob_client("raw", name).download_blob()
    dst = blobs.get_blob_client("curated", "checkout_" + name)
    blocks, kept = [], 0
    for lines in whole_lines(src):               # chunk by chunk
        keep = [l + b"\n" for l in lines if l.endswith(b",checkout")]
        kept += len(keep)
        blocks.append(BlobBlock(f"{len(blocks):05d}"))
        dst.stage_block(blocks[-1].id, b"".join(keep))   # one block
    dst.commit_block_list(blocks)       # the file appears only now
    logging.info("%s: %d bytes in %d chunks, kept %d, %d blocks",
                 name, src.size, len(blocks), kept, len(blocks))
files_app/chunks.py
def whole_lines(download):
    """Yield the complete lines of each downloaded chunk."""
    rest = b""
    for chunk in download.chunks():           # one chunk in memory
        *lines, rest = (rest + chunk).split(b"\n")
        yield lines                           # rest: a partial line
    if rest:
        yield [rest]                          # no newline at the end
Terminal
$ cd files_app && fhost start
host started, functions:
  filter_clicks: queueTrigger
$ python make_big.py
raw/clicks_1101.csv: 13638910 bytes, 1000000 clicks
queued: clicks_1101.csv
$ cd files_app && fhost wait "Executed 'Functions.filter_clicks'" 120 && fhost log
Executing 'Functions.filter_clicks' (Reason='New queue message detected on 'big-files'.')
clicks_1101.csv: 13638910 bytes in 4 chunks, kept 250000, 4 blocks
Executed 'Functions.filter_clicks' (Succeeded)

The file is 13638910 bytes and 1000000 clicks. It came in 4 chunks, and the function kept 250000 checkout clicks in 4 blocks. It only ever held about one chunk in memory.

Tip

Keep clients at module level so invocations reuse them, and set the azure logger to WARNING, or the SDK logs every request.

Scenario 7: Partners drop CSV files all day and each must be validated, converted to Parquet and loaded. How do you build it?

Event-driven, one file at a time. Partners write to their own folders, an Event Grid blob trigger validates each file, and a queue hands good files to a convert function that writes Parquet for the load.

Here is the flow:

  • incoming/: the landing container, a folder per partner
  • validate_file: Event Grid blob trigger
  • to-parquet: storage queue for good files
  • quarantine/: bad files, the reason in metadata
  • convert_to_parquet: queue trigger, writes curated/orders/

Validation lives in a plain module, checks.py, with the partner contract: the header, at least one row, and numeric quantity and amount. We run python partners_drop.py, which uploads three files and posts one event each, then python report.py.

files_app/function_app.py
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="incoming/{name}",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
    name = blob.name.split("/", 1)[1]           # acme/orders_1101.csv
    why = problem(blob.read().decode())
    if why is None:
        convert.set(json.dumps({"blob": name}))  # to the converter
        logging.info("valid %s -> to-parquet queue", name)
        return
    bad = blobs.get_blob_client("quarantine", name)
    bad.start_copy_from_url(blob.uri, metadata={"reason": why})
    logging.info("rejected %s: %s -> quarantine", name, why)
files_app/checks.py
import pyarrow as pa

HEADER = "order_id,sku,qty,amount"
TYPES = {"order_id": pa.int64(), "sku": pa.string(),   # for Parquet
         "qty": pa.int64(), "amount": pa.float64()}


def problem(text: str):
    """None when the partner file is valid, else the reason."""
    head, *rows = text.strip().splitlines() or [""]
    if head != HEADER:
        return "bad header"
    if not rows:
        return "no rows"
    try:
        [(int(r.split(",")[2]), float(r.split(",")[3])) for r in rows]
    except (IndexError, ValueError):
        return "bad qty or amount"
    return None
Terminal
$ cd files_app && fhost start
host started, functions:
  validate_file: blobTrigger
$ python partners_drop.py
uploaded incoming/acme/orders_1101.csv
uploaded incoming/globex/orders_1101.csv
uploaded incoming/initech/orders_1101.csv
Event Grid delivery: 202
Event Grid delivery: 202
Event Grid delivery: 202
$ cd files_app && fhost wait "Executed 'Functions.validate_file'" 60 3
$ cd files_app && fhost log | grep -E '^(valid|rejected)' | sort
rejected globex/orders_1101.csv: bad header -> quarantine
rejected initech/orders_1101.csv: bad qty or amount -> quarantine
valid acme/orders_1101.csv -> to-parquet queue
$ python report.py
to-parquet queue: ['acme/orders_1101.csv']
quarantine: globex/orders_1101.csv | reason: bad header
quarantine: initech/orders_1101.csv | reason: bad qty or amount
quarantine: invoices/inv_0902.csv | reason: poison
$ cd files_app && fhost stop
host stopped

Acme's file is valid and goes to the to-parquet queue. Globex used semicolons, bad header, and Initech has bad qty or amount, so both go to quarantine with the reason. The invoices/inv_0902.csv line is left over from question 5.

Now we add the convert blueprint and start the host again. It reads the CSV with pyarrow using the contract's types, not guessed ones, and writes Parquet with a fixed name, so a rerun overwrites.

files_app/function_app.py
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
from convert import bp

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
app.register_functions(bp)
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="incoming/{name}",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
    name = blob.name.split("/", 1)[1]           # acme/orders_1101.csv
    why = problem(blob.read().decode())
    if why is None:
        convert.set(json.dumps({"blob": name}))  # to the converter
        logging.info("valid %s -> to-parquet queue", name)
        return
    bad = blobs.get_blob_client("quarantine", name)
    bad.start_copy_from_url(blob.uri, metadata={"reason": why})
    logging.info("rejected %s: %s -> quarantine", name, why)
files_app/convert.py
import io, logging, os
import azure.functions as func
import pyarrow.csv as pv
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
from checks import TYPES                # the same partner contract

bp = func.Blueprint()               # registered in function_app.py
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])
OPTS = pv.ConvertOptions(column_types=TYPES)


@bp.queue_trigger(arg_name="msg", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def convert_to_parquet(msg: func.QueueMessage) -> None:
    name = msg.get_json()["blob"]              # acme/orders_1101.csv
    src = blobs.get_blob_client("incoming", name).download_blob()
    table = pv.read_csv(io.BytesIO(src.readall()),
                        convert_options=OPTS)    # no type guessing
    out = io.BytesIO()
    pq.write_table(table, out)
    target = "orders/" + name.replace(".csv", ".parquet")
    blobs.get_blob_client("curated", target).upload_blob(
        out.getvalue(), overwrite=True)        # rerun: same file
    logging.info("%s -> curated/%s: %d rows", name, target,
                 table.num_rows)
    logging.info("schema: %s", ", ".join(
        f"{f.name} {f.type}" for f in table.schema))
Terminal
$ cd files_app && fhost start
host started, functions:
  convert_to_parquet: queueTrigger
  validate_file: blobTrigger
$ cd files_app && fhost wait "Executed 'Functions.convert_to_parquet'" 60 && fhost log
Executing 'Functions.convert_to_parquet' (Reason='New queue message detected on 'to-parquet'.')
acme/orders_1101.csv -> curated/orders/acme/orders_1101.parquet: 2 rows
schema: order_id int64, sku string, qty int64, amount double
Executed 'Functions.convert_to_parquet' (Succeeded)
$ python read_parquet.py
orders/acme/orders_1101.parquet | 2 rows
   {'order_id': 5001, 'sku': 'tee-m', 'qty': 2, 'amount': 19.98}
   {'order_id': 5002, 'sku': 'mug', 'qty': 1, 'amount': 8.5}
$ cd files_app && fhost stop
host stopped

We get curated/orders/acme/orders_1101.parquet: 2 rows, with order_id int64, sku string, qty int64, amount double, typed as agreed.

Around it

A processed-files table keyed by name and content hash, alerts on the poison queues, and a note to the partner for each quarantined file. Very large files go to Databricks instead.

Key points to remember

  • Blob trigger with the Event Grid source: lower latency, ADLS Gen2, the only option on Flex Consumption
  • Polling scans the container and logs: best effort, up to 10 minutes on an idle Consumption app
  • Receipts are per function, blob and ETag: an overwrite runs again, so loads must be idempotent
  • Five tries, then webjobs-blobtrigger-poison: add a handler that quarantines and alerts
  • Big files: the InputStream holds the whole blob; stream with chunks in and blocks out

The full script

Here is a Python script that writes every file and runs every command in order with Core Tools and Azurite; run pip install azure-functions azure-functions-durable azure-storage-blob azure-storage-queue azure-data-tables first.

Show the full script that writes the files and runs every step
Full script - writes the files and runs every step with Azure Functions Core Tools and Azurite (pip install azure-functions azure-functions-durable azure-storage-blob azure-storage-queue azure-data-tables), run it with Python
# ========================================================================
# Azure Functions Interview Q&A #3 - Blob and file processing
# TechBrothersIT. Names and sample data are invented for teaching.
# Every step below was run with Azure Functions Core Tools 4.16.0 (Functions host 4.x, Python 3.13 worker,
# programming model v2, extension bundle 4.x) and Azurite 3.37.0; the outputs in the video and the blog post are the real outputs.
# To run it yourself (Linux or macOS shell, or WSL on Windows):
#   1. Install Azure Functions Core Tools 4 and Azurite (npm install -g azurite), then start Azurite:
#      azurite --skipApiVersionCheck --location ./azurite-data   (blob 10000, queue 10001, table 10002)
#   2. pip install azure-functions azure-functions-durable azure-storage-blob azure-storage-queue azure-data-tables
#   3. Set AZURE_STORAGE_CONNECTION_STRING to "UseDevelopmentStorage=true", then run python Functions_Interview_QA_03_Code.py.
# "fhost" in the commands is a small helper written by Step 0 (fhost.py): it runs `func start` in the background,
# waits until the functions are loaded, and prints the host log lines without timestamps and invocation ids.
# Run it in an empty folder and on a freshly started emulator: it creates its files (and any local data
# files) there, and a second run on old data gives different output.
# Steps that restart the emulator say so in a comment: stop and start it by hand at that point.
# ========================================================================

import os, subprocess, time

def write(path, text):
    if text is None:
        if os.path.exists(path): os.remove(path)
        return
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        f.write(text.strip("\n") + "\n")

def run(cmd, expect_error=False, allow_error=False, env=None):
    print("\n$ " + cmd, flush=True)
    rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
    if rc != 0 and not (expect_error or allow_error):
        raise SystemExit("stopped: this command was expected to succeed")

# Step 0: the fhost helper and the starting files
write("fhost.py", '"""fhost: run the Azure Functions host (func start) in the background of the current folder, for scripted demos.\n  fhost start [port] [-- extra func args]   start `func start`, wait until the functions are indexed, print them\n                                            (and any indexing errors)\n  fhost wait PATTERN [secs] [count]         wait until `count` (default 1) NEW host log lines (not yet shown by\n                                            `fhost log`) match the regex PATTERN (default timeout 60 s)\n  fhost log [--all] [--details] [--since-start]\n                                            print the host log lines written since the last `fhost log` (function logs,\n                                            Executing / Executed lines, exceptions); --details also keeps the\n                                            "Trigger Details" lines, --all keeps every line, --since-start starts at the\n                                            first line after the function list. Timestamps, invocation ids, durations and\n                                            other per-run values are removed.\n  fhost stop                                stop the host\nThe log is kept in .fhost.log in the current folder (one host per folder)."""\nimport os, re, signal, subprocess, sys, time\nLOG, PID, OFF, START = \'.fhost.log\', \'.fhost.pid\', \'.fhost.off\', \'.fhost.start\'\nTS = re.compile(r\'\\[\\d{4}-\\d\\d-\\d\\dT[\\d:.]+Z\\] \')\n# host messages to hide (anchored at the start of the cleaned line, so user log lines that merely contain these words stay)\nNOISE = re.compile(r\'^(Worker process started|Host lock lease|Found Python version|Azure Functions Core Tools|Core Tools Version|\'\n                   r\'Function Runtime Version|For detailed output|Using cached version|When you have network connectivity|\'\n                   r\'Warning: Unable to download extension|Host started|Job host started|Host initialized|Loading functions metadata|\'\n                   r\'Reading functions metadata|Starting Host|Initializing Warmup|Generating \\d+ job function|Functions:$|\'\n                   r\'Debugger listening|Host Status|Starting worker process|Worker failed to load|Retrieving function|\'\n                   r\'\\d+ functions? (found|loaded)|Successfully (found|loaded|registered)|Application Insights|WARNING: )\')\nDETAILS = re.compile(r\'^Trigger Details:\')\n\ndef clean(l):\n    l = TS.sub(\'\', l.rstrip(\'\\n\'))\n    l = re.sub(r\', Id=[0-9a-f-]{36}\', \'\', l)\n    l = re.sub(r\'Id=[0-9a-f-]{36}\', \'\', l)\n    l = re.sub(r\', Duration=\\d+ms\', \'\', l)\n    l = re.sub(r\'\\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\\b\', \'<id>\', l)\n    l = re.sub(r\'\\b[0-9a-f]{32}\\b\', \'<instance-id>\', l)\n    l = re.sub(r\'\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(Z|[+-]\\d\\d:\\d\\d)?\', \'<time>\', l)\n    l = re.sub(r\'"?0x[0-9A-F]{15,16}"?\', \'<etag>\', l)\n    l = re.sub(r\'([?&]code=)[^&\\s"]+\', r\'\\1<key>\', l)\n    l = re.sub(r\'Threadpool id: \\d+\', \'Threadpool id: <n>\', l)\n    l = re.sub(r\'(processUptime\\W+)\\d+\', r\'\\1<n>\', l)\n    return l.replace(\' ()\', \'\').rstrip()\n\ndef read_from(off):\n    """complete lines from byte offset `off`; returns (lines, new_offset). A glued line ([ts] a [ts] b) is split."""\n    if not os.path.exists(LOG): return [], off\n    with open(LOG, \'rb\') as f:\n        f.seek(off); data = f.read()\n    end = data.rfind(b\'\\n\')\n    if end < 0: return [], off\n    chunk = data[:end + 1].decode(\'utf-8\', \'replace\')\n    out = []\n    for l in chunk.split(\'\\n\')[:-1]:\n        parts = [p for p in re.split(r\'(?=\\[\\d{4}-\\d\\d-\\d\\dT[\\d:.]+Z\\] )\', l) if p.strip()] or [\'\']\n        out += parts\n    return out, off + end + 1\n\ndef get(f, d=0):\n    return int(open(f).read()) if os.path.exists(f) else d\n\ndef filtered(lines, all_=False, details=False):\n    out, in_stack, last_exc = [], False, None\n    for l in lines:\n        raw = TS.sub(\'\', l)\n        if in_stack:\n            if raw.startswith((\' \', \'\\t\')): continue\n            in_stack = False\n        c = clean(l)\n        if not c: continue\n        if c.startswith(\'Stack:\') and not all_: in_stack = True; continue\n        if not all_ and (c in (\'Type:\', \'.\') or c == last_exc): continue\n        if c.startswith(\'Exception: \'): last_exc = c[len(\'Exception: \'):]\n        if all_ or (details and DETAILS.search(c)): out.append(c); continue\n        if NOISE.search(c) or DETAILS.search(c): continue\n        out.append(c)\n    return out\n\ndef start(*args):\n    port, extra = \'7071\', []\n    if args and args[0] != \'--\': port = args[0]; args = args[1:]\n    if args and args[0] == \'--\': extra = list(args[1:])\n    if os.path.exists(PID): stop(quiet=True)\n    for f in (LOG, OFF, START):\n        if os.path.exists(f): os.remove(f)\n    out = open(LOG, \'w\')\n    p = subprocess.Popen([\'func\', \'start\', \'--port\', port] + extra, stdout=out, stderr=subprocess.STDOUT,\n                         stdin=subprocess.DEVNULL, start_new_session=True)\n    open(PID, \'w\').write(str(p.pid))\n    t0 = time.time()\n    while time.time() - t0 < 120:\n        txt = open(LOG, encoding=\'utf-8\', errors=\'replace\').read()\n        if p.poll() is not None:\n            print(\'\\n\'.join(filtered(txt.split(\'\\n\'), all_=True)[-30:])); sys.exit(\'host exited\')\n        m = re.search(r\'\\nFunctions:\\n(.*?)\\n(For detailed output[^\\n]*\\n)\', txt, re.S)\n        if m and \'Worker process started and initialized\' in txt:\n            errs = [clean(l) for l in txt[:m.start()].split(\'\\n\')\n                    if re.search(r\'in error|failed indexing|Error indexing|will be disabled\', l)]\n            names = [clean(l).strip() for l in m.group(1).split(\'\\n\') if l.strip()]\n            boundary = len(txt[:m.end()].encode(\'utf-8\'))\n            open(START, \'w\').write(str(boundary)); open(OFF, \'w\').write(str(boundary))\n            time.sleep(1.5)   # let triggers start listening\n            for e in errs: print(e)\n            print(\'host started, functions:\')\n            for n in names: print(\'  \' + re.sub(r\'http://localhost:\\d+\', \'http://localhost:\' + port, n))\n            return\n        if not m and re.search(r\'\\n\\[[^\\]]*\\] (Job host started|Host started)\', txt) and \'Worker process started and initialized\' in txt:\n            time.sleep(1.5)\n            boundary = len(txt.encode(\'utf-8\')); open(START, \'w\').write(str(boundary)); open(OFF, \'w\').write(str(boundary))\n            for l in txt.split(\'\\n\'):\n                if re.search(r\'in error|failed indexing|is disabled\', l): print(clean(l))\n            print(\'host started, no enabled functions listed\'); return\n        if re.search(r\'No job functions found|A host error has occurred\', txt):\n            time.sleep(1); print(\'\\n\'.join(filtered(txt.split(\'\\n\'), all_=True)[-30:])); stop(quiet=True); sys.exit(\'host error\')\n        time.sleep(0.3)\n    print(\'\\n\'.join(filtered(open(LOG).read().split(\'\\n\'), all_=True)[-30:])); stop(quiet=True)\n    sys.exit(\'host did not start in 120 s\')\n\ndef wait(pat, secs=\'60\', count=\'1\'):\n    rx, t0, need = re.compile(pat), time.time(), int(count)\n    while time.time() - t0 < float(secs):\n        lines, _ = read_from(get(OFF))\n        if sum(1 for l in lines if rx.search(clean(l))) >= need: return\n        time.sleep(0.2)\n    lines, _ = read_from(get(OFF))\n    print(\'\\n\'.join(filtered(lines)[-20:])); sys.exit(\'timed out waiting for %r\' % pat)\n\ndef log(all_=False, details=False, since_start=False):\n    off = get(START) if since_start else get(OFF)\n    lines, new = read_from(off)\n    if not since_start or new > get(OFF): open(OFF, \'w\').write(str(new))\n    for c in filtered(lines, all_, details): print(c)\n\ndef stop(quiet=False):\n    if not os.path.exists(PID): return\n    pid = int(open(PID).read())\n    for sig in (signal.SIGTERM, signal.SIGKILL):\n        try: os.killpg(pid, sig)\n        except Exception: break\n        for _ in range(50):\n            try: os.killpg(pid, 0); time.sleep(0.1)\n            except Exception: break\n    os.remove(PID)\n    if not quiet: print(\'host stopped\')\n\nif __name__ == \'__main__\':\n    a = sys.argv[1:] or [\'help\']\n    if a[0] == \'start\': start(*a[1:])\n    elif a[0] == \'wait\': wait(*a[1:])\n    elif a[0] == \'log\': log(\'--all\' in a, \'--details\' in a, \'--since-start\' in a)\n    elif a[0] == \'stop\': stop()\n    else: print(__doc__)\n')
os.environ["FHOST"] = os.path.abspath("fhost.py")   # commands below run it as: python "$FHOST" ...
write('files_app/host.json', """\
{
  "version": "2.0",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  },
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Function": "Information",
      "Azure.Core": "Error",
      "Host.Function.Console": "Information"
    }
  }
}
""")
write('files_app/local.settings.json', """\
{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "AzureWebJobsStorage": "UseDevelopmentStorage=true"
  }
}
""")
write('files_app/requirements.txt', """\
azure-functions
azure-storage-blob
azure-data-tables
pyarrow
""")
write('containers.py', """\
import os
from azure.storage.blob import BlobServiceClient

svc = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for name in ["landing", "drops", "orders", "invoices", "quarantine",
             "raw", "curated", "incoming"]:
    if not svc.get_container_client(name).exists():
        svc.create_container(name)
print("containers ready")
""")
write('upload.py', 'import os, sys\nfrom azure.storage.blob import BlobServiceClient\n\nsvc = BlobServiceClient.from_connection_string(\n    os.environ["AZURE_STORAGE_CONNECTION_STRING"])\ncontainer, name, text = sys.argv[1:4]          # "|" = new line\nsvc.get_blob_client(container, name).upload_blob(\n    text.replace("|", "\\n") + "\\n", overwrite=True)\nprint("uploaded", f"{container}/{name}")\n')
write('grid.py', """\
import sys, requests

# Azurite raises no storage events, so this script plays Event Grid:
# it posts a BlobCreated event to the host's blob webhook.
container, blob = sys.argv[1].split("/", 1)
event = [{"id": "evt-" + blob, "eventType": "Microsoft.Storage.BlobCreated",
          "subject": f"/blobServices/default/containers/{container}"
                     f"/blobs/{blob}",
          "eventTime": "2026-11-01T09:00:00Z", "dataVersion": "",
          "data": {"api": "PutBlob", "blobType": "BlockBlob",
                   "url": "http://127.0.0.1:10000/devstoreaccount1/"
                          f"{container}/{blob}"}}]
r = requests.post("http://localhost:7071/runtime/webhooks/blobs",
                  params={"functionName": "Host.Functions." + sys.argv[2]},
                  headers={"aeg-event-type": "Notification"}, json=event)
print("Event Grid delivery:", r.status_code)
""")
write('receipts.py', """\
import os, sys
from azure.storage.blob import BlobServiceClient

hosts = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("azure-webjobs-hosts")
names = [b.name for b in hosts.list_blobs(name_starts_with="blobreceipts/")
         if f"/Host.Functions.{sys.argv[1]}/" in b.name]
for n in names:
    print(n)                           # .../<function>/<etag>/<blob>
print("receipts:", len(names), "| different ETags:",
      len({n.split("/")[3] for n in names}))
""")
write('where.py', """\
import os, sys
from azure.storage.blob import BlobServiceClient

svc = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"])
for c in sys.argv[1:]:
    names = sorted(svc.get_container_client(c).list_blob_names())
    print(f"{c}:", names)
""")
write('runs.py', """\
import re, sys

log = open("files_app/.fhost.log").read()          # the host console
runs = re.findall(f"Executing 'Functions.{sys.argv[1]}'", log)
print(f"{sys.argv[1]} runs so far:", len(runs))
""")
write('make_big.py', 'import os\nfrom azure.storage.blob import BlobServiceClient\nfrom azure.storage.queue import QueueClient, TextBase64EncodePolicy\n\nconn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]\npages = ["home", "product", "cart", "checkout"]\nrows = [f"{i},{pages[i % 4]}" for i in range(1, 1_000_001)]\ndata = ("click_id,page\\n" + "\\n".join(rows) + "\\n").encode()\nBlobServiceClient.from_connection_string(conn).get_blob_client(\n    "raw", "clicks_1101.csv").upload_blob(data, overwrite=True)\nprint("raw/clicks_1101.csv:", len(data), "bytes,", len(rows), "clicks")\nq = QueueClient.from_connection_string(\n    conn, "big-files", message_encode_policy=TextBase64EncodePolicy())\nq.create_queue()\nq.send_message("clicks_1101.csv")             # just the name\nprint("queued: clicks_1101.csv")\n')
write('partners_drop.py', """\
import subprocess, sys

H = "order_id,sku,qty,amount"
files = {"acme/orders_1101.csv": f"{H}|5001,tee-m,2,19.98|5002,mug,1,8.50",
         "globex/orders_1101.csv": "order_id;sku;qty;amount|7001;cap;1;12",
         "initech/orders_1101.csv": f"{H}|9001,hoodie,two,45.00"}
for name, text in files.items():
    subprocess.run([sys.executable, "upload.py", "incoming", name, text])
for name in files:                         # Event Grid: one event each
    subprocess.run([sys.executable, "grid.py", "incoming/" + name,
                    "validate_file"])
""")
write('report.py', """\
import json, os
from azure.storage.blob import BlobServiceClient
from azure.storage.queue import QueueClient, TextBase64DecodePolicy

conn = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
q = QueueClient.from_connection_string(
    conn, "to-parquet", message_decode_policy=TextBase64DecodePolicy())
todo = sorted(json.loads(m.content)["blob"]
              for m in q.peek_messages(max_messages=32))
print("to-parquet queue:", todo)
box = BlobServiceClient.from_connection_string(
    conn).get_container_client("quarantine")
for b in sorted(box.list_blobs(include=["metadata"]), key=lambda b: b.name):
    print("quarantine:", b.name, "| reason:", b.metadata["reason"])
""")
write('read_parquet.py', """\
import io, os
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient

box = BlobServiceClient.from_connection_string(
    os.environ["AZURE_STORAGE_CONNECTION_STRING"]
).get_container_client("curated")
for name in sorted(box.list_blob_names(name_starts_with="orders/")):
    table = pq.read_table(io.BytesIO(box.download_blob(name).readall()))
    print(name, "|", table.num_rows, "rows")
    for row in table.to_pylist():
        print("  ", row)
""")
run('python containers.py')

# Step 1: A blob trigger on landing/{name}.csv: upload an orders file and the function runs
write('files_app/function_app.py', """\
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
    rows = blob.read().decode().strip().splitlines()[1:]
    logging.info("%s: %d rows, %d bytes", blob.name, len(rows),
                 blob.length)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py landing orders_0901.csv "order_id,amount|1001,25.50|1002,14.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_landing\'" 60 && python "$FHOST" log')

# Step 2: The Event Grid source: an upload alone runs nothing; a BlobCreated event posted to the webhook runs it
write('files_app/function_app.py', """\
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="landing/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_landing(blob: func.InputStream) -> None:
    logging.info("polling source: %s", blob.name)


@app.blob_trigger(arg_name="blob", path="drops/{name}.csv",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
def on_drop(blob: func.InputStream) -> None:
    logging.info("event grid source: %s, %d bytes", blob.name,
                 blob.length)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py drops orders_0902.csv "order_id,amount|1003,9.50"')
run('sleep 5 && python runs.py on_drop')
run('python grid.py drops/orders_0902.csv on_drop')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_drop\'" 60 && python "$FHOST" log')

# Step 3: The same file uploaded twice with the same content: two runs, two receipts
write('files_app/function_app.py', """\
import logging
import azure.functions as func

app = func.FunctionApp()


@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
    logging.info("load %s, %d bytes", blob.name, blob.length)
    # ... append the rows to the orders table ...
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60')
run('python upload.py orders daily_1031.csv "order_id,amount|1001,25.50"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 2 && python "$FHOST" log')
run('python receipts.py on_order_file')

# Step 4: Idempotent: a content hash in a processed-files table; resend skipped, corrected file loaded
write('files_app/function_app.py', """\
import hashlib, logging, os
import azure.functions as func
from azure.core.exceptions import ResourceNotFoundError
from azure.data.tables import TableServiceClient

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
done = TableServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"]
).create_table_if_not_exists("processedfiles")


@app.blob_trigger(arg_name="blob", path="orders/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_order_file(blob: func.InputStream) -> None:
    key = blob.name.split("/", 1)[1].replace("/", "|")  # no / in keys
    digest = hashlib.sha256(blob.read()).hexdigest()
    try:
        seen = done.get_entity("orders", key)["sha256"]
    except ResourceNotFoundError:
        seen = None                                 # never loaded
    if seen == digest:
        logging.info("skip %s: same content", blob.name)
        return
    # ... load, overwriting this file's own rows ...
    done.upsert_entity({"PartitionKey": "orders", "RowKey": key,
                        "sha256": digest})
    logging.info("loaded %s", blob.name)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,30.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 2')
run('python upload.py orders daily_1101.csv "order_id,amount|2001,32.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_order_file\'" 60 3 && python "$FHOST" log')

# Step 5: A bad invoice fails five times, goes to webjobs-blobtrigger-poison, and a handler parks it in quarantine
write('files_app/function_app.py', """\
import logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="invoices/{name}.csv",
                  connection="AzureWebJobsStorage")
def on_invoice(blob: func.InputStream) -> None:
    header = blob.read().decode().splitlines()[0]
    if header != "invoice_id,amount":
        raise ValueError(f"bad header {header!r} in {blob.name}")
    logging.info("loaded %s", blob.name)


@app.queue_trigger(arg_name="msg", connection="AzureWebJobsStorage",
                   queue_name="webjobs-blobtrigger-poison")
def park_poison(msg: func.QueueMessage) -> None:
    info = msg.get_json()        # FunctionId, ContainerName, BlobName
    box, name = info["ContainerName"], info["BlobName"]
    src = blobs.get_blob_client(box, name)
    dst = blobs.get_blob_client("quarantine", f"{box}/{name}")
    dst.start_copy_from_url(src.url, metadata={"reason": "poison"})
    if dst.get_blob_properties().copy.status == "success":
        src.delete_blob()                           # same account
    logging.info("parked %s/%s in quarantine", box, name)
""")
run('cd files_app && python "$FHOST" start')
run('python upload.py invoices inv_0901.csv "invoice_id,amount|1,50.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.on_invoice\'" 60')
run('python upload.py invoices inv_0902.csv "id;amount|2;75.00"')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.park_poison\'" 90 && python "$FHOST" log')
run('python where.py invoices quarantine')

# Step 6: Stream a 13 MB file: 4 MiB chunks in, one staged block per chunk out
write('files_app/function_app.py', 'import logging, os\nimport azure.functions as func\nfrom azure.storage.blob import BlobBlock, BlobServiceClient\nfrom chunks import whole_lines\n\nlogging.getLogger("azure").setLevel(logging.WARNING)\napp = func.FunctionApp()\nMiB = 1024 * 1024\nblobs = BlobServiceClient.from_connection_string(\n    os.environ["AzureWebJobsStorage"],\n    max_single_get_size=4 * MiB, max_chunk_get_size=4 * MiB)\n\n\n@app.queue_trigger(arg_name="msg", queue_name="big-files",\n                   connection="AzureWebJobsStorage")\ndef filter_clicks(msg: func.QueueMessage) -> None:\n    name = msg.get_body().decode()               # clicks_1101.csv\n    src = blobs.get_blob_client("raw", name).download_blob()\n    dst = blobs.get_blob_client("curated", "checkout_" + name)\n    blocks, kept = [], 0\n    for lines in whole_lines(src):               # chunk by chunk\n        keep = [l + b"\\n" for l in lines if l.endswith(b",checkout")]\n        kept += len(keep)\n        blocks.append(BlobBlock(f"{len(blocks):05d}"))\n        dst.stage_block(blocks[-1].id, b"".join(keep))   # one block\n    dst.commit_block_list(blocks)       # the file appears only now\n    logging.info("%s: %d bytes in %d chunks, kept %d, %d blocks",\n                 name, src.size, len(blocks), kept, len(blocks))\n')
write('files_app/chunks.py', 'def whole_lines(download):\n    """Yield the complete lines of each downloaded chunk."""\n    rest = b""\n    for chunk in download.chunks():           # one chunk in memory\n        *lines, rest = (rest + chunk).split(b"\\n")\n        yield lines                           # rest: a partial line\n    if rest:\n        yield [rest]                          # no newline at the end\n')
run('cd files_app && python "$FHOST" start')
run('python make_big.py')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.filter_clicks\'" 120 && python "$FHOST" log')

# Step 7: Partner files: validate each one, good files to the to-parquet queue, bad files to quarantine
write('files_app/function_app.py', """\
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="incoming/{name}",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
    name = blob.name.split("/", 1)[1]           # acme/orders_1101.csv
    why = problem(blob.read().decode())
    if why is None:
        convert.set(json.dumps({"blob": name}))  # to the converter
        logging.info("valid %s -> to-parquet queue", name)
        return
    bad = blobs.get_blob_client("quarantine", name)
    bad.start_copy_from_url(blob.uri, metadata={"reason": why})
    logging.info("rejected %s: %s -> quarantine", name, why)
""")
write('files_app/checks.py', 'import pyarrow as pa\n\nHEADER = "order_id,sku,qty,amount"\nTYPES = {"order_id": pa.int64(), "sku": pa.string(),   # for Parquet\n         "qty": pa.int64(), "amount": pa.float64()}\n\n\ndef problem(text: str):\n    """None when the partner file is valid, else the reason."""\n    head, *rows = text.strip().splitlines() or [""]\n    if head != HEADER:\n        return "bad header"\n    if not rows:\n        return "no rows"\n    try:\n        [(int(r.split(",")[2]), float(r.split(",")[3])) for r in rows]\n    except (IndexError, ValueError):\n        return "bad qty or amount"\n    return None\n')
run('cd files_app && python "$FHOST" start')
run('python partners_drop.py')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.validate_file\'" 60 3')
run('cd files_app && python "$FHOST" log | grep -E \'^(valid|rejected)\' | sort')
run('python report.py')
run('cd files_app && python "$FHOST" stop')

# Step 8: The convert function (a blueprint): the queued file becomes Parquet with pyarrow; read it back
write('files_app/function_app.py', """\
import json, logging, os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
from checks import problem
from convert import bp

logging.getLogger("azure").setLevel(logging.WARNING)
app = func.FunctionApp()
app.register_functions(bp)
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])


@app.blob_trigger(arg_name="blob", path="incoming/{name}",
                  connection="AzureWebJobsStorage",
                  source=func.BlobSource.EVENT_GRID)
@app.queue_output(arg_name="convert", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def validate_file(blob: func.InputStream, convert: func.Out[str]):
    name = blob.name.split("/", 1)[1]           # acme/orders_1101.csv
    why = problem(blob.read().decode())
    if why is None:
        convert.set(json.dumps({"blob": name}))  # to the converter
        logging.info("valid %s -> to-parquet queue", name)
        return
    bad = blobs.get_blob_client("quarantine", name)
    bad.start_copy_from_url(blob.uri, metadata={"reason": why})
    logging.info("rejected %s: %s -> quarantine", name, why)
""")
write('files_app/convert.py', """\
import io, logging, os
import azure.functions as func
import pyarrow.csv as pv
import pyarrow.parquet as pq
from azure.storage.blob import BlobServiceClient
from checks import TYPES                # the same partner contract

bp = func.Blueprint()               # registered in function_app.py
blobs = BlobServiceClient.from_connection_string(
    os.environ["AzureWebJobsStorage"])
OPTS = pv.ConvertOptions(column_types=TYPES)


@bp.queue_trigger(arg_name="msg", queue_name="to-parquet",
                  connection="AzureWebJobsStorage")
def convert_to_parquet(msg: func.QueueMessage) -> None:
    name = msg.get_json()["blob"]              # acme/orders_1101.csv
    src = blobs.get_blob_client("incoming", name).download_blob()
    table = pv.read_csv(io.BytesIO(src.readall()),
                        convert_options=OPTS)    # no type guessing
    out = io.BytesIO()
    pq.write_table(table, out)
    target = "orders/" + name.replace(".csv", ".parquet")
    blobs.get_blob_client("curated", target).upload_blob(
        out.getvalue(), overwrite=True)        # rerun: same file
    logging.info("%s -> curated/%s: %d rows", name, target,
                 table.num_rows)
    logging.info("schema: %s", ", ".join(
        f"{f.name} {f.type}" for f in table.schema))
""")
run('cd files_app && python "$FHOST" start')
run('cd files_app && python "$FHOST" wait "Executed \'Functions.convert_to_parquet\'" 60 && python "$FHOST" log')
run('python read_parquet.py')
run('cd files_app && python "$FHOST" stop')
Next in this series

Part 4: Queues, Event Hubs and Service Bus Triggers

The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run with Azure Functions Core Tools 4.16.0 (the real Functions host, Python 3.13 worker) and Azurite 3.37.0 for storage. Features the local host cannot show truthfully are shown as code. Part of the Real Time Azure Functions Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of November 2026; check the official Azure Functions documentation (learn.microsoft.com/azure/azure-functions) for anything you quote in an interview.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.