In this post we are going to cover ADF architecture and core concepts: pipelines, activities, datasets, linked services, integration runtimes and triggers, and how they connect. These are 7 real Azure Data Factory interview questions, asked the way they come up on a real project, and the last one is a scenario where a business user wants a sales table copied from on-premises SQL Server to the data lake every night.
Azure Data Factory, or ADF, is the Azure service we use to move data between stores and orchestrate the steps around it. Interviewers usually start here, because every other ADF question (runtimes, copy performance, incremental loads, CI/CD) builds on these few objects. They also like to ask where ADF stands now that Data Factory in Microsoft Fabric exists, so we cover that too.
Part 1 of the Real Time Azure Data Factory Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Question 1: A new teammate asks you to explain how Azure Data Factory is put together. What are pipelines, activities, datasets, linked services, integration runtimes and triggers, and how do they connect?
- Question 2: Your manager reads that Data Factory in Microsoft Fabric is the next generation and asks: should we stop building in ADF? What do you tell her?
- Question 3: A developer hard-codes the same server name in twenty pipelines. What is the difference between pipeline parameters, variables and global parameters, and which one fixes this?
- Question 4: You open a pipeline JSON and see expressions like @pipeline().RunId and @activity('Lookup1').output. How does the ADF expression language work, and what are system variables?
- Question 5: A pipeline that works in debug fails when the trigger runs it. What is the difference between a debug run, a trigger run and a published pipeline?
- Question 6: Your company already uses Synapse pipelines for one project and ADF for another. What is the difference between ADF and Synapse pipelines?
- Scenario 7: A business user asks you to copy a sales table from on-premises SQL Server to the data lake every night. Walk me through the ADF objects you create, in order.
The short answer
A trigger starts a pipeline, the pipeline runs activities, activities read and write datasets, datasets point to linked services, and linked services run on an integration runtime. ADF is fully supported; Fabric Data Factory is the next generation. Triggers run only the published version, while debug runs what is on your canvas.
Keep that chain in mind. Every question below is one piece of it.
Question 1: A new teammate asks you to explain how Azure Data Factory is put together. What are pipelines, activities, datasets, linked services, integration runtimes and triggers, and how do they connect?
It is a chain: a trigger starts a pipeline, the pipeline runs activities, activities use datasets, datasets point to linked services, and linked services run on an integration runtime.
Let's take the pieces one at a time.
- Pipeline. A logical group of activities that does one unit of work, like loading the sales data.
- Activities. The steps inside it: copy data, transform it, or control the flow with ForEach and If Condition.
- Linked service. It works like a connection string to a data store or a compute service.
- Dataset. A named view of data inside that store, like one table, file or folder. Activities use datasets as their inputs and outputs.
- Integration runtime. The compute that actually does the work. The linked service points to it with connect via. There are three types, Azure, self-hosted and Azure-SSIS, and they get Part 2 of this series.
- Trigger. It decides when the pipeline runs: on a schedule, on a tumbling window, or on a storage or custom event.
Each execution of a pipeline is a pipeline run, with its own run ID. That run ID is how you find it again in monitoring.
Draw the chain on the whiteboard: trigger, pipeline, activities, datasets, linked services, integration runtime. If you can draw it and explain each arrow, you've answered the question.
Question 2: Your manager reads that Data Factory in Microsoft Fabric is the next generation and asks: should we stop building in ADF? What do you tell her?
Facts first. Azure Data Factory is fully supported. It is not deprecated, there is no retirement date, and there is no forced upgrade.
What Microsoft does say is that Data Factory in Fabric is the next generation, new innovation lands there, and if you are new to data integration, you should start with Fabric Data Factory.
So here is what I tell her. Our existing factories keep running, and we plan an upgrade on our own timeline. The upgrade tooling, View in Fabric and assess and upgrade, is still in preview.
For a new project, I would look at Fabric first, unless we need something only ADF gives us:
- The Azure-SSIS integration runtime.
- A managed virtual network with managed private endpoints.
- Tumbling window backfill.
- Pay-per-use billing without a Fabric capacity.
Then I tell her what changes if we move. Let's put the two side by side.
| Azure Data Factory | Fabric Data Factory | |
|---|---|---|
| STATUS | fully supported, no retirement date | the next generation, new innovation |
| MODEL | PaaS, pay as you go | SaaS, Fabric capacity |
| DATA ACCESS | linked services and datasets | connections only, no datasets |
| DEPLOY | publish, ARM templates | save and run, deployment pipelines |
| ON-PREMISES | self-hosted IR | on-premises data gateway |
| CONSTANTS | global parameters | variable libraries |
In short: no datasets, only connections; no publish step; the self-hosted runtime becomes the on-premises data gateway; and global parameters become variable libraries.
Question 3: A developer hard-codes the same server name in twenty pipelines. What is the difference between pipeline parameters, variables and global parameters, and which one fixes this?
The fix is a global parameter. But let's first see what each of the three does.
- Pipeline parameters are inputs. They are passed in when the run starts, by a trigger, a parent pipeline or by hand, and they are read-only during the run. You can have up to 50 per pipeline.
- Variables are the pipeline's working state. You change them during the run with Set Variable and Append Variable.
- Global parameters are constants for the whole factory. You define them once in the Manage hub, and any pipeline in the factory reads them in an expression.
| Pipeline parameter | Variable | Global parameter | |
|---|---|---|---|
| SCOPE | one pipeline run | one pipeline run | the whole factory |
| CHANGES IN A RUN? | no, read-only | yes, Set Variable | no, a constant |
| SET BY | trigger, parent, manual run | Set or Append Variable | Manage hub, CI/CD |
Here is how each one is read in an expression:
// pipeline parameter: read-only input
@pipeline().parameters.TableName
// variable: changes during the run
@variables('RowCount')
// global parameter: one value per factory
@pipeline().globalParameters.SqlServerNameSo for the server name, create one global parameter and pass it into one parameterized dataset and linked service. Now a server change is one edit, not twenty.
In CI/CD, include the global parameters in the ARM template, so each environment gets its own server name. One catch: Synapse pipelines don't have global parameters.
Variables are scoped to the whole pipeline, not to one loop iteration. Never use Set Variable inside a parallel ForEach.
Question 4: You open a pipeline JSON and see expressions like @pipeline().RunId and @activity('Lookup1').output. How does the ADF expression language work, and what are system variables?
A value that starts with @ is an expression, evaluated at run time. Let's go through the parts you will see in real pipelines.
First, system variables. @pipeline().RunId is the unique ID of this run. Others give the pipeline name and the trigger time, name and type. The trigger type says Manual for an on-demand run.
Then trigger values, like the scheduled time, the window start or the file name. They live in the trigger, and the trigger maps them to pipeline parameters.
Then the activity function. It reads an earlier activity's output by name: firstRow, or value for all rows, or rowsCopied for a copy. It only sees activities that run before the current one.
// system variables
@pipeline().RunId
@pipeline().Pipeline
@pipeline().TriggerType
@pipeline().parameters.WindowStart // set by trigger
// activity output
@activity('Lookup1').output.firstRow.Watermark
@activity('Copy1').output.rowsCopied
// string interpolation always returns a string
sales_@{formatDateTime(utcNow(),'yyyyMMdd')}.csvFunctions do the real work: concat, formatDateTime, utcNow, addDays, coalesce. And a few syntax rules come up again and again:
@starts an expression.@{ }inside text is string interpolation, and it always returns a string.?.before a property that may be missing returns null instead of failing.- Two single quotes put one quote inside a string.
Question 5: A pipeline that works in debug fails when the trigger runs it. What is the difference between a debug run, a trigger run and a published pipeline?
A debug run runs the version on your canvas, unpublished changes included. A trigger run only ever runs the published version, the one that is live in the factory.
Publishing is what makes changes live. In Git mode, you merge to the collaboration branch and publish from there. Trigger now also runs the published version, not your canvas.
| Debug run | Trigger run | |
|---|---|---|
| RUNS | the canvas version, unpublished changes | the published version only |
| PARAMETERS | the values you typed | trigger values or defaults |
| TRIGGER METADATA | only what you typed | scheduled time, window, mapped to parameters |
| METRICS, ALERTS | none | yes |
Now the troubleshooting. These are the three causes I check, in order:
- Not published. Something like a linked service, a dataset or a child pipeline was never published. Debug used your new version, and the trigger used the old one. This is the most common cause.
- Parameters. In debug, you typed the values. The trigger passes its own, or the defaults. A parameter the trigger fills from its scheduled time or window gets that value only in a trigger run.
- Timing. At 2 AM the source may be busy, or the file may not be there yet.
Compare the failed run with your debug run in the Monitor hub.
Debug runs don't emit metrics, so alerts only see triggered runs.
Question 6: Your company already uses Synapse pipelines for one project and ADF for another. What is the difference between ADF and Synapse pipelines?
Synapse pipelines are based on Azure Data Factory. Same pipelines, activities, triggers and mapping data flows, so the skills transfer directly.
The difference is where they live. ADF is a standalone service. Synapse pipelines live inside a Synapse workspace, next to the SQL pools, Spark pools and notebooks they usually orchestrate.
Some features are ADF only: global parameters, sharing one self-hosted runtime across factories, a cross-region runtime for data flows, and deploying pipelines with the ADF style ARM templates. Synapse has its own strength too. Its managed virtual network can block outbound traffic, which is data exfiltration protection. In ADF, the managed network leaves outbound ports open.
| Azure Data Factory | Synapse pipelines | |
|---|---|---|
| GLOBAL PARAMETERS | yes | no |
| SHARED SELF-HOSTED IR | yes, across factories | no |
| CROSS-REGION DATA FLOW IR | yes | no |
| OUTBOUND BLOCKING | no, outbound ports open | yes, exfiltration protection |
Both have an upgrade path to Fabric, in preview.
With both in one company, keep the same patterns in both, like parameterized pipelines and a control table. Then moving between them, or to Fabric later, stays cheap.
Scenario 7: A business user asks you to copy a sales table from on-premises SQL Server to the data lake every night. Walk me through the ADF objects you create, in order.
This question checks if you know the objects and the order they depend on each other. Let's build it step by step.
- First, the integration runtime. The data is on-premises, so I install a self-hosted runtime on a Windows machine that can reach SQL Server, and register it with its key.
- Second, the linked services. SQL Server through the self-hosted runtime, with the password in Key Vault. ADLS Gen2 with the factory's managed identity, which needs Storage Blob Data Contributor on the lake.
- Third, the datasets. The sales table, and a Parquet dataset in the lake with the folder parameterized by date.
- Fourth, the pipeline. One copy activity. I debug it, then publish.
- Fifth, a schedule trigger. Every day at 1 AM in our time zone. I start it, and publish again.
- Last, an alert on failed pipeline runs.
So the flow is SQL Server, through the self-hosted runtime, into ADLS Gen2 as Parquet by date, with the SQL password in Key Vault and the lake reached by managed identity.
Parquet on a self-hosted runtime needs Java on that machine. Install it before the first run.
When the table grows, I switch the full copy to an incremental load, which we cover in Part 4 of this series.
Key points to remember
- Trigger, pipeline, activities, datasets, linked services, integration runtime
- ADF is fully supported; Fabric is the next generation
- Parameters are inputs, variables change, global parameters are factory constants
- Expressions start with @; system variables like RunId
- Triggers run only the published version
Part 2: Integration Runtimes and Security
Part of the Real Time Azure Data Factory Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the Microsoft Learn page for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.