In this post we are going to cover Data Factory in Microsoft Fabric: what changes from Azure Data Factory, Copy job versus pipelines versus Dataflow Gen2, gateways, incremental loads and migration from ADF. These are 7 real Microsoft Fabric interview questions, asked the way they come up on a real project, and the last one is a scenario where we build a nightly pipeline that loads bronze, runs notebooks for silver and gold, refreshes the model and alerts on failure.
Data Factory in Fabric is the data integration part of Fabric: pipelines, Copy job and Dataflow Gen2, all running on your Fabric capacity. Interviewers ask about it because most teams arrive with Azure Data Factory experience, and they want to know if you understand what carries over, what is new, and which tool fits which job.
Part 6 of the Real Time Microsoft Fabric Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Question 1: Your team knows Azure Data Factory. What is the same and what is different in Fabric Data Factory?
- Question 2: You need to load fifty tables from an on-premises SQL Server every night. Pipeline copy activity, copy job or Dataflow Gen2: which one?
- Question 3: A business analyst wants to clean data with Power Query and land it in the lakehouse. How does Dataflow Gen2 work, and when is it the wrong tool?
- Question 4: The source is behind the company firewall. How do the on-premises data gateway and the virtual network data gateway work?
- Question 5: You need only changed rows each night. How do you do incremental loads in Fabric pipelines and copy jobs?
- Question 6: Leadership wants all ADF pipelines moved to Fabric. What is the migration path, and what doesn't move?
- Scenario 7: The nightly pipeline must load bronze, run notebooks for silver and gold, refresh the model and alert on failure. How do you build it?
The short answer
Fabric Data Factory keeps ADF's pipelines and copy activity, but drops datasets, publish and integration runtimes: connections and gateways replace them. Use a Copy job for plain bulk and incremental copies, Dataflow Gen2 for Power Query shaping, and pipelines to orchestrate notebooks, refreshes and alerts.
Keep that in mind. Every question below is one piece of it.
Question 1: Your team knows Azure Data Factory. What is the same and what is different in Fabric Data Factory?
The core is the same, and the plumbing is simpler.
What stays the same: pipelines, activities, the expression language, parameters and the copy activity. About 90 percent of ADF activities are available in Fabric, so the team's skills carry over.
Now the differences. Let's put them side by side:
| Azure Data Factory | Fabric Data Factory | |
|---|---|---|
| DATASETS | separate objects | gone: set inline in the activity |
| CONNECT | linked services | connections |
| DEVELOP | publish, debug mode, Azure IR | save and run; no runtime to manage |
| ON-PREMISES | self-hosted IR | on-premises or VNet data gateway |
| SCHEDULE | tumbling window trigger | interval schedule |
| DEPLOY | ARM templates | Git and deployment pipelines |
Datasets are gone, and their properties sit inline in the activity. Linked services become connections. There is no publish step and no debug mode: you save and run. And there is no Azure integration runtime to manage. For on-premises sources, the self-hosted integration runtime is replaced by the on-premises data gateway or the virtual network data gateway.
Fabric also adds things ADF never had:
- Copy job and Dataflow Gen2
- Mirroring, in place of Synapse Link
- Activities for Teams, Outlook and semantic model refresh
And it all runs on your Fabric capacity.
The Azure-SSIS integration runtime and managed virtual network private endpoints aren't there in the same form, and mapping data flows only map to a preview feature.
Question 2: You need to load fifty tables from an on-premises SQL Server every night. Pipeline copy activity, copy job or Dataflow Gen2: which one?
For fifty tables of plain copying, my first choice is a Copy job. It's the go-to data mover without a pipeline: pick the tables, choose full or incremental, and it creates the destination tables.
It also keeps state for each table. It resumes from the last successful run, you can reset one table to a full copy, and the update method can be append, overwrite, merge or SCD Type 2.
| Copy job | Pipeline copy | Dataflow Gen2 | |
|---|---|---|---|
| BEST FOR | bulk tables, full or incremental | orchestration, control table | Power Query shaping |
| STATE | kept per table, resumes | you build it | incremental refresh |
| 50 TABLES | first choice | when you need control | wrong tool |
Choose a pipeline with a copy activity when you need orchestration around the copy: a control table, a ForEach over the tables, a custom query per table, and notebooks or stored procedures after it. That's the metadata-driven pattern from ADF.
Dataflow Gen2 is the wrong tool here. It's built for shaping data with Power Query, not for moving fifty tables as they are.
All three reach the SQL Server through the on-premises data gateway, and land the data in the bronze lakehouse.
A Copy job lands bronze, and pipelines run the transformations.
Question 3: A business analyst wants to clean data with Power Query and land it in the lakehouse. How does Dataflow Gen2 work, and when is it the wrong tool?
Dataflow Gen2 is Power Query in Fabric: over 300 transforms, low code, and a data destination like a lakehouse table. It fits an analyst who already knows Power Query from Excel or Power BI.
Let's look at what happens under the hood. A dataflow reads from a source (files, databases, SaaS apps), shapes the data, and writes to the destination. Along the way it can stage:
- Intermediate results go to a hidden staging lakehouse.
- Transforms run on a staging warehouse SQL engine.
- For large sources, fast copy, when enabled, switches to the copy engine automatically, above about 100 MB of files or 5 million database rows.
Since April 2026, every new Dataflow Gen2 comes with Git and deployment pipeline support. Saved changes apply on the next refresh. And incremental refresh needs a date column and a source that folds.
Now when it is the wrong tool:
| Right fit | Wrong tool | |
|---|---|---|
| WHO | analyst shaping with Power Query | heavy, complex transforms: notebook |
| TABLE | only this dataflow writes it | other writers, like Spark |
Very large or complex transformations belong in a Spark notebook, which gives you more control. And incremental refresh expects no other writer, like Spark, on the same destination table.
Dataflow Gen2 for analyst-owned shaping, notebooks for heavy engineering transforms.
Question 4: The source is behind the company firewall. How do the on-premises data gateway and the virtual network data gateway work?
There are two gateways, and the source decides which one you use.
The on-premises data gateway is software you install on a machine inside your network that can reach the source. It connects out to Fabric, so you don't open inbound firewall ports. Pipelines, Copy job and Dataflow Gen2 all use it.
It replaces the self-hosted integration runtime from ADF. A few things to know:
- Keep it current. Pipelines need a minimum gateway version, so an outdated gateway is the first thing to check when a pipeline can't use the connection.
- For high availability, install it on more than one machine as a gateway cluster, close to the source.
The virtual network data gateway is for Azure sources behind a virtual network or private endpoints, like an Azure SQL database with only a private endpoint. There's no machine to patch: you create it in Fabric and link it to a delegated subnet of your virtual network.
On-premises sources: the on-premises gateway. Azure sources with public access off: the virtual network data gateway, with no server to maintain.
Question 5: You need only changed rows each night. How do you do incremental loads in Fabric pipelines and copy jobs?
The easiest way is a Copy job in incremental mode.
| Copy job | How it works |
|---|---|
| WATERMARK | rowversion, datetime, date or integer column; remembers the last value |
| CDC | reads the changes from the source, including deletes |
| UPDATE METHOD | append, overwrite, merge, SCD Type 2 |
You pick a watermark column per table: a rowversion, datetime, date or integer. The Copy job remembers the last value and copies only newer rows. If the source has change data capture enabled, a Copy job can read the changes instead, and that's how you catch deletes too. With the merge update method, changed rows are updated in the destination, not appended again.
In a pipeline, it's the classic ADF pattern, in four steps:
- A Lookup reads the old watermark from a control table.
- A second Lookup reads the new maximum.
- The copy source query selects the rows in between.
- A last step, a script or stored procedure, saves the new watermark.
Here is the copy source query:
SELECT order_id, amount, status, modified_at
FROM dbo.Orders
WHERE modified_at >
'@{activity('OldWM').output.firstRow.wm}'
AND modified_at <=
'@{activity('NewWM').output.firstRow.wm}'Now the pitfalls:
- Use greater than the old value, not greater than or equal, or the boundary rows load twice.
- Save the watermark only after the copy succeeds.
- Land into a MERGE, not a plain append, so a rerun can't duplicate rows.
Start with Copy jobs, and keep pipeline watermarks for the sources they can't handle.
Question 6: Leadership wants all ADF pipelines moved to Fabric. What is the migration path, and what doesn't move?
Don't start by rebuilding. Start with an assessment.
First, in ADF Studio, View in Fabric, in preview, opens an assess and upgrade experience. The readiness assessment is read-only and covers the whole factory.
Then upgrade pipeline by pipeline: map each linked service to a Fabric connection, and the tool converts the pipeline. A PowerShell module also exists, but Microsoft now recommends the built-in experience for most cases.
If you need time, mount the factory first. The Azure Data Factory item shows your existing factory inside a Fabric workspace, live. Nothing is rebuilt, and you can unmount it any time.
What doesn't move as is:
| In ADF | In Fabric | |
|---|---|---|
| DATASETS | separate objects | inline settings |
| RUNTIME | self-hosted IR | data gateway |
| TRIGGERS | tumbling window | interval schedule |
| DATA FLOWS | mapping data flows | eligible ones: Dataflow Gen2 (preview) |
| SSIS | Azure-SSIS IR | not there; Invoke SSIS activity (preview) |
Mapping data flows convert only where eligible, to a preview feature. And the Azure-SSIS runtime isn't there; an Invoke SSIS package activity is in preview.
My plan is waves:
- Wave 1: simple copy pipelines.
- Wave 2: pipelines that need a gateway.
- Wave 3: SSIS and data flow pipelines.
- Run side by side: compare row counts, then switch off ADF.
Scenario 7: The nightly pipeline must load bronze, run notebooks for silver and gold, refresh the model and alert on failure. How do you build it?
I build one pipeline on a schedule. Let's build it step by step.
- Bronze. A copy activity, or a ForEach over a control table, lands the raw data in the bronze lakehouse.
- Silver. On success, a Notebook activity builds silver.
- Gold. Then a second Notebook activity builds gold.
- Model refresh. When gold succeeds, a semantic model refresh activity updates the model for tonight's data.
I pass the run date as a notebook parameter, so a rerun for a past day is just a parameter change. And I set retries with backoff on the copy and notebook steps.
For alerts, a Teams or Outlook activity goes on the failure path. Here is the trap: wire several fail outputs into one alert, and it runs only when all of them fail.
So I put the whole load in a child pipeline, call it with Invoke Pipeline, waiting on completion, and alert once on its failure:
"dependsOn": [
{
"activity": "Invoke_Nightly_Load",
"dependencyConditions": [ "Failed" ]
}
]Finally, log each run's row counts to a table, and test the failure path on purpose once.
An alert you've never seen fire is an alert you can't trust.
Key points to remember
- Same pipelines; no datasets, publish or IRs: connections and gateways
- Copy job for bulk and incremental copies; pipelines to orchestrate
- Dataflow Gen2 for Power Query shaping, not heavy transforms
- Incremental: watermark or CDC, then MERGE; watermark on success only
- Migrate: assess, upgrade or mount; SSIS and data flows last
Part 7: Incremental Loads, MERGE and SCD
Part of the Real Time Microsoft Fabric Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the Microsoft Learn page for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.