In this post we are going to cover lineage in Microsoft Purview: how Purview shows where your data comes from and where it goes, across Azure Data Factory, Microsoft Fabric, Databricks and Power BI. These are 7 real Purview interview questions, asked the way they come up on a real project, and the last one is a scenario where a source column is going to be renamed and you need to find everything that will break.
Lineage is a map of how data moves: which source feeds which table, which table feeds which model, and which model feeds which report. Interviewers ask about it because auditors and managers ask for it all the time, and because every tool gives you a different level of lineage. Knowing where the gaps are is what separates a real answer from a sales pitch.
Part 5 of the Real Time Microsoft Purview Interview Questions and Answers series on TechBrothersIT.
- The short answer
- Question 1: An auditor asks where the revenue number on the board report comes from. How does lineage in Purview answer that?
- Question 2: Your Azure Data Factory pipelines load the lakehouse. How does ADF lineage get into Purview, and which activities are covered?
- Question 3: The data team builds everything in Microsoft Fabric. What lineage does Purview show for Fabric items, and at what level?
- Question 4: Your transformations run in Databricks notebooks. When do you get table and column lineage from Unity Catalog, and when don't you?
- Question 5: A Power BI report reads from Azure SQL through a semantic model. What lineage do you see, and when is it column level?
- Question 6: Half of your ETL runs in a custom Python tool that Purview doesn't know. How do you add custom lineage?
- Scenario 7: A source column is going to be renamed next week. How do you use lineage to find everything that will break?
The short answer
Lineage lives in the Data Map: pushed at run time by ADF and Synapse (Copy, Data Flow, SSIS only) and scanned from sources like Databricks, Power BI and Fabric. Fabric is item level, Databricks column lineage needs cluster runs, and Power BI goes to column level only from Azure SQL. Fill the gaps with custom lineage through the Atlas API.
Keep that in mind. Every question below is one piece of it.
Question 1: An auditor asks where the revenue number on the board report comes from. How does lineage in Purview answer that?
Lineage lives in the Data Map, as a graph of assets and the processes between them. You open it from an asset's Lineage tab in Unified Catalog.
So let's start at the board report and walk upstream:
- The board report, in Power BI, reads a semantic model with the Revenue measure.
- The semantic model reads a sales table in the lakehouse.
- An ADF copy, a pipeline run, loaded that table.
- The copy read the ERP database in Azure SQL.
That chain comes from two kinds of lineage. Push-based lineage is sent at run time by the tool that moves the data. Scan-based lineage is read when Purview scans the source.
| Push-based (run time) | Scan-based | |
|---|---|---|
| SENT BY | ADF, Synapse pipelines, Data Share, Airflow | Databricks, Snowflake, Oracle, Teradata |
| ALSO | Azure SQL stored procedures (preview) | BigQuery, SAP, Hive, Power BI, Fabric |
Where the source supports it, you drill down to column level, so you can show the auditor which source column feeds revenue.
Now the honest part. Lineage is only as complete as what you connected, and one custom script breaks the chain.
Lineage shows the path, not the business rule. Pair it with a glossary term that defines revenue.
Question 2: Your Azure Data Factory pipelines load the lakehouse. How does ADF lineage get into Purview, and which activities are covered?
You connect the data factory to your Purview account, and lineage is pushed every time a pipeline runs.
Let's go through it in order. First, connect the factory. Its managed identity gets the Data Curator role on the root collection, and the portal adds that for you when you connect. Then, when a pipeline runs, it sends its source, sink and process to the Data Map, a little while after the run. Nothing is scanned.
Only three activities are covered:
- Copy Data, for tables and views
- Data Flow, source and sink only
- Execute SSIS Package
And here is what gives you no lineage:
- Notebook, Lookup and Stored Procedure activities
- A query or a stored procedure as the source of a copy
- Flowlets in a data flow
So even inside the three covered activities there are limits. A data flow shows the source and the sink, not the transformations in between.
Synapse pipelines work the same way, with Copy and Data Flow only. Spark notebooks are not covered.
A pipeline that calls a notebook leaves a gap in the chain. Plan to fill it with custom lineage.
Question 3: The data team builds everything in Microsoft Fabric. What lineage does Purview show for Fabric items, and at what level?
For Power BI items you get the full Power BI lineage. For every other Fabric item, Purview shows item level lineage only.
First, register the Fabric tenant as a source in the Data Map and scan it. The scan brings in items and lineage, and it covers Power BI too, because Power BI items are part of the Fabric tenant.
Then let's look at what you get. Pipelines, notebooks, lakehouses and warehouses show up as items. So you see that a pipeline feeds a lakehouse, and the lakehouse feeds a semantic model, which feeds a report. The lakehouse tables and files are scanned as metadata, but there is no table to table lineage inside the lakehouse.
For items outside Power BI, here are the gaps to know:
- No table to table lineage inside a lakehouse
- No external upstream sources
- No lineage across workspaces
- No notebook to pipeline lineage
The Fabric admin must allow service principals to use the read-only admin APIs and turn on detailed metadata in the admin API responses. Then wait about 15 minutes before you scan.
Question 4: Your transformations run in Databricks notebooks. When do you get table and column lineage from Unity Catalog, and when don't you?
Purview scans Unity Catalog and reads lineage from its system tables: table lineage and column lineage, in the system.access schema. So the scan identity needs select on those two tables.
Now the key point: what you get depends on how the code runs.
| How it runs | Lineage in Purview |
|---|---|
| Notebook on a cluster | tables, views and columns |
| Notebook run as a job | column lineage might be missing |
| Query on a SQL warehouse | no column level lineage |
| External table | no lineage |
| Notebook triggered from ADF | pipeline link not captured |
So a notebook on a cluster gives you tables, views and columns. Run the same notebook as a job, and column lineage might be missing. Queries on a SQL warehouse give no column level lineage, external tables give no lineage, and when ADF triggers the notebook, the link from the pipeline isn't captured.
There are two more surprises on a real project:
- Notebooks show up as numeric IDs, not names.
- Lineage is partial unless you scan every workspace that touches those tables.
So in an interview I say it like this: run production transformations as notebooks on clusters, check column lineage for job runs, prefer managed tables, and add custom lineage for the hop from ADF.
Question 5: A Power BI report reads from Azure SQL through a semantic model. What lineage do you see, and when is it column level?
You see the item chain, and because the source is Azure SQL Database, you also get column level lineage.
Purview gets Power BI lineage from the Fabric tenant scan. The chain goes like this: the Azure SQL table, then the semantic model, then the report, and a dashboard if one is built from it. Dataflows show up in that chain too.
With Azure SQL Database as the source, you also get sub-artifact lineage: which table columns feed which model columns and measures. That is the only source where Power BI lineage goes down to column level.
| Source or setup | Column level lineage? |
|---|---|
| Azure SQL Database | yes: columns and measures |
| Any other source | no: item level at most |
| Dedicated SQL pool | not supported |
| Dynamic M query parameters | no lineage |
So read from any other source, and you lose column level. A dedicated SQL pool source isn't supported. And if the model uses dynamic M query parameters, there is no lineage at all.
One more detail: SQL views show up as tables in this lineage. If the model reads a view, don't expect to see the view's own logic.
If column lineage into reports matters for audit, that's an argument for a governed Azure SQL layer in front of Power BI.
Question 6: Half of your ETL runs in a custom Python tool that Purview doesn't know. How do you add custom lineage?
You create the lineage yourself, as Process entities through the Atlas API.
Purview lineage is built on Apache Atlas. A lineage step is a Process entity, with inputs and outputs that point to assets already in the Data Map. At scale, your tool calls the Atlas bulk entity API and posts one process per job, with its input tables and its output tables:
POST {endpoint}/datamap/api/atlas/v2/entity/bulk
?api-version=2023-09-01
{ "entities": [ {
"typeName": "Process",
"attributes": {
"qualifiedName": "pyetl://jobs/load_revenue",
"name": "load_revenue",
"inputs": [
{ "guid": "guid-of-erp-sales-table" } ],
"outputs": [
{ "guid": "guid-of-lakehouse-sales-table" } ]
} } ] }Here is what you need to know, step by step:
- Process entity: inputs and outputs, pointing to existing assets.
- Service principal: Data Curator role on the collection. Azure roles alone aren't enough; it has to be a Purview data plane role.
- Manual lineage: for a handful of links, under about 20 to 30, draw it in the portal instead.
- Manual column lineage: only between two assets, not through a process.
Also, types with automatic lineage don't allow manual lineage. That covers ADF, Synapse pipelines, Power BI semantic models and SQL stored procedures.
Find each input and output in the Data Map by its exact qualified name, and use that asset's unique ID. Otherwise the process won't connect to the scanned assets.
Scenario 7: A source column is going to be renamed next week. How do you use lineage to find everything that will break?
Let's do it in five steps.
- Lineage tab. Find the source table in Unified Catalog, open its Lineage tab and expand everything downstream. Switch to column level where it's available, and follow that one column.
- List every asset. Write down every downstream asset and process: ADF copy activities with column mappings, lakehouse and Databricks tables, semantic models and reports.
- Check the gaps by hand. Lineage won't show them, so check these yourself: data flow internals, Fabric items below item level, SQL warehouse queries in Databricks, and any custom code without custom lineage.
- Notify owners. Use the catalog to find the people. Assets and data products have owners, so you notify them, and check if the column is mapped to a critical data element, which is still in preview.
- Rescan and verify. After the change, rerun the pipelines, rescan, and check the lineage again.
This is where the earlier answers come together: the gaps from the other questions are exactly what you check by hand here.
Lineage is your starting list, not a guarantee. Test the rename in a lower environment first.
Key points to remember
- Lineage lives in the Data Map: pushed or scanned
- ADF and Synapse: Copy, Data Flow, SSIS only
- Fabric: item level; Databricks columns need clusters
- Power BI column level only from Azure SQL
- Gaps: custom lineage with the Atlas API
Part 6: Data Quality and Health Management
Part of the Real Time Microsoft Purview Interview Questions and Answers series on TechBrothersIT. Product names, features and their status (GA or preview) are current as of October 2026. Microsoft changes Purview every month, so check the Microsoft Learn page for anything you quote in an interview.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.