REST API Fetching only Required Records and Writing to Azure SQL DB in Azure Data Factory

 Topic: REST API Fetching only Required Records and Writing to Azure SQL DB in Azure Data Factory.

In this article, we are going to learn about REST API fetching only required records and writing them to the Azure SQL database in Azure Data Factory, in this scenario we are going to fetch data from a website, that provides the free Dummy data, so I have copied the web link and then get data from that link then write it to the Azure SQL database. let's start our demonstration.

Open the Azure Data factory studio, go to the Author tab, click on + sign and create a new pipeline, find and bring the Lookup activity, then go to the settings tab and click on the + New button to create a new source dataset.

Select Azure SQL database, then click on continue.

Name your dataset, select the linked service, select none for import schema, then click on Ok.

Next, create a table in SSMS, then in the Settings tab select the Query, provide the SQL query and uncheck the First row only.

Once we are done with our lookup activity, find and bring the ForEach loop activity, connect with the lookup activity, go to the settings tab, in the items click on add dynamic content, then select the lookup and in the end add ''.value''.

Go inside the ForEach loop activity, find and bring the copy data activity, go to the source tab, then click on the + New button to create a new source dataset.

Select the REST then click on continue.

Name your dataset, then create a new linked service, name your link service, provide the base URL which we already copied, then select the authentication type, then click on create and then click on Ok. 

In the source, tab click on the Open button then go to the parameter tab click on the + New button and create a new parameter.

In the connection, tab select the parameter which we have created, then back to the pipeline.

In the source tab map, this parameter with the ForEach loop and in the expression add ''.id'' in the end

Once we are done with our source tab, go to the sink tab and click on the + New button to create a new sink dataset.

Select Azure SQL database then click on continue.

Name your Sink dataset, select the linked service, select none for import schema, then click on Ok.

In the sink tab, click on the Open button then in the connection tab provide the table name and back to the pipeline.

Then, select auto create the table, in the sink tab, and then click on Debug to execute our pipeline.

Video Demo: REST API Fetching only Required Records and Writing to Azure SQL DB in Azure Data Factory.