What are Linked Services and DataSets in Azure Data Factory - ADF Tutorial 2021

 Issue: What are Linked Services and DataSets in Azure Data Factory - ADF Tutorial 2021

In this article, we are going to learn What is Linked Services and DataSets in the Azure data factory, the Linked services are much like connection strings, which define the connection information needed for the service to connect to external resources. and the datasets represent the structure of the data within the linked data stores, and the linked service defines the connection to the data source. 

Let's start our demonstration, open your Azure portal, and first of all, we need to create an Azure data factory, so in the Azure portal, we have a services tab on the top from where we can go to the Data factories, once we click on that it will open a new page click on Create data factory then Select the subscription, Resource group, Region, Name, and Version, then click on Review + Create and then click create. as shown in the picture below.


Fig-1: Create a new Azure Data Factory.



Once the Data factory is created open your Azure Data Factory Studio, once you ADF Studio opened let's make a Linked service, we can create a linked service by using pipeline, and also we can create linked service separately in the manage tab, let's create our linked service from the manage tab, go to the Manage tab and then in the connection tab we have linked services, click on +New then Select Azure SQL database as our connection, and then click on continue, then we will move to the new page which is New linked service, give the name as you can identify your linked service, in my case I gave it ''lnkazuresqltechbrothersdb''  then select the Azure subscription, select the database, Authentication type, username & password, Test the connection and click on create. as shown in the picture below.

Fig-2: Create a new Linked service.

Once we created the above-linked service, let's create a new database and then create a new linked service to the newly created database, go to the SQL Server Management Studio, and create a new database, in my case I have created a database named ''Sales'', and also create a table, then go to Azure Data factory Studio and create a new linked service, Click on + New, then select Azure SQL Database, then click continue, then name the database, in my case, it is ''lnksalesdb'', then select the Azure subscription, Authentication type, Username & password, test the connection and click on create, as shown in the picture below.


Fig-3: Create a new linked service.


Now we have two linked services, one is ''lnkazuresqltechbrothersdb'' and second is ''lnksalesdb'', so we can read the data from one and write to another one, now go to your blob storage in the Azure portal and click on the containers tab under the data storage, and create two new containers, one for input and another one for output, once we created the containers, go back to the Azure data factory studio, and create a new linked service, to connect the techbrotheritstorage where our containers are located,  Click on +New, then select the Azure Blob Storage as connection then click continue, and then name the linked service in my case it is ''lnktechbrotherstorage'', then select the Azure subscription then storage account and then test the connection and click create. as shown in the picture below.



Fig-4: Create a new linked service.

Now we have three linked services, and now we have to create pipelines & use them in our datasets,  go to the author in Azure Data Factory Studio, and click on pipelines, then click on the new pipeline, then click on the copy data activity and drag it in the working window,  click on the copy data activity and then click on the source, as you click on the source it will ask for the source dataset, from where we read the data, as we don't have any dataset so click on + New button to create a new dataset, once we click on New, it will ask for the data source, select the Azure SQL database as our data source, then name the dataset and select the linked service which we have created before, and then it will ask for the table, select the table which we have created before, then select import schema and then click ok.

As we have successfully created the dataset for read the data, now click on Sink and here we have to create a new dataset for write the data, so click on the + New button then select the connection source Azure SQL database, then click continue and then name the dataset, then select the linked service which we have created before, then table name and click ok, as shown in the picture below.


Fig-5: Create a new dataset.


Once we have successfully created our datasets, now go to the main window and click on ''debug'',  once the debug process will be completed it will show the results, and then you can go to your output folder and find the data files there.  

Video Demo: What are Linked Services and DataSets in Azure Data Factory





2 comments:

  1. “Very nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed browsing your blog posts. After all I will be subscribing to your feed and I hope you write again very soon!”
    https://www.onlinefastservice.com/product/buy-fiverr-reviews/

    ReplyDelete
  2. As I worked with databases for a long time, I found useful tool - unidac.

    ReplyDelete