Azure Data Factory 2021 | Azure Data Factory Tutorial For Beginners | Azure Data Factory Tutorial

Topic: Azure Data Factory 2021 | Azure Data Factory Tutorial For Beginners | Azure Data Factory Tutorial.

Azure Data Factory is Azure's cloud ETL service (Extract, transform, and load) for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management. You can also lift and shift existing SQL Server Integration Service packages to Azure and run them with full compatibility in Azure Data Factory.

In this article, we are going to learn about the Azure Data Factory and will explain some of its major functions, lets start our demonstration, first of all, log in to your Azure portal, where the Azure Data Factory exists.

Once we are on our Azure portal, here is a Tab named Data factories, click on that to create a new Azure Data Factory.

In the data factories, click on the + Create button then, in the basics tab select Azure subscription, then select resource group, then select your region, then give a name to data factory, and then click on the Review + create button then click on Configuration Git later and click on Create.

Once our Azure Data Factory is created, go to the resource and open then Azure data factory studio, in the Azure data factory studio we have four major tabs which are 1. Home, 2. Author, 3. Monitor, 4. Manage, in the Home tab we have many more options i.e, Create pipeline, in the pipeline we can create the ETL activities (Extract, transform, and load) A pipeline is a logical grouping of activities that performs a unit of work. Together, the activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingests data from an Azure blob and then runs a Hive query on an HDInsight cluster to partition the data Create Data flow Create Pipeline from the template, Copy data, Configure SSIS integration, Set up code repository.

Data Flow allows you to develop graphical data transformation logic that can be executed as activities within ADF pipelines. The intent of ADF Data Flows is to provide a fully visual experience with no coding required.

In the home Tab, the 3rd option is Create Pipeline from a template, An Azure Data Factory pipeline template is a predefined pipeline that provides you with the ability to create a specific workflow quickly, without the need to spend time in designing and developing the pipeline, using an existing Template Gallery that contains data Copy templates, External Activities templates, data Transformation templates, SQL Server Integration Service template or your custom pipelines templates.

.

Next is Copy Data, In Azure Data Factory and Synapse pipelines, you can use the Copy activity to copy data among data stores located on-premises and in the cloud. After you copy the data, you can use other activities to further transform and analyze it. You can also use the Copy activity to publish transformation and analysis results for business intelligence and application consumption.

Then we have (Configure SSIS Integration) Data Integration. SSIS performs data integration by combining the data from multiple sources and provides unified data to the users.

At last, we have Set up a code repository, Your Azure Repos code repository name. Azure Repos projects contain Git repositories to manage your source code as your project grows. You can create a new repository or use an existing repository that's already in your project.

Then we have the second tab which is Author, in this tab we can create pipelines and then we have Dataset, then Data flows, and then power Query, in this tab we can perform tons of activities.

Then we have the 3rd tab which is Monitor, in this tab you can monitor the pipeline that has been executed in your respective azure data factory account can be seen here. Here you have the filter available using which you can filter the pipeline run which you want to monitor. You can filter based on pipeline name, execution time, triggered by and many other filter criteria and see the performance of your Pipelines, Activities, and trigers,

Then we have our 4th and final tab which is Manage, in the manage tab, first of all, we have Linked services, linked services are the connection to your blob or Azure SQL server or any other data source.

Then we have integration runtime, The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory and Azure Synapse pipelines to provide the following data integration capabilities across different network environments.

Then we have Azure Purview (Preview), Connection in Azure Data factory, to Azure Purview will enable you to discover trusted and accurate data across your hybrid environment.

Under the Source control tab, we have Git Configuration, It is a very important tool, when you are going to create your Azure data factory, it is recommended to configure your Github because if in case you deleted your Azure data factory then your all pipelines, datasets and linked services whatever associated with your Azure data factory will be removed, now if you have configured your Github repository, you can easily recover your Azure data factory.

Then we have the ARM template, again it is a very important tool, in this tool you can export (Backup) your entire Data Factory, or you can Import your Backed up data factory.

In the Author, we have Triggers, as you know the triggers are the objects that can run your pipelines.  

Then we have global parameters, Global parameters are constants across a data factory that can be consumed by a pipeline in any expression. They're useful when you have multiple pipelines with identical parameter names and values.

Inside the Security tab, we have the customer-managed key, When you specify a customer-managed key, Data Factory uses both the factory system key and the Customer Managed Key to encrypt customer data. Missing either would result in Deny of Access to data and factory. Azure Key Vault is required to store customer-managed keys.

Video Demo: Introduction To Azure Data Factory | Azure Data Factory Tutorial For Beginners | Azure Data Factory Tutorial