How to Re-run Pipeline from Point of Failure in Azure Data Factory- Azure Data Factory Tutorial 2021.

Issue: How to Re-run Pipeline from Point of Failure in Azure Data Factory- Azure Data Factory Tutorial 2021.

In this article, we are going to learn, how to Rerun pipeline from point of failure in Azure Data Factory, as this is a very important issue, as you guys know that in SSIS tools we did not have this option, for example, if we are using data flow tas kor any other task and if it fails, we can not rerun the task from the point of failure, we will go back and start from the beginning. In the Azure data factory, we have this option, we can Rerun our pipeline from the point of failure, no need to start the task from the beginning, so in this article, we will explore the same. let's start our demonstration, first of all, open the Azure Data factory studio, go to the author, and click on pipeline and then select New pipeline, then give any specific name to the pipeline, in my case, it is ''Rerun_Demo'', so in this pipeline, we are going to use some activities, go to the Activities tab and select the wait activity and drag it into the working window, and give the name, then drag the copy data activity here, as shown in the picture below.

Fig-1: Drag the copy data activity.

We will read the file from blob storage and write it to the blob storage, click on the copy data activity, and go to the source tab and then click on the + New button to create a new source dataset, then select the azure blob storage, then click on continue and then select Delimited text as file format, then click on continue, then click on +New button to create new linked service, select the subscription, then select the storage account, click on test connection and then click on create as shown in the picture below.

Fig-2: Create a new linked service.

Once our linked service is created, we have to select the folder from where we need to read the file and then select the first row as header and then select the import schema as from connection/store and then click ok as shown in the picture below.

Fig-3: Select the file path and import schema.

 

Next click on the sink tab, and then click on the + New button to create a new sink dataset, then select the azure blob storage and then click on continue, then select Delimited text as a file format which is CSV, then click continue, and then name the dataset and select the linked service as we created before, because we are using the same blob storage, then select the output container where we have to write the file, give the file name and then select the first row as header and then select import schema as None and click ok as shown in the picture below. 

Fig-4: Create a new sink dataset.

Once our sink dataset is created go to the activities and drag another wait activity and connect it with the copy data activity, and name the wait activity, and click on publish all, once our pipelines are publish click on as trigger and click trigger now, once the process will be completed it will show the results as shown in the picture below.

Fig-5: Copy data activity pipeline completion results.

as you can see in the above picture all files are green, which means the task is completed successfully, now we are going to delete the input file from the input container and then trigger it again, once we will trigger again it will give the error as shown in the picture below,

Fig-6: Copy data activity pipeline error. 

as you can see in the above picture our first activity has been completed successfully, and the second activity which is copy data activity is failed, now we don't have to run this pipeline from the first wait activity, once we will restore our input file, we will just Rerun the pipeline from copy data activity, we don't have to start from the first wait activity. 

Video Demo: How to Re-run Pipeline from Point of Failure in Azure Data Factory.