Welcome To TechBrothersIT: How to Delete Files from Multiple Blob Containers by Using Controlled File in Azure Data Factory

Issue: How to delete files from multiple blob containers by using controlled file in Azure data factory.

In this article we are going to learn how to delete files from multiple blob containers by using controlled file in the Azure data factory, in this scenario I have 3 folders in my Blob storage named ''Input'' ''Output'' and ''Logs'', I have uploaded some files in these three folders, now I have to create a controlled file to delete the specific type of files, let's start our demonstration.

I have created the controlled file, in which I have two columns named folder name and wild card, and under these columns, I have folder names and in front of the folders, I have described the file type which I want to delete from these folders, and I have uploaded this master delete file in another folder which name is Mylogs.

Next, Open the Azure data factory, and go to the Author tab and click on the + Sign to create a new pipeline, then find and drag the lookup activity.

Now, click on the lookup activity, then go to the settings tab and click on the + New button to create a new source dataset.

Select the Azure Blob Storage, then click on continue.

Then select the CSV as file format, and click on continue.

Name the dataset, then select the linked service, then select the DeleteMaster file which we have uploaded in the Mylog folder, then select the first row as header, select none for import schema, and then click on ok

Please uncheck '' First row only''.

Once we are done with our lookup activity, now find and drag the ForEach loop activity and then connect with the lookup activity.

Click on the ForEach loop activity, then go to the settings tab then click on sequential, and in the items click on the Add dynamic content, to provide the list of the items from our Lookup activity.

Now go to our ForEach activity and click on the pencil sign to go inside the activity where we have to set up our delete activity.

Inside the ForEach Loop activity, find and drag the Delete activity, then go to the source tab and then click on the + New button to create a new source dataset.

Select the Azure Blob storage and click on continue.

Name the dataset, then select your linked service, and then click on ok

Inside the source, tab click on the Open button to provide the folder path from where we need to delete our files

Now go to the parameter tab and click on the + sign to create a new parameter.

Then go to the connection tab and in the file path provide this parameter.

Now, go back to the Delete activity, and provide the values, which we will get from our ForEach loop Activity.

Next, Click on the Wildcard file path and click on ''Add dynamic content''.

Then we have a very important thing to do, as we are going to delete our files so it is extremely recommended to save logs, so select the storage and container where you want to save your delete logs.