How to Copy data from an HTTP endpoint by using Azure Data Factory

Issue: Copy data from an HTTP endpoint by using Azure Data Factory


In this article, we are going to learn how to copy data from an HTTP endpoint by using Azure Data Factory, let's start our demonstration, I have taken the sample data from a free website that are providing free sample data for demo purposes, so go to any free sample data website and copy the URL which you want to read.
.


Open the Azure Data factory studio, then go to the Author tab and click the + button then click on the pipeline to create a new pipeline.



Find and drag the Copy data activity in the pipeline, then click on the Source tab and then click on the + New button to create a new source dataset.


Find and select the HTTP and click on continue.



Select the file format which you are reading from your source, in my case it is .JSON format, then click on continue.



Then it will ask for the linked service, click on the + New button to create a new linked service.



Provide the base URL which we copied from the Sample data website, then select the authentication type as I am using free sample data so I don't have any user name and password so I will select Anonymous, in your case you can select other options then test the connection and click on create.



Then select the request method, in my case I am getting the data so I will select Get.


Go to the sink tab and then click on the + New button to create a new Sink dataset.




Select the Azure blob storage and then click on continue.



Then select the file format, in my case I need CSV (Delimited Text) file, then click on continue



Then it will ask for the linked service click on the + New button to create a new linked service, then select the Azure subscription, then select the storage account and then click on create.


 Select the File path and give then file name, then select the First row as header, then Select import schemas as From connection/store then click on ok.




Go to the mapping tab then click on import schemas, it will bring all the schemas from json and we can select what we need, and also we can delete them.



Click on the Debug button, once the debug process will be completed go to the output folder and your file will be there.






Video Demo: How to Copy data from an HTTP endpoint by using Azure Data Factory

1 comment: