How to perform copy activity in azure data factory | creating a linked service in azure data factory | create dataset using azure data factory

I have two containers in azure data lake.

1. Raw

2. Output

Let us see how we can copy the file from one place to other place in azure cloud using azure data factory.

In this blog, Let us see how we can copy the file, which is located in raw container to output container

Please follow the below steps to perform copy activity using azure data factory pipelines.

STEP 1. You should have linked service established for your azure data lake from azure data factory (In this case, we are copying the file from raw container in data lake to output container in data lake).

. Please follow the below steps on how to create a linked services in azure data factory.

· Under manage tab click on new option and search for azure data Lake Gen2.

· Fill all the required details or select from the dropdown. Finally test the connection and click on create.

· You should see the below page after creating your linked service in azure data factory.

STEP 2. Create a dataset for your file that you are going to copy from azure data lake. (Source dataset).

· Please follow the below steps to know on how to create a dataset in azure data factory.

· Under author tab navigate to datasets and click on new dataset to create a dataset in azure data factory.

Search for gen2, select azure data lake storage gen2, and click to continue.

· Select appropriate file format(EG: if you are trying to copy the data from csv file select delimited text format or the best approach will be select the binary format it will support any file type.)

· In the next tab, select linked service, which we created from the drop down and select the respected file, which is in your azure data lake container or folder.

After selecting the file and click on ok.

· You can see the below page appears as soon as your datasets get created.

STEP 3. Create a dataset for your file that you are going to copy to azure data lake (Sink dataset).

Similarly follow the above steps which was mentioned under step 2 to create the dataset for sink.

STEP 4.Click the three dots and select create pipeline.

STEP 5. Search for copy activity in azure data factory or you can also find the copy activity under move and transform section.

STEP 6. You need to attach your source dataset under source option and attach sink dataset under sink option.

STEP 7. Click on debug option to test whether your copy activity pipeline working fine.

STEP 8. If everything works fine click on publish to save your changes.

STEP 9. Check if you have file copied in your destination location.

Please let me know if you are facing any issue’s in the comments . Lets connect on LinkedIn : https://www.linkedin.com/in/madhu-narayana-varma-m-08216919a/