Copying multiple files in azure data factory | How to copy multiple files using single copy activity in ADF
A Step-by-Step Guide to Copying multiple Files from Source to Sink in Azure Data Factory
In today’s data-driven world, efficiently moving data from source to destination is a crucial task for organizations of all sizes. Azure Data Factory (ADF) is a robust and scalable data integration service offered by Microsoft, designed to help you orchestrate and automate data workflows. In this blog post, we will explore how to copy multiple files from a source location to a sink using Azure Data Factory.
TO know how to create the linked services I recommend to read me other blog
I need to copy the files present under raw container to output container as shown below images .
Step 1: create a dataset in the below manner .
For detailed steps on how to create the parameterized datasets or dynamic datasets please read my other blog . https://medium.com/@madhunarayanavarma/how-to-parameterize-dataset-in-azure-data-factory-parameterization-in-azure-data-factory-4cae57d886aa
Step 2: Create a copy activity in your pipeline and attach your dataset under source. Please pass your source container name under dataset properties. Now select wildcard file path instead of file path in dataset option under file path type option. Under wildcard paths pass *.csv (Pass your file extension accordingly let’s say if you want to copy .json files then please pass *.json) for wildcard file name.
Step 3: Under sink properties please attach your dataset and pass your destination container name.
Step 4: Click on debug to test your pipeline in azure data factory.
Step 5: once you see the pipeline got succeeded then verify the files in output container .
Please follow the above steps to achieve the azure data factory pipeline to copy the multiple file from source to destination.
If you face any errors or need any help on azure data factory please feel free to reach out to me on LinkedIn : https://www.linkedin.com/in/madhu-narayana-varma-m-08216919a/
