This is the 12th post in a series of blog posts using a theme of “Tools of the trade”. The series targets software tools, statistical concepts, data science techniques, or related items. In all cases, the topic will contribute to accomplishing data science tasks. The target audience for the posts is engineers, analysts, and managers who want to build their knowledge and skills in data science, particularly those in the Microsoft Dynamics ecosystem.
This post documents a useful Azure utility for moving data around – AzCopy.
What is it?
AzCopy is a command line utility that “can copy blobs, directories, and containers between storage accounts”. You can get the link to download the utility and read more about it at Copy or move data to Azure Storage by using AzCopy v10 | Microsoft Docs.
How do I use it?
Moving data exported from Dynamics 365 is needed to hydrate proof of concept or test environments internally and externally. AzCopy is a simple to use tool to move data between storage accounts.
I’ve found the best way to do this is to create Secure Access Signatures (SAS) for the source and destination storage accounts. This time bounds the access and enables set up of the minimal required permissions. The source data access requires list and read permissions, for example.
To create the SAS, navigate to the container in the Azure Portal and select Shared access signature in Settings. Set the required permissions, the targeted time frame, and then click ‘Generate SAS token and URL’. You’ll need the Blob SAS URL for the AzCopy command. Do this for both the source and destination storage accounts.
Once the SAS keys are set up, the AzCopy command to move the data is as follows. Note that you may need to navigate to the folder where you downloaded the utility.
azcopy copy –recursive “Insert source Blob SAS URL here” “Insert destination Blob SAS URL here”
You can specify sub-paths of the source or destination container by simply editing the first part of the Blob SAS URL. In the URL mock-up below, you can edit the <Path> to narrow the scope of the copy.
https://<StorageAccountName>.blob.core.windows.net/<ContainerName>/<Path>?sp=…
Discussion
The Dynamics team is providing more options for moving data out of the SQL operational database and into Azure Data Lake Storage (ADLS). You can move Dataverse data to ADLS and/or Synapse as described in Azure Synapse Link – Power Apps | Microsoft Docs. Moving Dynamics 365 Finance and Operations data to ADLS is currently in preview and is described at Configure export to Azure Data Lake – Finance & Operations | Dynamics 365 | Microsoft Docs.
There are scenarios where you only need the exported data, but you don’t want to use the data where at its initial export location. This is a very handy tool for moving data to your personal playground. Give it at try!
References
See links in line.
Picture details: Cattail, 8/20/2020, Canon PowerShot G3 X, f/5, 1/80 s, ISO-250