102 thoughts on “Connecting to Azure Data Lake Storage Gen2 from PowerShell using REST API – a step-by-step guide”

1. Hey Michal.

   Very informative post. Helped me understand lot of semantics around the authorization header .
   Your powershell script to create a file system works like a charm. However due to environment restrictions, I am trying to emulate that in a bash script (using curl)
   and keep hitting this roadblock

   Error:
   curl: (6) Could not resolve host: PUT
   {“error”:{“code”:”AuthenticationFailed”,”message”:”Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e422daa5-901f-00d8-0231-54a8cc000000\nTime:2019-08-16T12:54:36.0101349Z”}}https://storageaccountname.dfs.core.windows.net/filesystemname?resource=filesystem

   ————————————————————————————————–
   #!/usr/bin/env bash
   STORAGE_ACCOUNT_NAME=${1}
   FILESYSTEM_NAME=${2}
   ACCESS_KEY=${3}

   request_date=$(TZ=GMT date “+%a, %d %h %Y %H:%M:%S %Z”)
   #request_date=$(date +%Y-%m-%d)
   storage_service_version=”2018-11-09″

   # HTTP Request headers
   x_ms_date_h=”x-ms-date:$request_date”
   x_ms_version_h=”x-ms-version:$storage_service_version”

   n=”\n”

   method=”PUT”

   string_to_sign+=”$method\n\n\n\n\n\n\n\n\n\n\n\n$x_ms_date_h\n$x_ms_version_h\n/$STORAGE_ACCOUNT_NAME/$FILESYSTEM_NAME\nresource:filesystem”

   # Decode the Base64 encoded access key, convert to Hex.
   shared_key=”$(echo -n $ACCESS_KEY | base64 -d -w0 | xxd -p -c256)”

   # Create the HMAC signature for the Authorization header
   signed_signature=$(printf “$string_to_sign” | iconv -t utf-8 | openssl dgst -sha256 -mac HMAC -macopt “hexkey:$shared_key” -binary | base64 -w0)

   auth_header=”Authorization: SharedKey ${STORAGE_ACCOUNT_NAME}:$signed_signature”

   uri=”https://$STORAGE_ACCOUNT_NAME.dfs.core.windows.net/$FILESYSTEM_NAME?resource=filesystem”

   curl -d -X $method -H “$x_ms_date_h” -H “$x_ms_version_h” -H “$auth_header” $uri

   if [ $? -eq 0 ]; then
   echo $uri
   exit 0;
   fi;
   exit 1
   —————————————————————————————————

   Reply

2. Hi Michal,
   Thank you for the blog. I’m able to create files and folders using your script. However I’m having trouble renaming the file. I used “x-ms-rename-source” in the header but could not get it to work.
   I keep getting error
   Here’s the code I have been using
   $stringToSign +=
   # SECTION: CanonicalizedHeaders + “\n” #
   “x-ms-date:$date” + $n +
   “x-ms-version:2018-11-09” + $n +
   “x-ms-rename-source:/adlsg2filesystemname/folderpath/filename” + $n
   # SECTION: CanonicalizedHeaders + “\n” #

   $stringToSign +=
   # SECTION: CanonicalizedResource + “\n” #
   “/$StorageAccountName/$FilesystemName” + $PathToCreate + $n
   # SECTION: CanonicalizedResource + “\n” #

   $sharedKey = [System.Convert]::FromBase64String($AccessKey)
   $hasher = New-Object System.Security.Cryptography.HMACSHA256
   $hasher.Key = $sharedKey

   $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

   $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

   $headers = @{“x-ms-date”=$date}
   $headers.Add(“x-ms-version”,”2018-11-09″)
   $headers.Add(“x-ms-rename-source”,”/adlsg2filesystemname/folderpath/filename”)
   $headers.Add(“Authorization”,$authHeader)
   $headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

   $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + $PathToCreate

   Could you please tell me what am I missing here?

   Reply

   1. Hi KD,

      Have you got any solution for this?

      Thanks,
      Charan

      Reply

   2. Hi Michal,

      Can you please help us in this issue.

      -Charan

      Reply

      1. Unfortunately, I don’t have much time for a full code review and writing it as a script.
         But I can point few interesting and important facts.

         In KD’s example, in canonicalized headers: x-ms-rename-source is AFTER x-ms-version, and this is against the rule of having lexicographical sort order. “x-ms-r” should be before “x-ms-v”.

         Now, regarding the “x-ms-rename-source” itself, look at the API documentation here, section request headers, under x-ms-rename-source:

         This value must be a URL percent-encoded string. Note that the string may only contain ASCII characters in the ISO-8859-1 character set.

         This is important. If you will sniff rename operation in Azure Storage Explorer, as i described it in my other blog post here, you will see an interesting situation:

         

         Well, I do not know why it is encoded in request URL and not in the x-ms-rename-source, BUT there is some kind of percent-encoding and you need to figure it out 🙂
         SO at least please try to change the path in Request URL, then if it will fail, maybe it should be modified also in canonicalized header?

         As I have limited time, it would be great if you will post your findings here. As soon as I have a little more time I will prepare a script based on this. Thanks!

         Reply

         1. Hi Michal,

            The issue was with canonicalized headers. I tried with x-ms-rename-source before x-ms-version and its working fine now.

            Thank you for your help.

            -Charan

            Reply

            1. 

               Reply

               1. Hi Michal,

                  I am executing the same code from Azure Powershell Function App but getting authorization exception.

                  Exception: “2019-11-01T06:35:00.249 [Error] ERROR: Invoke-RestMethod : {“error”:{“code”:”AuthorizationFailure”,”message”:”This request is not authorized to perform this operation.\nRequestId:77a8e674-f01f-0089-097e-90705c000000\nTime:2019-11-01T06:34:59.9031753Z”}}”

                  I am having contributor access to Azure Function. Do I need any more access to execute Invoke-RestMethod?

                  Please help me on this.

                  Thanks,
                  Charan

                  Reply

                  1. Have you tried with elevated privileges e.g Account Admin role ,just to ensure it is really RBAC issue ?

                     Reply

                     1. Hi Mohan,

                        Yes, we tried with Admin role and its working. Can you please provide some more insight on RBAC set up.

                        Thanks,
                        Charan

                        Reply

                        1. I created new folder in ADL file-system from Azure function powershell app and it went through smoothly. The contributor/owner role should be good enough.

                           Check the permissions of roles in use , https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-powershell.

                           Reply

            2. Hi Charan and Michal ,

               thanks to both of you for all the inputs , the post is extremely insightful and helpful.

               I need your inputs on renaming the folder pls , added the code here , i am consistently getting error , stating , “One of the headers specified in the request is not supported.” Spent almost 2 days but invain..

               However the code for creating file-system , folder and deleting it is working fine.

               function ADLRenameFolder
               {
               Param(
               [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
               [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
               [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey,
               [Parameter(Mandatory=$True,Position=4)] [string] $PathToCreate,
               [Parameter(Mandatory=$True,Position=5)] [string] $SourcePath
               )

               # Rest documentation:
               # https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create

               $PathToCreate =$PathToCreate.trim(“/”) # remove all “//path” or “path/”
               $SourcePath = $FilesystemName + “/” + $SourcePath.trim(“/”) # remove all “//path” or “path/”

               #$URL = Read-Host “Enter URL to Decode”
               $PathToCreate = [System.Web.HttpUtility]::UrlEncode($PathToCreate)
               Write-Host $PathToCreate

               $PathToCreate=[System.Web.HTTPUtility]::UrlEncode($PathToCreate)

               $date = [System.DateTime]::UtcNow.ToString(“R”) # ex: Sun, 10 Mar 2019 11:50:10 GMT

               $n = “`n”
               $method = “PUT”

               $stringToSign = “$method$n” #VERB
               $stringToSign += “$n” # Content-Encoding + “\n” +
               $stringToSign += “$n” # Content-Language + “\n” +
               $stringToSign += “$n” # Content-Length + “\n” +
               $stringToSign += “$n” # Content-MD5 + “\n” +
               $stringToSign += “$n” # Content-Type + “\n” +
               $stringToSign += “$n” # Date + “\n” +
               $stringToSign += “$n” # If-Modified-Since + “\n” +
               $stringToSign += “$n” # If-Match + “\n” +
               $stringToSign += “*” + “$n” # If-None-Match + “\n” +
               $stringToSign += “$n” # If-Unmodified-Since + “\n” +
               $stringToSign += “$n” # Range + “\n” +
               $stringToSign +=

               “x-ms-date:$date” + $n +
               “x-ms-rename-source:$SourcePath” + $n +
               “x-ms-version:2018-11-09” + $n #

               $stringToSign +=

               “/$StorageAccountName/$FilesystemName/” + $PathToCreate + $n +
               “resource:directory”#

               Write-Output $stringToSign

               $sharedKey = [System.Convert]::FromBase64String($AccessKey)
               $hasher = New-Object System.Security.Cryptography.HMACSHA256
               $hasher.Key = $sharedKey

               $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

               $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

               $headers = @{“x-ms-date”=$date}
               $headers.Add(“x-ms-rename-source”,$SourcePath)
               $headers.Add(“x-ms-version”,”2018-11-09″)
               $headers.Add(“Authorization”,$authHeader)
               $headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

               $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + “/” + $PathToCreate + “?resource=directory”

               Write-Output $URI

               Write-Output $headers
               try {
               Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
               $true
               }
               catch {
               $ErrorMessage = $_.Exception.Message
               $StatusDescription = $_.Exception.Response.StatusDescription
               $false

               Write-Output $ErrorMessage + ” ” + $StatusDescription
               #Throw $ErrorMessage + ” ” + $StatusDescription
               }
               }

               **************************** Invoking Function below *********************************************
               $StorageAccountName=”teststorageaccount”
               $FilesystemName=”testfilesystem”
               $AccessKey=””
               $PathToCreate=”/data/inv/researchrenamed”
               $SourcePath=”/data/inv/research2″

               Add-Type -AssemblyName System.Web

               ADLRenameFolder $StorageAccountName $FilesystemName $AccessKey $PathToCreate $SourcePath

               Reply

               1. Hmm, from what I can see above there is definitely a problem with:
                  $stringToSign += “*” + “$n” # If-None-Match + “\n” +

                  you cannot use If-None-Match in renaming, only x-ms-source-if-none-match is allowed.
                  Try to remove it:
                  $stringToSign += “$n” # If-None-Match + “\n” +

                  Also, you have:
                  “?resource=directory”

                  and in the documentation they mentioned resource like this:
                  Required only for Create File and Create Directory. The value must be “file” or “directory”.
                  So I think this should be also removed 🙂

                  Try it and return with a feedback please 🙂

                  By the way, looks like I need to take a while and prepare rename example :]

                  Reply

                  1. Thanks for quick reply , i tried both inputs but error is same. I have removed one at a time

                     When removed only $stringToSign += “$n” # If-None-Match + “\n” +
                     Same err:400 One of the headers specified in the request is not supported.

                     When both removed “?resource=directory”
                     Error 400 : The source URI is invalid.

                     Reply

                     1. The output values i have captured may give some insights ,
                        URI : https://.dfs.core.windows.net/testfilesystem/data%252finv%252fresearchrenamed?resource=directory
                        Headers:
                        Name Value
                        —- —–
                        x-ms-version 2018-11-09
                        Authorization SharedKey :JCEAKSIvntEY5rtL6oM5xY9QdZfvuNUcHnaHw5z1h8g=
                        x-ms-rename-source testfilesystem/data/inv/research2
                        x-ms-date Thu, 24 Oct 2019 15:55:03 GMT
                        x-ms-client-request-id 18429f61-2cf5-4075-96ce-c4a254e84771

                        Reply

                        1. Uhh, it was easier to write it myselft than to debug in the comment 🙂
                           Here it is, working renaming (without nomach, which did not work for me and I do not have time to analyze it, so watch out when you will try to rename folders to something that already exists 😀 )
                           Just remember to encode URI in canonical headers and resource, do not use resource parameters and everything will work 🙂

                           Example link: Rename file/folder (or move to the other path)

                           Reply

                           1. Hi Michal ,

                              Really appreciate your quick response and contribution to community.

                              When i compared with your new code , i realized i was not encoding the source path( PathToRename) but only value of RenameTo.

                              The third change of encoding i did post addressing the previous two inputs you have provided.

                              Cheers !!!

                              Here goes updated function…..
                              function ADLRenameFolder
                              {
                              Param(
                              [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
                              [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
                              [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey,
                              [Parameter(Mandatory=$True,Position=4)] [string] $PathToCreate,
                              [Parameter(Mandatory=$True,Position=5)] [string] $SourcePath
                              )

                              # Rest documentation:
                              # https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create

                              $requestid = [guid]::NewGuid().ToString()
                              Write-Output $requestid

                              $PathToCreate =$PathToCreate.trim(“/”) # remove all “//path” or “path/”

                              $SourcePath = “/” + $FilesystemName + “/” + $SourcePath.trim(“/”) # remove all “//path” or “path/”

                              #$URL = Read-Host “Enter URL to Decode”
                              $PathToCreate_encoded = [System.Web.HttpUtility]::UrlEncode($PathToCreate)
                              Write-Host “New Path” + $PathToCreate

                              $SourcePath_encoded=[System.Web.HttpUtility]::UrlEncode($SourcePath)

                              $date = [System.DateTime]::UtcNow.ToString(“R”) # ex: Sun, 10 Mar 2019 11:50:10 GMT

                              $n = “`n”
                              $method = “PUT”

                              $stringToSign = “$method$n” #VERB
                              $stringToSign += “$n” # Content-Encoding + “\n” +
                              $stringToSign += “$n” # Content-Language + “\n” +
                              $stringToSign += “$n” # Content-Length + “\n” +
                              $stringToSign += “$n” # Content-MD5 + “\n” +
                              $stringToSign += “$n” # Content-Type + “\n” +
                              $stringToSign += “$n” # Date + “\n” +
                              $stringToSign += “$n” # If-Modified-Since + “\n” +
                              $stringToSign += “$n” # If-Match + “\n” +
                              $stringToSign += “$n” # If-None-Match + “\n” +
                              $stringToSign += “$n” # If-Unmodified-Since + “\n” +
                              $stringToSign += “$n” # Range + “\n” +
                              $stringToSign +=

                              “x-ms-client-request-id:$requestid” + $n +
                              “x-ms-date:$date” + $n +
                              “x-ms-rename-source:$SourcePath_encoded” + $n +
                              “x-ms-version:2018-11-09” + $n #

                              $stringToSign +=

                              “/$StorageAccountName/$FilesystemName/” + $PathToCreate_encoded

                              Write-Output $stringToSign

                              $sharedKey = [System.Convert]::FromBase64String($AccessKey)
                              $hasher = New-Object System.Security.Cryptography.HMACSHA256
                              $hasher.Key = $sharedKey

                              $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

                              $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

                              $headers = @{“x-ms-date”=$date}
                              $headers.Add(“x-ms-client-request-id”,$requestid)
                              $headers.Add(“x-ms-rename-source”,$SourcePath_encoded)
                              $headers.Add(“x-ms-version”,”2018-11-09″)
                              $headers.Add(“Authorization”,$authHeader)
                              #$headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

                              $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + “/” + $PathToCreate_encoded

                              Write-Output $URI

                              Write-Output $headers
                              try {
                              Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
                              $true
                              }
                              catch {
                              $ErrorMessage = $_.Exception.Message
                              $StatusDescription = $_.Exception.Response.StatusDescription
                              $false

                              Write-Output $ErrorMessage + ” ” + $StatusDescription
                              #Throw $ErrorMessage + ” ” + $StatusDescription
                              }
                              }

                              #**************************** Invoking Function below *********************************************
                              $StorageAccountName=”yourAccountName”
                              $FilesystemName=”testfilesystem”
                              $AccessKey=”yourAccountNameKey”
                              $PathToCreate=”/data/inv/research_renamed”
                              $SourcePath=”/data/inv/research”

                              Add-Type -AssemblyName System.Web

                              ADLRenameFolder $StorageAccountName $FilesystemName $AccessKey $PathToCreate $SourcePath

                           2. Thanks. I’m glad I could help 🙂

3. hi Michał Pawlikowski ,

   You above code works just fine hierarchical folder creation .
   I my case i need to assign service Principal at Folder Level. Say top Level and assign service Prinicpal.
   Can you suggest how to assign already created service Principal while time of folder Creation.

   Regards
   Deepak

   Reply

4. Also I tried “Set permissions on filesystem or path” . In place of $PermissionString i used Key of service Principal but it did not helped.

   thanks

   Reply

5. Hi to all! I apologize for the long absence, but I had to get some treatment in the hospital. Do you still need help or have you dealt with your problems?

   Reply

6. Hi Michal,
   Hope you are doing well.
   I am trying hierarchical folder creation in adls gen 2 with service principal. Fir you solution in “set solution on file system or path” with what component I have to replace $Permissionstring.

   Reply

   1. Hi Deepak.
      The answer is just in the comments above 🙂
      Look for my reply to Mark’s question:

      

      Just do it manually in Azure Storage Explorer and see for yourself how the string is made there.
      Just reuse it within your code. Yeap, simple as that 🙂

      Reply

7. Hi Michal,

   Hope you are doing well.

   Any sample script are there to delete folders or files from adlsgen2.
   i am following your above script for the same with method “Delete” and changed some script systax.
   but its not working.

   Reply

   1. Hi Shreenivas,

      thanks, I’m doing great 🙂

      To construct a proper header you need to have recursive parameter and send DELETE command to the rest api.
      I’ve created an example for you, look for “Delete folder or file (simple, without continuation)” in the examples above.
      Please bear in mind, that I did not include optional parameter “continuation” that handles the situation when you are trying to delete a lot of files (nested, so recursively under the folder that you will try to delete, see details in the example)

      Regards,
      m.

      Reply

      1. Hi Michal,

         Thanks lot for suggestion and sample code.
         I was not passing the recursive value and script was failing.

         Now it’s working through your help

         Regards,
         Shreenivas Mirajkar

         Reply

8. Hi Michal

   You ROCK ! 🙂
   VERY nice article and helped me a LOT 🙂

   THANK YOU.

   Reply

   1. Thank you!

      

      Reply

9. Hi Michal,

   Thanks much mate. Your post saved me lots of research time.

   Lackshu

   Reply

   1. It’s satisfying. Thank you!

      Reply

10. Hi Michal,

    I was referring this blog for creation of Filesystem as well as creating Folder under that Filesystem.
    The Filesystem creation go success, but the folder creation is resulting in error

    The remote server returned an error: (403) Forbidden. Server failed to authenticate the request. Make sure the value
    of Authorization header is formed correctly including the signature.
    At C:\Users\goswams1\Documents\WIP\Documents\Powershell-Commands\CreateFolder.ps1:68 char:5
    + Throw $ErrorMessage + ” ” + $StatusDescription
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OperationStopped: (The remote serv… the signature.:String) [], RuntimeException
    + FullyQualifiedErrorId : The remote server returned an error: (403) Forbidden. Server failed to authenticate the
    request. Make sure the value of Authorization header is formed correctly including the signature.

    I have just copied the piece of code mentioned at the section – Create directory (or path of directories).
    I am giving the input for Filesystem as the one created with your script and Folder Path as /Landing/Import. But this fails.

    I am just clueless in this. Please help me here.

    Reply

    1. Unfortunately, you need to share your code anyway. I’ve tested it myself and folder creation example works like a charm.. If you did not make any mistake in access key or sotrage account name, the only reason is something changed in the code. Please use portals like this: https://paste.ofcode.org/ to share your code.

       Reply

       1. Hi Michal,

          This actually worked. I just re-initiated the session and this work fine. If I get stuck again, will reach out to you.
          I would like to convey a big thanks for creating this blog. This solved a major chunk of work for me.

          Regards,
          Sudipta.

          Reply