Connecting to Azure Data Lake Storage Gen2 from PowerShell using REST API – a step-by-step guide

This article describes how to connect using the access key. I also prepared another article showing how to connect to ADLS Gen2 using OAuth bearer token and upload a file. It is definitely easier (no canonical headers and encoding), although it requires the application account. You can read more here: http://sql.pawlikowski.pro/2019/07/02/uploading-file-to-azure-data-lake-storage-gen2-from-powershell-using-oauth-2-0-bearer-token-and-acls/

 

 

Introduction


Azure Data Lake Storage Generation 2 was introduced in the middle of 2018. With new features like hierarchical namespaces and Azure Blob Storage integration, this was something better, faster, cheaper (blah, blah, blah!)  compared to its first version – Gen1.

Since then, there has been enough time to prepare the appropriate libraries, thanks to which we could connect to our data lake.

Is that right? Well, not really…

Ok, it’s August 2019 and something finally has changed 🙂

Ok, it’s February 2020 and finally, MPA is in GA!

Microsoft introduced a public preview of Multi Protocol Access (MPA) which enables BLOB API on hierarchical namespaces.

Currently it works only in West US 2 and West Central US regions works in all regions and of course has some limitations (click here).

You can read about the details below:

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-multi-protocol-access

ADLS Gen2 is globally available since 7th of February 2019. Thirty-two days later, there is still no support for the BLOB API, and it means no support for az storage cli or REST API. So you gonna have problems here:

“Blob API is not yet supported for hierarchical namespace accounts”

Are you joking?

And good luck with searching for a proper module in PS Gallery. Of course, this will change in the future.

As a matter of fact – I hope, that this article will help someone to write it 🙂 (yeah, I’m too lazy or too busy or too stupid to do it myself 😛 )

So for now, there is only one way to connect to Azure Data Lake Storage Gen2… Using native REST API calls. And it’s a hell of the job to understand the specification and make it work in the code. But it’s still not rocket science 😛

And by the way. I’m just a pure mssql server boy, I need no sympathy 😛 So let me put it in that way: web development, APIs, RESTs and all of that crap are not in my field of interest. But sometimes you have to be your own hero to save the world… I needed this functionality in the project, I read all the documentation and a few blog posts. However, no source has presented how to do it for ADLS Gen2 😮

So now I’ll try to fill this gap and explain this as far as I understood it, but not from the perspective of a professional front-end/back-end developer, which I am definitely not!

 

ADLS Gen REST calls in action – sniffing with Azure Storage Explorer


I wrote an entire article about How to sniff ADLS Gen2 storage REST API calls to Azure using Azure Storage Explorer.

Go there, read it, try it for yourself. If you need to implement it in your code just look at how they are doing it.

 

Understanding Data Lake REST calls


If you want to talk to ADLS Gen2 endpoint in his language, you have to learn “two dialects” 😛

  • REST specification dedicated only for Azure Data Lake Storage Gen2 (URL with proper endpoints and parameters), documented HERE.
  • GENERAL specification for authenticating connections to any Azure service endpoint using “Shared key”, documented HERE.

Knowing the first one gives you the ability to invoke proper commands with proper parameters on ADLS. Just like in console, mkdir or ls.

Knowing the second – just how to encrypt those commands. Bear in mind that no bearer token (see the details in a green box at the beginning of this page…), no passwords or keys are used for communication! Basically, you prepare your request as a bunch of strictly specified headers with their proper parameters (concatenated and separated with new line sign) and then you “simply” ENCRYPT this one huge string with your Shared Access Key taken from ADLS portal (or script).

Then you just send your request in pure http(s) connection with one additional header called “Authorization” which will store this encrypted string, along with your all headers!

You may ask, whyyyyy, why we have to implement such a logic?

The answer is easy. Just because they say so 😀

But to be honest, this makes total sense.

ADLS endpoint will gonna receive your request. Then it will also ENCRYPT it using our secret Shared Key (which once again, wasn’t transferred in the communication anywhere, it’s a secret!). After encryption, it will compare the result to your “Authorization” header. If they will be the same it means two things:

  • you are the master of disaster, owner of the secret key that nobody else could use to encrypt the message for ADLS
  • there was no in-the-middle injection of any malicious code and that’s the main reason why this is so tortuous…

How to encrypt – it is just a different kettle of fish… Because you have to use keyed-Hash Message Authentication Code (HMAC) prepared with SHA256 algorithm converted to Base64 value using a hash table generated from your ADLS shared access key  (which, by the way, is the Base64 generated string :D) But no worries! It’s not so complicated as it sounds, and we have functions in PowerShell that can do that for us.

 

Authorization Header specification


Ok, let’s look closer to the Azure auth rest specification here: https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key

We are implementing an ADLS gen2 file system request, which belongs to “Blob, Queue, and File Services” header. If you are going to do this for Table Service or if you would like to implement it in “light” version please look for a proper paragraph in docs.

Preparing  Signature String


Quick screen from docs:

We can divide it into 4 important sections.

  1. VERB – which is the kind of HTTP operation that we are gonna invoke in REST (in.ex. GET will be used to list files, PUT for creating directories or uploading content to files or PATCH for changing permissions)
  2. Fixed Position Header Values – which must occur in every call to REST, obligatory for sending BUT you can leave them as an empty string (just “” with an obligatory end of line sign)
  3. CanonicalizedHeaders – what should be there is also specified in the documentation. Basically, you have to put all available “x-ms-” headers here. And this is important – in lexicographical order by header name! For example: "x-ms-date:$date`nx-ms-version:2018-11-09`n"
  4. CanonicalizedResourcealso specified in docs. Here come all lowercase parameters that you should pass to adls endpoint according to its own specification. They should be ordered exactly the same as it is in the invoked URI. So if you want to list your files recursively in root directory invoking endpoint at https://[StorageAccountName].dfs.core.windows.net/[FilesystemName]?recursive=true&resource=filesystem" you need to pass them to the header just like this: "/[StorageAccountName]/[FilesystemName]`nnrecursive:true`nresource:filesystem"

Bear in mind that every value HAS TO BE ended with new line sign, except for the last, that will appear in the canonicalized resource section. They also need to be provided as lowercase values!

In examples above, I’m using `n which in PowerShell is just a replacement for \n

 

So if I want to list files in the root directory first I declare parameters:

 

Then I need to prepare a date, accordingly to specification. And watch out! Read really carefully this “Specifying the Date Header” section in the docs! It’s really important to understand how Date headers work! The most important part is:

Ok, creating it in PowerShell, also I’m preparing new line and my method as variables ready to be used later:

 

So what about fixed headers? For listing files in ADLS and with defined x-ms-date I can leave them all empty. It would be different if you would like to implement PUT and upload the content to Azure. Then you should use at least Content-Length and Content-Type . For now it looks like this:

 

  • Implementing file upload is much more complex than only listing them. According to the docs, you have to first CREATE file then UPLOAD a content to it (and it looks like you can do this also in parallel 😀 Happy threading/forking!)
  • Don’t forget about the required headers. UPDATE gonna need more than listing.
  • You can also change permissions here, see setAccessControl
  • There is also a huuuge topic about conditional headers. Don’t miss it!

Encrypting Signature String


Let’s have fun 🙂 Part of the credits goes to another blog with an example of a connection to storage tables.

First, we have to prepare an Array of Base64 numbers converted from our Shared Access Keys:

 

Then we create HMAC SHA256 object, we gonna fill it with our numbers from access key.

 

Now we can actually encrypt our string using ComputeHash function of our HMAC SHA256 object. But before encryption, we have to convert our string into the byte stream.

As specification requires it from us, we have to encode everything once again into Base64.

 

And use it as auth header, along with storage account name before it.

Also here we have to add ordinary headers with date of sending the request and API version.

Remember to use proper version (a date) which you can find in ADLS Gen2 REST API spec:

 

That’s it!

 

Now you can invoke request to your endpoint:

 

Result:

It works!

 

Now small debug in Visual Studio Code, adding a watch for $result  variable to see a little more than in console output:

From here you can see, that recursive=true did the trick, and now I have a list of 4 objects. Three folders in root directory and one file in FOLDER2, sized 49MB. Just rememebr that API has a limit of 5000 objects.

Easy peasy, right? 😉 Head down to the example scripts and try it for yourself!

Example Scripts


 

 

List files


This example should list the content of your root folder in Azure Data Lake Storage Gen2, along with all subdirectories and all existing files recursively.

 

 

 

List files in directory, limit results, return recursive


This example should list the content of your requested folder in Azure Data Lake Storage Gen2. You can limit the number of returned results (up to 5000) and decide if files in folders should be returned recursively.

 

 

 

Create directory (or path of directories)


This example should create folders declared in PathToCreate variable.

Bear in mind, that creating a path here, means creating everything declared as a path. So there is no need to create FOLDER1 as first, then FOLDER1/SUBFOLDER1 as second. You can make them all at once just providing the full path "/FOLDER1/SUBFOLDER1/"

 

 

Rename file/folder (or move to the other path)


This example should rename given path to another, you can also move files with that command (hierarchical namespaces, path to file is just only metadata)

Example parameters:

 

And the most requested code 😀

 

 

 

Delete folder or file (simple, without continuation)


This example should delete file or folder, giving you also an option to delete folder recursively.

Bear in mind, that this example uses simple delete without continuation. To learn what it is and how to handle it please refer section describing parameter “continuation” in the documentation here (click)

Usage example:

 

 

List filesystems


This example lists all available filesystems in your storage account. Refer to $result.filesystems  to get the array of filesystem objects.

 

 

Create filesystem


This example should create filesystem provided in FilesystemName parameter. Remember, that filesystem names cannot have big letters or special chars!

 

 

Get permissions from filesystem or path


It will return a raw value of x-ms-acl property like: user::rwx,group::r-x,other::—
In Path parameter:
If you want get permission for filesystem just use “/”.
If you want to get permissions for a given path use it without leading “/” in.ex.: “Folder1/Folder2/File.csv”
 

Set permissions on filesystem or path


In Path parameter:
If you want set permission for filesystem just use “/”.
If you want to set permissions for a given path use it without leading “/” in.ex.: “Folder1/Folder2/File.csv”
Example of setting “Other” the Default –X (execute) permission (set it as $PermissionString  var):
user::rwx,group::r-x,other::--x,default:other::--x
 

Good luck!

102 thoughts on “Connecting to Azure Data Lake Storage Gen2 from PowerShell using REST API – a step-by-step guide

  1. Hey Michal.

    Very informative post. Helped me understand lot of semantics around the authorization header .
    Your powershell script to create a file system works like a charm. However due to environment restrictions, I am trying to emulate that in a bash script (using curl)
    and keep hitting this roadblock

    Error:
    curl: (6) Could not resolve host: PUT
    {“error”:{“code”:”AuthenticationFailed”,”message”:”Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:e422daa5-901f-00d8-0231-54a8cc000000\nTime:2019-08-16T12:54:36.0101349Z”}}https://storageaccountname.dfs.core.windows.net/filesystemname?resource=filesystem

    ————————————————————————————————–
    #!/usr/bin/env bash
    STORAGE_ACCOUNT_NAME=${1}
    FILESYSTEM_NAME=${2}
    ACCESS_KEY=${3}

    request_date=$(TZ=GMT date “+%a, %d %h %Y %H:%M:%S %Z”)
    #request_date=$(date +%Y-%m-%d)
    storage_service_version=”2018-11-09″

    # HTTP Request headers
    x_ms_date_h=”x-ms-date:$request_date”
    x_ms_version_h=”x-ms-version:$storage_service_version”

    n=”\n”

    method=”PUT”

    string_to_sign+=”$method\n\n\n\n\n\n\n\n\n\n\n\n$x_ms_date_h\n$x_ms_version_h\n/$STORAGE_ACCOUNT_NAME/$FILESYSTEM_NAME\nresource:filesystem”

    # Decode the Base64 encoded access key, convert to Hex.
    shared_key=”$(echo -n $ACCESS_KEY | base64 -d -w0 | xxd -p -c256)”

    # Create the HMAC signature for the Authorization header
    signed_signature=$(printf “$string_to_sign” | iconv -t utf-8 | openssl dgst -sha256 -mac HMAC -macopt “hexkey:$shared_key” -binary | base64 -w0)

    auth_header=”Authorization: SharedKey ${STORAGE_ACCOUNT_NAME}:$signed_signature”

    uri=”https://$STORAGE_ACCOUNT_NAME.dfs.core.windows.net/$FILESYSTEM_NAME?resource=filesystem”

    curl -d -X $method -H “$x_ms_date_h” -H “$x_ms_version_h” -H “$auth_header” $uri

    if [ $? -eq 0 ]; then
    echo $uri
    exit 0;
    fi;
    exit 1
    —————————————————————————————————

  2. Hi Michal,
    Thank you for the blog. I’m able to create files and folders using your script. However I’m having trouble renaming the file. I used “x-ms-rename-source” in the header but could not get it to work.
    I keep getting error
    Here’s the code I have been using
    $stringToSign +=
    # SECTION: CanonicalizedHeaders + “\n” #
    “x-ms-date:$date” + $n +
    “x-ms-version:2018-11-09” + $n +
    “x-ms-rename-source:/adlsg2filesystemname/folderpath/filename” + $n
    # SECTION: CanonicalizedHeaders + “\n” #

    $stringToSign +=
    # SECTION: CanonicalizedResource + “\n” #
    “/$StorageAccountName/$FilesystemName” + $PathToCreate + $n
    # SECTION: CanonicalizedResource + “\n” #

    $sharedKey = [System.Convert]::FromBase64String($AccessKey)
    $hasher = New-Object System.Security.Cryptography.HMACSHA256
    $hasher.Key = $sharedKey

    $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

    $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

    $headers = @{“x-ms-date”=$date}
    $headers.Add(“x-ms-version”,”2018-11-09″)
    $headers.Add(“x-ms-rename-source”,”/adlsg2filesystemname/folderpath/filename”)
    $headers.Add(“Authorization”,$authHeader)
    $headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

    $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + $PathToCreate

    Could you please tell me what am I missing here?

      1. Unfortunately, I don’t have much time for a full code review and writing it as a script.
        But I can point few interesting and important facts.

        In KD’s example, in canonicalized headers: x-ms-rename-source is AFTER x-ms-version, and this is against the rule of having lexicographical sort order. “x-ms-r” should be before “x-ms-v”.

        Now, regarding the “x-ms-rename-source” itself, look at the API documentation here, section request headers, under x-ms-rename-source:

        This value must be a URL percent-encoded string. Note that the string may only contain ASCII characters in the ISO-8859-1 character set.

        This is important. If you will sniff rename operation in Azure Storage Explorer, as i described it in my other blog post here, you will see an interesting situation:

        ase sniffing

        Well, I do not know why it is encoded in request URL and not in the x-ms-rename-source, BUT there is some kind of percent-encoding and you need to figure it out 🙂
        SO at least please try to change the path in Request URL, then if it will fail, maybe it should be modified also in canonicalized header?

        As I have limited time, it would be great if you will post your findings here. As soon as I have a little more time I will prepare a script based on this. Thanks!

        1. Hi Michal,

          The issue was with canonicalized headers. I tried with x-ms-rename-source before x-ms-version and its working fine now.

          Thank you for your help.

          -Charan

            1. Hi Michal,

              I am executing the same code from Azure Powershell Function App but getting authorization exception.

              Exception: “2019-11-01T06:35:00.249 [Error] ERROR: Invoke-RestMethod : {“error”:{“code”:”AuthorizationFailure”,”message”:”This request is not authorized to perform this operation.\nRequestId:77a8e674-f01f-0089-097e-90705c000000\nTime:2019-11-01T06:34:59.9031753Z”}}”

              I am having contributor access to Azure Function. Do I need any more access to execute Invoke-RestMethod?

              Please help me on this.

              Thanks,
              Charan

              1. Have you tried with elevated privileges e.g Account Admin role ,just to ensure it is really RBAC issue ?

                1. Hi Mohan,

                  Yes, we tried with Admin role and its working. Can you please provide some more insight on RBAC set up.

                  Thanks,
                  Charan

          1. Hi Charan and Michal ,

            thanks to both of you for all the inputs , the post is extremely insightful and helpful.

            I need your inputs on renaming the folder pls , added the code here , i am consistently getting error , stating , “One of the headers specified in the request is not supported.” Spent almost 2 days but invain..

            However the code for creating file-system , folder and deleting it is working fine.

            function ADLRenameFolder
            {
            Param(
            [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
            [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
            [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey,
            [Parameter(Mandatory=$True,Position=4)] [string] $PathToCreate,
            [Parameter(Mandatory=$True,Position=5)] [string] $SourcePath
            )

            # Rest documentation:
            # https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create

            $PathToCreate =$PathToCreate.trim(“/”) # remove all “//path” or “path/”
            $SourcePath = $FilesystemName + “/” + $SourcePath.trim(“/”) # remove all “//path” or “path/”

            #$URL = Read-Host “Enter URL to Decode”
            $PathToCreate = [System.Web.HttpUtility]::UrlEncode($PathToCreate)
            Write-Host $PathToCreate

            $PathToCreate=[System.Web.HTTPUtility]::UrlEncode($PathToCreate)

            $date = [System.DateTime]::UtcNow.ToString(“R”) # ex: Sun, 10 Mar 2019 11:50:10 GMT

            $n = “`n”
            $method = “PUT”

            $stringToSign = “$method$n” #VERB
            $stringToSign += “$n” # Content-Encoding + “\n” +
            $stringToSign += “$n” # Content-Language + “\n” +
            $stringToSign += “$n” # Content-Length + “\n” +
            $stringToSign += “$n” # Content-MD5 + “\n” +
            $stringToSign += “$n” # Content-Type + “\n” +
            $stringToSign += “$n” # Date + “\n” +
            $stringToSign += “$n” # If-Modified-Since + “\n” +
            $stringToSign += “$n” # If-Match + “\n” +
            $stringToSign += “*” + “$n” # If-None-Match + “\n” +
            $stringToSign += “$n” # If-Unmodified-Since + “\n” +
            $stringToSign += “$n” # Range + “\n” +
            $stringToSign +=

            “x-ms-date:$date” + $n +
            “x-ms-rename-source:$SourcePath” + $n +
            “x-ms-version:2018-11-09” + $n #

            $stringToSign +=

            “/$StorageAccountName/$FilesystemName/” + $PathToCreate + $n +
            “resource:directory”#

            Write-Output $stringToSign

            $sharedKey = [System.Convert]::FromBase64String($AccessKey)
            $hasher = New-Object System.Security.Cryptography.HMACSHA256
            $hasher.Key = $sharedKey

            $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

            $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

            $headers = @{“x-ms-date”=$date}
            $headers.Add(“x-ms-rename-source”,$SourcePath)
            $headers.Add(“x-ms-version”,”2018-11-09″)
            $headers.Add(“Authorization”,$authHeader)
            $headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

            $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + “/” + $PathToCreate + “?resource=directory”

            Write-Output $URI

            Write-Output $headers
            try {
            Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
            $true
            }
            catch {
            $ErrorMessage = $_.Exception.Message
            $StatusDescription = $_.Exception.Response.StatusDescription
            $false

            Write-Output $ErrorMessage + ” ” + $StatusDescription
            #Throw $ErrorMessage + ” ” + $StatusDescription
            }
            }

            **************************** Invoking Function below *********************************************
            $StorageAccountName=”teststorageaccount”
            $FilesystemName=”testfilesystem”
            $AccessKey=””
            $PathToCreate=”/data/inv/researchrenamed”
            $SourcePath=”/data/inv/research2″

            Add-Type -AssemblyName System.Web

            ADLRenameFolder $StorageAccountName $FilesystemName $AccessKey $PathToCreate $SourcePath

            1. Hmm, from what I can see above there is definitely a problem with:
              $stringToSign += “*” + “$n” # If-None-Match + “\n” +

              you cannot use If-None-Match in renaming, only x-ms-source-if-none-match is allowed.
              Try to remove it:
              $stringToSign += “$n” # If-None-Match + “\n” +

              Also, you have:
              “?resource=directory”

              and in the documentation they mentioned resource like this:
              Required only for Create File and Create Directory. The value must be “file” or “directory”.
              So I think this should be also removed 🙂

              Try it and return with a feedback please 🙂

              By the way, looks like I need to take a while and prepare rename example :]

              1. Thanks for quick reply , i tried both inputs but error is same. I have removed one at a time

                When removed only $stringToSign += “$n” # If-None-Match + “\n” +
                Same err:400 One of the headers specified in the request is not supported.

                When both removed “?resource=directory”
                Error 400 : The source URI is invalid.

                  1. Uhh, it was easier to write it myselft than to debug in the comment 🙂
                    Here it is, working renaming (without nomach, which did not work for me and I do not have time to analyze it, so watch out when you will try to rename folders to something that already exists 😀 )
                    Just remember to encode URI in canonical headers and resource, do not use resource parameters and everything will work 🙂

                    Example link: Rename file/folder (or move to the other path)

                    1. Hi Michal ,

                      Really appreciate your quick response and contribution to community.

                      When i compared with your new code , i realized i was not encoding the source path( PathToRename) but only value of RenameTo.

                      The third change of encoding i did post addressing the previous two inputs you have provided.

                      Cheers !!!

                      Here goes updated function…..
                      function ADLRenameFolder
                      {
                      Param(
                      [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
                      [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
                      [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey,
                      [Parameter(Mandatory=$True,Position=4)] [string] $PathToCreate,
                      [Parameter(Mandatory=$True,Position=5)] [string] $SourcePath
                      )

                      # Rest documentation:
                      # https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create

                      $requestid = [guid]::NewGuid().ToString()
                      Write-Output $requestid

                      $PathToCreate =$PathToCreate.trim(“/”) # remove all “//path” or “path/”

                      $SourcePath = “/” + $FilesystemName + “/” + $SourcePath.trim(“/”) # remove all “//path” or “path/”

                      #$URL = Read-Host “Enter URL to Decode”
                      $PathToCreate_encoded = [System.Web.HttpUtility]::UrlEncode($PathToCreate)
                      Write-Host “New Path” + $PathToCreate

                      $SourcePath_encoded=[System.Web.HttpUtility]::UrlEncode($SourcePath)

                      $date = [System.DateTime]::UtcNow.ToString(“R”) # ex: Sun, 10 Mar 2019 11:50:10 GMT

                      $n = “`n”
                      $method = “PUT”

                      $stringToSign = “$method$n” #VERB
                      $stringToSign += “$n” # Content-Encoding + “\n” +
                      $stringToSign += “$n” # Content-Language + “\n” +
                      $stringToSign += “$n” # Content-Length + “\n” +
                      $stringToSign += “$n” # Content-MD5 + “\n” +
                      $stringToSign += “$n” # Content-Type + “\n” +
                      $stringToSign += “$n” # Date + “\n” +
                      $stringToSign += “$n” # If-Modified-Since + “\n” +
                      $stringToSign += “$n” # If-Match + “\n” +
                      $stringToSign += “$n” # If-None-Match + “\n” +
                      $stringToSign += “$n” # If-Unmodified-Since + “\n” +
                      $stringToSign += “$n” # Range + “\n” +
                      $stringToSign +=

                      “x-ms-client-request-id:$requestid” + $n +
                      “x-ms-date:$date” + $n +
                      “x-ms-rename-source:$SourcePath_encoded” + $n +
                      “x-ms-version:2018-11-09” + $n #

                      $stringToSign +=

                      “/$StorageAccountName/$FilesystemName/” + $PathToCreate_encoded

                      Write-Output $stringToSign

                      $sharedKey = [System.Convert]::FromBase64String($AccessKey)
                      $hasher = New-Object System.Security.Cryptography.HMACSHA256
                      $hasher.Key = $sharedKey

                      $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))

                      $authHeader = “SharedKey ${StorageAccountName}:$signedSignature”

                      $headers = @{“x-ms-date”=$date}
                      $headers.Add(“x-ms-client-request-id”,$requestid)
                      $headers.Add(“x-ms-rename-source”,$SourcePath_encoded)
                      $headers.Add(“x-ms-version”,”2018-11-09″)
                      $headers.Add(“Authorization”,$authHeader)
                      #$headers.Add(“If-None-Match”,”*”) # To fail if the destination already exists, use a conditional request with If-None-Match: “*”

                      $URI = “https://$StorageAccountName.dfs.core.windows.net/” + $FilesystemName + “/” + $PathToCreate_encoded

                      Write-Output $URI

                      Write-Output $headers
                      try {
                      Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
                      $true
                      }
                      catch {
                      $ErrorMessage = $_.Exception.Message
                      $StatusDescription = $_.Exception.Response.StatusDescription
                      $false

                      Write-Output $ErrorMessage + ” ” + $StatusDescription
                      #Throw $ErrorMessage + ” ” + $StatusDescription
                      }
                      }

                      #**************************** Invoking Function below *********************************************
                      $StorageAccountName=”yourAccountName”
                      $FilesystemName=”testfilesystem”
                      $AccessKey=”yourAccountNameKey”
                      $PathToCreate=”/data/inv/research_renamed”
                      $SourcePath=”/data/inv/research”

                      Add-Type -AssemblyName System.Web

                      ADLRenameFolder $StorageAccountName $FilesystemName $AccessKey $PathToCreate $SourcePath

  3. hi Michał Pawlikowski ,

    You above code works just fine hierarchical folder creation .
    I my case i need to assign service Principal at Folder Level. Say top Level and assign service Prinicpal.
    Can you suggest how to assign already created service Principal while time of folder Creation.

    Regards
    Deepak

  4. Also I tried “Set permissions on filesystem or path” . In place of $PermissionString i used Key of service Principal but it did not helped.

    thanks

  5. Hi Michal,
    Hope you are doing well.
    I am trying hierarchical folder creation in adls gen 2 with service principal. Fir you solution in “set solution on file system or path” with what component I have to replace $Permissionstring.

    1. Hi Deepak.
      The answer is just in the comments above 🙂
      Look for my reply to Mark’s question:

      post

      Just do it manually in Azure Storage Explorer and see for yourself how the string is made there.
      Just reuse it within your code. Yeap, simple as that 🙂

  6. Hi Michal,

    Hope you are doing well.

    Any sample script are there to delete folders or files from adlsgen2.
    i am following your above script for the same with method “Delete” and changed some script systax.
    but its not working.

    1. Hi Shreenivas,

      thanks, I’m doing great 🙂

      To construct a proper header you need to have recursive parameter and send DELETE command to the rest api.
      I’ve created an example for you, look for “Delete folder or file (simple, without continuation)” in the examples above.
      Please bear in mind, that I did not include optional parameter “continuation” that handles the situation when you are trying to delete a lot of files (nested, so recursively under the folder that you will try to delete, see details in the example)

      Regards,
      m.

      1. Hi Michal,

        Thanks lot for suggestion and sample code.
        I was not passing the recursive value and script was failing.

        Now it’s working through your help

        Regards,
        Shreenivas Mirajkar

  7. Hi Michal,

    I was referring this blog for creation of Filesystem as well as creating Folder under that Filesystem.
    The Filesystem creation go success, but the folder creation is resulting in error

    The remote server returned an error: (403) Forbidden. Server failed to authenticate the request. Make sure the value
    of Authorization header is formed correctly including the signature.
    At C:\Users\goswams1\Documents\WIP\Documents\Powershell-Commands\CreateFolder.ps1:68 char:5
    + Throw $ErrorMessage + ” ” + $StatusDescription
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OperationStopped: (The remote serv… the signature.:String) [], RuntimeException
    + FullyQualifiedErrorId : The remote server returned an error: (403) Forbidden. Server failed to authenticate the
    request. Make sure the value of Authorization header is formed correctly including the signature.

    I have just copied the piece of code mentioned at the section – Create directory (or path of directories).
    I am giving the input for Filesystem as the one created with your script and Folder Path as /Landing/Import. But this fails.

    I am just clueless in this. Please help me here.

    1. Unfortunately, you need to share your code anyway. I’ve tested it myself and folder creation example works like a charm.. If you did not make any mistake in access key or sotrage account name, the only reason is something changed in the code. Please use portals like this: https://paste.ofcode.org/ to share your code.

      1. Hi Michal,

        This actually worked. I just re-initiated the session and this work fine. If I get stuck again, will reach out to you.
        I would like to convey a big thanks for creating this blog. This solved a major chunk of work for me.

        Regards,
        Sudipta.

Leave a Reply