AuthenticationFailed attempting to use SAS URL

[ckcollab]

First of all: thanks for all of the hard work on this project!

Problem Description

I am attempting to upload to a SAS URL, but I'm a bit confused on how exactly to specify the storage account name, bucket and full path.

Azure blobxfer parameters output

============================================
         Azure blobxfer parameters
============================================
         blobxfer version: 1.3.1
                 platform: Darwin-16.7.0-x86_64-i386-64bit
               components: CPython=2.7.15-64bit azstor.blob=1.3.0 azstor.file=1.3.0 crypt=2.3 req=2.19.1
       transfer direction: local -> Azure
                  workers: disk=16 xfer=32 md5=0 crypto=0
                 log file: None
                  dry run: False
              resume file: None
                  timeout: connect=10 read=19 max_retries=1000
                     mode: StorageModes.Auto
                  skip on: fs_match=False lmt_ge=False md5=False
        delete extraneous: False
                overwrite: True
                recursive: True
            rename single: False
              access tier: None
         chunk size bytes: 0
           one shot bytes: 0
         strip components: 0
         store properties: attr=False md5=False
           rsa public key: None
       local source paths: reference_data.zip

Steps to Reproduce

Here's what I'm doing now:

blobxfer upload \
    --storage-account azure-storage-account-name \
    --sas "?sr=b&sp=w&sv=2014-02-14&st=2018-08-01T22%3A25%3A26Z&sig=some_signature=&se=2018-11-29T22%3A25%3A26Z" \
    --remote-path "bucket_name/and/then/the/path" \
    -v \
    --enable-azure-storage-logger \
    --local-path reference_data.zip

Expected Results

To upload the file

Actual Results

2018-08-01 20:35:42,328 INFO - blobxfer start time: 2018-08-01 20:35:42.328140-07:00
2018-08-01 20:35:42,328 DEBUG - spawning 16 disk threads
2018-08-01 20:35:42,332 DEBUG - spawning 32 transfer threads
2018-08-01 20:35:44,022 ERROR - Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

[alfpark]

This is because the SAS given is a write-only SAS. blobxfer by default attempts to compare the remote file, if it exists, to execute options given.

As a workaround, please provide a read/write SAS.

[ckcollab]

Thank you so much for the quick help!

--sas "?sr=b&sp=rw&sv=2014-02-14&st=2018-08-02T06%3A54%3A33Z&sig=the_signature=&se=2019-01-29T06%3A54%3A33Z"

I generated rw permissions here, but received the same error.

[alfpark]

Can you post the stack trace? I want to verify that it's throwing at the same place in the code.

A few other things to try:

1. Can you drop the st parameter of the SAS? (re-generate the SAS without a start time).
2. Try creating a container-level SAS with permissive rwl permissions (also don't use a start time).

[ckcollab]

Stack trace from previous attempt

2018-08-02 00:00:18,154 ERROR blobxfer.operations.upload:start:1215 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

Traceback (most recent call last):
  File "/usr/local/bin/blobxfer", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/blobxfer_cli/cli.py", line 1040, in upload
    ctx.general_options, ctx.credentials, spec
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

Attempting rwl without start

--sas "?sr=b&sp=rwl&sig=the_signature&sv=2014-02-14&se=2018-10-01T07%3A27%3A08Z"

2018-08-02 00:21:39,115 ERROR blobxfer.operations.upload:start:1215 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

Traceback (most recent call last):
  File "/usr/local/bin/blobxfer", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/blobxfer_cli/cli.py", line 1040, in upload
    ctx.general_options, ctx.credentials, spec
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

Attempting rw without start

--sas "?sr=b&sp=rw&sig=the_signature&sv=2014-02-14&se=2018-10-01T07%3A27%3A08Z"

2018-08-02 00:30:40,617 ERROR blobxfer.operations.upload:start:1215 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

Traceback (most recent call last):
  File "/usr/local/bin/blobxfer", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/blobxfer_cli/cli.py", line 1040, in upload
    ctx.general_options, ctx.credentials, spec
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1208, in start
    self._run()
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 1102, in _run
    self._generate_destination_for_source(src)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 931, in _generate_destination_for_source
    ase = self._check_for_existing_remote(sa, cont, name)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/upload.py", line 872, in _check_for_existing_remote
    sa.block_blob_client, cont, name, self._spec.options.mode)
  File "/usr/local/lib/python2.7/site-packages/blobxfer/operations/azure/blob/__init__.py", line 83, in get_blob_properties
    container_name=container, blob_name=prefix, timeout=timeout)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py", line 1530, in get_blob_properties
    return self._perform_request(request, _parse_blob, [blob_name, snapshot])
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/azure/storage/common/storageclient.py", line 381, in _perform_request
    raise ex
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed

I also attempted these without expirations as well.. the stack traces look the same, apologies for spamming them, will edit them out if they're not useful.

[alfpark]

Thanks, they do look the same.

Can you try the second suggestion, using a container-level SAS without a start time?

[ckcollab]

I will try this, but for my case I need to just give permissions for a specific file only? Would attempting it on the folder containing the single file be worthwhile as well?

[alfpark]

Agreed, it's non-optimal. This is just a test for your scenario to confirm a workaround in the meantime.