Connection reset by peer

[BossensM]

Problem Description

When uploading using Blobxfer, I get an azure exception ('Connection aborted.', error(104, 'Connection reset by peer')) on a random basis when uploading a lot of small files to Azure blob storage (from a VM in Azure)

2018-06-29 04:46:58,221 DEBUG - 410955 local files processed, waiting for upload completion of approx. 41268.6124 MiB
2018-06-29 04:46:58,222 ERROR - exceptions encountered while uploading
2018-06-29 04:46:58,222 ERROR - ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
Traceback (most recent call last):
 File "/usr/local/lib/python2.7/dist-packages/blobxfer/operations/upload.py", line 1208, in start
 self._run()
 File "/usr/local/lib/python2.7/dist-packages/blobxfer/operations/upload.py", line 1171, in _run
 raise self._exceptions[0]
AzureException: ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
Traceback (most recent call last):
 File "/usr/local/bin/blobxfer", line 11, in <module>
 sys.exit(cli())
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
 return self.main(*args, **kwargs)
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
 rv = self.invoke(ctx)
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
 return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
 return ctx.invoke(self.callback, **ctx.params)
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
 return callback(*args, **kwargs)
 File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 64, in new_func
 return ctx.invoke(f, obj, *args[1:], **kwargs)
 File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
 return callback(*args, **kwargs)
 File "/usr/local/lib/python2.7/dist-packages/blobxfer_cli/cli.py", line 1040, in upload
 ctx.general_options, ctx.credentials, spec
 File "/usr/local/lib/python2.7/dist-packages/blobxfer/operations/upload.py", line 1208, in start
 self._run()
 File "/usr/local/lib/python2.7/dist-packages/blobxfer/operations/upload.py", line 1171, in _run
 raise self._exceptions[0]
azure.common.AzureException: ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

Azure blobxfer parameters output

blobxfer version: 1.3.0
 platform: Linux-4.15.0-1013-azure-x86_64-with-Ubuntu-16.04-xenial
 components: CPython=2.7.12-64bit azstor.blob=1.3.0 azstor.file=1.1.0 crypt=2.2.2 req=2.19.1
 transfer direction: local -> Azure
 workers: disk=16 xfer=8 md5=0 crypto=0
 log file: None
 dry run: False
 resume file: None
 timeout: connect=100.0 read=200.0 max_retries=1000000
 mode: StorageModes.Auto
 skip on: fs_match=False lmt_ge=False md5=False
 delete extraneous: False
 overwrite: True
 recursive: True
 rename single: False
 access tier: None
 chunk size bytes: 0
 one shot bytes: 0
 strip components: 0
 store properties: attr=False md5=False
 rsa public key: None
 local source paths: /datadisk/working

Steps to Reproduce

CLI usage of blobxfer, uploading large amounts of (small) files.

Expected Results

Successful upload or retry

Actual Results

Failing upload

Additional Logs

INSERT ADDITIONAL LOGS HERE

Additonal Comments

[alfpark]

Looks like you may be getting intermittent errors from storage. blobxfer should be retrying these cases, but it's possible the newer version of requests have mutated the exception (again). You can also create a resume db file that will allow you to resume after failures (re-run the command again with the resume db option) as a temporary workaround.

[BossensM]

Thanks! I'll be updating the version and adding the resume file, and see how it goes.

Also sometimes I get the following error, on the same type of data uploads.

SSLError: The write operation timed out    

Is this also an exception with can be retried automatically ?

[alfpark]

Please see: https://github.com/Azure/blobxfer/blob/master/docs/98-performance-considerations.md#pyopenssl

If possible, please install blobxfer for Python3 as you are using Ubuntu 16.04. Also ensure that you are running in a virtual env and not directly using the system Python packages.