This repository was archived by the owner on Jul 18, 2024. It is now read-only.
This repository was archived by the owner on Jul 18, 2024. It is now read-only.
blobxfer fails on download when empty blob exist #92
Closed
Description
Problem Description
When downloading a blob and an empty blob with the same prefix exists, blobxfer breaks.
e.g.:
/root/blob1/data (non zero bytes)
/root/blob1 (zero bytes)
those "empty" blobs are created whenever I mount the blob with "blobfuse" and execute a "mkdir blob1" on my mount.
Command:
blobxfer download --storage-account X --local-path . --remote-path phillytools/pt-results/2019-02-27/application_1549393859033_5094
Azure blobxfer parameters output
============================================
Azure blobxfer parameters
============================================
blobxfer version: 1.7.0
platform: Linux-4.15.0-44-generic-x86_64-with-debian-buster-sid
components: CPython=3.6.7-64bit azstor.blob=1.5.0 azstor.file=1.4.0 crypt=2.5 req=2.21.0
transfer direction: Azure -> local
workers: disk=4 xfer=3 (msoc=8) md5=0 crypto=0
log file: None
dry run: False
resume file: None
timeout: connect=10 read=200 max_retries=1000
mode: StorageModes.Auto
skip on: fs_match=False lmt_ge=False md5=False
delete extraneous: False
overwrite: True
recursive: True
rename single: False
chunk size bytes: 0
strip components: 0
compute file md5: False
restore properties: attr=False lmt=False
rsa private key: None
local destination: .
============================================
2019-02-27 18:26:45.545 INFO - blobxfer start time: 2019-02-27 18:26:45.545830-05:00
2019-02-27 18:26:45.658 DEBUG - dest is_dir=True for 1 specs
2019-02-27 18:26:45.658 INFO - downloading blobs/files to local path: .
2019-02-27 18:26:45.658 DEBUG - spawning 3 transfer threads
2019-02-27 18:26:45.669 DEBUG - spawning 4 disk threads
2019-02-27 18:26:45.696 INFO - MD5: SKIPPED, phillytools/pt-results/2019-02-27/application_1549393859033_5094 None <L..R> 1B2M2Y8AsgTpgAmY7PhCfg==
2019-02-27 18:26:45.696 DEBUG - 0 files 0.0000 MiB filesize and/or lmt_ge skipped
2019-02-27 18:26:45.696 DEBUG - 2 remote files processed, waiting for download completion of approx. 0.0846 MiB
2019-02-27 18:26:45.696 ERROR - exceptions encountered while downloading
2019-02-27 18:26:45.696 ERROR - [Errno 17] File exists: 'pt-results/2019-02-27/application_1549393859033_5094'
Traceback (most recent call last):
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/operations/download.py", line 871, in start
self._run()
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/operations/download.py", line 831, in _run
raise self._exceptions[0]
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/operations/download.py", line 493, in _worker_thread_transfer
self._process_download_descriptor(dd)
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/operations/download.py", line 524, in _process_download_descriptor
offsets, resume_bytes = dd.next_offsets()
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/models/download.py", line 556, in next_offsets
self._allocate_disk_space()
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/blobxfer/models/download.py", line 416, in _allocate_disk_space
mode=0o750, parents=True, exist_ok=True)
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/pathlib2/__init__.py", line 1464, in mkdir
_try_except_filenotfounderror(_try_func, _exc_func)
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/pathlib2/__init__.py", line 90, in _try_except_filenotfounderror
try_func()
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/pathlib2/__init__.py", line 1455, in _try_func
self._accessor.mkdir(self, mode)
File "/home/alsordon/miniconda/envs/py36/lib/python3.6/site-packages/pathlib2/__init__.py", line 541, in wrapped
return strfunc(str(pathobj), *args)
FileExistsError: [Errno 17] File exists: 'pt-results/2019-02-27/application_1549393859033_5094'
Activity