base path its self included in transfer when using --strip-components

[krezreb]

Problem Description

I am transferring between two containers in two storage accounts, let's call them a and b.
container a is a backup archive containing paths to different app backups, e.g.

a
   app1/data/
   app2/data/
   etc...

Container b is used directly by app1. Its root structure is the same as a/app1/data

So when I transfer between the two, I am using --strip-components 2 to remove app1/data from the blob paths.

Below is my exact command:

export SRC_CONTAINER=a
export SRC_PATH=app1/data
export DEST_CONTAINER=b

blobxfer synccopy \
--server-side-copy --progress-bar --delete \
--skip-on-md5-match \
--overwrite \
--remote-path $SRC_CONTAINER/$SRC_PATH \
--storage-account $SRC_STORAGE_ACCOUNT \
--sync-copy-dest-storage-account  $DEST_STORAGE_ACCOUNT \
--sync-copy-dest-remote-path $DEST_CONTAINER \
--strip-components 2  

I am unable to do the actual sync, see #118

However, when using --dry-run, it seems that that blobxfer is trying to create the parent folder its self on the destination.

2020-04-24 14:24:12.280 INFO - blobxfer start time: 2020-04-24 14:24:12.280271+02:00
2020-04-24 14:24:12.280 DEBUG - spawning 32 transfer threads
2020-04-24 14:24:20.144 INFO - [DRY RUN] synccopy: a/app1/data -> b/data
2020-04-24 14:24:20.197 INFO - [DRY RUN] synccopy: a/app1/data/CFS -> b/CFS
2020-04-24 14:24:20.247 INFO - [DRY RUN] synccopy: a/app1/data/CFS/2019 -> b/CFS/2019

In other words, a/app1/data -> b/data should not be there.

If I exclude it with --exclude $SRC_PATH , it looks ok, but I shouldn't have to do this in the first place.

Azure blobxfer parameters output

============================================
         Azure blobxfer parameters
============================================
         blobxfer version: 1.9.4
                 platform: Linux-5.4.0-26-generic-x86_64-with-glibc2.29
               components: CPython=3.8.2-64bit azstor.blob=2.1.0 azstor.file=2.1.0 crypt=2.8 req=2.22.0
       transfer direction: Azure -> Azure
                  workers: disk=0 xfer=32 md5=0 crypto=0
                 log file: None
                  dry run: True
              resume file: None
                  timeout: connect=10 read=200 max_retries=1000
              source mode: StorageModes.Auto
                dest mode: StorageModes.Auto
         server side copy: True
                  skip on: fs_match=False lmt_ge=False md5=True
                   delete: extraneous=True only=False
                overwrite: True
                recursive: True
            rename single: False
         strip components: 2
              access tier: None
============================================

[alfpark]

I've created a container with the same path as yours (a/app1/data/...) and specified the same options with explicit remote path to the exact a/app1/data along with strip components of 2. I fail to repro:

...
2021-04-16 19:47:45.969 INFO - blobxfer start time: 2021-04-16 19:47:45.969537+00:00
2021-04-16 19:47:45.969 DEBUG - spawning 16 transfer threads
2021-04-16 19:47:46.159 INFO - [DRY RUN] synccopy: syncsrc/app1/data/r.txt -> syncdst/r.txt
2021-04-16 19:47:46.188 INFO - [DRY RUN] synccopy: syncsrc/app1/data/s.PNG -> syncdst/s.PNG
2021-04-16 19:47:46.188 DEBUG - 2 remote files to sync, waiting for copy completion of approx. 0.0000 MiB
...

Synccopy sources are populated straight from Azure Storage, so objects being iterated are real. Due to the oddity of the behavior in #118, my guess is that you actually have a blob with the same name as your virtual directory in Azure storage. When I create this condition in my storage account, I can repro what you reported:

...
2021-04-16 19:52:05.072 DEBUG - spawning 16 transfer threads
2021-04-16 19:52:05.231 INFO - [DRY RUN] synccopy: syncsrc/app1/data -> syncdst/data
2021-04-16 19:52:05.256 INFO - [DRY RUN] synccopy: syncsrc/app1/data/r.txt -> syncdst/r.txt
2021-04-16 19:52:05.280 INFO - [DRY RUN] synccopy: syncsrc/app1/data/s.PNG -> syncdst/s.PNG
2021-04-16 19:52:05.281 DEBUG - 3 remote files to sync, waiting for copy completion of approx. 0.0000 MiB
...

This is exacerbated with prefix matching as noted in the limitations doc. I'll be making a fix to this logic to ignore sources that result in a component count of less than the strip count.