Values assigned to `exclude` and/or `include` parameters not being consumed by `download`, `synccopy`, or `upload` commands when using a YAML configuration file

[zhodowanec]

Problem Description

The exclude and include parameters for the download, synccopy, and upload commands do not work when using a YAML configuration file.

Azure blobxfer parameters output

============================================
         Azure blobxfer parameters
============================================
         blobxfer version: 1.9.4
                 platform: Linux-4.19.128-microsoft-standard-x86_64-with
               components: CPython=3.7.5-64bit azstor.blob=2.1.0 azstor.file=2.1.0 crypt=2.8 req=2.22.0
       transfer direction: Azure -> Azure
                  workers: disk=0 xfer=24 md5=0 crypto=0
                 log file: None
                  dry run: False
              resume file: None
                  timeout: connect=10 read=200 max_retries=1000
              source mode: StorageModes.Auto
                dest mode: StorageModes.Auto
         server side copy: True
                  skip on: fs_match=False lmt_ge=False md5=True
                   delete: extraneous=True only=False
                overwrite: True
                recursive: True
            rename single: False
         strip components: 0
              access tier: None
============================================

Steps to Reproduce

1. Create $PWD/config/config.yaml file

2. Add the following configuration to $PWD/config/config.yaml:

   version: 1
   
   options:
     log_file: blobxfer.log
     progress_bar: true
     verbose: true
   
   azure_storage:
     endpoint: core.windows.net
     accounts:
       devstoreaccount1: XXXXXXXXXXXXX
   
   synccopy:
     - source:
         - devstoreaccount1: src
       destination:
         - devstoreaccount1: dest
       include:
         - "*"
       exclude:
         - "*.csv"
       options:
         mode: auto
         dest_mode: auto
         access_tier: null
         delete_extraneous_destination: true
         delete_only: false
         overwrite: true
         recursive: true
         rename: false
         server_side_copy: true
         skip_on:
           filesize_match: false
           lmt_ge: false
           md5_match: true

3. Upload files matching both include and exclude file matching patterns to the source storage account(s) defined in $PWD/config/config.yaml config

4. Run the following command:

   docker run -v $PWD/config:/config mcr.microsoft.com/blobxfer synccopy --config /config/config.yaml

Expected Results

Only files that match the include file matching pattern, and not those matching the exclude file matching pattern, be copied from devstoreaccount1: src container to the devstoreaccount1: dest container.

Actual Results

All files from the devstoreaccount1: src container, including those explicitly defined to be excluded via the exclude parameter, are copied to the devstoreaccount1: dest container

Additional Logs

INFO - blobxfer start time: 2021-02-04 00:28:08.993581+00:00
DEBUG - spawning 24 transfer threads
DEBUG - 5 remote files to sync, waiting for copy completion of approx. 0.0000 MiB
DEBUG - attempting to delete extraneous blobs/files from: devstoreaccount1;core.windows.net;publish
INFO - deleted 0 extraneous blobs/files
INFO - elapsed copy time and throughput of 0.0000 GiB: 0.862 sec, 0.0000 Mbps (0.000 MiB/sec)
INFO - blobxfer end time: 2021-02-04 00:28:10.566358+00:00 (elapsed: 1.573 sec)

Additional Comments

The issue appears to be a result of using type multiple for the _exclude_option and _include_option objects on cli.py, in conjunction with the condition in the _merge_setting function on settings.py. For reference, When passing a default with multiple=True, the default value must be a list or tuple, otherwise it will be interpreted as a list of single characters.

[SailingYYC]

@alfpark, how do we get this approved? This has prevented our use of the tool for months.

I suspect on the last published release, the click dependency was upgraded, they changed the output for cli args defined as multiple to be an empty tuple when none are specified on the cli invocation.

[SailingYYC]

@rems75 @alexandair As contributors on this project, are either of you aware of how we can get PRs/issues updated?

[zhodowanec]

I have added the below discussion topic to the Microsoft Tech Community.

https://techcommunity.microsoft.com/t5/azure-storage/blobxfer-support/m-p/2141138#M112

[alfpark]

Thanks for submitting your PR and apologies for the delay. After doing more tests on your PR change, there is a regression on bool values. A fix will be made a top your PR to account for this.

[SailingYYC]

Thank you @alfpark