Skip to content

juicesync result differs with rsync when include two stars. #4445

Closed
@zhoucheng361

Description

**What happened**:
root@bench-01:~/juicefs# git pull && git checkout test-sync-option
root@bench-01:~/juicefs# python3 .github/scripts/hypo/syncrand_test.py -k test_sync1
setup_logger ./syncrand.log
2024-03-04 22:31:49,405 - INFO - do_mkdir /tmp/sync_src/a with mode 0o0 succeed
2024-03-04 22:31:49,405 - INFO - do_mkdir /tmp/sync_src2/a with mode 0o0 succeed
2024-03-04 22:31:49,405 - INFO - do_create_file /tmp/sync_src/a/a with mode w succeed
2024-03-04 22:31:49,405 - INFO - do_create_file /tmp/sync_src2/a/a with mode w succeed
2024-03-04 22:31:49,407 - INFO - rsync -r -vvv /tmp/sync_src/ /tmp/rsync/ --include aa/*** --exclude a?**
2024-03-04 22:31:49,410 - INFO - ./juicefs sync --dirs -v /tmp/sync_src/ /tmp/juicesync/ --include aa/*** --exclude a?**
Only in /tmp/rsync/a: a
Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.
E
======================================================================
ERROR: test_sync2 (__main__.TestFsrand2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".github/scripts/hypo/syncrand_test.py", line 17, in test_sync2
    state.sync(options=[{'option': '--include', 'pattern': 'aa/***'},
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 110, in sync
    )
  File "/root/hypothesis/hypothesis-python/src/hypothesis/stateful.py", line 681, in rule_wrapper
    return f(*args, **kwargs)
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 123, in sync
    raise e
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 120, in sync
    subprocess.check_call(['diff', '-r', self.DEST_RSYNC, self.DEST_JUICESYNC])
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 0.044s

FAILED (errors=1)
root@bench-01:~/juicefs# tree /tmp/sync_src /tmp/juicesync /tmp/rsync/
/tmp/sync_src
└── a
    └── a
/tmp/juicesync
└── a
/tmp/rsync/
└── a
    └── a

root@bench-01:~/juicefs# python3 .github/scripts/hypo/syncrand_test.py -k test_sync2
setup_logger ./syncrand.log
2024-03-04 22:40:47,440 - INFO - do_create_file /tmp/sync_src/a with mode w succeed
2024-03-04 22:40:47,440 - INFO - do_create_file /tmp/sync_src2/a with mode w succeed
2024-03-04 22:40:47,442 - INFO - rsync -r -vvv /tmp/sync_src/ /tmp/rsync/ --exclude **/***
2024-03-04 22:40:47,444 - INFO - ./juicefs sync --dirs -v /tmp/sync_src/ /tmp/juicesync/ --exclude **/***
Only in /tmp/juicesync: a
Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.
E
======================================================================
ERROR: test_sync1 (__main__.TestFsrand2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".github/scripts/hypo/syncrand_test.py", line 9, in test_sync1
    state.sync(options=[{'option': '--exclude', 'pattern': '**/***'}])
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 110, in sync
    )
  File "/root/hypothesis/hypothesis-python/src/hypothesis/stateful.py", line 681, in rule_wrapper
    return f(*args, **kwargs)
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 123, in sync
    raise e
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 120, in sync
    subprocess.check_call(['diff', '-r', self.DEST_RSYNC, self.DEST_JUICESYNC])
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 0.044s

root@bench-01:~/juicefs# tree /tmp/sync_src /tmp/rsync /tmp/juicesync
/tmp/sync_src
└── a
/tmp/rsync
/tmp/juicesync
└── a

root@bench-01:~/juicefs# python3 .github/scripts/hypo/syncrand_test.py -k test_sync5
setup_logger ./syncrand.log
2024-03-04 23:16:08,585 - INFO - rsync -r -vvv /tmp/sync_src/ /tmp/rsync/ --include a
2024-03-04 23:16:08,587 - INFO - ./juicefs sync --dirs -v /tmp/sync_src/ /tmp/juicesync/ --include a
2024-03-04 23:16:08,620 - INFO - do_mkdir /tmp/sync_src/a with mode 0o0 succeed
2024-03-04 23:16:08,621 - INFO - do_mkdir /tmp/sync_src2/a with mode 0o0 succeed
2024-03-04 23:16:08,621 - INFO - do_create_file /tmp/sync_src/a/a with mode w succeed
2024-03-04 23:16:08,621 - INFO - do_create_file /tmp/sync_src2/a/a with mode w succeed
2024-03-04 23:16:08,623 - INFO - rsync -r -vvv /tmp/sync_src/ /tmp/rsync/ --include aa --exclude a?**
2024-03-04 23:16:08,625 - INFO - ./juicefs sync --dirs -v /tmp/sync_src/ /tmp/juicesync/ --include aa --exclude a?**
Only in /tmp/rsync/a: a
Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.
E
======================================================================
ERROR: test_sync5 (__main__.TestFsrand2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".github/scripts/hypo/syncrand_test.py", line 42, in test_sync5
    state.sync(options=[{'option': '--include', 'pattern': 'aa'},
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 113, in sync
    )
  File "/root/hypothesis/hypothesis-python/src/hypothesis/stateful.py", line 681, in rule_wrapper
    return f(*args, **kwargs)
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 126, in sync
    raise e
  File "/root/juicefs/.github/scripts/hypo/syncrand.py", line 123, in sync
    subprocess.check_call(['diff', '-r', self.DEST_RSYNC, self.DEST_JUICESYNC])
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['diff', '-r', '/tmp/rsync', '/tmp/juicesync']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 0.080s

root@bench-01:~/juicefs# tree /tmp/sync_src /tmp/rsync /tmp/juicesync
/tmp/sync_src
└── a
    └── a
/tmp/rsync
└── a
    └── a
/tmp/juicesync
└── a

**What you expected to happen**:

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**

**Environment**:
- JuiceFS version (use `juicefs --version`) or Hadoop Java SDK version:
- Cloud provider or hardware configuration running JuiceFS:
- OS (e.g `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Object storage (cloud provider and region, or self maintained):
- Metadata engine info (version, cloud provider managed or self maintained):
- Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
- Others:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions