Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows)

[minusdavid]

I've tried to provision a few times now using vars-singlenode-qa.yml instead of vars-singlenode-1.6.yml, and I keep getting the same error.

TASK [artefactual.archivematica-src : Checkout archivematica-storage-service code] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x", "failed": true, "msg": "Failed to checkout qa/0.x", "rc": 1, "stderr": "error: Your local changes to the following files would be overwritten by checkout:\n\t
storage_service/static/js/vendor/base64.js\nPlease, commit your changes or stash them before you can switch branches.\nAborting\n", "stderr_lines": ["error: Your local changes to the following files would be overwritten by checkout:", "\tstorage_service/static/js/vendor/
base64.js", "Please, commit your changes or stash them before you can switch branches.", "Aborting"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/vagrant/singlenode.retry

When I look at src/archivematica-storage-service, I see that branch stable/0.10.x is checked out even though archivematica_src_ss_version is set to "qa/0.x".

roles/artefactual.archivematica-src/tasks/ss-main.yml looks good. The version corresponds to the yml file and force is set to "yes".

Investigating...

[minusdavid]

Here's the output from git after I try /usr/bin/git checkout --track -b qa/0.x origin/qa/0.x by hand:

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git status
On branch stable/0.10.x
Your branch is up-to-date with 'origin/stable/0.10.x'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   osdeps/RedHat-7.json
    modified:   osdeps/Ubuntu-16.json
    modified:   storage_service/static/js/vendor/base64.js

no changes added to commit (use "git add" and/or "git commit -a")
vagrant@am-local:/vagrant/src/archivematica-storage-service$ git diff
diff --git a/osdeps/RedHat-7.json b/osdeps/RedHat-7.json
--- a/osdeps/RedHat-7.json
+++ b/osdeps/RedHat-7.json
@@ -1 +0,0 @@
-CentOS-7.json
\ No newline at end of file
diff --git a/osdeps/Ubuntu-16.json b/osdeps/Ubuntu-16.json
--- a/osdeps/Ubuntu-16.json
+++ b/osdeps/Ubuntu-16.json
@@ -1 +0,0 @@
-Ubuntu-14.json
\ No newline at end of file
diff --git a/storage_service/static/js/vendor/base64.js b/storage_service/static/js/vendor/base64.js
--- a/storage_service/static/js/vendor/base64.js
+++ b/storage_service/static/js/vendor/base64.js
@@ -1 +0,0 @@
-../../../external/base64-helpers/base64-helpers.js
\ No newline at end of file

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls -l storage_service/static/js/vendor/base64.js
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 storage_service/static/js/vendor/base64.js -> ../../../external/base64-helpers/base64-helpers.js

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls ../../../external/base64-helpers/base64-helpers.js
ls: cannot access ../../../external/base64-helpers/base64-helpers.js: No such file or directory

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls -l osdeps
total 3
-rwxrwxrwx 1 vagrant vagrant 1032 Aug 7 02:16 CentOS-7.json
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 RedHat-7.json -> CentOS-7.json
-rwxrwxrwx 1 vagrant vagrant 895 Aug 7 02:16 Ubuntu-14.json
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 Ubuntu-16.json -> Ubuntu-14.json

--

I'm guessing that maybe this is the Windows symlink issue raising its head again... even though in the case of "osdeps", the symlinks have been created successfully...

[minusdavid]

If I look at it using Git Bash, everything looks OK. It's only when I go into the VM that I see those differences and even using "git checkout -- " won't get rid of them.

[minusdavid]

Looks like the ../../../external/base64-helpers/base64-helpers.js might be submodule?

I'm going to try stable again.....

[minusdavid]

Using vars-singlenode-1.6.yml, and I'm still in the process of provisioning, but the archivematica-storage-service seems to have been installed correctly.

Here's the view from Windows:

$ ls -l
total 8
-rw-r--r-- 1 dcook 197121 1032 Aug 7 13:07 CentOS-7.json
lrwxrwxrwx 1 dcook 197121 13 Aug 7 13:07 RedHat-7.json -> CentOS-7.json
-rw-r--r-- 1 dcook 197121 895 Aug 7 13:07 Ubuntu-14.json
lrwxrwxrwx 1 dcook 197121 14 Aug 7 13:07 Ubuntu-16.json -> Ubuntu-14.json

$ ls -l storage_service/static/js/vendor/base64.js
lrwxrwxrwx 1 dcook 197121 50 Aug 7 13:07 storage_service/static/js/vendor/base64.js -> ../../../external/base64-helpers/base64-helpers.js

Going to see if I can vagrant ssh in during provisioning...

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git status
On branch stable/0.10.x
Your branch is up-to-date with 'origin/stable/0.10.x'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   osdeps/RedHat-7.json
    modified:   osdeps/Ubuntu-16.json
    modified:   storage_service/static/js/vendor/base64.js

I'm seeing the same thing as last time... however git didn't fail when using vars-singlenode-1.6.yml for the archivematica-storage-service...

[minusdavid]

Hmm I notice that core.symlinks isn't set on the VM-side whereas I do have it set on the Windows side. I might experiment with that for a moment...

[minusdavid]

Hmm doesn't seem to make a difference at a glance.

[minusdavid]

I imagine this doesn't affect vars-singlenode-1.6.yml because git doesn't need to switch branches since the git repositories use the stable branches as the repository HEAD.

[minusdavid]

Because everything looks OK on Windows, I'm wondering if this is actually a vboxfs issue?

[minusdavid]

I noticed the Ansible command was "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x" which doesn't mention force... so I just tried the following in "archivematica":

"git checkout --force qa/1.x"

And that worked. The "git status" output has changed from:

[minusdavid]

    modified:   src/MCPClient/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/Ubuntu-16.json
    modified:   src/archivematicaCommon/osdeps/RedHat-7.json
    modified:   src/archivematicaCommon/osdeps/Ubuntu-16.json
    modified:   src/dashboard/osdeps/RedHat-7.json
    modified:   src/dashboard/osdeps/Ubuntu-16.json
    modified:   src/dashboard/src/fpr
    modified:   src/dashboard/src/templates/fpr

to:

    modified:   src/MCPClient/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/Ubuntu-16.json
    modified:   src/archivematicaCommon/osdeps/RedHat-7.json
    modified:   src/archivematicaCommon/osdeps/Ubuntu-16.json
    modified:   src/dashboard/osdeps/RedHat-7.json
    modified:   src/dashboard/osdeps/Ubuntu-16.json

[minusdavid]

@sevein mentioned that qa/1.x moved fpr out of a submodule and instead gets fpr using pip so that makes sense that it's gone. I imagine the others stayed the same across versions.

And they're only showing up as modified because vboxsf or at least vboxsf on Windows is maybe cloning them as a file with the symlink target as its content before creating the symlinks? I don't know enough about Git internals to know exactly what's going on here...

[minusdavid]

So I just tried the following:

cd /vagrant
git clone https://github.com/artefactual/archivematica.git test
touch /home/vagrant/test
ln -s ~/test /vagrant/test/test
git add test

At the end of a git diff, I see this:

diff --git a/test b/test
--- a/test
+++ b/test
@@ -1 +0,0 @@
-/home/vagrant/test
\ No newline at end of file

[minusdavid]

Looks like someone else noted this years ago:

https://stackoverflow.com/questions/34225522/files-changed-directly-after-git-clone-into-shared-folder-using-vagrant-on-windo

I'm thinking it must be something like... it gets cloned into the shared folder as a file containing text to the target, and then underneath Virtualbox Windows is creating a symlink after Git has already recorded it as a regular file?

Of course, it's not an issue on Windows. Git seems to work correctly there.

On Windows, I'm using git version 2.13.2.windows.1. Whereas in Virtualbox, I'm using git version 1.9.1.

Using the following to install a newer version of git:
sudo apt-add-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

Now I'm using git version 2.13.0 on Virtualbox guest...

Still a problem.

--

Using "git update-index --assume-unchanged" might be an option, but not really in this case...
https://stackoverflow.com/questions/18771926/persuading-git-to-handle-symlinks-correctly-on-vboxsf-filesystem

[minusdavid]

"Support for symbolic links across synced folder implementations and host/guest combinations is not consistent. Vagrant does its best to make sure symbolic links work by configuring various hypervisors (such as VirtualBox), but some host/guest combinations still do not work properly. This can affect some development environments that rely on symbolic links.

The recommendation is to make sure to test symbolic links on all the host/guest combinations you sync folders on if this is important to you."

https://www.vagrantup.com/docs/synced-folders/basic_usage.html

But the symlinks do work... it's just git which is having an issue with it...