Deploying from nix-shell under root

[danbst]

I was doing this:

1. install nixos on ec2 from nixos AMI
2. git clone nixops on ec2 machine
3. run ./dev-shell
4. nixops deploy

building path(s) в??/nix/store/6slz3xj7ski2jyza20m27l6dg85wyysy-system-unitsв?T
/nix/store/jm26xg0h3jcrg4bbrwiqx3jpirscdk0p-stdenv/setup: line 456: /run/user/0/nix-build-system-units.drv-0/env-vars: Permission denied
builder for в??/nix/store/sjhxwnljr0bh8rwjvqawcwypir82rqci-system-units.drvв?T failed with exit code 1
cannot build derivation в??/nix/store/gv7fsdkh6x1rmqfh1v7jhr4m8p6k7l8z-etc.drvв?T: 1 dependencies couldn't be built
cannot build derivation в??/nix/store/j1k7ggpaswv2v4kx5if8fq46kzm7rdc5-nixos-15.06pre64647.a73d25d.drvв?T: 1 dependencies couldn't be built
cannot build derivation в??/nix/store/gjk9p7hbrv95kjf9pf878sfd9j7xy9mf-nixops-machines.drvв?T: 1 dependencies couldn't be built
error: build of в??/nix/store/gjk9p7hbrv95kjf9pf878sfd9j7xy9mf-nixops-machines.drvв?T failed
error: unable to build all machine configurations

in IRC we concluded that the problem is in temp directory and solution is to

mkdir /tmp/foo
export TMPDIR=/tmp/foo

This is quite non-obvious behaviour for those, who want to hack on nixops, and when running nixops deploy without dev-shell, everything is ok.

[nh2]

I'm suffering from this, because I use nixops from a nixops clone directly. Workaround works though.

@aszlig is this tag backends/ec2 right?

[aszlig]

@nh2: Hm, doesn't seem so, I'll remove the label.

[aszlig]

So am I correctly assuming that $TMPDIR is /run/user/0 here?

[nh2]

It seems the TMPDIR workaround is not enough:

While it fixes the error message appearing, it corrupts files in the nix store. Some just end up empty (0 bytes) when really they should not be.

For example, when deploying I get the error:

/nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml:1: parser error : Document is empty

And indeed that file is empty. But the same path is not empty on my Ubuntu machine (where I also run nix-shell -p nixops, but not as root) and has sensibel contents there.

@cleverca22 told me how I can spot the reason:

On the good (Ubuntu) machine:

nix-store -l /nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml has no output.

But on the problematic machine:

# nix-store -l /nix/store/pbhxc864gqach922mgl4v03d9dff93h4-grub-config.xml
/nix/store/685n25b9yc8sds57vljk459ldly1xyhn-stdenv/setup: line 500: /run/user/0/nix-build-grub-config.xml.drv-0/env-vars: Permission denied

So that's our error again!

@aszlig This also mentions the /run/user/0 that you were mentioning.

[danbst]

Also, I don't know how to fix your error (I don't run nixops from under nix-shell from under root anymore, I actually rebuild it after changes), I have encountered those errors outside of nixops context:

1. I had a zero-length file in store, which happend to be used in /boot
2. I often had /tmp/env-vars: Permission denied when doing nix-shell builds with two different users. In particular, sometimes /tmp/env-vars is created by user deployer, and user danbst can't do any builds in nix-shell anymore (insufficient rights to overwrite /tmp/env-vars).

So now I think, this isn't nixops issue, but a Nix one.

[dhruvio]

For posterity, see NixOS/nix#568 (comment)