working instance can be killed before new one is ready

[wpietri]

Issue #18 exposes another problem. The creation of a new box failed while running the user-data script. But the working VPN was killed anyhow. Is there some way for it to fail safely, so that the working VPN stays up until a working replacement is ready?

[atrull]

It's a downside to this with the dynamic infra. Protocol is UDP and not especially sticky. I actually wish the wg client detected a failure like this and renegotiated. As a form of improvement for this case I'm thinking we could put the new config into s3 and then have the server sync it. @jmhale ?

[jmhale]

So, the issue here is that the new box came up, and was technically healthy from the ASG's perspective, because the EC2 instance booted fine. The basic EC2 health checks don't care if cloud-init succeeds or not (at least, not the part of cloud-init that is running our user-data script).

I'm thinking that we might be able to use a lifecycle hook to "hold" the new box from being declared as good until cloud-init runs all of our stuff ok, If it doesn't, the hold isn't released and the old box doesn't get terminated. That'll create another issue of new boxes boot-looping, but at least the overall system will be stable.

I'll look into that this weekend, as well as the PPA removal and the grub hang PR. Sorry about the delays on those. I've been spending free non-work time getting caught up on PRs in another project and now I'm getting back to this one.