Implement persistent caching in SourceMgr

[fabulous-gopher]

From @sdboyer on December 30, 2016 11:46

With additional testing being done as part of the new tool, it's increasingly clear that we really, really need to get persistent, disk-backed caching into place. There are four basic things we need to cache:

2. Version lists - all the versions available for a source
3. Manifest and Lock - for a given rev and a particular analyzer, these should be permanently cacheable
4. PackageTree - for a given rev and gps version, these should be permanently cacheable
5. Project root computations - those that require a go-get metadata query

The biggest immediate question is how to actually store it on disk. For the first three, at least, discrete files would probably work. However, I'm inclined to store everything together using bolt. That'll cover the last case more easily, especially because it is likely to require prefix queries; it'll also probably make it a bit cleaner to store TTLs.

(supercedes #8)

Copied from original issue: sdboyer/gps#130

[fabulous-gopher]

From @sdboyer on April 4, 2017 14:36

With #196 finally done, the road to this is pretty clear. There's even a TODO exactly where a handle to it should be dropped in 😄

We do need to pick a storage lib for it. LevelDB would be fine, as would Bolt.

[jmank88]

I can work on this issue.

I've started sketching out a Bolt singleSourceCache implementation, and the main open question is how to encode the complex structs/interfaces. @sdboyer Do you have any ideas/preferences for serializing: Manifest/RootManifest, Lock, pkgtree.PackageOrErr, and UnpairedVersion (and their underlying types - e.g. Constraint, maybe typedString())?

[jmank88]

Looking at dep's rawManifest for inspiration makes me think that most of the underlying types can be cleanly converted to string fields anyways, so I'll experiment with how far I can get with native Bolt structure (nested buckets, and/or composite keys ending in field name suffixes), before introducing another encoding layer.

[sdboyer]

AMAZING!! 🎉 🎉 🎉 🎊 - this is something that would make a huge difference for users, and i've got nothing even close to the time required to work on it.

Looking at dep's rawManifest for inspiration makes me think that most of the underlying types can be cleanly converted to string fields anyways

Unfortunately, while that's adequate for a written file manifest/lock's purposes (by design), it likely won't be for gps' internal purposes. There are two issues I can think of off the top of my head:

• branchVersion has the default field, which is not expressed in even typedString() output. We could add it, but...
• semver.Version has an original field that...well, i guess that the String() method prints it out, but that field has bitten me so many times in subtle ways that i just don't trust it anymore. (OTOH, i'm not sure there's anything we can do EXCEPT stringify that, as semver doesn't export those fields).

And that's just with the versions. I generally tried not to allow such gotchas, but it wasn't a design goal that I explicitly kept track of, so I can't say with certainty that there aren't more such issues.

Notwithstanding all that, I think it's still the right call to prototype this with strings, where it works. Layering in an encoding system later is not difficult. And, more generally, I'd strongly prefer to keep this proceeding in smaller chunks with smaller PRs (even if they have no immediate effect), rather than building up something huge and unreviewable.