Fix module issues once and for all

[ugorji]

The use of modules in go-codec introduced more headaches than necessary.

It has been a disaster, and I really wish I had waited until the dust settled before making this module aware, especially since so many people depended on this package and it has the atypical "package name differs from directory name" issue.

I regret jumping on modules that early, and having to constantly try to correct my mistakes.

It seems the dust has settled now and we have the tools to:

• support the simulation of a multi-module repository

We have 2 options:

1. Make go/codec its own module using a tag: codec/vX.Y.Z
2. Keep go as the only module, using a tag: vX.Y.Z
3. Have both as modules using "same'ish" parallel tags (vX.Y.Z and codec/vX.Y.Z) and corresponding go.mod files

I am not sure if 3. above is supported by go modules. I read that a module doesn't include modules within it.

I will appreciate any thoughts on how best to resolve this. I am including references to places where I have seen folks have issues. I hope that you folks can respond and help me settle this once and for all with the best solution.

https://github.com/golang/go/wiki/Modules#publishing-a-release
https://golang.org/wiki/Modules#faqs--multi-module-repositories

gin-gonic/gin#1897

@dmitshur
@arizvisa
@bvisness
@HZ89
@bcmills
@amarox
@thepudds

[trivigy]

@ugorji i am pretty sure that option 3 is a valid solution is probably preferred to support backward compatibility with those folks that depend on the library from the main module.

It is just unfortunate that you will have to maintain two sets of tags requiring making sure that they are always on the same commits.

[ugorji]

@trivigy

https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

Each module starts at the directory containing its go.mod file, and contains all packages from that directory and its subdirectories recursively, excluding any subtree that contains another go.mod file.

This is why I'm not sure if 3. is a valid option, as this suggest that .../go movie will NOT contain .../go/codec

[HZ89]

I think option 1 is a more practical choice. In gin-gonic, just use go/codec decode msgpack.

[dmitshur]

Option 3 will not work easily. If the codec directory contains a go.mod file, its contents are "carved out" from the outer module and won't be included in it. As far as I know, you can only make it work by maintaining two parallel git histories: one for inner module (with go.mod in subdirectory) and another for outer module (without go.mod in subdirectory). I don't think that's practical.

[thepudds]

I suspect you could make any of those three options work...

But the simplest in the long-term might be option 2, which is what the remainder of my comment focuses on:

2. Keep go as the only module, using a tag: vX.Y.Z

Having a single go.mod in the root of a repo is the most common situtation, and it avoids needing to prefix tags and is generally simpler and more widely understood than either of the other two options.

That also happens to be the current state of the latest release here, as far as I've followed -- it seems v1.1.4 has a single go.mod in the repo root: https://github.com/ugorji/go/blob/v1.1.4/go.mod

However, it seems the main issue with the current state is multiple consumers have the old github.com/ugorji/go/codec module path in their go.mod files. One approach to resolving that might be something like adding an FAQ on this repo asking people to upgrade to point to latest of the github.com/ugorji/go module and remove any references to old github.com/ugorji/go/codec module path from their go.mod files. I suspect that might be enough to resolve spf13/viper#658, for example, but I am also not sure until someone actually tries it and sees it work.

In general, moving go.mod files within a repo (which is in effect what happened here) is a very, very tricky situation, but as far as I understand, the best way to handle it is to create a requirement cycle for the moved/added/removed modules within the repo so that any consumer that points to one of the modules automatically gets ratcheted forward to a consistent version of the other module(s) within that same repo.

The modules wiki covers this in a fair amount of detail if you are adding a new module: https://github.com/golang/go/wiki/Modules#is-it-possible-to-add-a-module-to-a-multi-module-repository

However, removing a module from a multi-module repo (which I think is what happened here most recently) is covered much more briefly -- it says:

Is it possible to remove a module from a multi-module repository?
Yes, with the same two cases and similar steps as above.

On the other hand, there is a more precise but older description of removing a module that lives within the code itself. The full section is here, and worth reading:
https://github.com/golang/go/blob/release-branch.go1.12/src/cmd/go/internal/modfetch/repo.go#L106-L125

where this snippet here might be the piece that is most applicable in terms of next steps:

// The gf256 package could move back into the
// original by another simultaneous release of rsc.io/qr v0.4.0 including
// the gf256 subdirectory and an rsc.io/qr/gf256 v0.2.0 with no code
// in its root directory, along with a new requirement cycle.

That might be an approach here -- prepare a new github.com/ugorji/go/codec module that does not have any code in it, set up a requirement cycle between the new github.com/ugorji/go/codec module and a new release of the github.com/ugorji/go module, and then release both. The new github.com/ugorji/go/codec module probably could be on a different branch.

But, I've never done that myself, so take that as a theoretical possibility that might or might not be correct.

Also, there might be a simpler solution.

CC @jadekler (the main author of the multi-module repo documentation)

[ugorji]

@thepudds Thanks much for this detailed explanation. Much much appreciated.

I messed things up by not respecting how tricky this would be, and trying to fix it quickly by moving go.mod around, which made it worse. Now there are frustrated users on both the .../go module and .../go/codec module. It will be nice if I could get option 3. to work somehow.

When I read https://github.com/golang/go/blob/release-branch.go1.12/src/cmd/go/internal/modfetch/repo.go#L106-L125 , it suggests to me that I can support both, by having 2 modules that are kept in sync.

• github.com/ugorji/go/codec v1.1.5
• github.com/ugorji/go v1.1.5 depends on github.com/ugorji/go/codec v1.1.5

This way, folks can use either of the module versions and work.

Do you (or does anyone else with more module experience) see a reason why this wouldn't work?

^^ @jadekler

[jeanbza]

I did a light review of the issues involved. It seems like you had added a module below the root, creating a multi-module repository, and then removed it? I agree with @thepudds that the easiest solution is for all your dependors to update to latest version of github.com/ugorji/go and remove the old, non-exsistent github.com/ugorji/go/codec reference.

That might be an approach here -- prepare a new github.com/ugorji/go/codec module that does not have any code in it, set up a requirement cycle between the new github.com/ugorji/go/codec module and a new release of the github.com/ugorji/go module, and then release both. The new github.com/ugorji/go/codec module probably could be on a different branch.

Do you (or does anyone else with more module experience) see a reason why this wouldn't work?

I'm not sure this works: users still would need to upgrade to the later version of the module, no?

---

Sorry I never got around to fleshing out Is it possible to remove a module from a multi-module repository?. I should go back and add more detail to that.