Allow consuming cobra as a library without picking up viper dependencies

[liggitt]

Status

1. https://github.com/spf13/cobra-cli has been created with an extracted copy of github.com/spf13/cobra/cobra/...
2. Initialize cobra-cli repo cobra-cli#1 renamed the command to cobra-cli
3. https://github.com/spf13/cobra-cli/releases/tag/v1.3.0, matching github.com/spf13/cobra v1.3.0 (except for the binary name)
4. PRs to update public references to cobra/cobra in go imports and scripted references
   • go imports
     • Switch to standalone cobra-cli dependency vmware-tanzu/kubeapps#4373
     • Switch tools to standalone cobra-cli dependency wormhole-foundation/wormhole#936
     • Switch to standalone cobra-cli dependency allaboutapps/go-starter#169
     • Switch tools to standalone cobra-cli dependency shihanng/gig#56
   • scripted installs
     • Update reference to github.com/spf13/cobra/cobra codeedu/imersao-fullstack-fullcycle#7
     • Switch to standalone cobra-cli dependency blacktop/ipsw#86
     • Update reference to github.com/spf13/cobra/cobra codeedu/imersao-fullcycle-3#3
     • Remove reference to standalone cobra cli GoogleCloudPlatform/kafka-pubsub-emulator#40
     • Remove reference to standalone cobra cli ezbuy/tgen#74
     • Remove reference to standalone cobra cli blacktop/graboid#12
     • Switch to standalone cobra-cli dependency ahmetb/dotfiles#2
     • Switch to standalone cobra-cli dependency cisco-sso/kdk#218
     • Switch to standalone cobra-cli dependency codeedu/fc2-arquitetura-hexagonal#4
     • Switch to standalone cobra-cli dependency awslabs/tecli#14
   • doc references
     • Remove reference to standalone cobra cli SimonWaldherr/golang-examples#63
     • Switch example to github.com/spf13/cobra-cli Kevin-fqh/learning-k8s-source-code#4
     • Switch to standalone cobra-cli dependency armadaproject/armada#888
5. Remove CLI from this repo and update this repo's readme:
   • Removes viper dependency by removing cobra/ CLI tool #1604

---

(original description follows)

First, many thanks for such a useful command library. The widespread adoption demonstrates how useful it is. A consequence of being very popular is that any dependencies of this module are picked up as transitive dependencies by a lot of consumers. The github.com/spf13/viper dependency of the cobra command line tool is particularly problematic, as it pulls in large numbers of large dependencies.

It would greatly help downstream consumers if this library could be consumed without picking up the viper dependencies.

---

History

Relevant existing issues:

• Cobra pull a lot of dependencies through viper #1138
• 1.2.0 deps explosion?  #1435 (comment)
• Reorganise the cobra library and the CLI as siblings #1240

There was a prior attempt at isolating the CLI command and viper dependency via a submodule:

• modules: add a secondary go.mod to segregate CLI dependencies #1139

That resulted in the following issues:

• failed to install cobra generator and its dependencies #1191
• "ambiguous import"-error when trying to "go get" cobra #1215
• go get error: unexpected module path "go.etcd.io/bbolt" #1219

There were a few problems with the previous attempt:

• No new tag was created for github.com/spf13/cobra once the submodule was created, so the @latest version of both github.com/spf13/cobra and github.com/spf13/cobra/cobra provided the github.com/spf13/cobra/cobra import path. This caused the "ambiguous import" error
• The submodule used a local filesystem replace ... => ../ directive, which would break go get github.com/spf13/cobra/cobra and go install github.com/spf13/cobra/cobra@latest

---

Proposal

2022-03-10 note: rather than the approach proposed here, the Cobra CLI was split to a separate repo at https://github.com/spf13/cobra-cli. See discussion starting at #1597 (comment)

Possible solutions for reorganizing this library to isolate the viper dependency are discussed in #1240. I agree with @spf13 that we should avoid solutions that disrupt current consumers of the github.com/spf13/cobra import path.

I think the previous attempt to create a dedicated module for the cobra command packages (github.com/spf13/cobra/cobra/...) was the right direction, and could resolve the problems with the previous attempt with two changes:

1. Tag a github.com/spf13/cobra release after adding the sub go.mod so github.com/spf13/cobra@latest resolves to a version that does not provide the github.com/spf13/cobra/cobra package.
2. Update github.com/spf13/cobra/cobra to refer to the tagged github.com/spf13/cobra release created in step 1 so that it works with go get and go install

---

Testing

To test this approach, I:

1. cloned this repo to github.com/liggitt/cobra, and created tags at various points in cobra's history with the import paths renamed (v1.0.0, v1.1.3, and v1.3.0)
2. created a go.mod file at github.com/liggitt/cobra/cobra, committed and tagged as github.com/liggitt/cobra v1.5.0 - liggitt@0c686b5
3. updated the go.mod file at github.com/liggitt/cobra/cobra to refer to github.com/liggitt/cobra@v1.5.0, committed and tagged github.com/liggitt/cobra/cobra cobra/v1.5.0 - liggitt@458f3c5

Now, all of the following still work with no complaints about ambiguous import paths:

go get:

go get github.com/liggitt/cobra
go get github.com/liggitt/cobra@v1.3.0
go get github.com/liggitt/cobra@v1.5.0
go get github.com/liggitt/cobra@latest
go get -u github.com/liggitt/cobra
go get -u github.com/liggitt/cobra@v1.3.0
go get -u github.com/liggitt/cobra@latest

go install:

go install github.com/liggitt/cobra/cobra@v1.3.0
go install github.com/liggitt/cobra/cobra@v1.5.0
go install github.com/liggitt/cobra/cobra@latest

---

Impact on library consumers

To see what the impact would be on downstream consumers of this library, I took Kubernetes' use as an example (test commits visible in https://github.com/liggitt/kubernetes/commits/cobra130):

1. made copies of Kubernetes dependencies that refer to spf13/cobra@{v1.0.0,v1.1.3,v1.3.0}, and adjusted Kubernetes and the dependencies to use liggitt/cobra at the same versions
2. updated Kubernetes to use liggitt/cobra@v1.5.0
3. saw a significant number of transitive dependencies drop out of our dependency graph:
   • cloud.google.com/go/firestore
   • github.com/DataDog/datadog-go
   • github.com/armon/go-metrics
   • github.com/armon/go-radix
   • github.com/bgentry/speakeasy
   • github.com/circonus-labs/circonus-gometrics
   • github.com/circonus-labs/circonusllhist
   • github.com/fatih/color
   • github.com/hashicorp/consul/api
   • github.com/hashicorp/consul/sdk
   • github.com/hashicorp/errwrap
   • github.com/hashicorp/go-cleanhttp
   • github.com/hashicorp/go-hclog
   • github.com/hashicorp/go-immutable-radix
   • github.com/hashicorp/go-msgpack
   • github.com/hashicorp/go-multierror
   • github.com/hashicorp/go-retryablehttp
   • github.com/hashicorp/go-rootcerts
   • github.com/hashicorp/go-sockaddr
   • github.com/hashicorp/go-syslog
   • github.com/hashicorp/go-uuid
   • github.com/hashicorp/golang-lru
   • github.com/hashicorp/hcl
   • github.com/hashicorp/logutils
   • github.com/hashicorp/mdns
   • github.com/hashicorp/memberlist
   • github.com/hashicorp/serf
   • github.com/magiconair/properties
   • github.com/mattn/go-colorable
   • github.com/mattn/go-isatty
   • github.com/miekg/dns
   • github.com/mitchellh/cli
   • github.com/mitchellh/go-homedir
   • github.com/mitchellh/go-testing-interface
   • github.com/pascaldekloe/goe
   • github.com/pelletier/go-toml
   • github.com/posener/complete
   • github.com/ryanuber/columnize
   • github.com/sagikazarmark/crypt
   • github.com/sean-/seed
   • github.com/spf13/cast
   • github.com/spf13/jwalterweatherman
   • github.com/spf13/viper
   • github.com/subosito/gotenv
   • github.com/tv42/httpunix
   • gopkg.in/ini.v1

---

Impact on library consumers of github.com/spf13/cobra/cobra/...

There don't appear to be very many consumers of these packages (which is not surprising, since their purpose is to support the cobra command line):

• https://grep.app/search?q=%22github.yoyoweb123.workers.dev/spf13/cobra/cobra%22&filter[lang][0]=Go (4 public references in .go files)
• https://grep.app/search?q=%22github.yoyoweb123.workers.dev/spf13/cobra/cobra/cmd%22 (0 public references outside of vendored copies of github.com/spf13/cobra)
• https://grep.app/search?q=%22github.yoyoweb123.workers.dev/spf13/cobra/cobra/tpl%22 (0 public references outside of vendored copies of github.com/spf13/cobra)

Any consumers using these packages beyond v1.3.0 would need to depend on the github.com/spf13/cobra/cobra module instead.

---

Impact on development

Changes that span the two modules would become more difficult to make, and could not be committed/pushed in a single PR. This is because the two modules are effectively behaving like two distinct modules that happen to live in the same repo, and a release of the library module would be required before the command module could start relying on the changes in it via a publicly referenced version.

---

Impact on release mechanics

Currently, to release github.com/spf13/cobra:

1. create and push a v1.x.y git tag

With multiple modules, effectively, two releases would be required (one for the library and one for the command), which would require two tags and a go.mod change:

1. create and push a v1.x.y git tag
2. update the github.com/spf13/cobra/cobra/go.mod file to reference that version of the github.com/spf13/cobra library, go mod tidy, commit and push
3. create and push a cobra/v1.x.y git tag

[jpmcb]

Thanks for the in depth research on this!

Pinning since I think this issue is the most fresh in regards to how we pull in viper and it's deep dependency chain.

Let's also get @marckhouzam and @umarcor eyes on this 👀

[jpmcb]

FYI @spf13 - I think we will plan to move forward with this since we are getting more and more requests from kubernetes/kubernetes, helm, vmware-tanzu, etc to decouple the dependencies.

We've also seen the rise of a prominent fork which is the same as cobra but removes the interlinking of the cobra CLI and the cobra library. So, I'd also like to invite @muesli to provide their feedback and thoughts on this proposal.

[jpmcb]

Proposal review:

It would greatly help downstream consumers if this library could be consumed without picking up the viper dependencies.

100% agree. This is a long standing problem and something that has been challenging for downstream consumers and maintainers to keep ontop of. Further, it only exacerbates a challenging secure supply chain issue: we shouldn't be bringing unnecessary dependencies into kubernetes/kubernetes, helm, tanzu, etc.

History

Also note that at the time we attempted this, cobra did not have a release cycle and it was expected that users would pull off of the main branch. If I remember correctly, this was even before go mod v2. I think that now, we are in a better place to tag the appropriate release, and get the necessary bits out in the wild where they can be consumed.

I agree with spf13 that we should avoid solutions that disrupt current consumers of the github.com/spf13/cobra import path.

Yes, I 100% agree. There are far too many large projects downstream that rely on the library import path and we do not have the open source bandwidth to advocate those projects change their import path.

There don't appear to be very many consumers of these packages

There shouldn't be since it's primary purpose is to bootstrap a new cobra program via it's CLI. I'd say anyone that is using it should transition to using the library directly.

the two modules are effectively behaving like two distinct modules that happen to live in the same repo
...
With multiple modules, effectively, two releases would be required

⚠️ This is my only concern. This effectively makes spf13/cobra a library distribution and a consumer of it's own library. I agree this would make development work more challenging and almost creates a circular dependency. Part of me wonders if this is a better chance to remove the cobra CLI from this project and move it to it's own repo. That way, the upstream, downstream model makes much more sense. But if that isn't what we want to do, I don't think we should block on that.

The release cadences are already difficult (given we need an injection of engineering and maintainer resources). So If at all possible, it'd be great if this wasn't made more difficult.

My only suggestion is that we consider moving the cobra CLI to it's own repository. If @spf13 doesn't have bandwidth to do that, I'd be happy to provide a repo and also maintain it along side this library. And if we don't want to remove it, I think your proposal is extremely solid and would work in the long run.

[liggitt]

This effectively makes spf13/cobra a library distribution and a consumer of it's own library.

it would be a multi-module repo (https://go.dev/doc/modules/managing-source#multiple-module-source), where each module gets its own lifecycle/tags/etc, and only refers to the other modules in the same repo via committed/published tags

I agree this would make development work more challenging and almost creates a circular dependency.

Fortunately, the dependency is still directed, not circular (command depends on library), but yeah, it could make development across the two modules more difficult (though no different from multi-repo development, if a release of the library is required before library changes are picked up and used by the command)

Part of me wonders if this is a better chance to remove the cobra CLI from this project and move it to it's own repo. That way, the upstream, downstream model makes much more sense. But if that isn't what we want to do, I don't think we should block on that.

The release cadences are already difficult (given we need an injection of engineering and maintainer resources). So If at all possible, it'd be great if this wasn't made more difficult.

My only suggestion is that we consider moving the cobra CLI to it's own repository.

Note that the "two release" aspect isn't required to happen in lockstep, only when you want the command module to pick up and start making use of changes added to the library module

The only real pros I can think of for keeping the command in the same repo are:

• preserving the ~4 importers of the go path
• preserving all the places people have scripted go get (or install) github.com/spf13/cobra/cobra to install the command

That second one (the install command) might be significant… I suspect you'd get a lot of reports of breakage if the current install method stopped working

[spf13]

I fully support moving to having viper not imported by default.

Honestly that decision was made very VERY early in the lifecycle of Go and I don't think any of us understood how dependencies would have played out. I also had no idea how popular this little library of mine would become.

FWIW, The viper folks are working on a version that dramatically reduces it's dependencies as well. This should minimize, but not eliminate the problem that we're addressing here. cc: @sagikazarmark for his input and visibility.

For separating the CLI from the library, I'm not sure what the right answer is but have a few requirements.

1. It should be as easy to work with as possible for the Cobra maintainers.
2. It should not introduce any additional dependencies for users (or any additional complications)
3. Let's try to not change things as much as possible.

My guess is that the option that addresses these the requirements the best is to move it to it's own repo. Very little coordination happens between the CLI and the library because the CLI is very simple. Not having to worry about multiple modules in a repo is simpler. The only disadvantage I see is that all documentation around Cobra CLI will point to the wrong install string. This is unfortunate, but I think we can message this okay. I'm much more concerned about breaking dependencies that are automated things vs something that a human is manually doing. We can inform the human of a new installation command. If we go down this route we should put it at github.com/spf13/cobra-cli (or similar) so it's still discoverable when people look for spf13/cobra.

Am I missing anything or not thinking about it correctly?

[liggitt]

From a library consumer and cobra maintainer perspective, moving the CLI to a separate repo is certainly easy to reason about, accomplishes the dependency reduction goals for the library, and is simpler to maintain.

If we're confident all (or the majority) of installation invocations are done by humans we can inform, and not scripted in automation we'll break, that sounds reasonable. I'm not familiar enough with use of the CLI side of this repo to know if that's the case.

If we go down this route we should put it at github.com/spf13/cobra-cli (or similar) so it's still discoverable when people look for spf13/cobra.

I assume we'd still want something that produced a cobra binary. Does that require the last path segment to be "cobra"?

[muesli]

My guess is that the option that addresses these the requirements the best is to move it to it's own repo.

I believe this is the right decision here. Introducing a sub-go.mod is typically causing more issues, not only on a technical level, but also with regards to awareness and obviousness. With the CLI living in a separate repository these concerns would be eliminated.

Very little coordination happens between the CLI and the library because the CLI is very simple. Not having to worry about multiple modules in a repo is simpler. The only disadvantage I see is that all documentation around Cobra CLI will point to the wrong install string. This is unfortunate, but I think we can message this okay. I'm much more concerned about breaking dependencies that are automated things vs something that a human is manually doing. We can inform the human of a new installation command. If we go down this route we should put it at github.com/spf13/cobra-cli (or similar) so it's still discoverable when people look for spf13/cobra.

I completely agree with this 👍

I'd also like to point out that in my (admittedly limited and personal) experience only a fraction of Cobra's users are actually using the CLI tool. It's definitely nice to have when experimenting with Cobra in the beginning, but once you figured out how this library works most everyone I work with gets by without using the CLI. If that's not just my "personal anecdote" it's probably another good indicator the CLI should be moved to a separate repository.

[caarlos0]

Another good candidate to be a separated module: the docs and man page generators.

IMHO, ideally they should be interfaces the user can implement themselves, or import packages that does it for them. This way we could drop even more transitive dependencies, and also allow for more customization.

[sagikazarmark]

Thanks for pinging me @spf13 !

I think I agree with most of what's been said before. I only have a couple additional thoughts:

• The Cobra-Viper integration is regularly misunderstood by consumers and they often come to the Viper repo opening issues that can't be helped, so personally I fully support removing Viper as a direct dependency.
• It's been a long road, but finally we have a direction at Viper: the plan is to reduce the dependencies by moving out some of the components to separate packages. I agree with @spf13 that this itself wouldn't solve the dependency problem for cobra though.
• Those separate packages I mentioned will be under a go-viper/viper-go organization on GitHub (at least that's the current plan). If cobra ends up being split up as well, we might want to consider moving things under a single, CLI focused organization (even if they won't directly depend on each other)

I'd be happy to work with you to improve the situation (Cobra and Viper belong roughly to the same ecosystem after all).