Rev of proto-gen-go to ProtoPackageIsVersion3 causing breakage

[edwarnicke]

I am hopeful that there is a simple solution to get around this issue :)

A few hours ago, this commit introduced proto.ProtoPackageIsVersion3

and set const generatedCodeVersion = 3

Our CI system (and I suspect many other peoples) do a:

go get github.com/golang/protobuf/protoc-gen-go

as part of setting up the system, and also do a:

go generate ./...

and check to make sure that regenerated files match the files in the commit (basically, checking for regen on updating the .proto files).

This has been going swimmingly until a few hours ago. In trying to debug this problem, I came to realize that the fact this worked at all is a testament to how good you guys are at backward compatibility 👍

What I discovered digging in is that we have been vendoring github.com/golang/protobuf/ v1.2.0 (because of other dependencies we have), but of course go get pulls the latest from master.

v1.2.0 lacks proto.ProtoPackageIsVersion3, so our generated code suddenly doesn't compile.

So I fell back to using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

in our CI. Unfortunately, this seems to result in some generation differences. If I run it locally, push, and then our CI runs it, the generated code doesn't quite match.

I get diffs like:

-var fileDescriptor_networkservice_d4de7ff5f5769d53 = []byte{
+var fileDescriptor_networkservice_5b085879d0d87bdf = []byte{

I don't precisely consider this a bug on your end, you guys are super good about backward compatibility, and at some point you do have to make the change.

Do you have any ideas about how I might navigate all of this? I suspect lots and lots of folks will shortly have similar issues...

[edwarnicke]

@dsnet do you have any insight?

[dsnet]

I'm not sure what we can do on our end, this seems to be an issue really with the Go toolchain that there isn't an easy way (or at least I don't know of one) to build a binary at a specific revision.

For the time being, you can do something like this:

$ git clone git@github.com:golang/protobuf.git && (cd protobuf && git checkout v1.2.0 && go build -o $BIN_PATH/protoc-gen-go ./protoc-gen-go) && rm -r protobuf

Be sure to have $BIN_PATH be the output path where the binary should end up.

[dsnet]

I'm going to close this as a duplicate of #751 since the general issue is that the instructions for how to install protoc-gen-go are not ideal. However, the broader issue at hand is golang/go#24250

[edwarnicke]

@dsnet Yeah... I can build the protoc-gen-go from my vendored dir (v1.2.0) but that version (unlike more recent ones) doesn’t seem to always generate the same go code for identical .proto.

At what point did proto-gen-go start generating identical go for identical proto?

[dsnet]

The output of protoc-gen-go should be deterministic. If it isn't, then that's a bug.

That said, the most likely cause of non-deterministic output is due to how protoc is invoked. The descriptor protos that are embedded in each .pb.go file contains the filepath of the proto file and depending on the -I flags to protoc, it is possible that those change.

[edwarnicke]

@dsnet Also, I don’t think is the same as #751 as it doesn’t have anything to do with modules and we are seeing it with go 1.10 (pre modules)

[edwarnicke]

@dsnet OK, that’s super helpful to know. Are the determined by the fully qualified path of included proto files? I ask, because we also just started using -I ../.../vendor in our protoc invocations.

[edwarnicke]

Which is to say, the fully qualified path of the imported protofiles (things like github.com/golang/protobufs/ptypes/empty/empty.proto) will different depending on where they are built, as we are pulling them in from vendor/ )

[dsnet]

Errr, sorry, I mispoke. It's not the -I flag. It's the how the proto file to compile was specified when given to protoc.

For example:

$ protoc --go_out=paths=source_relative:. ptypes/any/any.proto
$ mv ptypes/any/any.pb.go ptypes/any/any1.pb.go
$ (cd ptypes && protoc --go_out=paths=source_relative:. any/any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any2.pb.go 
$ (cd ptypes/any && protoc --go_out=paths=source_relative:. any.proto)
$ mv ptypes/any/any.pb.go ptypes/any/any3.pb.go 
$ cat any*.pb.go | grep RegisterFile
func init() { proto.RegisterFile("ptypes/any/any.proto", fileDescriptor_any_fea4429e4270d3cd) }
func init() { proto.RegisterFile("any/any.proto", fileDescriptor_any_f2863f9fa4d3af49) }
func init() { proto.RegisterFile("any.proto", fileDescriptor_any_0a174ad2e4d7ef58) }

Notice that all three have different values?

This behavior is surprising, but is an artifact of how protobufs were originally designed. That is, they assume a single root where all source files universally lived under in a build system similar to Bazel. It's not great for today's world of independently distributed build processes, but such is the status quo.

[edwarnicke]

OK.. so that’s a super helpful hint... let me go look for those differences and see if I can think of a solution.

[edwarnicke]

@dsnet OK... so I can't quite square this with my understanding of your explanation:

-	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_efee0c6a43fef699)
+	proto.RegisterFile("networkservice.proto", fileDescriptor_networkservice_1d42c3056c917b94)

[dsnet]

So clearly something else is feeding into the hash. What versions of protoc-gen-go were used to produce those .pb.go files? Also what versions of the Go toolchain were used to build those binaries?

[edwarnicke]

dep has our vendored version of github.com/golang/protobuf/protoc-gen-go/ at version v1.2.0.

we are building protoc-gen-go using:

go install ./vendor/github.com/golang/protobuf/protoc-gen-go/

So I'm going to go out on a limb and say its proto-gen-go version v1.2.0 (I couldn't find any way for the protoc-gen-go binary to output a version, would love to know if there's a trick there :) ).

go version yields go1.10.5 linux/amd64

[dsnet]

And what are the versions going into the other protoc-gen-go binary being used?