Include Testing Example

[marwan-at-work]

I am unsure how to test a protoc plugin that I've written with this library (awesome work btw!).

It would be great if the example folder had a _test.go file to show how one can test their plugin by providing a .proto file. The docs mention using the debug plugin to create a .bin file but I'm still unsure on how to proceed next.

[rodaine]

Thanks for writing in! Yes, we are looking to add a testutil subpackage to make this easier/clearer with prebuilt mocks and what-have-you, as well as more docs in this vein.

In the meantime, as for protoc-gen-debug, it emits the CodeGeneratorRequest as a binary file for the specified run. You can then read the bin file in test, and have it emit the AST that you can use in your plugins. Here's some example helpers that we are using internally, and where it's leveraged in the pgsgo subpackage.

[F21]

I would definitely love to see an example of how this is done as well.
I generated a code_generator_request.pb.bin containing some test protobuf (both valid and invalid for my module) files.

I then load the files and run my module using this:

pgs.Init(
   pgs.ProtocInput(req),  // use the pre-generated request
   pgs.ProtocOutput(res), // capture CodeGeneratorResponse
   pgs.FileSystem(fs),    // capture any custom files written directly to disk
).RegisterModule(New()).Render()

For some reason, I am unable to see the errors outputted by my module when checking the protobuf inputs for correctness. Is there currently a way to do this?

[pdecks]

For some reason, I am unable to see the errors outputted by my module when checking the protobuf inputs for correctness. Is there currently a way to do this?

Have you tried using the Debug Mode functionality? It would look like:

pgs.Init(
   pgs.ProtocInput(req), 
   pgs.ProtocOutput(res),
   pgs.DebugMode(),
   pgs.FileSystem(fs),
).RegisterModule(New()).Render()

or

pgs.Init(
   pgs.ProtocInput(req), 
   pgs.ProtocOutput(res),
   pgs.DebugEnv("PGS_DEBUG"), // non-empty environment variable
   pgs.FileSystem(fs),
).RegisterModule(New()).Render()

[pdecks]

Thanks for the feedback! Will make a note to add some documentation / a walkthrough. The basic steps are:

1. Create the .bin using protoc-gen-debug
2. In your test file, load the binary and pass it to the generator. The entities associated with the AST can be manipulated and at the module level, if generating CustomTemplateFile artifacts written to disk, you can assert these expected artifacts have been generated.

module-level test, e.g. module_test.go

func MyModuleTest(t *testing.T) {
	in, err := os.Open("<path/to/generated.bin>")
	require.NoError(t, err)

	fs := &logging_fs.FS{AF: afero.NewMemMapFs()} // using github.com/spf13/afero filesystem
	g := pgs.Init(
		pgs.FileSystem(fs),
                 pgs.ProtocInput(in), // pass the binary to the generator
                 pgs.DebugEnv("PGS_DEBUG"), // non-empty environment variable
	)
 
        // if your module also consume sthe AST
        ast := g.AST()

        g.RegisterModule(New(ast)).Render()

        // helper function for testing artifact generation
       // destination path is only needed if generating files outside the normal protoc flow, e.g. when the Artifact is a CustomTemplateFile
        myHelperFunction(t, fs, g, "path/to/destination")} 
}

func myHelperFun(t *testing.T, fs afero.Fs, g *pgs.Generator, destPkg string) {
        // example of expected CustomTemplateFiles we want to check the existence of
	var arts []string
	arts = append(arts, fmt.Sprintf("%s/go.mod", destPkg))
	arts = append(arts, fmt.Sprintf("%s/.env", destPkg))

        // use afero to check the existence or absence of generated files
	for _, a := range arts {
		if ok, err := afero.Exists(fs, a); err != nil {
			g.Failf(fmt.Sprintf("error on check for Exists(%s)", a), err)
		} else if exp {
			assert.True(t, ok)
		} else {
			assert.False(t, ok)
		}
	}
}

model-level test, e.g. module/model_test.go

func loadAST(t *testing.T) pgs.AST {
	in, err := os.Open("path/to/generated.bin")
	require.NoError(t, err)
	fs := &logging_fs.FS{AF: afero.NewMemMapFs()}
	return pgs.Init(
		pgs.FileSystem(fs), pgs.ProtocInput(in), pgs.DebugEnv("PGS_DEBUG"),
	).AST()
}

func TestModel_Methods(t *testing.T) {
	ast := loadAST(t)

	for _, target := range ast.Targets() {
		m := New(target.Package())

		assert.Equal(t, expectedValue, m.SomeModelMethod())
        }
}

[F21]

Thanks for the examples! They are immensely helpful 😄

I have a few more questions for my situation. I am not passing the AST to my module; I don't think I need to do this as I am embedding *pgs.ModuleBase in my module and my module implements Execute(targets map[string]pgs.File, pkgs map[string]pgs.Package) []pgs.Artifact and is passed the protoc request.

For testing, I have a bunch of protoc files as test cases, some are valid and should generate expected output. Some are invalid (according to the module) and should return a error.

The protoc-gen-debug plugin combines all the .proto files into a single code_generator_request.pb.bin.

• How do I test each .proto file individually as a test case without passing the AST to my module?
• When testing an invalid .proto file, how should I assert the error? Currently I use pgs.ProtocOutput(res) and can see print the output to see the errors. However, it doesn't seem very correct to attempt to string-match the error in the output.

[pdecks]

You're welcome!

I am not passing the AST to my module; I don't think I need to do this as I am embedding *pgs.ModuleBase in my module and my module implements Execute(targets map[string]pgs.File, pkgs map[string]pgs.Package) []pgs.Artifact and is passed the protoc request.

Correct -- I noted you only need to generate the AST at the generator-level if passing it to the AST 😄

• How do I test each .proto file individually as a test case without passing the AST to my module?

AFAIK, you can only test individual files on the module (here, Module) with something like the following, where you still need to generate the AST because you want to access its targets and you can use ast.Lookup(name string) to return an Entity from the graph by using its fully-qualified name (FQN) say if you wanted to VisitMethod instead of VisitFile:

func loadModule() (*Module, pgs.AST, pgs.MockDebugger) {
	req, err := os.Open("./path/to/code_generator_request.pb.bin")
	if err != nil {
		panic(err)
	}

	ast := pgs.Init(pgs.ProtocInput(req)).AST()

        // create the module to be tested
	m := New(<args...>)

        // used for BuildContext below
	d := pgs.InitMockDebugger()

         // any required context for the module (pgs.BuildContext in pgs.ModuleBase)
	bc := pgs.Context(d, pgs.Parameters{
		paramName:         "<paramValue>",
	}, ".")
	m.InitContext(bc)

	return m, ast, d
}

func TestModuleUsingSomeVisitMethod(t *testing.T) {
	m, ast, _ := loadModule()
	_, _ = m.VisitFile(ast.Targets()["path/to/desired.proto"].File())

        // if valid case, where here m.errors is a custom-defined set of errors
       	require.Empty(t, m.errors)
      
        // if invalid case
        // custom logic for checking actual vs expected errors
        require.NotEmpty(t, errors, "no errors were returned")
	for _, err := range m.errors {
             ...
	}
}

• When testing an invalid .proto file, how should I assert the error? Currently I use pgs.ProtocOutput(res) and can see print the output to see the errors. However, it doesn't seem very correct to attempt to string-match the error in the output.

This depends on what kind of plugin you've written. For example, for a linter you would be looking up the error and asserting it matches whatever custom error you've defined for that invalid case. For a code generator, it could be checking side effects of the Visit methods.

[F21]

I noticed the use of m.VisitFile() in your example. I am not using a Visitor in my module, so the method is not available. Are visitors the only way to test a single proto file? Is it possible to test Execute with a single proto file that's in code_generator_request.pb.bin