proposal: testing: add -json flag for json results

[rsc]

The idea is to make the test results available in JSON form so that other tools like
build servers can read the results easily.

[alberts]

Comment 1:

Many build servers also like the JUnit XML format. Might be worth it to do both. We've
had good success with running go test -v and translating to the JUnit format (it even
has a place to put stdout and stderr).

[rsc]

Comment 2:

Labels changed: added go1.1.

[rsc]

Comment 3:

Let's do this. Plan of attack:
Structs for output:
type Result struct {
    Tests []TestResult
    Benchmarks []BenchmarkResult
    About struct {
        Version string
        OS string
        Arch string
        Hostname string
        StartTime time.Time
        EndTime time.Time
    }
}
type BenchmarkResult +=
    Package string
    Name string
type TestResult struct {
    Package string
    Name string
    Pass bool
    Output string
    Duration time.Duration
}
Add flag -test.package=math/rand (import path of package), for use in setting Package in
TestResult and BenchmarkResult.
Add -test.json (bool flag) to generate json instead of standard output.
Generated json should be indented.

[rsc]

Comment 4:

Labels changed: added size-l.

[rsc]

Comment 5:

Labels changed: added suggested.

[alberts]

Comment 6:

Any interest in generalising this slightly to allow a JUnit format to be contributed
later?
E.g.
-test.output=std (default)
-test.output=json
-test.output=junitxml

[rsc]

Comment 7:

I would like to make sure we have the information JUnit needs, but I'm
not interested in generating the XML directly. I'd rather that
separate tools took care of externally-defined formats.
Russ

[alberts]

Comment 8:

Looking over the junit stuff:
http://pzolee.blogs.balabit.com/2012/11/jenkins-vs-junit-xml-format/
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd
Here's a few notes:
1. JUnit supports separate stdout and stderr, although we've found that this sometimes
makes it more difficult to understand what happened, so Output is probably fine.
2. JUnit distinguishes between errors and failures. Failures are probably t.* stuff and
errors are panics, but we write a lot of tests that simply panic instead of passing t
around everywhere, so maybe the distinction isn't important.
So I think the model above is probably fine.

[tebeka]

Comment 9:

wrote https://bitbucket.org/tebeka/go2xunit. Willl gladly rewrite to work with JSON
output (specially in parallel testing).

[cespare]

Comment 10:

Russ, what is the purpose of -test.package? Shouldn't 'go test' pass the import path of
the package under test to package testing? Would {Benchmark,Test}Result.Package be blank
when -test.package isn't set?

[rsc]

Comment 11:

The idea is that 'go test' would specify -test.package on the command line when running
the test.
However, it could also compile it in, by adding Package string fields to
testing.InternalTest and testing.InternalBenchmark. That's probably better than the
command-line option.

[cespare]

Comment 12:

I've got the beginning sketches of an implementation at
https://golang.org/cl/7061048/. I'm wondering, though, what the interaction of
-test.json with cmd/go/test should be.
- If we print out summary info (e.g., ok    archive/tar 0.006s), then the output of 'go
test' is not valid json -- you'd have to pre-process it a bit.
- Ditto for multiple packages, if we just print out the results sequentially for each
package.
But, the flag is called -test.json, so 'go test' doesn't directly handle it, right?
Similarly, what should happen if -test.v and -test.json are both set? I'd suggest that
-test.json disable chattiness because the output is part of the json result anyway.

[rsc]

Comment 13:

In json mode the testing framework should not print about starting and
stopping each test. It cannot clear test.v, though, because tests
might use testing.Verbose() (which looks at test.v) to decide how much
log output to generate, and we do want to generate that output (and
save it into the JSON).
The go command will need to know about the json flag regardless, in
order to turn -json into -test.json. When in that mode, it should
synthesize its own Result built containing all the TestResults and
BenchmarkResults from the individual packages. The ability to do this
is the reason that Package appears in each TestResult/BenchmarkResult
instead of in the top-level Result.
Russ

[rsc]

Comment 14:

I spent a while looking into this. I think it's too complex to get into Go 1.1. The hard
part is making sure that every possible case ends up printing json and not text intended
for humans. 
First there is the problem that all test output needs to be caught, even output
generated with fmt.Printf or the builtin println or panic traces, and even if the test
binary exits unexpectedly. The only truly feasible way to do this is to make the go
command interpret the test binary output and generate the JSON from that interpretation. 
I started to try to make that a little easier by adding a -test.outputprefix flag to
specify a prefix string that the test binary will put at the beginning of each line that
it generates, so that the structure of the test output can be seen more readily. That's
in CL 7678044.
But then the next problem is making the go command parse that output and generate JSON.
And the go command must also catch all its own output and convert it to JSON
equivalents, not just for running tests but also for build failures and the like. That's
a bigger task, and at some point it starts to seem like maybe we should make the go
command just have a little bit of framing and have a separate program whose only job is
to invoke the go command (or, perhaps, also a test binary directly) and parse the output
and turn it into JSON.
This is too big a change for what time we have remaining for Go 1.1. Perhaps we can
figure out what the story is by Go 1.2. For now, an external tool that parses the go
command output (and therefore also the test binary output, if using go test -v) seems
like the best choice. And then once that exists we can think about small framing changes
to make the parsing less confusable.

Labels changed: added go1.2, removed go1.1.

[rsc]

Comment 15:

[The time for maybe has passed.]