Failing input for new fuzz test

[n-peugnet]

Hi, I am currently trying to add a fuzz target to the test suite, currently I wrote this test (based on TestInfStream)...

func FuzzStream(f *testing.F) {
	testcases := []string{
		`{id: 1, key: "object number 1"}`,
		"hello\n  \n\tworld",
	}

       for _, tc := range testcases {
               f.Add(tc) // Use f.Add to provide a seed corpus
       }
       f.Fuzz(func(t *testing.T, orig string) {
               origBytes := []byte(orig)
               buffer := bytes.NewBuffer(origBytes)
               tokenizer := New()
               commaKey := TokenKey(10)
               colonKey := TokenKey(11)
               openKey := TokenKey(12)
               closeKey := TokenKey(13)
               dquoteKey := TokenKey(14)
               tokenizer.DefineTokens(commaKey, []string{","})
               tokenizer.DefineTokens(colonKey, []string{":"})
               tokenizer.DefineTokens(openKey, []string{"{"})
               tokenizer.DefineTokens(closeKey, []string{"}"})
               tokenizer.DefineStringToken(dquoteKey, `"`, `"`).SetEscapeSymbol('\\')

               stream := tokenizer.ParseStream(buffer, 100)
               var actual []byte
               for stream.IsValid() {
                       current := stream.CurrentToken()
                       // t.Logf("%#v", current)
                       actual = append(actual, current.Indent()...)
                       actual = append(actual, current.Value()...)
                       stream.GoNext()
               }
               // t.Logf("%#v", stream.CurrentToken())

               // As we only concatenate the indents of each token, the trailing
               // whitespaces and token separators are lost, so we trim these
               // characters on the right of both actual and expected slices.
               trimset := ". \t\r\n\x00"
               expected := bytes.TrimRight(origBytes, trimset)
               actual = bytes.TrimRight(actual, trimset)
               if !bytes.Equal(expected, actual) {
			t.Errorf("input:\n%q\nexpected:\n%q\nactual:\n%q", orig, expected, actual)
               }
       })
}

...and ran it using go test -fuzz FuzzStream, which gave me the following output:

fuzz: elapsed: 0s, gathering baseline coverage: 0/120 completed
failure while testing seed corpus entry: FuzzStream/22babe8c5ca7133b
fuzz: elapsed: 0s, gathering baseline coverage: 0/120 completed
--- FAIL: FuzzStream (0.03s)
    --- FAIL: FuzzStream (0.00s)
        stream_test.go:491: input:
            "0E"
            expected:
            "0E"
            actual:
            "0"
    
FAIL
exit status 1
FAIL	github.com/bzick/tokenizer	0.042s

It seems to me that the result is indeed a bug in tokenizer, as the E character is completely ignored, but I am not exactly sure how it should be handled. Should it be a single token {"0E"} or two tokens {"0", "E"}

[bzick]

Yep, sequence 0E has a bug. 0 parses as int, but E switches parser to float mode. And code has invalid condition for situation where is no numbers after E. By this reason parser loses E.
Also fuzz detected other problems, which I will fix in other issues. Thank you for your fuzz-code!

[n-peugnet]

Thanks for the explanation. So the parser should return two tokens, one int 0 and one keyword E?

I don't know if you are aware, but you should be able to keep the fuzz test in the source even if all the inputs don't pass. When running tests with just go test, only the known test cases are run (the one manually added in the code and the ones from testdata, generated when an input fail). When you fix an input, you don't necessarily need to add a new test, you can simply commit the input file from the testdata directory, and it will be part of the test-suite.

It is only with the -fuzz option that go test will try new inputs.

See https://go.dev/doc/security/fuzz/ for more info.

Also I am not entirely sure that the test I wrote is correct, for example, should the concatenation of Indent() and Value() really exactly return the exact same thing as the input string? I think it would be helpful to be able to do it, but maybe Indent() is not the correct way.

[bzick]

Done in #29

• added fuzz tests
• fixed all bugs from fizz tests

[bzick]

I'll extend fuzz testing later, after investigation