Description
We have many tests that generate random inputs and then check the result using some other equivalent computation, for example ecc_randomized and the ECC fuzzers perform very much the same operation. The only difference is that ecc_randomized tests with random inputs (albeit with some bias to try to trigger errors) while the fuzzers are choosing the input via coverage feedback. We also have fuzzers for some operations without a corresponding test, and also randomized tests without a fuzzer.
Ideally these notions could be merged in some way.
The first obvious choice is to redefine the fuzzers from taking a bytestring to taking an RNG instance. This RNG instance is just the input to the fuzzer from LLVMFuzzerTestOneInput and loops as required. This would already be a nice improvement for the fuzzers, and remove a lot of special casing done around input sizes as well as the fixed seed ChaCha20 RNG which likely confuses clusterfuzz somewhat.
After that it gets tricky. Conceptually all of these operations are bool test_it(RandomNumberGenerator& rng); with the difference being that a fuzzer crashes on failure while a test outputs some nice report. We don't want reporting (eg in the style of Test::Result) to add overhead to the fuzzers). I guess this can be resolved via templating the result type.
That raises a second step which likewise gets things a bit closer to the end state, and would be a nice improvement on its own:
- Instead of janky macros like
FUZZER_ASSERT_EQUALdefine something minimal in the style ofTest::Resultthat compares its arguments and instead of logging anything just prints the issue to stderr and crashes if the test fails.
Also from a build perspective this is a bit non-trivial. Best I can come up with here is define such tests/fuzzers in a header and have the fuzzer mains include the relevant files.
Not a fully formed idea, this ticket is to document the situation and suggest the first steps that would allow for later convergence.
Ref #4569 (comment)
Activity