Non-deterministic encodes on Speed 10 on fast-scene detection

[vibhoothi]

When we encode a clip at Speed 10 with a config, we are getting different results at each runs. On quick check with the encoded clips, we can see that, the keyframe placement is changing in each runs. For example, in one encode Keyframe was in Frame 90, while second time when we run, keyframe was on Frame 99. Making non-reproducible encodes. At some tests the number of keyframes are also changing.

Easy way to reproduce is, encode a clip with speed 10, and also have --psnr for getting metrics,
$ ./target/release/rav1e ~/awcy/media/vimeo-corpus-10s-qcif/snowboard.y4m -o snowboard.ivf --psnr --limit=100 -y -s 10

Prior to #2710, things was normal. So we need to dig down the rabbit-hole for figure out what is wrong. This was noticed when reviewing #2765.

AWCY Runs:
current-master,
https://beta.arewecompressedyet.com/?job=master-s10-vimeo-run1%402021-08-25T15%3A06%3A10.173Z&job=master-s10-vimeo-run2%402021-08-25T15%3A05%3A40.907Z
Pre 2710,
https://beta.arewecompressedyet.com/?job=pre-sc-master-s10-run1%402021-08-25T21%3A03%3A55.665Z&job=pre-sc-master-s10-run2%402021-08-25T21%3A04%3A16.653Z

[shssoichiro]

This seems to be related to the use of downscale and the parallelism within it. If I change this line to let threads = 1;, it works around the issue and I get consistent output, although I haven't yet determined why the parallelism is causing non-deterministic results here.

[shssoichiro]

Downscale doesn't appear to be the issue (or at least not the only issue). Even with threading on downscale forced to 1 thread, there is still some inconsistency. Debugging is showing me that deltas are significantly different from one run to the next, even in cases where it coincidentally doesn't affect keyframe decision.

I'm going to continue investigating this until I have an answer.