Loading an MKV in AviSynth+ has wrong framerate and strange out of sync problems

[Roemer]

Hello all
I am trying to digitalize some VHS tapes with an USB-Grabber.
My workflow is as follows:

1. Record the video with ffmpeg in mkv with a low compression (almost lossless)
2. Load the mkv with avisynth+ and perform some filters for fixes
3. Recode the avs with ffmpeg to the final format

Now my issue is that the audio is way off. Upon further investigation I see two problems:

1. The FPS shown in ffmpeg/ffplay is off. It shows like 29.5 instead of 30.
2. When playing the video in any player (like vlc), the first frame appears at around 21 second. But when playing the mkv with ffplay with a simple avisynth file which just loads the file (FFmpegSource2("video.mkv", vtrack = -1, atrack = -1)), the first frame appears already at second 15.

I tried to shorten the mkv with ffmpeg to the first 30 seconds which then shows an even worse framerate of 25 but the first frame appears at second 18? Still wrong but not that much as when using the full videofile.

I now also converted the video to uncompressed avi/pcm (ffmpeg -i "video.mkv" -acodec copy -vcodec rawvideo -y E:\Recordings\output.avi) and this video seems to be totally correct in avisynth when using AviSource to load it.

Can anyone help me here figuring out if this is a bug? I can also provide the video file if needed, just not here in github as the 30 second sample is still wrong but different as the full one.

Thank you

Edit1: I also noticed that the video duration is off by about 3 seconds (this are the 3 seconds which are missing and it starts at 18 instad of 21). The length when opening the MVK with avisynth is 00:27:14.01 and when using the raw avi (made from the same mkv) or when played with vlc it is 00:27:17.57.

Edit2: When re-encoding, I found out that the exact same encoding with mp4 or with mkv leads to the issue with the mkv encode but not with the mp4 encode.
ffmpeg -i "orig.mkv" -c:v libx264 -crf 18 -aspect 4:3 -pix_fmt yuv420p -c:a libmp3lame -b:a 320k output.mp4 -> This one is fine, shows 30 fps , has the correct duration and starts the first frame at second 21 and the audio is sync.
ffmpeg -i "orig.mkv" -c:v libx264 -crf 18 -aspect 4:3 -pix_fmt yuv420p -c:a libmp3lame -b:a 320k output.mkv -> This one is bad, shows the wrong fps, wrong duration and the first frame starts at second 15 and the audio is async.

[myrsloik]

You've most likely created a huge mess without realizing. I suspect that your USB-grabber actually drops frames which leads to the mkv output being full variable framerate. Using avi as the container and manually specifying the framerate should yield something more managable for intermediate processing.
The "bug" (documented quirk is the correct term in this case) is that you then use avisynth to open VFR material which it doesn't support. FFMS2 tries to be helpful and simply reports the average framerate (total frames/total time) to still make things work. Sometimes.

[Roemer]

Hmm when I recorded with ffmpeg, it showed 3 dropped frames or is this something else? What would be a good format to record from the usb grabber in order to work with it afterwards? Currently I use:
ffmpeg -f dshow -rtbufsize 100M -video_size 720x480 -i video="AV TO USB2.0":audio="Mikrofon (USB2.0 MIC)" -t 00:00:10 -c:v libx264 -crf 12 -preset veryfast -c:a libmp3lame -b:a 320k -f tee -map 0:v -map 0:a -flags +global_header "vid.mkv|vi.mp4"

I use the mp4 for processing afterwards as that one seems to work and the mkv is just used that I can duplicate it while encoding and open it, which does not work with the mp4 until it is finished.

[Roemer]

Using the mp4 as input instead of the mkv solves the video issue but now with veeeery long videos (3 hours), the audio is getting more and more async as the video progresses.
The best fix I found so far is just to adjust the framerate to the framecount / audio rate. This gives a framerate of 29.99 (instead of 30) but at least then the audio seems to be async even at the end of the video. The Avisynth code I use for this is: AssumeFPS(FrameCount()/AudioLengthF()*AudioRate(), False. Is that ok or should I use something else?
It is still strange because playing the mp4 in vlc or even in ffplay does not show any async problems, only when using it as input for FFmpegSource2.

[Roemer]

Unfortunately the answer above did not help. I now ended up setting the framerate as above but then manually play around with the TimeStrech for the audio track at a very late part of the video and adjust it (mostly just very small values like + or - 0.008) and just figure out a good number. I checked the source files vor dynamic fps but the dynamic part is shown as 0.0000001% so maybe 1 frame so that is also not the issue.
Something seems very off with the plugin for me.

Edit: To check the framerate, I am using
ffmpeg -i "<videofile>" -vf vfrdet -f null -
Which shows for example
[Parsed_vfrdet_0 @ 000002425539f880] VFR:0.000000 (0/53131)

Edit2: Not sure if this is related to #350, but I have the issue also with mp4 files and also with version 2.23.1.

Use -vsync cfr -r 30 -x264opts force-cfr -c:v libx264 -async 1 out.mkv options in ffmpeg.

"It is still strange because playing the mp4 in vlc or even in ffplay does not show any async problems, only when using it as input for FFmpegSource2"
Players work correctly with non-linear container timecodes. AviSynth not. Use the command above to fix it.

[Roemer]

@artenaki Thank you a lot for your tips, I tried it and it seems better but the audio is still a bit async when played with avisynth compared to vlc.

Edit: Now it just seems slightly during the whole video, so it is at least not getting worse the longer the video.
In an 4h example, I printed the Audio duration (c.AudioLengthF() / Float(c.AudioRate())) and the video duration (c.FrameCount() / Float(c.FrameRate()) which show 14763.765325 for audio and 14763.799805 for video.
If I now add a small audio delay with their difference (AudioDuration - VideoDuration) the sound seems to be fairly ok now.
Edit2: I tried the above with multiple videos, doesn't work either. So I'm still back at manually testing DelayAudio / TimeStretch to find the best variant...