RGB DPX 16-bit sequences (4950x3764, 900+GB) not initiating FFmpeg transcode

[digitensions]

Hi Jérôme and all,

I'm running some parallel transcoding tests at the moment across several storage devices. I'm having repeated issues with very large files reaching the end of the first RAWcooked analysis, and then stalling. This is mainly being noticed against the slower storage devices. The outputs reach over 100% with the last amount finding multiple files:
Analyzing files (100.32%), 5 file/s
Analyzing files (100.33%), 16 file/s
The files I assume are probably my required attachments. But at this point the FFmpeg command does not launch, and no further warnings/errors are output to the log. The RAWcooked launch scripts remain open though, blocking any further process from launching in GNU parallel.

The files are RGB 16-bit, 4950x3764 and generally between 800-950GB. We're running on our usual Linux server from Ubuntu 20.04 LTS, using FFmpeg version 4.2.7-0ubuntu0.1, and using RAWcooked snapshot 21.09.20221208. MediaInfoLib is 22.06.

Please let me know if you'd like samples of the files or reversibility data, etc.
Thanks so much!

[digitensions]

I should add these pauses date back to 10th February and I don't think our IO would be the cause, but I'll check with iftop tomorrow.

Apart from this, some excellent MKV transcodes are cooking here! Thanks so much.

[JeromeMartinez]

I'm having repeated issues with very large files reaching the end of the first RAWcooked analysis, and then stalling.

You mean that this is not always the case i.e. for a same dir sometimes it works, sometimes not?

Analyzing files (100.32%), 5 file/s

More than 100%!!! and slow...

Could you provide the output of ls -lR on the dir?
So I could try to reproduce the same kind of attachments / content without the files.

It will be long to debug :(.
But also I plan to rework this part (multithreading for not depending on I/O latency), with some luck it would remove this issue, but no ETA :(.

[digitensions]

Hi Jérôme, thanks for quick response.

I've been investigating the sibling folders as they're split from the same original sequence. There are error messages featuring in the logs for the RAWcooked, but MKV files are building:
[ffv1 @ 0x55a89be576c0] Cannot allocate worst case packet size, the encoding could fail speed = 0.00141x

I attach the ls output for a folder which has frozen.
Happy to send any sample DPX you may need.
Thanks,
Joanna

ls_lr_N_464631_06of12.txt

[JeromeMartinez]

There are error messages featuring in the logs for the RAWcooked, but MKV files are building:
[ffv1 @ 0x55a89be576c0] Cannot allocate worst case packet size, the encoding could fail speed = 0.00141x

Not error message, only warning, due to size of image, "worst case" do not happen in practice (and if so, the encoding would fail so visible).

I attach the ls output for a folder which has frozen.

Very few and small attachments, classic layout, very weird.

I've been investigating the sibling folders as they're split from the same original sequence.

So same kind, but different content "only", weird!
I am afraid that I need the full content for reproducing, and/or I need to have more log. I bet I could not reproduce so maybe the way to go is to have more logging, I'll implement more logging so you can see when it is frozen where it is frozen.

BTW, if you try again with the same folder, is it fine, i.e. is this freeze easily reproducible? If so, even without using GNU Parallel?

[digitensions]

Thanks Jérôme, good to know that the warning isn't critical.
More logging sounds good, I'll await a new snapshot.

I'll will try cooking this problem item directly now without Parallel and let you know how it goes.

Many thanks,
Joanna

[digitensions]

The folder is RAWcooked encoding using a direct command Jérôme, I'll monitor my IO for a while today also and see if we have issues there. Not sure if it's relevant but the ffmpeg command being used has a framerate of 24.000002, when all rest are straight 24.000000.

[digitensions]

Morning, I can confirm that transcode completed manually over the weekend okay Jérôme. Any more I encounter with this issue I'll cook manually, it could be a conflict with GNU parallel. Many thanks!

[JeromeMartinez]

I plan to refactor the analysis part, as it is not blocking your work it seems less urgent, I think I'll wait for this work before adding more logs and so on.
Let me know if it is too much often.

[stephenmcconnachie]

Thanks both for investigating / troubleshooting. Jérôme if a budget is required to do refactoring I could attempt to raise that, as these workflows are now so critical and positive for our preservation. Please consider if that would help, and let me know? All the best, Stephen

…

________________________________ From: Jérôme Martinez ***@***.***> Sent: 20 February 2023 10:51 To: MediaArea/RAWcooked ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [MediaArea/RAWcooked] RGB DPX 16-bit sequences (4950x3764, 900+GB) not initiating FFmpeg transcode (Issue #396) Attention. This email originated outside the BFI. Please be extra vigilant when opening attachments or clicking links. I plan to refactor the analysis part, as it is not blocking your work it seems less urgent, I think I'll wait for this work before adding more logs and so on. Let me know if it is too much often. — Reply to this email directly, view it on GitHub<https://url4.mailanyone.net/scanner?m=1pU3lX-0001XK-40&d=4%7Cmail%2F90%2F1676890200%2F1pU3lX-0001XK-40%7Cin4h%7C57e1b682%7C10717690%7C7962607%7C63F350AF0DAA4ABC5D028F74B5AE7BE8&o=%2Fphti%3A%2FgtsmbthMco%2Fu.aaedRre%2FiAdoAWike%2Fco9ess%23%2F36usmuisecomse71nt8363-4656&s=T6_4K41dg196UOI30A0n1KDLxk0>, or unsubscribe<https://url4.mailanyone.net/scanner?m=1pU3lX-0001XK-40&d=4%7Cmail%2F90%2F1676890200%2F1pU3lX-0001XK-40%7Cin4h%7C57e1b682%7C10717690%7C7962607%7C63F350AF0DAA4ABC5D028F74B5AE7BE8&o=%2Fphti%3A%2Fgtsmbthnco%2Fu.tfotocaiiibunscsus%2Fnteri%2Fauhb-BKADQAZ6354R5HBLHUTTCWVEANET3YAFNCAM6ANSGUAAQ36HA3&s=clHsGN-iPfhWXtGjps8QC84IciI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> The British Film Institute is governed by Royal Charter and is a charity registered in England and Wales number 287780. The contents of this e-mail are confidential and may be legally privileged. If you are not the intended recipient, kindly notify the sender that you have received this message in error and immediately delete it. Unless you are the intended recipient, you may not forward this e-mail to anybody, nor make any use of its contents.

[JeromeMartinez]

as these workflows are now so critical

☺️

Please consider if that would help, and let me know?

For the moment I am blocked by MediaInfo related projects during the next months, but with no other big projects in the pipe (except... FFV1 fast encoder/decoder! But also related to RAWcooked).
I drop an email with more details.