GPU is reset on Raspberry Pi 3

[carlonluca]

Is this the right place for my bug report?
It is very difficult to say what component is causing the issue, but maybe someone here can help.

Describe the bug
I ported a system including Qt apps based on eglfs/dispmanx to fkms in order to support rpi4. Apps are also used on Windows, rpi3, mac os etc... with small variations. I have issues with animations that I cannot explain: some scaling animations seem to cause distortion of the scene. The distortion appears a bit like a vsync issue (just trying to describe it, not sure if it is related to vsync). The distortion often persists after the animation completes. If I set the animation to have duration 0, no issue appear.
The distortion seems to only appear when resolution is set to values "near" 1366x768 (I could also reproduce on 1360x768) and on fkms. 1920x1080 seems to be ok with both drivers. 1280x720 seems to be ok with both drivers. 1366x768 was ok on old dispmanx drivers but is not on fkms.

In some cases, when the animation starts on 1366x768, I get this from the kernel:

[  262.072366] [drm] Resetting GPU.
[  270.952406] [drm] Resetting GPU.
[  276.952461] [drm] Resetting GPU.
[  278.952417] [drm] Resetting GPU.
[  281.112422] [drm] Resetting GPU.
[  283.032194] [drm] Resetting GPU.

To reproduce
I tried to write a minimal example showing the problem, but I failed. Even in the same software, not every scaling animations show the problem.
What I see is that I can reproduce by running some scaling animations on fkms by setting the resolution to 1366x768.

Expected behaviour
No distortion, like with the legacy drivers.

Actual behaviour
Distortion appears.

System

• Which model of Raspberry Pi?

Pi3B+

• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2019-06-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 150e25c4f8123a4c9c63e8dca1b4737fa6c1135c, stage2

• Which firmware version (vcgencmd version)?

Aug 15 2019 12:08:48
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start_x)

• Which kernel version (uname -a)?

Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

Logs
Dmesg output:

[  262.072366] [drm] Resetting GPU.
[  270.952406] [drm] Resetting GPU.
[  276.952461] [drm] Resetting GPU.
[  278.952417] [drm] Resetting GPU.
[  281.112422] [drm] Resetting GPU.
[  283.032194] [drm] Resetting GPU.

Not sure if other logs exist somewhere. I tried to get logs with vcdbg but I got an error:

sudo vcdbg log msg
Failed to allocate -505734863 bytes for message buffer

Additional context
My application runs without X11. Difficult to say if this is a problem in Qt, my app, drivers etc... Any advice on how to investigate further?

[carlonluca]

I'm not getting any answer here. May I have more luck in the firmware repo? Thanks.

[JamesH65]

I doubt anyone has had time to look at this. Note that 1366x768 is a very strange resolution that does cause problems in the HDMI system due to having odd numbered timings. Might be related.

EDIT: Sorry, I was thinking of the Pi4. Probably not relevant.

[carlonluca]

Thank you for answering. That is the resolution of the screen I'm targeting. To temporarily "fix", I simply forced a resolution of 720p when auto detection sets fb to 1366x768 or 1360x768, but I'm afraid people may hit the same issue with other resolutions.

Do you think someone will have time to look into this (I perfectly understand you have priorities to respect)? Do you suggest to leave the message here or in the firmware repo? I'm testing a rpi3b+ atm.
Thanks again.

[6by9]

This would appear to be related to KMS and/or Mesa, so this is the right place.

Is the behaviour the same with both vc4-kms-v3d(*) and vc4-fkms-v3d on your Pi3? That would imply that it is more in the 3D rendering side than the composition and output side.
[drm] Resetting GPU. would seem to say that it is 3D related anyway.

The behaviour may be slightly different on a Pi4 anyway, as the 3D block is 2 generations further on than Pi3.

(*) NB That vc4-kms-v3d is currently NOT available on Pi4.

[carlonluca]

To test I simply switched from dtoverlay=vc4-fkms-v3d to dtoverlay=vc4-kms-v3d, is this sufficient to switch from fkms to kms?

I ran the same test and the issue is there in both cases. However, there is a slight difference: with fkms the scale animation stutters and then, for some time, I get a corruption similar to horizontal tearing, which is very noticeable as there is a scrollable content in my UI.
On kms, the same scale animation stutters but then I can see no noticeable "corruption" when scrolling the content. So it seems the only issue is the scaling animation on kms.

[carlonluca]

I could reproduce the issue also on other resolutions, like 1080p and 720p only on rpi3. Rpi4 seems to work properly.

[timemaster5]

I am having the same issue. Can't find a solution for a longer time. I also tried to get back to the userland drivers instead of fkms and the problem persists.

[JamesH65]

The code here has been undergoing a lot of changes. Is you system completely up to date?

[timemaster5]

Thank you for your response. After today's deep dive, I suspect some low-level issue is going on there, like device tree configuration or so. I tried multiple kernels, from 4.19.127 to 5.10, then kms, fkms and userland. I was also fiddling with gpu_mem and cma. Nothing helped. Then I thought, maybe it is a mesa problem or libdrm, so I updated them too to the 21.x version without any luck at the end :(

The issue is nicely summarized here: https://forums.raspberrypi.com/viewtopic.php?t=274293. I am also on CM3 and see exactly the same behaviour. I will try Rpi 3b+ tomorrow to be sure that there is nothing wrong with my side.

What leads me to think about the device tree problem is that when this happens ([drm] Resetting GPU..), and I reboot the machine by issuing a reboot command, it won't boot again. I can see kernel messages on the serial interface, but no picture, only display blinking from time to time when it is supposed to change resolution. Then the whole system is stuck, and I need to do a complete power cycle to boot again.

Any suggestions on what I should try? I am on Yocto, so I can precisely build a specific commit of some component if needed. But so far, my list of things I wanted to try is shrinking fast :(

[timemaster5]

This would appear to be related to KMS and/or Mesa, so this is the right place.

Is the behaviour the same with both vc4-kms-v3d(*) and vc4-fkms-v3d on your Pi3? That would imply that it is more in the 3D rendering side than the composition and output side. [drm] Resetting GPU. would seem to say that it is 3D related anyway.

The behaviour may be slightly different on a Pi4 anyway, as the 3D block is 2 generations further on than Pi3.

(*) NB That vc4-kms-v3d is currently NOT available on Pi4.

It is related to 3D only, yes. It happens a few secs after some rendering starts. On 2D like Weston desktop, I see no problems at all, but if I open some for example WebGL content, it goes bad after a few seconds of nice rendering.

I would like to gather some debug logs, but I wasn't successful with these environment variables:

export G_MESSAGES_DEBUG=all
export MESA_DEBUG=1
export EGL_LOG_LEVEL=debug
export LIBGL_DEBUG=verbose
export WAYLAND_DEBUG=1

[timemaster5]

Ok, I can confirm it is the same on RPi3b+. But I am moving forward slowly. Is it possible that this is caused by the code of the 3D app which uses DRM? I try some WebGL stuff, and one demo loads fine and renders without any issue and the other causes crash every time. So I am wondering what is wrong, not enough memory, some unsupported instruction or some bug?

Even if it is an application issue, I would expect the application to segfault instead of full system lockup with the power cycle needed.