VC4 and Mesa Drivers Update Plan

[FrankGould]

I have been working with a team of Kivy and Mesa driver engineers who have identified that Raspbian graphics need to be updated and a new version tested and released. Below is more information about this issue that was first reported on Kivy Github Oct 7, 2016, link below.
https://github.com/kivy/kivy/issues/4662

Below is the Final Word from one of the engineers:

What is needed to fix this would be a updated VC4 (VideoCore IV) and then update the graphic drivers. I saw a VC4 version here on Github but latest changes are about two years old.

This is a dead end unless the foundation themself (Raspberry Pi) marks this as important. When I've been testing, I've tried using Gl4es and Mesa.

Kivy only in Fullscreen or openGL segmentation fault:
viewtopic.php?p=1173191

Kivy only in Fullscreen or openGL segmentation fault:
viewtopic.php?t=185675

Below are the issues we've reported and the support responses:

Kivy apps crash - need mesa version to fix #106:
https://github.com/anholt/mesa/issues/106

Full KMS - #Kivy with Mesa #107:
https://github.com/anholt/mesa/issues/107

Because of this Kivy driver problem, I tried to migrate my app to pi3d but here's what I reported in May 2018 and have not had any responses:

pi3d and streaming video crash:
viewtopic.php?f=28&t=214284&p=1319167

Where should we go to request Raspbian be scheduled for graphic drivers updates, tests, and release?

[fruitoftheloom]

I have been working with a team of Kivy and Mesa driver engineers who have identified that Raspbian graphics need to be updated and a new version tested and released. Below is more information about this issue that was first reported on Kivy Github Oct 7, 2016, link below.
https://github.com/kivy/kivy/issues/4662

Below is the Final Word from one of the engineers:

What is needed to fix this would be a updated VC4 (VideoCore IV) and then update the graphic drivers. I saw a VC4 version here on Github but latest changes are about two years old.

This is a dead end unless the foundation themself (Raspberry Pi) marks this as important. When I've been testing, I've tried using Gl4es and Mesa.

Kivy only in Fullscreen or openGL segmentation fault:
viewtopic.php?p=1173191

Kivy only in Fullscreen or openGL segmentation fault:
viewtopic.php?t=185675

Below are the issues we've reported and the support responses:

Kivy apps crash - need mesa version to fix #106:
https://github.com/anholt/mesa/issues/106

Full KMS - #Kivy with Mesa #107:
https://github.com/anholt/mesa/issues/107

Because of this Kivy driver problem, I tried to migrate my app to pi3d but here's what I reported in May 2018 and have not had any responses:

pi3d and streaming video crash:
viewtopic.php?f=28&t=214284&p=1319167

Where should we go to request Raspbian be scheduled for graphic drivers updates, tests, and release?

http://www.anholt.net

http://www.anholt.net/resume.html

https://github.com/anholt

https://anholt.github.io/twivc4/

[Gavinmc42]

I run Gentoo64 and have noticed the same 106 issue, glxgears gets jerky if you increase the window size.
I compiled the full mesa3d demo examples and the same thing happens with those examples.
Default window is not bad but increasing the size and smooth movements become jerky.

Tim's Star Citizen videos shows how well OpenGLES can work and gives me hope

I am not sure which drivers Sakaki has used in Gentoo64 but they seem very recent, but it is only 4.14.50 at the moment.
I tried to update Raspbian to compile the mesa3d examples but it crashed when I was messing about with the newest driver sources from Eric's github.

Using Eric's github I regard as bleeding edge, but eventually it does need testing.
I think the mesa3d demos are a good place to start for bench marking and bug detection and even learning OpenGL.
Including them in standard Raspbian would help for 32bit debugging.
I also wanted to compare the mesa3d OpenGLES compiled versions against the OpenGL ones.

Currently it is easier to do OpenGL and mesa3d in Gentoo64.
Do the mesa3d demos/examples work on Gentoo64 when configured for OpenGLES?
https://mesa3d.org/opengles.html
Will the VC4 32bit interface cause issues now/still?

Since I am no expert in drivers or compiling or 3D, I decided to wait and spend my time elsewhere.

I try to keep up with Eric's posts every few weeks or so, hoping one day I will understand them
Reading between the lines, he is slowly killing bugs.
There is the issue of Eric working on drivers ahead of current Raspbian too.
I think Eric is now up to 4.19 and Raspbian is 4.14?
So really, most user testing is not been done on the latest version anyway.

Perhaps a wiki on how to update the kernel/driver to the latest for those who want to use it to test the latest for bugs?
I go to the Raspberry Linux github and see Raspbian has a branch at 4.18
Someone who knows what they are doing could probably download the zip and compile 4.18 and test the latest kernel.
Someone who doesn't know how may need a wiki with instructions

As Eric says, just relying on Dom to test stuff is slow work.
Perhaps many hands make light work?

[6by9]

What is needed to fix this would be a updated VC4 (VideoCore IV) and then update the graphic drivers. I saw a VC4 version here on Github but latest changes are about two years old.

This is a dead end unless the foundation themself (Raspberry Pi) marks this as important. When I've been testing, I've tried using Gl4es and Mesa.

Sorry, that's a very confused report.

VC4 (VideoCore 4) is a set of hardware blocks. So you want VideoCore5 as your upgrade? That won't help you on any existing Pis.
Updated graphics drivers? Which bit of the stack? You've got mentions dotted around of GLES (firmware), GL (ARM kernel drivers, not appropriate for Pi0 & 1), and various other bits. Which drivers do you fancy?
VC4 version of what on Github? Where?

GL4es? What's that when it's at home? https://github.com/ptitSeb/gl4es? In which case it's a third party library that isn't shipped with Raspbian.

Mesa. Final something that makes sense and is supported.
The fun and games is that Mesa goes hand-in-hand with the kernel gpu drivers. We take the Long Term Support (LTS) kernel releases because investing the effort in bringing up a kernel every 3 months just isn't worth it. That means currently we're on 4.14, whilst the latest and greatest is 4.18, currently on release candidate 2.
So we need a Mesa to work with 4.14, and can't necessarily just take thei master branch. Looking at their recent releases they throw out a new version fairly regularly, so choose an arbitrary point and now need a test suite to confirm it works. What test apps do you use? And how do you resource it when people complain that their random app doesn't work? It becomes "fun".
And which of the 59 packages that mention Mesa do you need to update? Do they have dependencies? Time to head down the rabbit hole, and fully appreciate why Debian takes the approach of "stable" that it does.

We are trying to shift to the DRM/KMS driver by default, but there are numerous things in the way that either require dropping support for APIs, or significant reworking. And we're talking fairly fundamental things like putting video overlays on the screen, and due to DRM only permitting a single client you can't have omxplayer or raspistill slapping images on the screen should X11 be running to display the GUI (It can work fine via libDRM with a console boot, but we now need two apps, one for command-line and one for GUI).

[Gavinmc42]

We are trying to shift to the DRM/KMS driver by default, but there are numerous things in the way that either require dropping support for APIs, or significant reworking. And we're talking fairly fundamental things like putting video overlays on the screen, and due to DRM only permitting a single client you can't have omxplayer or raspistill slapping images on the screen should X11 be running to display the GUI (It can work fine via libDRM with a console boot, but we now need two apps, one for command-line and one for GUI).

Ouch, I know just enough to feel for you.
Still MMAL and camera V4L2 etc to be done and tested, if I have read between the line in Eric's posts.
Transition will not be fun, there will be complaints, but that's progress.

I am lucky, I am in the position, "oh, does not work yet", I will come back in 6 months or a year, ""oh look it works now".
Time is magic, enough time will fix anything, or enough time I might learn enough to understand it a year later
I can now go back two years to my old posts, oh that works now, time to do it.

Oh wow, 5 years since I wanted to do OpenVG UI's
Does not feel that long, time flies when you are having fun?
They say it takes 10 years to get good at anything, I am half way there, RPF is not much older.
I think you guys are on schedule for a pretty good OS by year 10.
The ecosystem for Pi's is now big enough there are many options now for doing things now if you don't need bleeding edge stuff.

[6by9]

We are trying to shift to the DRM/KMS driver by default, but there are numerous things in the way that either require dropping support for APIs, or significant reworking. And we're talking fairly fundamental things like putting video overlays on the screen, and due to DRM only permitting a single client you can't have omxplayer or raspistill slapping images on the screen should X11 be running to display the GUI (It can work fine via libDRM with a console boot, but we now need two apps, one for command-line and one for GUI).

Ouch, I know just enough to feel for you.
Still MMAL and camera V4L2 etc to be done and tested, if I have read between the line in Eric's posts.
Transition will not be fun, there will be complaints, but that's progress.

MMAL is working via libDRM. See https://github.com/6by9/drm_mmal
MMAL is working via X11 (if you've also update Mesa). See https://github.com/anholt/drm_mmal/tree/x11
Camera V4L2 is working, but it needs a set of kernel changes merging (current task).
Codecs via V4L2 is working, but needs kernel changes merging. LibreElec folk are trying to move to this by default.

Further entertainment that the above are mainly using CMA allocated by the Linux kernel, not gpu_mem. So how do we move people off setting gpu_mem=X? Current move is for the VPU to also ask the kernel for CMA via a new driver (also working), so we could make gpu_mem=X set up cma=X, but that is then likely to overwrite whatever config a user may have chosen to add.

But all of this is new API stuff, so then have to update/modify raspistill, raspivid, omxplayer, Chromium, etc to be able to use them, or write wrappers around lumps so that old pipelines still work. DRM could be made to look like DispmanX, but still has the issue of single client. The V4L2 stuff is wrapping MMAL, so no changes really needed there. etc. It all takes time.

[FrankGould]

Thanks 6by9 for your response. If what I wrote is confusing, I apologize because I'm new to this environment and may be out of context with reality.

What we're trying to accomplish is to get Kivy's screenmanager to work with transitions and to stop crashing RPi. In our research into the symptoms, we have tried compiling many Mesa versions to see if something broke with an earlier release. That's why we are trying to determine if it is VC4 and Mesa drivers that need updating for Raspbian. We asked Eric Anholt about an update, he responded, "If the current VC4 drivers in Mesa master work, then there's not much I can do for you. Raspbian has ancient software, and I have no control over their update schedule."

https://github.com/anholt/mesa/issues/107

It appears Linus Torvalds now manages the VC4 modules but we don't know how to contact him or ask.

If you have any other suggestions, please let me know. Thanks for your time!

[anholt]

@6by9 On Mesa master and current releases, the vc4 driver works with all kernel versions that Mesa has ever worked with. I don't believe we've ever had a kernel backwards compatibility regression.

[6by9]

@6by9 On Mesa master and current releases, the vc4 driver works with all kernel versions that Mesa has ever worked with. I don't believe we've ever had a kernel backwards compatibility regression.

I'll see what traction I can get in the office then. There will be a fairly heavy reliance on that "no kernel backwards compatibility regression" though.

We really do need to get that CI testing sorted for the stack. I'll talk to Gordon as to whether it is worth getting Maxine over to help set it up - spare resource is rather limited at the moment.

[FrankGould]

Simon Long says: 1st Jul 2018 at 7:07 pm

As mentioned previously, we are in the process of developing a replacement for our existing 3D driver. I cannot tell from your bug reports whether the problem you are experiencing is with the old driver or with the (still experimental) new driver.

If it is a problem with the old driver, it is unlikely to be fixed, as the plan is to move to the new driver and that is where all development work is concentrated.

If it is a problem with the new driver, you already seem to have reported it to Eric Anholt, who is responsible for the development work on said driver, so the ball is in his court for fixing it.

I do not currently have a timescale for the release of the experimental driver, but it can be enabled in existing releases under Advanced settings in raspi-config if you would like to try it to see if it fixes your problem (but you may already have done so).

https://www.raspberrypi.org/blog/raspbi ... nt-1462095

[feelslikeautumn]

You could try an alternative operating system with a more recent version of mesa/etc.

[Gavinmc42]

I am pretty happy with Gentoo64 OpenGL so far, but I have not pushed it hard.
Hoping to get Blender working for more stress testing

Locking up GPU? Looks like another bug got squished or was that a VC5 bug?

The final result was a CLIF file I sent to the HW team of a simple transform feedback test that was locking up the GPU. They reported that I was missing a wait packet that was required, beyond what the HW specs said. (I had actually earlier tried emitting the wait, but typoed the condition for enabling it!) With that, I think I’ve cleared up all of the GPU hangs in piglit runs of khr_gles3.

[6by9]

So we're looking at the version that ships with Stretch, but stability is still the core aim there so are being cautious.

However the testing branch for Buster is already at 2.4.92 (9 May 2018) for libDRM and 18.1.2 (June 15 2018) for Mesa and you can mix and match repos. The instructions I've been given:

I don't know about easy, but this is the cleanest:

https://serverfault.com/questions/22414 ... om-testing

You'd want to edit sources.list to duplicate the deb line, but change one from stretch to buster.
Then add testing.pref and stable.pref as described.

Run apt update. Since this pins 'stable' above 'testing', apt upgrade shouldn't want to upgrade anything. If it does, something is wrong.

Then you should be able to run apt install -t=testing some_package.

Haven't done this in a long time, so I can't guarantee this will just work as described.

I'm on other stuff at the moment, but will be giving it a whirl as soon as I get a chance.

[ShiftPlusOne]

Quick heads up that I wouldn't recommend following those instructions without a backup on an install you expect to be usable afterwards.

https://wiki.debian.org/DontBreakDebian ... nkenDebian

[FrankGould]

Thanks ShiftPlusOne and 6by9 for the advice and recommendation. Unfortunately, I'm not educated enough to know how to build these different modules other than a simple install script. I'm working with someone who is familiar but he couldn't quite translate the recommendation. The advice about FrankenDebian sure makes sense, especially for future stability.

We are hoping to learn more about this issue and are ready to try anything you all recommend.

[FrankGould]

Today, I ran the same screenmanagertest.py code on Gentoo and experienced the same OS locked up happened. If I recall correctly, it's the same on Raspbian, KivyPie, and now Gentoo OS. Does anyone have any suggestions how to get these OS builds to support kivy screenmanager? Everything we see appears to do with mesa/VC4 but we don't know what to do to find a solution, on any OS.

[FrankGould]

Yesterday, I ran ubuntu mate on my RPi 3B and it appears their VC4/Mesa drivers work. I tested using the screenmanagertest.py code and it ran for an hour and never locked up or crashed.

So, it appears ubuntu has the right drivers. Is there any way for raspbian to release the right drivers or can I help by testing their drivers? I'll need some guidance but am willing to try their drivers on raspbian.

At this point, I have a solution for kivy. However, in the future, I'd like to use raspbian since it is foundation approved.

[Gavinmc42]

I though Ubuntu Mate was still an old version?
Is this the one?
https://ubuntu-mate.org/raspberry-pi/
Maybe Kivy needs the old drivers?

[FrankGould]

Gavinmc42, I tried to find the command to determine the driver versions but was unsuccessful (I even asked on #ubuntu-mate).

That is the link where I downloaded ubuntu-mate image.

Any suggestions you have about kivy using the old drivers versus new would be appreciated. All I know is that ubuntu-mate works with screenmanager but does not with raspbian.

[Gavinmc42]

Mesa has some utils perhaps glxinfo -B
es2_info?
I am still new at learning this stuff.

[FrankGould]

You're doing pretty good. glxinfo -B worked and the summaries are below:

Ubuntu-Mate Versions Summary: https://pastebin.com/a2bbCstw
Device: llvmpipe (LLVM 6.0, 128 bits) (0xffffffff)
Version: 18.0.5

Raspbian Versions Summary: https://pastebin.com/eFT7UGuS
Device: llvmpipe (LLVM 3.9, 128 bits) (0xffffffff)
Version: 13.0.6

[6by9]

You're doing pretty good. glxinfo -B worked and the summaries are below:

Ubuntu-Mate Versions Summary: https://pastebin.com/a2bbCstw
Device: llvmpipe (LLVM 6.0, 128 bits) (0xffffffff)
Version: 18.0.5

Raspbian Versions Summary: https://pastebin.com/eFT7UGuS
Device: llvmpipe (LLVM 3.9, 128 bits) (0xffffffff)
Version: 13.0.6

https://www.mesa3d.org/

June 3, 2018
Mesa 18.0.5 is released. This is a bug-fix release.
NOTE: It is anticipated that 18.0.5 will be the final release in the 18.0 series. Users of 18.0 are encouraged to migrate to the 18.1 series in order to obtain future fixes.

March 20, 2017
Mesa 13.0.6 and Mesa 17.0.2 are released. These are bug-fix releases from the 13.0 and 17.0 branches, respectively.
NOTE: It is anticipated that 13.0.6 will be the final release in the 13.0 series. Users of 13.0 are encouraged to migrate to the 17.0 series in order to obtain future fixes.

I'm actually stunned that Ubuntu has taken such a recent drop as they normally also follow Debian stable. By the looks of it it's only hit as part of Ubuntu 18.0.5 - http://changelogs.ubuntu.com/changelogs ... /changelog

We are discussing it, but there are a fair number of dependencies to cover.

[gkreidl]

...
I'm actually stunned that Ubuntu has taken such a recent drop as they normally also follow Debian stable. By the looks of it it's only hit as part of Ubuntu 18.0.5 - http://changelogs.ubuntu.com/changelogs ... /changelog
...

Ubuntu is based on Debian unstable ....

[Gavinmc42]

My 4.14.44-v8 Gentoo64 is reporting version 18.1.2
It is not the current kernel which I think is 4.14.54?
Gentoo64 is new to me too, trying to keep up can be tricky with new OS's

Interesting, your Raspbian is reporting the old 0.4 Gallium driver.
Sakaki's Gentoo64 is reporting the much newer VC4 V3D 2.1 renderer. but only the 1.2 shading language.

.

[Neex]

Just putting my hand up saying that we've invested heavily in RasPi for our product, and Kivy as the provider for our UI. As we near product launch, and discover that our UI is failing because of a memory leak, we are extremely concerned. Can anyone provide a status, schedule and workaround? Many thanks.