Raspbian vs Wheezy (beta) Mono support

[emg]

Anyone see issues running .Net applications under Mono 2.10 with the recently released Raspbian OS? I have a small test application that runs great on all the armel releases (soft float) including the latest Wheezy beta. When I try the same application under previous and current Raspbian version I get errors like:

Code: Select all

root@raspberrypi:~# mono hello.exe
Stacktrace:

  at System.Drawing.Font.CreateFont (string,single,System.Drawing.FontStyle,System.Drawing.GraphicsUnit,byte,bool) <0x00147>
  at System.Drawing.Font..ctor (string,single,System.Drawing.FontStyle,System.Drawing.GraphicsUnit,byte,bool) <0x0007f>
  at System.Drawing.Font..ctor (string,single,string) <0x00057>
  at (wrapper remoting-invoke-with-check) System.Drawing.Font..ctor (string,single,string) <0xffffffff>
  at System.Drawing.SystemFonts.get_DefaultFont () <0x0007b>
  at System.Windows.Forms.Theme..ctor () <0x00033>
  at System.Windows.Forms.ThemeWin32Classic..ctor () <0x00013>
  at System.Windows.Forms.ThemeEngine..cctor () <0x000bb>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at System.Windows.Forms.SystemInformation.get_MenuAccessKeysUnderlined () <0x00007>
  at System.Windows.Forms.Control..ctor () <0x001fb>
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control..ctor () <0xffffffff>
  at System.Windows.Forms.WindowsFormsSynchronizationContext..cctor () <0x0002f>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at System.Windows.Forms.Control..ctor () <0x0004b>
  at System.Windows.Forms.ScrollableControl..ctor () <0x00013>
  at System.Windows.Forms.ContainerControl..ctor () <0x00023>
  at System.Windows.Forms.Form..ctor () <0x00077>
  at hello.frmMain..ctor () <0x00013>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>
  at System.Reflection.MonoCMethod.Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) <0x001ff>
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) <0x00053>
  at System.Reflection.ConstructorInfo.Invoke (object[]) <0x0005f>
  at System.Activator.CreateInstance (System.Type,bool) <0x001ab>
  at System.Activator.CreateInstance (System.Type) <0x00017>
  at System.Activator.CreateInstance<T> () <0x00017>
  at hello.My.MyProject/MyForms.Create__Instance__<T> (T) <0x00133>
  at hello.My.MyProject/MyForms.get_frmMain () <0x00023>
  at hello.My.MyApplication.OnCreateMainForm () <0x0001b>
  at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.OnRun () <0x00043>
  at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.Run (string[]) <0x00023>
  at hello.My.MyApplication.Main (string[]) <0x0003f>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:


Debug info from gdb:

Cannot access memory at address 0x0
Cannot access memory at address 0x0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x408b4460 (LWP 1799)]
0x40135f30 in read () from /lib/arm-linux-gnueabihf/libpthread.so.0
  Id   Target Id         Frame 
  2    Thread 0x408b4460 (LWP 1799) "mono" 0x40134d84 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
* 1    Thread 0x40053000 (LWP 1798) "mono" 0x40135f30 in read () from /lib/arm-linux-gnueabihf/libpthread.so.0

Thread 2 (Thread 0x408b4460 (LWP 1799)):
#0  0x40134d84 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1  0x00195a30 in mono_sem_wait ()
#2  0x000ea388 in ?? ()
#3  0x000ea388 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x40053000 (LWP 1798)):
#0  0x40135f30 in read () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1  0x40135770 in __pthread_enable_asynccancel () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2  0x00000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

Aborted

Any idea what the differences in fonts between the armel and armhf images?

Thanks!

[jheizer]

Seeing the same thing here as well.

[DonOregano]

I'm seeing the same exception when running resgen (the mono program that converts resx files to resource files) on some resx files. It appears to happen when there is an ImageStream in the file. (Can't attach an example, forum wont allow me...)

I'm running on a system based on Hexxeh's image, what images are you using?

Can't really see that using a different image should make much difference, unless there is some package dependency problem where some very basic font package is not installed on the Hexxeh image.

[jheizer]

I am using the "official" image and I just happen to get my pi the day before it was released. I tried installing ms ttf to see if that helped. Also ran fc-cache -fv and it completed without errors.

[emg]

None of the armhf images have worked for me (Hexxeh's latest, piecies and latest 'Official' Raspbian). Same error, consistent. No problems with the original Debian and the latest Wheezy beta armel. A also tried adding ms fonts, no effect. Is there a way to get the Raspbian team to take a look at the Mono build/dependencies? (or at least someone who knows what they are doing) What about installing Mono from the armel Wheezy beta repository onto Raspbian, can it be done? Would it help?

[mpthompson]

This is probably going to take the help of someone who understands the internals of Mono to help debug what the general issue is. When I have some time I'll look to see what build options that might have been overlooked to configure the package. Sometimes I get lucky and find a combination of configuration settings that work. A complete guess on my part is that the Mono JIT (I'm assuming it has one) is producing ARMv7 code that passes tests on our build servers, but fails when run on an ARMv6 device such as the Raspberry Pi.

Right now, the system I generally use for working with builds is busy with a compile of Chromium. When it's done (hopefully within a day or so), I'll take a look at this.

[emg]

Mike, many thanks for the info and help!

[jonskeet]

Glad to find this topic... I only got started with my Pi yesterday, and installed the Wheezy image. Getting Mono going "at all" was incredibly simple, and I managed to build my pet date/time project (Noda Time) easily - but a lot of the unit tests failed. I didn't have time to investigate that yesterday, but it could be related to some of the problems reported here.

What actually brought me here was a tweet I received:
http://twitter.com/devlead/status/225728523803963392

Including: "FYI I got best experience with mono using Arch. Raspbian mono is broken."

I don't know what the differences are, but I'll probably give it a whirl on a separate SD card to see if it works any better for me.

[emg]

If you want to try Debian Wheezy instead of Arch, try the beta found here:
http://www.raspberrypi.org/archives/1435
Mono works great and this is the distro I'm using until (hopefully) Mono under Raspbian gets sorted.

[mpthompson]

What actually brought me here was a tweet I received:
http://twitter.com/devlead/status/225728523803963392

Including: "FYI I got best experience with mono using Arch. Raspbian mono is broken."

I don't know what the differences are, but I'll probably give it a whirl on a separate SD card to see if it works any better for me.

Does anyone know if Arch builds Mono with hard float? If so, we could compare configuration settings to see if what needs to be done to get it working under Raspbian.

[mpthompson]

I'm taking a look at the Mono issues right now.

Looking at the Mono debian package source code I'm not seeing anything that really specifies the CPU or floating point configuration within the build rules files. This indicates to me that the target CPU and FPU options are determined dynamically during the actual build process rather than pre-specified with configuration settings are they are with tools such as the Gnu compilers. Since our build systems are armv7 systems rather than armv6 systems I'm guessing that the resulting CPU configuration is incorrect for the armv6 Raspberry Pi.

Unless someone has a better idea, I think my next steps is to try to build Mono under Raspbian on an actual Raspberry Pi or in a QEMU environment. Either way, this could be a bit daunting as the source code is 47MB compressed. Yeesh. Anyway, this will take a while to setup my Pi for compilation and building. However, I do want to get Mono working as it is an important package.

I'll post back with more information as progress is made. In the meantime it would help if someone could confirm that Mono builds under Arch with hard float and what settings they use to accomplish the build.

[amigarulez]

No hard float on Arch yet.

[jui-feng]

Maybe this is a hint, starting from line 578.

http://anonscm.debian.org/gitweb/?p=pkg ... .c;hb=HEAD

Setting the MONO_CPU_ARCH environment variable to "armv6" could help. AFAICT from the build log and debian/rules, it is currently not set, so that code will do the /proc/cpuinfo parsing, which results in ARMv7 JIT-compiler optimizations getting enabled when building on the ARMv7 build servers.

Disclaimer: I have never looked at the mono source or the mono debian package before. I also haven't tested this. So it's very possible that setting the environment variable is not enough, or doesn't even help at all.

[mpthompson]

jui-feng, very good find.

Hmmm. Maybe the armv6 and armv7 is a red herring based upon my own prejudices of what might be wrong. It could be a font or some other issue after all.

Can someone send me the definitive steps to reproduce the error? Basically, what specific packages do I need to install and what commands do I need to run (including any files) that will trigger the fault. I would like to run this on my Pi and a build server to see if it reproduces under Debian Wheezy armhf. If not, it will help me identify what the difference between Debian Wheezy armhf and Raspbian armhf might be.

[emg]

packages:
- mono-runtime
- mono-vbnc (if you want VB support)

This will allow you to run any C# or VB .Net exe. Let me know if you need a test executable and I can email you one.

to run: mono <your exe name> from terminal. You should get a stack dump in terminal window on error similar to my thread starter post.

[emg]

Mike,
Here is a small VB application that will generate the error. I use the file transfer feature of UVNC or TightVNC to copy the file to the Pi.

[mpthompson]

Thank you for the test data. I'm able to reproduce the fault on my Pi and in a Raspbian chroot environment on one of our armv7 build servers. Next, I'm going to create an Debian Wheazy armhf chroot environment and see if the issue occurs in that. At least that will tell me it's something specific to Raspbian rather than some more general to Debian armhf.

Something I can tell is if the "Debug info from gdb:" regarding libpthread.so.0 is relevant or not. For example, is the fault in that native library, or is that something just brought in by gdb when presenting the fault information.

[DonOregano]

I do think that it is something more than just a font issue. I got some very strange results when I used mono to call a native library with floating point stuff in the interface.

[amigarulez]

Also think it's more than a font issue. I.e. I can't use the Mono native posix syslog APIs in my Daemons. Also thread mutexes and locks acting wierd (releasing direct or not at all). Perhaps it's some optimization in the JIT kicking in.

[mpthompson]

The plot thickens...

On our Debian Wheezy armhf build servers I created a Debian Wheezy armhf chroot environment so I can test Mono without altering the configuration of the server itself. Also note that this is a distinct port from Debian Wheezy armel (soft float) where Mono does seem to work on the Raspberry Pi. Using the binaries produced by Debian within the this chroot environment, I received the exact same error:

Code: Select all

mike@build00:~/mono$ mono test.exe
Stacktrace:

  at System.Drawing.Font.CreateFont (string,single,System.Drawing.FontStyle,System.Drawing.GraphicsUnit,byte,bool) <0x0013f>
  at System.Drawing.Font..ctor (string,single,System.Drawing.FontStyle,System.Drawing.GraphicsUnit,byte,bool) <0x0007f>
  at System.Drawing.Font..ctor (string,single,string) <0x00057>
  at (wrapper remoting-invoke-with-check) System.Drawing.Font..ctor (string,single,string) <0xffffffff>
  at System.Drawing.SystemFonts.get_DefaultFont () <0x0005b>
  at System.Windows.Forms.Theme..ctor () <0x0002b>
  at System.Windows.Forms.ThemeWin32Classic..ctor () <0x00013>
  at System.Windows.Forms.ThemeEngine..cctor () <0x0008b>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at System.Windows.Forms.SystemInformation.get_MenuAccessKeysUnderlined () <0x00007>
  at System.Windows.Forms.Control..ctor () <0x001e3>
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control..ctor () <0xffffffff>
  at System.Windows.Forms.WindowsFormsSynchronizationContext..cctor () <0x00027>
  at (wrapper runtime-invoke) object.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at System.Windows.Forms.Control..ctor () <0x0004b>
  at System.Windows.Forms.ScrollableControl..ctor () <0x00013>
  at System.Windows.Forms.ContainerControl..ctor () <0x00023>
  at System.Windows.Forms.Form..ctor () <0x00067>
  at test.Form1..ctor () <0x00013>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>
  at System.Reflection.MonoCMethod.Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) <0x001f7>
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) <0x00053>
  at System.Reflection.ConstructorInfo.Invoke (object[]) <0x00057>
  at System.Activator.CreateInstance (System.Type,bool) <0x0018f>
  at System.Activator.CreateInstance (System.Type) <0x00017>
  at System.Activator.CreateInstance<T> () <0x00017>
  at test.My.MyProject/MyForms.Create__Instance__<T> (T) <0x0012b>
  at test.My.MyProject/MyForms.get_Form1 () <0x0001b>
  at test.My.MyApplication.OnCreateMainForm () <0x0001b>
  at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.OnRun () <0x00043>
  at Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase.Run (string[]) <0x00023>
  at test.My.MyApplication.Main (string[]) <0x0003f>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:


Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted
mike@build00:~/mono$

Note I didn't have gdb installed so that portion of the bug messages is missing.

The next steps should be is someone submitting this bug against Debian Wheezy armhf version of Mono and we should be able to get the bug fixed upstream from people who are qualified to debug and fix the issue. I know this doesn't help those who immediately want to use Mono with Raspbian, but it at least can get things on the path to getting fixed.

Also, it's possible this bug has already been reported and a fix may already exist as a set of patches. If so, we can apply the patches against our build of Mono and perhaps get a fix before it appears in upstream Debian Wheezy. We've done this with a few of other packages.

So this is where things are at for now. I hope others in the Raspberry Pi community can take it from here and continue to update this thread until this issue is closed.

[amigarulez]

Yes may be an general Debian hard-float issue, read on the mailing list that the ArmV7 port has issues with libffi variadic functions.

A Google found more mentions of the issue and some Raspbian praise:)
http://www.mail-archive.com/debian-arm@ ... 13722.html

[jui-feng]

jui-feng, very good find.

I'm afraid it's actually not really a good find. I was in a hurry yesterday and a few minutes after posting this, it occured to me that the code is obviously not part of the mono build system, it's part of the JIT compiler. So it is not run on the build servers (only for tests and probably some stuff that is perfectly okay to use ARMv7 for).

When running on the pi, that code will parse /proc/cpuinfo (unless the environment variable is set) and set
v5_supported = TRUE;
v6_supported = TRUE;
thumb_supported = TRUE;

So.. forget about that. Unless the code then uses thumb_supported to assume that thumb2 is supported, it should be fine according to my limited understanding of ARM architectures.

[emg]

Is this relevant?
http://www.mail-archive.com/debian-arm@ ... 13218.html

Submitting bug to Debian is way outside my expertise so I will have to rely on the kindness of others. Is there a way to check if other distro's have solved this issue already?

See here:
https://build.opensuse.org/package/file ... tory%3AARM

BTW, Thanks Mike for taking the time to have a look at this.

[ProjectMayu]

Did anyone else notice the type conversion errors when installing mono, several of which are font related, and others that could cause threading issues.

[emg]

Sorry, Did not look, just saw that it installed... Did you capture it?