Kernel paging error during heavy network and disk load

[crozone]

Hi, I'm running into some issues while using rtorrent on my raspberrypi.

I have an external, self powered 300gb NTFS drive mounted using ntfs-3g to /mnt/sda1. I am also running rtorrent in a background screen session using the recommended init.d startup script. rtorrent is configured to use ~/rtorrent/watch and ~/rtorrent/session as it's watch and session directories (which are located on the sd card), and /mnt/sda1/rtorrent/downloads as the primary download directory.

After 30 mins or so of downloading a single torrent at ~1MB/s,
the Raspi crashes and I get the message:

Unable to handle kernel paging request at virtual address 8e78b8a2
pgd =cb9e4000
*pgd=00000000
Internal error: Oops: 805 [#1] PREEMPT
Entering kdb (current0xca4ec520, pid 1313 Oops: (null) due to oops @ 0xc024d3d4

along with a whole bunch of other data on registers etc.

I am using a 4gb class 4 SD card with the paging file extended to fill. It does not matter whether I use the 192/64 or 224/32 memory split, it only buys a little more time before the error occurs. CPU usage stays around 80-100% and free memory jumps around 10-50mb the entire time rtorrent is running. ntfs-3g and rtorrent are using all the cpu and memory.

From the appearance of the OK status led, the SD card is getting accessed in short pulses and fairly frequently, but its not getting "hammered" in any way to suggest heavy swap use. Swap Free stays around 92MB, so the RaspberryPi does not appear to be running out of swap.

I'm running the Raspian (Debian wheezy 2012-07-15) image and have my raspberrypi powered using the apple ipad charger, and the voltages seem to stay at 4.9V stable, so I don't think it's a power issue. either. I'm running stock clocks and the cpu and usb chips are warm but not overly hot to the touch.

At this stage the only thing I can think of is that swap read/write requests are somehow timing out (is this even possible?), but other than that I have no idea. The terminal (local, not ssh) does seem very sluggish at times and input does get delayed a bit. I've googled for similar problems but come up with very little. Are there any log files I could check to try and find out more about why this occurred? Any help would be greatly appreciated.

[dom]

Do you back a backtrace? (a list of function names where the crash occurred). Can you capture that (a photo of screen is one way).

[crozone]

Currently I can only post an image of the screen that is given when rtorrent crashes, (the rest of the error stated above), and the amount of memory/cpu used at the time of the crash (via an ssh top session), although that probably won't be much help. I'm trying to compile rtorrent with debug symbols so that I can try to reproduce the error while using gdb but I'm having some real trouble getting it to build.

I did notice that it was using over 150MB of virtual mem the last time it crashed, although there was 70MB free swap left...not sure if that helps at all. Sometimes I get another error where screen reports that it can't find a process to attach to, and rtorrent exits gracefully.

However, when the kernel paging crash does occur, it's exactly the same every time,

[fridgemagnet]

IMG_20120802_225134.jpg (51.89 KiB) Viewed 14458 times

I may be seeing the same thing. Been playing with the MPD music daemon. If I request it build the database, from files over an NFS (wireless) link, the CPU gets busy for a bit, plenty of network traffic and presumably periodic SD accesses as it writes the database then falls over with a paging fault. See attached for a nice screenshot.
This is with the Raspbian OS build.

[javatmn]

I may have the same crash issue. I am using ml-donkey to download large files, after downloading about 500M file content, then the PI crashes. It does happened 3 times till now. I suspect the kernel might be not stable enough. Do you have same issue with old debian squeeze version?

[Goldy]

I've got the same problem.

IIn the past, i was fixed this problem by adding vm.min_free_kbytes = 8192 in /etc/sysctl.conf.

The problem was gone, but I don't know why I did that, I updated the kernel with rpi-update, and now it crashes when the load is increased to 5 or more.

Is there any way to backtrace the crash when the raspberry is used without screen?

[ubergrog]

I am experiencing this exact same issue. I'm running transmission-daemon & sharing a drive via samba & have experienced this error several times now. I'm using the 2012-07-15 image which already has the "vm.min_free_kbytes = 8192" in /etc/sysctl.conf.

Any help would be greatly appreciated. TX

[dom]

@ubergrog
I'd try doubling the 8192.

[ubergrog]

Running with "vm.min_free_kbytes = 16384" for an uptime of over 13 hours so far, no issues.

Is there anything I can be tracking this? What information might be useful leading up to this issue if it happens again?

Thanks dom.

[secretagent]

I can post instructions on how to run Kdump on the Pi, which would let users experiencing kernel crashes, such as this, to send complete memory dumps of the crashed kernel for developers to look at. However there is one caveat: this requires a bit of RAM to be reserved for the crashkernel, so that RAM will not be available during normal operation. I can check to see how small can that be made while still working. Also, for this to be useful the original kernel will need to be compiled with debugging symbols and the vmlinux file will need to be made available, so that the memory dumps can actually be used. Dom, any thoughts on this? Or has this been considered already?

[fridgemagnet]

Running with vm.min_free_kbytes = 16384 has made no difference to the problem I'm seeing. I have another SD card with the old Debian Squeeze image on it but I'm still getting things installed (ie. the wifi adaptor) on it to be able to report back on that.

[fridgemagnet]

Quick update - I can confirm a similar thing happens with the Debian/Squeeze build however I'm now fairly sure my problem is related to power. Up till now I have been using a USB WiFi dongle plugged straight into the Pi. Just tried it with the onboard Ethernet and I've not seen the crash, it's been running a good deal longer but I am getting repeated characters from the keyboard which I've read elsewhere is a sign of power issues. Odd because I'm using a Nokia PSU rated at 1.2A but looks like I may need to investigate powered hub and maybe another PSU.

[elatllat]

...

Repeated characters is a software bug not a power issue.

related: http://www.raspberrypi.org/phpBB3/viewt ... 24&t=14394

[dom]

Repeated characters is a software bug not a power issue.

No, it can be either. Repeating characters are particularly common when plugging in >100mA USB devices, which causes the keyboard to get significantly less than 5V.

[uberscubajim]

I too have been running into this problem (see posting at http://www.raspberrypi.org/phpBB3/viewt ... 71&t=15207)

[ubergrog]

FWIW. I have had one freeze during the previous week, along with the running transmission-daemon I had two terminals open, both copying files from one usb drive to another when it happened. I also share the drives via samba but wasn't accessing them that way at the time.

To clarify I am powering the Pi via a 1A usb adapter & the two usb-powered drives I am currently using are on a self-powered usb hub, so I wouldn't think it's a power issue. I did try doubling the vm.min_free_kbytes value again to 32768 with little effect, I suspect it's a case of diminishing return. I've learned to just limit the number of concurrent tasks.

[crozone]

This issue appears to be caused by the USB drivers having trouble with the built in hub, there's a major discussion about it over here: http://www.raspberrypi.org/phpBB3/viewt ... p?p=153956

I eventually fixed the issue by switching over to Chris Boot's custom armel kernel, which can be found here http://www.bootc.net/projects/raspberry-pi-kernel/

I've been running transmission-daemon and an apache server without any issues. However, it seems to break omxplayer, probably due to the fact that it's only soft float, rather than the official armhf hard float build.

[Goldy]

I stopped to use the usb hdd and the raspberry is now very stable. The problem seems to be coming from the usb driver.

[uberscubajim]

There do seem to be some common themes occurring.

Firstly, a lot of us are running bittorrent clients when the machine fails and secondly we seem to be using a Wi-Fi based USB dongle.

There has been a lot of talk about issues with the USB interface. One of the things that occurs when downloading using a bittorrent client via Wi-Fi is that you can receive a lot of data over the USB with no real flow control. Increasing the minfree gives the kernel some extra space to work with but at times it isn't enough.

Or perhaps there's just an issue with the USB drivers that has yet to be fixed (or worked around).....

[elatllat]

Repeated characters is a software bug not a power issue.

No, it can be either. Repeating characters are particularly common when plugging in >100mA USB devices, which causes the keyboard to get significantly less than 5V.

All I know is I get repeated characters using either of my 1A power supplies and no USB plugged in (keyboard over ethernet) and no other power related problems. The problem is consistent regardless of CUP/GPU/NIC usage.

[ubergrog]

To clarify I am powering the Pi via a 1A usb adapter & the two usb-powered drives I am currently using are on a self-powered usb hub, so I wouldn't think it's a power issue. I did try doubling the vm.min_free_kbytes value again to 32768 with little effect, I suspect it's a case of diminishing return. I've learned to just limit the number of concurrent tasks.

I've updated to the latest firmware using "apt-get dist-upgrade && rpi-update" so am now running the 3.2.27 kernel & have reverted back to "vm.min_free_kbytes = 8192". So far so good, usb hard drive access is much more stable.

[uberscubajim]

I can confirm that a quick kernel upgrade has indeed solved my problem too.

Looking good guys!

[fridgemagnet]

Just to really add/confirm what is probably already known - swapping over power supplies, hubs etc. did not yield anything positive on the problem I was seeing. However I finally got round last week to upgrading the kernel (via rpi-update) and things are much better now. The original fault I reported has gone as have the other oddities relating to network drop outs and usb stability.

Cheers all.