Constant crashing [obsolete]

[coldguy]

I can't get my Raspberry Pi to run without crashing for any significant length of time. Under light loads, it seems to do fine, but if I wander off and then come back later, it's often unresponsive. Under moderate loads it crashes pretty quickly. I've been trying to "apt-get build-dep xbmc" tonight and I can't get all the way through it without a crash. It is very frustrating.

When it crashes when the screen is active, I get some sort of endlessly scrolling kernel error on the screen. Unfortunately, it never stops, and it's moving rather faster than I can read with any clarity. I can make out the phrase "may have been dropped." There appears to be some sort of stack trace, but sadly I can't figure out how to get it to sit still long enough to figure out what it is saying.

Any clues on what might be causing this, what a solution might be, or just how to capture the kernel error so that I can ask a more intelligent question? I don't have the hardware to attempt a serial connection. Is this level of instability just the current state of affairs with regard to the Pi? It would seem not, since it appears other people are actually able to get stuff done with theirs. Is it possible that I have a bad board?

I'm running Raspbian, although I'm not sure what's going on in userspace is relevant here. I've tried both the offical (3.1.9+) and Chris Boot's kernel - it appears that I get an identical error with both (at least, as far as I can tell; it's SHAPED the same )

[RaTTuS]

power supply
what have you got and what have you got plugged in

[coldguy]

I've got it hooked up to a 1 amp adapter designed to charge an iPad 3. I think it ought to be sufficient... I did have a USB keyboard and a mini Bluetooth module plugged into it, and I suspected one of them might be the source of the crash, since they're both $5 eBay specials, but I tried unplugging both of those and booting without them and was still able to reproduce the crash more or less immediately.

Is this the kind of problem I might expect to see with insufficient power, or are you just guessing there? Are there examples out there of what that kind of crash looks like that I can try and compare notes with?

[mahjongg]

Very hard to tell what the problem might be from just your description, and yes blaming power is just a "educated guess". But there could be numerous other problems causing such behaviour.
One thing that might be helpful would be having a picture of the screen, but the "may have been dropped" could be a number of things, I'm thinking of dropped ethernet packages, or maybe SD-card problems.

I'm still suspect of your power supply, it may say 1A, but that doesn't necessarily mean it has a stable 5V, charging circuits are a lot less susceptible to input voltage problems than the PI, and certainly your comment that if the CPU works harder (thus consumes more power) the problem increases does point to input voltage drops. Any case where the GPU has to do work should also "make matters worse".

My recommendation would be to give a good hard look at the 5V voltage on board, which can be measured between testpoints TP1 and TP2 (TP2 is ground). It should be between 4.7 and 5.2 Volt DC, also measure the ripple (AC on top of the DC, measured with a voltmeter in AC setting), which should be very small, a tiny fraction of a volt.

Another possible cause would be a bad SD-card.

read the troubleshooting guide on the wiki for more information.

[coldguy]

Well, so far I've been able to reproduce it on either port of this adapter, and off my Kindle charger. I've got a Samsung charger around here somewhere that I'll try once I find it. Hopefully later tonight I'll be able to get someone to bring their authentic OEM iPad charger by so I can try it; surely that ought to have enough grunt to run it, but if it's still happening there, I'll go ahead and meter the points. I'm kind of skeptical that it's a power issue, though; this supply ought to be good, it got glowing reviews on Amazon.

I had the thought that it might be an SD card issue as well, but I've actually already replaced that as well. The old one has worked fine in my phone for years and the new one is a brand-new Sandisk in retail packaging from Newegg, I'd be a little bit shocked if they were both bad, but I can scan for bad blocks on the new one and see what happens.

Any thoughts on how I can capture the error without a serial connection? I feel like it would be a lot easier to debug things intelligently if I could capture exactly what was going wrong. I tried loading the netconsole module, but I don't think the ethernet driver supports the necessary API:

Code: Select all

[  224.589086] netconsole: eth0 doesn't support polling, aborting.

[coldguy]

It just blew up on the same way running off the iPad charger, which is rated at 2.1 amps, 5.1 volts... I guess I'll dig out the meter later just to double check, but given the variety of adapters I've tried it on and since I'm unable to find anyone else having what sounds like a similar problem, I'm increasingly doubtful that this is power-related.

[TimG]

Have you tried booting without the telly connected? Connect only the ethernet and power, and see how long it responds to pings.

[coldguy]

Went back to my original power supply, plugged it directly into the wall, switched USB cables, and it still crashed in the same way. Tried starting it up with no display, and it still crashed in the same way. I dug out my meter and stuck it on the test points and came up with a pretty consistent 4.85-4.86 volts, although it did dip down to 4.8v at one point. That's as low as I saw it go, otherwise no major swings. That didn't seem to coincide with the time it crashed. I've switched my stress test to cloning the xbmc-rbp repo, as apt-get build-dep was able to get enough done incrementally each time to eventually finish. The git clone kills it dead in short order every time.

This smells more like a kernel problem than a hardware defect to me, but I'm surprised more people aren't experiencing something similar. The crash doesn't get written to any logs, unfortunately. If I could just find a way to get it to stop scrolling long enough to read it... I've ordered a USB to 3.3v serial adapter off of eBay, so I can capture the crash, but I'm slightly worried about running out of my return window with element14 before it gets here if it turns out I actually do have a defective board.

It sounds like https://github.com/raspberrypi/linux/issues/52 or possibly https://github.com/raspberrypi/linux/issues/56 might be a manifestation of the same thing?

[coldguy]

I should note that I tried the sysctl changes to net.core. suggested in this thread, and it still crashed in the same way: http://www.raspberrypi.org/phpBB3/viewt ... 66&t=11418

I thought 4.85 volts or so might be just a little too close to marginal, so I dug up the supply that came with my phone. This is rated at 5.0v 700mA. When hooked up to the Pi, the meter reads the board voltage as solidly just above 5 volts (around 5.05.) Sadly, it still crashes in the same way. Can I rule this out as being a power issue yet, or would the amperage of that supply be a concern? I don't have any USB peripherals hooked up now; just HDMI and ethernet. From what I've read I think really want at least 1000mA; I confess that I'm much more of a software than a hardware guy and when it comes to these things I may be a bit dense.

[alexchamberlain]

I assume your kernel is up to date?

[coldguy]

Yup, tried it on both latest "official" and the 3.2 port. Same thing.

[alexchamberlain]

Tried different SD cards?

[coldguy]

Yes...

[alexchamberlain]

Check /var/log/syslog

[coldguy]

As I said, nothing in the logs.

NOTE: Upon further inspection, it looks like I might not even be running a syslog daemon! Might have been missing from the image I used. Lemme see...

[alexchamberlain]

I'm clearly not very good at reading! Install syslog

[coldguy]

Installed rsyslogd. Sadly, still nothing in the logs. I tried taking a picture of the screen but it didn't come out very good. I can only make out a few bits:

Code: Select all

eth0: kevent 2 may have been dropped

...and at the end of a line a few lines above it (which may be the end of the previous looping message):

Code: Select all

delete 114, find 1/2

[RaTTuS]

that looks more like a SD card issue.
now try
logging in
sudo tail -f /var/log/syslog
then wait for it to crash - what do you see?

[jamesh]

Can I confirm if you are suing the latest Raspian/Wheezy release (as of yesterday)?

[coldguy]

I started with one of Hexxeh's images, but I've been keeping up-to-date with apt-get and rpi-update.

[coldguy]

RaTTuS: I see the same endlessly scrolling error in the console, and I see nothing at all on my SSH sessions, because they become unresponsive the moment the crash happens. When I reboot, I've only got boot messages in /var/log/syslog, nothing else.

[jamesh]

I started with one of Hexxeh's images, but I've been keeping up-to-date with apt-get and rpi-update.

Can you try the latest Raspbian and see if the issues still occurs. Ta.

[edg3]

I have been having similar crash situations (though no scrolling text), so I did another full update (distro and rpi-update), if I havent crashed by tomorrow morning will post back here, but if it does crash Im going to look at adding some extra logging of my own for system states to hopefully work towards having a "state" I can give to help reproduce the issue.

Some questions that might make sense to ask (and I will answer for myself too):
Distro? Raspbian
Initial image? Pisces R3
Up-to-date? yes
Things installed before the problem began? midori, gnome-terminal, vim, irssi

I think that I might have a slightly different issue though (can you try running an IRC client and watching if it is still "there" after the crash? i.e. unplug your ethernet cable till it times out and then replug it to reconnect? I apparently stayed on IRC despite my RPi becoming unresponsive some time during the night)

[coldguy]

I'm extremely pleased to report that the problem appears to have been resolved by switching to the Foundation's new Raspbian image. First I tried putting it on a different (from the previous two) SD card, and I was able to clone the XBMC repo into two different directories on two different terminals at the same time without crashing, which has previously been my surefire way to kill it. I then reimaged the card that was in there previously, and I was able to repeat the feat, again without crashing! So, so far so good I'll have the serial cable eventually so I can capture any future errors of this sort.

@edg3: When it died, the network died completely. My ssh sessions would just hang until I killed them, or until I restarted the box and got connection resets. I wouldn't be surprised if such a situation would leave a ghost on IRC.

[edg3]

My problem was also fixed.

@coldguy: I had the same problem with my ssh connection dying, but I would still be on IRC. I just figured if we get the same ghosting behaviour the problem would be much similiar.