User avatar
micksulley
Posts: 210
Joined: Sat Mar 03, 2012 11:48 am
Location: Melton Mowbray, England

ssh connection problem

Wed Nov 30, 2022 2:24 pm

I have a Pi4 which holds my family tree and runs headless. It runs from a hard drive (no SD card) and I can connect via ethernet or wi-fi. It is up to date.
When I set it up it ran for a few months and then failed, when I tried to ssh into it I got "kex_exchange_identification: read: Connection reset by peer". I power cycled it and it ran fine again. This has now happened again and again power cycle fixes it. When it fails I get the same error on ethernet and wi-fi.
What could cause it?

User avatar
FTrevorGowen
Forum Moderator
Forum Moderator
Posts: 7078
Joined: Mon Mar 04, 2013 6:12 pm
Location: Bristol, U.K.

Re: ssh connection problem

Wed Nov 30, 2022 2:38 pm

micksulley wrote:
Wed Nov 30, 2022 2:24 pm
I have a Pi4 which holds my family tree and runs headless. It runs from a hard drive (no SD card) and I can connect via ethernet or wi-fi. It is up to date.
When I set it up it ran for a few months and then failed, when I tried to ssh into it I got "kex_exchange_identification: read: Connection reset by peer". I power cycled it and it ran fine again. This has now happened again and again power cycle fixes it. When it fails I get the same error on ethernet and wi-fi.
What could cause it?
Sorry, can't comment on your SSH issue but, OOC, what "family history software/tools" are you using? (I have Gramps and a geneweb database running on one of my P4's . I had to import it from an old Windows-based Family History program's "export".)
Trev.
Still running Raspbian Jessie or Stretch on some older Pi's (an A, B1, 2xB2, B+, P2B, 3xP0, P0W, 2xP3A+, P3B, B+, and a A+) but Buster on the P3B+, P4B's & P400. See: https://www.cpmspectrepi.uk/raspberry_pi/raspiidx.htm

MiscBits
Posts: 1285
Joined: Wed Jan 27, 2021 12:48 pm

Re: ssh connection problem

Wed Nov 30, 2022 3:13 pm

The error infers SSH has crashed or is being blocked - I've seen this after the switch or router has been reset due to a reset (possibly a power issue).

I find that two connections to the same network can give problems and SSH over WiFi can be picky now and then...

Check the logs to see if there is anything for SSH in them - it could have happened at anytime though and the log may have been archived but I would start with:

Code: Select all

grep "ssh" /var/log/syslog
less /var/log/auth.log
Running your SSH command with debugging on can possibly help identify the cause of the error if its not in the logs - on a Mac

Code: Select all

ssh -v pi@ipaddress
will give you more connection information and

Code: Select all

ssh -vv
ssh -vvv
will give you even more (someone had an off day when they though 'vee vee' was OK and not get mixed up with 'double you'...).

Be aware just pulling the power on a Pi can lead to SD Card / disk corruption so make sure your backups are solid and up to date - you may want to look at adding a button on the GPIO to safely power the Pi down...

swampdog
Posts: 972
Joined: Fri Dec 04, 2015 11:22 am

Re: ssh connection problem

Wed Nov 30, 2022 3:17 pm

If something causes the filesystem to go readonly this will happen. Typically a failing sdcard but hard drives can fail also. Check its connections and 'fsck' it.

Assuming your filesystem is ok..

Code: Select all

$ sudo cat /var/log/syslog
..for hints. Other main reason 'sshd' fails like that, is the network has vanished so look for dhcp stuff as well as ssh in there.

There are a ton of possible causes though. 'sshd' is robust so it not working tends to be a sign of another prior issue corrupting it.

User avatar
micksulley
Posts: 210
Joined: Sat Mar 03, 2012 11:48 am
Location: Melton Mowbray, England

Re: ssh connection problem

Wed Nov 30, 2022 8:59 pm

Thanks for the replies

Trev - I use geneweb, been using it for many years, better functionality than anything else I have seen, but can be tricky to install.

MiscBits -
I looked in syslog and it's archives, I found a few instances like -
Nov 26 10:17:12 pi-gene-eth systemd[1]: Condition check resulted in Turn on SSH if /boot/ssh is present being skipped.
Nov 26 10:17:21 pi-gene-eth systemd[594]: Listening on GnuPG cryptographic agent (ssh-agent emulation).

I don't think these are errors, but I may be wrong :)

auth.log doesn't have anything that looks relevant.

I am connecting from my desktop, which runs Linux Mint. Connecting with -v (and -vv -vvv) does give a load of info but nothing that looks like an error

I agree with your point on pulling the power, not something I would normally do but when it is running headless (and having not added a button) there was no other option. I do have backups however.

swampdog

Checked for dhcp references in syslog I found a couple of instances of this message -
Nov 21 16:09:43 pi-gene-eth kernel: [169017.823920] ICMPv6: process `dhcpcd' is using deprecated sysctl (syscall) net.ipv6.neigh.wlan0.retrans_time - use net.ipv6.neigh.wlan0.retrans_time_ms instead

But I don't know what to do about it.

Thanks
Mick

swampdog
Posts: 972
Joined: Fri Dec 04, 2015 11:22 am

Re: ssh connection problem

Thu Dec 01, 2022 12:59 am

Those messages appear harmless. Have a look in the relevant syslog for the info last time you had to hit the power. Find where it is booting due to you hitting reset and look backward what was logged prior to that. Don't be surprised to find nothing because sadly if the kernel panicked (or went read-only) it won't have been able to update syslog/kern log etc and you may only have timestamps on those files to work with - ie: next time, remove the disk and attach it to your Mint, "mount -r" and "ls -ltr" on the rpi's "/var/log" to see last time it wrote anything. Whilst there, "umount" it and have Mint run an 'fsck'.

The smartmon tool typically has difficulty working through usb so if you can, physically attach the rpi disk to the Mint box via a sata cable and run 'smartctl' on it to see if it's failing (sudo smartctl --xall /dev/blah). Then force a test with "smartctl -t short /dev/blah". That won't take long but "smartctl -t long /dev/blah" will take considerably longer (the estimate it spits out when you invoke it are fairly accurate) and iirc: "smartctl -all /dev/blah" will indicate when complete. The "-xall" spits out all attributes including ones smartmon doesn't understand for that disk. If you google the precise hard disk model you can often discover the meaning of some of the unknowns. For instance, my crucial ssd's have an unknown option which turns out to be the ssd wear status. Spinning rust disks are typically obvious by looking at the "pending sector" value - ideally zero.

If you can leave a monitor attached or leave a connection over a serial interface running constantly until it happens again, you will be able to see what couldn't be logged. The kernel should dump something to the console. I did once have issues with leaving the rpi gui running. Eventually it would lock up - disabling the screensaver extended that to weeks. Having it boot to command line made the problem go away completely.

If there's nothing there on the console then start to suspect power glitches.

User avatar
micksulley
Posts: 210
Joined: Sat Mar 03, 2012 11:48 am
Location: Melton Mowbray, England

Re: ssh connection problem

Thu Dec 01, 2022 9:36 am

swampdog
many thanks for your advice, it is currently running fine , so I guess I just have to wait until it fails again and investigate then. I will feed back if/when I find anything.
Thanks
Mick

Return to “Troubleshooting”