pacman -Syu - operation too slow

[battletux]

So,

Fresh install of Arch and on running pacman -Syu to update it I keep getting the following issue:

Code: Select all

[root@alarmpi ~]# pacman -Syu
:: Synchronizing package databases...
 core is up to date
error: failed retrieving file 'extra.db' from eu.mirror.archlinuxarm.org : Operation too slow. Less than 1024 bytes/sec transferred the last 10 seconds
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from eu.mirror.archlinuxarm.org : Operation too slow. Less than 1024 bytes/sec transferred the last 10 seconds
error: failed to update community (download library error)
 alarm is up to date
 aur is up to date
:: Starting full system upgrade...
 there is nothing to do
[root@alarmpi ~]# 

I've added "ipv6.disable=1" to /boot/cmdline.txt to disable ipv6 and it's no good, I'm still getting the issue. Both repos get to 96% before the issue happens.

Are there any other things I can try to resolve this? i have no USB devices connected when trying to perform this as I'm doing it over SSH. I have also connected a kb to the pi and tried to update still with no success.

And no, my PSU is not the issue. I've used this one many times with various distros and usb devices on my 2 pi and never had an issue before now.

[flyinghappy]

I run into this from time to time, usually all I have to do is just run it again and it catches.

[pepedog]

Might be the geo located server is too slow, manually pick one

Code: Select all

nano /etc/pacman.d/mirrorlist
Comment out Server = http://mirror.archlinuxarm.org/armv6h/$repo and uncomment one of the others
fr is fast for me

[battletux]

It is odd. I've tried a number of times through the day and I've also tried different servers. It seems my pi just does not like those repos. And it is only those listed in the 1st post that it has an issue with.

[battletux]

I've managed to mitigate the issue slightly, by manually using wget to download the repo db files and then over writing the existing entries. After running pacman -Sy it all seems to go ok and I can now install programs from these repos without it erroring.

[finalduty]

I came across this myself and came up with the following little script:
http://chiron.net.nz/useful-links/code- ... estrepo.sh

Code: Select all

mirrors=$(cat /etc/pacman.d/mirrorlist | grep "Server =" | sed 's/^.*\=//')

for i in ${mirrors[@]}; do
echo -n ">" && echo -n $i >> .massping.tmp
ping -fc5 $(echo $i | awk -F '/' '{print $3}') | tail -1| awk -F '/' '{print " " $5 " ms"}' >> .massping.tmp
done

echo && sort -k2 .massping.tmp | grep -v "repo ms"

rm .massping.tmp

Essentially what it does is ask the /etc/pacman.d/mirrorlist file for the lines it has for "Server =" and tidied them up to plain urls. This is then added into a tmp file and the urls are stripped further and pinged 5 times to return an average speed again being written into the .tmp file. After that the file is sorted and output with the fastest one up top.

I then manually picked the fastest one and enabled that. Currently looking at doing this programmatically for my provisioning script.

[flyinghappy]

Very nice script. Would like to see it again if you get the part about choosing the mirror automated

[sdjf]

Arch also has a page with mirrow status, and ranks them according to performance:

https://www.archlinux.org/mirrors/status/

[finalduty]

Very nice script. Would like to see it again if you get the part about choosing the mirror automated

Today's your lucky day It took me a little while of fiddling with this to get it just the way I wanted, but here is the result. I've commented it all for you, including the part at the bottom where it actually writes the change to the file. I've added a disclaimer because noobs, but I've tested it myself and didn't run into any issues.

As always, make sure you take a backup.

Code: Select all

cp /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.bak

If you want to remove the comments from the script and strip it down, use this sed command

Code: Select all

sed '/^## /d' fastestmirror.sh && sed '/^$/d' fastestmirror.sh

And finally, the script itself:

Code: Select all

#####
## fastestmirror.sh
## finalduty 19/12/12
##
## This script is used to test for the fastest ping to ALARMPI mirrors.
## You can also have it automatically write to your mirrorlist
#####

#!/bin/bash

## Creates .tmp file if it doesn't exist and writes a blank line into it. (Useful if the script is cancelled before the rm function)
touch .massping.tmp
echo -n > .massping.tmp

## Collect the mirrors from /etc/pacman.d/mirrorlist, strip them and enter them into the $mirrors array
mirrors=$(cat /etc/pacman.d/mirrorlist | grep "Server =" | sed 's/^.*\=//')

## Loop through the array values and query them and ping them.
for i in ${mirrors[@]}; do

        ## Show > symbol to indicate progress and write the URL to temporary file without a newline
        echo -n ">" && echo -n $i >> .massping.tmp

        ## Ping the actual server, retrieve the average from the last line, strip it and write to the temporary file
        ping -fc5 $(echo $i | awk -F '/' '{print $3}') | tail -1| awk -F '/' '{print " " $5 " ms"}' >> .massping.tmp
done

## Set variable $fastest by sorting the temp file based off numerical pings, removes blanks/timeouts and returns the top value.
fastest=$(sort -nk2 .massping.tmp | grep -v "repo  ms" | head -1)

## Prints to screen the fastest average ping in green and clears color formatting, ready for next output
echo && echo -e "\e[0;32m"$fastest && tput sgr0

## Sorts the temporary file and outputs all but the fastest result (because it has already been printed in colour)
sort -nk2 .massping.tmp | grep -v "repo  ms" | tail -n+2

## Tidy up - Remove Temporary File
rm .massping.tmp

## Strip the ping time from the $fastest variable so we can search for it in the mirrors list
fast=$(echo $fastest | sed 's/ .*$//')


# WARNING - UNCOMMENTING BELOW THIS LINE WILL CAUSE CHANGES TO BE WRITTEN TO YOUR MIRRORLIST
# PLEASE MAKE SURE THAT YOU UNDERSTAND THE SCRIPT BEFORE ALLOWING THIS.
# USE AT YOUR OWN RISK AND MAKE SURE YOU TAKE A BACKUP. I WILL NOT CONSOLE YOU WHEN YOU CRY

## Comments all uncommented "Server =" entries in /etc/pacman.d/mirrorlist
#sed -i 's/^.*Server/\# Server/' /etc/pacman.d/mirrorlist

## Uncomments only the fastest server in /etc/pacman.d/mirrorlist
#sed -i "s|^# Server = $fast|Server = $fast|" /etc/pacman.d/mirrorlist && echo && echo "Server "$fast" has been set as the default mirror"

Please let me know your thoughts

EDIT: I forgot to put in a sample output, this is what it gives me (down here on the bottom of the world).

Code: Select all

[root@alarmpi ~]# ./fastestmirror.sh
>>>>>>>>>>>
http://ca.us.mirror.archlinuxarm.org/armv6h/$repo 161.534 ms
http://mirror.archlinuxarm.org/armv6h/$repo 162.066 ms
http://va.us.mirror.archlinuxarm.org/armv6h/$repo 222.342 ms
http://us.mirror.archlinuxarm.org/armv6h/$repo 257.483 ms
http://nl.mirror.archlinuxarm.org/armv6h/$repo 316.731 ms
http://eu.mirror.archlinuxarm.org/armv6h/$repo 320.869 ms
http://de.mirror.archlinuxarm.org/armv6h/$repo 328.618 ms
http://fr.mirror.archlinuxarm.org/armv6h/$repo 347.176 ms
http://cn.mirror.archlinuxarm.org/armv6h/$repo 364.516 ms
http://fi.mirror.archlinuxarm.org/armv6h/$repo 370.989 ms

Server http://ca.us.mirror.archlinuxarm.org/armv6h/$repo has been set as the default mirror

[finalduty]

I can no longer edit my last post, but the following section:

If you want to remove the comments from the script and strip it down, use this sed command

Code: Select all

sed '/^## /d' fastestmirror.sh && sed '/^$/d' fastestmirror.sh

Should read:

Code: Select all

sed '/##/d' fastestmirror.sh | sed '/^$/d

[maksverver]

I experienced this too, and I believe this is not caused by poor network connectivity, but rather by stalled writes to the SD card.

(Warning: long post incoming. Solutions at the bottom.)

Pacman uses cURL to download packages, and sets a lower bound on the speed limit. This feature is described here: https://curl.haxx.se/libcurl/c/CURLOPT_ ... LIMIT.html. cURL generates the "operation too slow" error which causes the download to fail if the download speed falls below the speed limit.

So how is this related to SD card writes? Many SD cards buffer writes, so that writing small amounts of data is very fast: the buffered data will be persisted in the background. Large sustained writes on the other hand, are slow. Periodically, the Linux filesystem driver will try to flush the write buffer. When this happens, all writes are stalled for several seconds, up to a few minutes!

This is exactly what happens when downloading multiple packages. When cURL downloads files, it needs to write them to disk. When enforcing the lower speed limit, cURL doesn't distinguish between reading from the network and writing to disk. It just notices that the download has stalled and it errors out. This is stupid, of course: the network is perfectly fine, and the only reasonable solution is just to wait for the SD card to recover.

The reason that retrying works is that most packages are not too large, so usually you can download a few packages before buffers are full and pacman fails. Then you wait for a while and rerun, and pacman can download a few more packages, etc. until you have them all.

Now for the solution:
If you know your network connection and mirror are fine, but you're using a cheap SD card that has the above problem (and if you're running Arch Linx on a Raspberry Pi, that's likely to be the case), then the solution is just to disable the low-speed limit. There are two ways to do it:

1. Add a line DisableDownloadTimeout to /etc/pacman.conf (perhaps under "Misc options")
2. Run Pacman with the --disable-download-timeout option (e.g. pacman -Syu --disable-download-timeout).