rsync files/folders from a source Pi to a destination Pi

[RDK]

I'm running a Postfix/Dovecot e-mail server (EM_PiA) on a Raspberry Pi 4B using Buster. It is working fine

Now we want to setup a parallel Pi (same basic image) and want to use rsync to transfer critical files to this Pi (EM_PiB) using a CRON job on a regular schedule (daily, hourly, ...). I have read through this posting (viewtopic.php?t=255425) which has thoroughly confused me to the point where I really don't know how to proceed. The "without password" criteria in that posting seemed to be integral to the complexity? And maybe necessary?

Anyway, today EM_PiA (source) and EM_PiB (destination) are essentially the same with only the hostname changed. Both are on the same internal network, in fact side-by-side, with SSH access and configured with SAMBA so we can setup shares if necessary.

We want to setup a CRON job to copy a folder, for example /var/vmail, from EM_PiA to EM_PiB, removing destination files/folders which are no longer on the source (EM_PiA).

We assumed that rsync would provide the tools to do this operation, but as I worked my way through this posting (viewtopic.php?t=255425) it seemed to be VERY complicated and left me with more questions than answers.

The goal here is to have a "backup" server up-to-date, ready and configured if the primary server has a serious failure. Can someone assist? Thanks...RDK

[rpdom]

The basics are quite simple. The rsync can be run from either end. Let's call them server A and server B to make it simple.

Firstly, you probably don't want to do things as root, so has the user that owns the files on server A got passwordless ssh access to server B?

I.e. if you are logged in to server A as "user", can you then "ssh B" without having to enter a password? If so, you're half way there.

To sync /home/user/data/ from A to B, running on A:

Code: Select all

user@a:~% rsync --archive --delete-after /home/user/data/ B:/home/user/data/

The --delete-after option is to get rid of files on B that have been deleted from A, otherwise these files will get left on B. The "-after" part means the delete happens at the end of the run. There is also --delete-before and --delete-during.

You can add a --verbose and --progress to see more output if you want.

I suggest you try it across some temporary locations first, just to get the hang of it. Also, don't forget to write the output of the cron to a log file so you can check it.

[RDK]

@rpdom WOW, a WHOLE lot simpler than the other reference I was struggling with!!! I'll give it try this afternoon.

I suspect /var/vmail will have some special permissions, but first I will experiment on those folders which seem simple. Many thanks...RDK

[RDK]

First off, thanks for your help.

OK, I guess I've hit the first issue, I can not "ssh B" from server A without getting asked for the password. Both boxes have the same password for "user". Is there a configuration folder which needs to be modified? Is it an issue if I'm accessing server A initially via SSH(PUTTY) using SSH Private Key for logon authentication?

And then your other requirement was that the "user" has to own the file/folder being transferred. Well, that is an issue as files from the "/etc/" folder seem to be owned by "root" and those for the "/var/vmail" folder which are owned by the user "vmail".

In the case of the /etc folder the transfers, ie /etc/nginx, they work but the owner is changed from root to user. For the files in /var/vmail the transfers are rejected due to permission errors.

Perhaps this is why the other posting was so complicated? Thanks again for the ideas, etc. Any ideas or next steps?....RDK

[rpdom]

Right, unfortunately that means it starts to get a bit complicated.

Running it as root (sudo rsync) would solve the permissions, but that creates security holes as root on system A would need passwordless ssh access to system B which isn't recommended.

Setting it so user vmail can ssh from A to B would allow /var/mail to be copied over, but that doesn't help with /etc/ files. Do those files get updated much? Could you manually copy them when they do? You probably don't want the whole of /etc copied over. The hostanem should be different for a start.

An alternative is to run an rsync server on B, set up a restricted configuration that allows access to just the files required and tell A to use a direct rsync connection instead of the default ssh.
It's been a while since I've used that method so I'll have to go have a think about it and probably run a test or two in the morning.

[bls]

An alternative is to run an rsync server on B, set up a restricted configuration that allows access to just the files required and tell A to use a direct rsync connection instead of the default ssh.
It's been a while since I've used that method so I'll have to go have a think about it and probably run a test or two in the morning.

You could save a bit of think time and start with this: viewtopic.php?p=1558100#p1558100

[thagrol]

Gonna throw an alternative but slightly more complex solution into the mix: rather than use rsync's network mode use NFS on Pi B to export the required directorie(s) with the no_root_squash option, mount the export onto Pi A then use sudo rsync -axrvH source destination.

a is archive mode. Amoung other things it preserves owner, group, and permissions.
x don't cross filesystem boundaries (so you don't recursively try to sync the destination to itself and so you don't try to sync virtual filesystems like /proc, /sys, /dev, etc)
r recursive
v verbose
H preserve hard links

An example configuration:

• On Pi B:
  1. Install nfs_kernel_server on Pi B
  2. Add something like the following to /etc/exports on Pi B:

     Code: Select all

     /srv/backups PiA(rw,sync,no_subtree_check,no_root_squash)

     "PiA" restricts access to clients whose hostname is "PiA".
  3. Run the following on PiB:

     Code: Select all

     sudo exportfs -r

• On Pi A:
  1. Mount the export from Pi B e.g.

     Code: Select all

     sudo mount -t nfs PiB:/srv/backups /mnt

  2. Run you rsync:

     Code: Select all

     sudo rsync -axrvH / /mnt

  3. Optionally unmount /mnt

I've used the above method many times to transition a PI from SD card boot to network boot.

That's not a perfect solution but rsync won't be prompting for passwords anymore and you won't need a user with root/sudo rights on PiB.

Two major drawbacks:

1. Any user with root/sudo privilleges on PiA will be able to mount the export and will have root access to files/directories within it.
2. Numeric user and groups IDs must be the same for a given user or group on both machines. Users/groups that exist only on one machine must have a numeric ID that is unused on the other one. If that isn't the case owner, group, and permissions will be wrong when viewd from one of the two machines. That said, this could also happen when using the network mode of rsync.

[RDK]

@bls Interesting as I started from that reference and got totally confiused, which I why I created this posting. I am studying it again...RDK

@thagrol OK, I'll study this technique. To be clear I want to do this as a regularly scheduled job (CRON ??) and I want to transfer files/folders from PiA to PiB.

Is this effectively setting up a share on PiB?

On Pi B:
Install nfs_kernel_server on Pi B
Add something like the following to /etc/exports on Pi B:

/srv/backups PiA(rw,sync,no_subtree_check,no_root_squash)

OK, more studying....Thanks...RDK

[thagrol]

@thagrol OK, I'll study this technique. To be clear I want to do this as a regularly scheduled job (CRON ??) and I want to transfer files/folders from PiA to PiB.

Is this effectively setting up a share on PiB?

More or less. But as you're copying system files you should use NFS and a Linux filesystem on PiB. If you use Samba/smb/cifs and a non Linux filesystem you'll lose all the metadata (owner, group, permissions, etc.).

[bls]

@bls Interesting as I started from that reference and got totally confiused, which I why I created this posting. I am studying it again...RDK

If there's something confusing, ask about it. Happy to help.

I will say, however, that thagrol's suggestion of using NFS is a good approach as well. NFS between Linux systems is super-nice, since it is a natural extension of the file system.

Either way will work of course. Use the one you're most comfortable in working with, and take good notes for next time

[mac587]

I wanted a similar solution for syncing PiHole on a primary and secondary raspberry pi.

1. I found this blog post: https://jejje.net/2021-01-30-sync-two-p ... r-failover
2. And the supporting GitHub Page: https://github.com/jejje/pihole-rsync

It is a simple bash script. You define a number of variables up front - Source, Destination, Files to sync and CRON frequency. I think it is easy enough to understand, and mod if for the files you want replicated. It is not specific to PiHole except for the files that get synchronized. I tweaked it to back up local HTML content.

It uses sshpass to handle the prompting for your password when trying to ssh access. If this is to be run locally at home, this is a nice solution (Not for anything that is exposed to internet access!).

Good Luck.