System restoration with REAR

 Boot from recovery iso on pihole

if replacing a drive, smart array will detect a change and disable logical drive and boot will hang at smart array initialisation.  

Make sure you reboot with text mode and select F2 to reenable drive array

Log in using standard root password over network

Recover einstein option

Should see 3 drives sda/sdb/sdc

mount the backup partition - it will be the one with the single partition

Device     Start        End    Sectors  Size Type

/dev/sdc1   2048 6251167743 6251165696  2.9T Linux filesystem

RESCUE einstein:~ # mkdir /backup

RESCUE einstein:~ # mount /dev/sdc1 /backup

Of the other 2, run pvs to determine which one is not the volume with all the guest lvms on it

RESCUE einstein:~ # pvs

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda2  rhel_einstein lvm2 a--  <50.00g <25.75g

  /dev/sdb   vg_guests     lvm2 a--    2.91t   1.86t

In this case it's sda that is the OS disk so ensure rear uses that

RESCUE einstein:~ # rear recover

Relax-and-Recover 2.6 / 2020-06-17

Running rear recover (PID 814)

Using log file: /var/log/rear/rear-einstein.log

Running workflow recover within the ReaR rescue/recovery system

Using backup archive '/backup/host/einstein/backup.tar.gz'

Will do driver migration (recreating initramfs/initrd)

Calculating backup archive size

Backup archive size is 4.6G     /backup/host/einstein/backup.tar.gz (compressed)

Comparing disks

Ambiguous possible target disks need manual configuration (more than one with same size found)

Switching to manual disk layout configuration

Using /dev/sdb (same name and same size 3200598237184) for recreating /dev/sdb

Current disk mapping table (source => target):

  /dev/sdb => /dev/sdb

Confirm or edit the disk mapping

1) Confirm disk mapping and continue 'rear recover'

2) Confirm identical disk mapping and proceed without manual configuration

3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)

4) Use Relax-and-Recover shell and return back to here

5) Abort 'rear recover'

It's picked the wrong one - so change with option 3

Current disk mapping table (source => target):

  /dev/sdb => /dev/sda

Confirm or edit the disk mapping

1) Confirm disk mapping and continue 'rear recover'

2) n/a

3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)

4) Use Relax-and-Recover shell and return back to here

5) Abort 'rear recover'

It will probably crash out

User confirmed disk mapping

Failed to apply layout mappings to /var/lib/rear/layout/config/df.txt for /dev/sda (probably no mapping for /dev/sda in /var/lib/rear/layout/disk_mappings)

Failed to apply disk layout mappings to /var/lib/rear/layout/config/df.txt

ERROR: Failed to apply disk layout mappings

Some latest log messages since the last called script 320_apply_mappings.sh:

  2026-09-27 16:22:17.927699482 Including layout/prepare/default/320_apply_mappings.sh

  2026-09-27 16:22:18.003792169 Failed to apply layout mappings to /var/lib/rear/layout/config/df.txt for /dev/sda (probably no mapping for /dev/sda in /var/lib/rear/layout/disk_mappings)

  2026-09-27 16:22:18.006684154 Failed to apply disk layout mappings to /var/lib/rear/layout/config/df.txt

Aborting due to an error, check /var/log/rear/rear-einstein.log for details

Exiting rear recover (PID 814) and its descendant processes ...

Running exit tasks

Terminated

So run it again and you should see this time it has the correct disk selected

Current disk mapping table (source => target):

  /dev/sda => /dev/sda

So proceed with recovery, option 1 will need to be selected multiple times

It will crash out with this error, but that's just a bug. reboot and all will be well.

there will be a double reboot for an SELINUX relabel.

ERROR:

====================

BUG in /usr/share/rear/lib/framework-functions.sh line 138:

'Directory /tmp/rear.4QIY3CICzHmVnSd/outputfs not empty, can not remove'

--------------------

after reboot check grub params

[root@einstein ~]# grub2-editenv - list

saved_entry=f32f63b9b0ac4d68aa0ab7e6a76c2f35-5.14.0-687.52.1.el9_8.x86_64

menu_auto_hide=1

boot_success=0

boot_indeterminate=0

if you don't see a menu_auto_hide at it back in

Rear recreates a brand new config rather than copying an existing one and this option isn't set by default.

sudo grub2-editenv - set menu_auto_hide=1

That should do it but if not do this as well, but shouldn't be necessary.

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Backing Up My Hypervisor - the return

Following on from my previous post on this my hypervisor has been upgraded to Centos 7.3.

REAR is still working well for the backups, here's a few optimisations worth noting....

My new /etc/rear/local.conf:

TMPDIR="/backup/tmp"
export TMPDIR
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR="/backup/host"
BACKUP=NETFS
BACKUP_URL=file:///backup/host
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

The TMPDIR makes REAR use my backup disk rather than /tmp on the very slow USB stick for creating the backup before moving it to /backup/host/einstein

OUTPUT_URL & ISO_DIR also stop REAR creating the rescue ISO on the USB stick and then copying it over. It's created direct into /backup/host

Recovery is as before. Don't be put off by these warnings, it will still work.

initramfs creation for Kernel 3.10.0-514.10.2.el7.x86_64 failed

You also wont need to recreate the ISO mount point or modify fstab. Everything just works.

Regarding recovery to a larger disk. A simpler solution is to just edit the disk size line in /var/lib/rear/layout/disklayout.conf with the size of the new disk. This will make REAR think the disk matches the original and it will restore without further modification. It will only use the original partition table so the extra space is wasted but for these purposes that's good enough.

disk /dev/sdc 7864320000 msdos

(A 16GB stick would be 16018046976)

Backing Up My Hypervisor

So following on from my last post on Backing Up My Virtual Guests here's a long overdue post on how I'm backing up my hypervisor.

The hypervisor is KVM running on a USB stick with Centos 6.8 and I'm making use of Relax and Recover, which the good folks at the upstream vendor have chosen to add into version 6.8.

Setup is straight forward, install the rear rpm and edit the configuration file /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///backup
BACKUP=NETFS
BACKUP_URL=file:///backup
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

So this will create a backup of the OS installed on the USB stick to my backup drive in /backup/einstein. It won't touch the RAID volume with the LVMs containing my guests but their configuration will be saved as part of backup so they will be available again on restore.
The exclude options stop it backing up some temporary areas to save space and also the isos directory as that's on the RAID volume.

The output will be an  ISO (/backup/einstein/rear-einstein.iso) which can be used to boot the machine for restoration and a large tar ball in the same directory containing the files.

And that's it.
I additionally removed the daily cron job in /etc/cron.d/rear and replaced with the following to perform a backup twice a month as this is sufficient for my needs and matches what I do for my guests.

30 3 2,16 * *   /usr/sbin/rear mkbackup > /dev/null 2>&1

So how do I restore then?

The USB stick has gone pop and if I've put another same sized one in then restoration is easy.

Boot off the ISO, select the Recover einstein.at.home option, mount up the backup drive and run the recover command.

# mkdir /backup

# mount /dev/sdd1 /backup

# rear recover

Once completed (and it will take quite a while!!) recreate the isos mount point (mkdir /mnt/local/isos) and remove any erroneous blank lines that may have got inserted into /etc/fstab (I don't know why this happens but it will generate warnings during boot if you don't fix).

Reboot and bingo you have a working system again.

I used a larger USB stick, it wont recover?

Yes it will, you just need to do a little bit more work. Rear can't find an identical disk to recover to so asks whether you want to map the recovery to another disk. If you pick option 6 (do not map disks) and then option 2 (edit disk layout) you can tell it to force partition the new stick the same way as the old one by uncommenting these lines.

• disk /dev/sde 7864320000 msdos
• part /dev/sde 524288000 1048576 primary boot /dev/sde1
• part /dev/sde 7338983424 525336576 primary lvm /dev/sde2
• lvmdev /dev/vg_einstein /dev/sde2 EC1aq1-z2Sv-8ARK-IrvJ-cGVv-eLFp-JvwXLe 14333952
• lvmgrp /dev/vg_einstein 4096 1749 7163904
• lvmvol /dev/vg_einstein lv_root 1562 12795904
• lvmvol /dev/vg_einstein lv_swap 187 1531904
• fs /dev/mapper/vg_einstein-lv_root / ext4 uuid=c2c66e88-641a-484f-b574-3e0de6170d4b label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16353 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• fs /dev/sde1 /boot ext4 uuid=c5e1b99c-e039-4f76-8947-67ec8e728f6a label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• swap /dev/mapper/vg_einstein-lv_swap uuid=e4e02117-935e-43d3-a7fc-fae005581f68 label=

Now you could get clever here and amend the partition sizes to make use of the extra space but in my situation there isn't any point!!

Next select option 5 (continue recovery) twice and the recovery should then continue as above.