System restoration with REAR

 Boot from recovery iso on pihole

if replacing a drive, smart array will detect a change and disable logical drive and boot will hang at smart array initialisation.  

Make sure you reboot with text mode and select F2 to reenable drive array

Log in using standard root password over network

Recover einstein option

Should see 3 drives sda/sdb/sdc

mount the backup partition - it will be the one with the single partition

Device     Start        End    Sectors  Size Type

/dev/sdc1   2048 6251167743 6251165696  2.9T Linux filesystem

RESCUE einstein:~ # mkdir /backup

RESCUE einstein:~ # mount /dev/sdc1 /backup

Of the other 2, run pvs to determine which one is not the volume with all the guest lvms on it

RESCUE einstein:~ # pvs

  PV         VG            Fmt  Attr PSize   PFree

  /dev/sda2  rhel_einstein lvm2 a--  <50.00g <25.75g

  /dev/sdb   vg_guests     lvm2 a--    2.91t   1.86t

In this case it's sda that is the OS disk so ensure rear uses that

RESCUE einstein:~ # rear recover

Relax-and-Recover 2.6 / 2020-06-17

Running rear recover (PID 814)

Using log file: /var/log/rear/rear-einstein.log

Running workflow recover within the ReaR rescue/recovery system

Using backup archive '/backup/host/einstein/backup.tar.gz'

Will do driver migration (recreating initramfs/initrd)

Calculating backup archive size

Backup archive size is 4.6G     /backup/host/einstein/backup.tar.gz (compressed)

Comparing disks

Ambiguous possible target disks need manual configuration (more than one with same size found)

Switching to manual disk layout configuration

Using /dev/sdb (same name and same size 3200598237184) for recreating /dev/sdb

Current disk mapping table (source => target):

  /dev/sdb => /dev/sdb

Confirm or edit the disk mapping

1) Confirm disk mapping and continue 'rear recover'

2) Confirm identical disk mapping and proceed without manual configuration

3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)

4) Use Relax-and-Recover shell and return back to here

5) Abort 'rear recover'

It's picked the wrong one - so change with option 3

Current disk mapping table (source => target):

  /dev/sdb => /dev/sda

Confirm or edit the disk mapping

1) Confirm disk mapping and continue 'rear recover'

2) n/a

3) Edit disk mapping (/var/lib/rear/layout/disk_mappings)

4) Use Relax-and-Recover shell and return back to here

5) Abort 'rear recover'

It will probably crash out

User confirmed disk mapping

Failed to apply layout mappings to /var/lib/rear/layout/config/df.txt for /dev/sda (probably no mapping for /dev/sda in /var/lib/rear/layout/disk_mappings)

Failed to apply disk layout mappings to /var/lib/rear/layout/config/df.txt

ERROR: Failed to apply disk layout mappings

Some latest log messages since the last called script 320_apply_mappings.sh:

  2026-09-27 16:22:17.927699482 Including layout/prepare/default/320_apply_mappings.sh

  2026-09-27 16:22:18.003792169 Failed to apply layout mappings to /var/lib/rear/layout/config/df.txt for /dev/sda (probably no mapping for /dev/sda in /var/lib/rear/layout/disk_mappings)

  2026-09-27 16:22:18.006684154 Failed to apply disk layout mappings to /var/lib/rear/layout/config/df.txt

Aborting due to an error, check /var/log/rear/rear-einstein.log for details

Exiting rear recover (PID 814) and its descendant processes ...

Running exit tasks

Terminated

So run it again and you should see this time it has the correct disk selected

Current disk mapping table (source => target):

  /dev/sda => /dev/sda

So proceed with recovery, option 1 will need to be selected multiple times

It will crash out with this error, but that's just a bug. reboot and all will be well.

there will be a double reboot for an SELINUX relabel.

ERROR:

====================

BUG in /usr/share/rear/lib/framework-functions.sh line 138:

'Directory /tmp/rear.4QIY3CICzHmVnSd/outputfs not empty, can not remove'

--------------------

after reboot check grub params

[root@einstein ~]# grub2-editenv - list

saved_entry=f32f63b9b0ac4d68aa0ab7e6a76c2f35-5.14.0-687.52.1.el9_8.x86_64

menu_auto_hide=1

boot_success=0

boot_indeterminate=0

if you don't see a menu_auto_hide at it back in

Rear recreates a brand new config rather than copying an existing one and this option isn't set by default.

sudo grub2-editenv - set menu_auto_hide=1

That should do it but if not do this as well, but shouldn't be necessary.

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Fixing frozen keyboard in ILO virtual console

On newer versions of RHEL (9+) the keyboard on the ILO virtual console will die shortly after reboot.

$ journalctl -k | egrep -i 'dmar|iommu|remap'

Reports lots of INTR-REMAP errors

and 

$ lsusb

Does not show the ILO virtual keyboard as connected when the console is open.

The fix is to add this parameter to the kernel command line

#  grubby --update-kernel=ALL --args="intremap=nosid"

# grubby --info=ALL

to verify it has been added, then reboot.

Opening and closing console should now work and you'll see the keyboard connected

[paul@einstein ~]$ lsusb

Bus 003 Device 010: ID 03f0:7029 HP, Inc Virtual Keyboard

And the following lines in the system messages file

Sep 20 15:53:57 einstein kernel: usb 3-1: new full-speed USB device number 10 using uhci_hcd

Sep 20 15:53:57 einstein kernel: usb 3-1: New USB device found, idVendor=03f0, idProduct=7029, bcdDevice= 0.02

Sep 20 15:53:57 einstein kernel: usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0

Sep 20 15:53:57 einstein kernel: usb 3-1: Product: Virtual Keyboard

Sep 20 15:53:57 einstein kernel: usb 3-1: Manufacturer: BMC

Sep 20 15:53:57 einstein kernel: input: BMC Virtual Keyboard  as /devices/pci0000:00/0000:00:1c.7/0000:01:00.4/usb3/3-1/3-1:1.0/0003:03F0:7029.0011/input/input21

Sep 20 15:53:57 einstein kernel: hid-generic 0003:03F0:7029.0011: input,hidraw0: USB HID v1.01 Keyboard [BMC Virtual Keyboard ] on usb-0000:01:00.4-1/input0

Sep 20 15:53:57 einstein kernel: input: BMC Virtual Keyboard  as /devices/pci0000:00/0000:00:1c.7/0000:01:00.4/usb3/3-1/3-1:1.1/0003:03F0:7029.0012/input/input22

Sep 20 15:53:57 einstein kernel: hid-generic 0003:03F0:7029.0012: input,hidraw1: USB HID v1.01 Mouse [BMC Virtual Keyboard ] on usb-0000:01:00.4-1/input1

Sep 20 15:53:57 einstein systemd-logind[999]: Watching system buttons on /dev/input/event1 (BMC Virtual Keyboard )

MFA with Google Authenticator

 Set server up so user can log in with an ssh key or Google authenticator (won't need both). 

If a user doesn't have key access setup they'll have to use Google Authenticator, they won't be able to log in with a password

The user still needs to type the password to log into cockpit

/etc/ssh/sshd_config

ChallengeResponseAuthentication yes

# Uncomment this to force google authenticator AND a public key

#AuthenticationMethods publickey,keyboard-interactive

/etc/pam.d/cockpit

# google authenticator for two-factor

auth  required  pam_google_authenticator.so secret=/home/${USER}/.ssh/.google_authenticator

/etc/pam.d/sshd

#Commented out to not allow passwords

#auth       substack     password-auth

# Use Google authenticator for login

auth       required     pam_google_authenticator.so nullok secret=/home/${USER}/.ssh/.google_authenticator

To setup a user with Google Authenticator make sure to use the -s option

 google-authenticator -s /home/test/.ssh/.google_authenticator

Expand a logical volume used inside a VM

This expands a LV used as a virtual disk inside a virtual machine.

First expand the LV on the hypervisor

# lvextend -L500g /dev/vg_guests/lv_myvideo

# lvs|grep myvideo

  lv_myvideo      vg_guests     -wi-ao---- 500.00g

Reboot the hypervisor so all disks are rescanned. There's probably a way of doing this without a reboot but I couldn't figure out how to get that to work;

Check the VM can see the new disk size

# fdisk -l /dev/vdc

Disk /dev/vdc: 500 GiB, 536870912000 bytes, 1048576000 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x60117149

Device     Boot Start       End   Sectors  Size Id Type

/dev/vdc1        2048 838860799 838858752  400G 83 Linux

Before continuing, stop any processes using the disk and unmount it.

The partition is still 400G so, using fdisk, delete the primary partition and recreate accepting all defaults, except...

Do not remove the existing disk signature

Do you want to remove the signature? [Y]es/[N]o: N

If you accidentally do remove it, repairing the disk should get it back.

# xfs_repair -L /dev/vdc1

Verify the new size

# fdisk -l /dev/vdc1

Disk /dev/vdc1: 500 GiB, 536869863424 bytes, 1048573952 sectors

Finally remount and grow the file system.

# xfs_growfs /dev/vdc1

Replacing A Failed Disk in a software Raid 1 array

 2 Disk array, /dev/sda & /dev/sdc where /dev/sdc has failed.

• Mark disk as failed 
• # mdadm --manage /dev/md0 --fail /dev/sdc1

• Remove disk from array  
• # mdadm --manage /dev/md0 --remove /dev/sdc1

• Remove physical disk

• Copy partition table from working disk 
• # sfdisk -d /dev/sda | sfdisk /dev/sdc

• Add disk to mirror 
• # mdadm --manage /dev/md0 --add /dev/sdc1

• Verify & check on rebuild status
• # mdadm --detail /dev/md0
• # cat /proc/mdstat

Non-root access to system logs

 Access to some system logs in /var/log is restricted to root only. 

Here's a handy way to grant access read-only access to all system logs without all that mucking around with sudo and file permissions.

• Install bindfs from the EPEL repo.
• Create a group (eg: logs) and add all users to it who need logfile access
• Create a mountpoint for the read-only view of /var/log (eg: /log)
• Create a bind mount in /etc/fstab
• /var/log  /log  fuse.bindfs     ro,force-group=logs,perms=g+rD  0 0
• Mount /log

System logs can now be viewed in /log by any user who is a member of the logs group.

This works because bindfs forces group read access on all files in the bind mount  

NordVPN access from RHEL 8

We can use OpenVPN to get access to NordVPN endpoints, EPEL has an NMCLI plugin to do this.
RHEL 8 doesn't yet have a dedicated EPEL repo so we can temporarily use the RHEL 7 repo.

• yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
• yum install NetworkManager-openvpn

Pull down all the NordVPN endpoint OpenVPN templates

• wget https://downloads.nordcdn.com/configs/archives/servers/ovpn.zip

Pick the one(s) you want to connect with and import it into NetworkManager

• nmcli connection import type openvpn file uk1089.nordvpn.com.udp.ovpn

It needs modifying to include your NordVPN username & password

• nmcli connection modify uk1089.nordvpn.com.udp +vpn.data "username=user@email.com,password-flags=0" vpn.secrets password="MyPassword"

With SELinux enabled you'll need to restore some SELinux contexts on the directory nmcli stores the certs in

• restorecon -R /root/.cert

To start / stop a connection:

• nmcli connection up uk1089.nordvpn.com.udp
• nmcli connection down uk1089.nordvpn.com.udp

To show available connections:

• nmcli connection show

To delete a connection:

• nmcli connection delete uk1089.nordvpn.com.udp

If you want a user other than root to be able to initiate VPN connections, create a polkit file:

/etc/polkit-1/localauthority/50-local.d/50-nmcli-remote-access.pkla

[Remote nmcli SSH access]Identity=unix-user:paulAction=org.freedesktop.NetworkManager.network-controlResultAny=yesResultInactive=yesResultActive=yes

User session messages in the system log on Centos 7

Here's how to get rid of those annoying systemd messages that flood /var/log/messages on Centos 7.

Filter them out with an rsyslog filter script in /etc/rsyslog.d -

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

Then restart rsyslog

systemctl restart rsyslog

Shamelessly stolen from Red Hat here

Centos 7 - systemd messages flooding logs

The systemd process floods /var/log/messages with notifications whenever a user logs in or a cron job runs.

Mar 27 03:40:01 einstein systemd: Created slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Starting user-0.slice.
Mar 27 03:40:01 einstein systemd: Started Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Starting Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Removed slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Stopping user-0.slice.


Given sar runs every 10 minutes there is a lot of crud in the messages file.

You can create a filter for rsyslogd to tell it to junk the messages:

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Removed slice user-" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

systemctl restart rsyslog.service 


I take no credit for this, thanks to this article on the excellent Redhat Knowledgebase.
(I slightly adapted the solution to include the Removed & Stopping messages too) 

Hypervisor Build on MicroServer - Part 2

So I'm assuming you've read the first part and also set up the RAID 5 array

We're going to be using logical volumes & LVM on our RAID 5 array for the storage of the virtual guests

# pvcreate /dev/md0

Physical volume "/dev/md0" successfully created

# vgcreate vg_guests /dev/md0

Volume group "vg_guests" successfully created

Also create a volume to hold the ISOs for building the guests

# lvcreate -L 25G vg_guests -n lv_isos

# mkfs.ext4 /dev/vg_guests/lv_isos

Auto-mount it in /etc/fstab

/dev/mapper/vg_guests-lv_isos /isos ext4 defaults 0 2

Mount and set permissions so 'paul' can upload ISOs directly to it

# mount /isos

# chmod 755 /isos

# chgrp paul /isos

Finally upload some ISOs...

Using virt-manager, connect to the local QEMU

• Remove the virtual network created inside the Virtual Network tab ; we'll be using my home lan for the guests
• Add a storage pool for the ISOs so they are available to the guests for mounting
• Name: ISOs
• Type: dir Filesystem Directory
• Target Path: /isos
• Add a storage pool for the RAID 5 LVM array to hold the virtual guest storage
• Name: Guests
• Type: logical: LVM Volume Group
• Target Path: /dev/vg_guests
• Source Name: vg_guests

So now we're set up, here's a run through of creating a typical guest:

Before getting into virt-manager create a logical volume for the storage, eg:

# lvcreate -L 10G vg_guests -n lv_myguestname

Now from within virt-manager:

• Give it a name
• Use Local install media / Use ISO image / Select an ISO from your ISO storage pool
• Set OS type & version, set applicable RAM & CPU
• Select managed or other existing storage & pick your newly created logical volume from the Guests storage pool.
• You'll notice no networking is available. This is fine just make sure to tick the 'Customize configuration before install' box and this can be added next.
• Add Hardware and select Network -  Host device eth0: macvtap
• Select the newly created NIC and make sure source mode is set to Bridge otherwise the interface will not work.

Now hit 'Begin Installation' and install your lovely new guest.

Once built, if it's a Linux guest, configure the virtual serial port as detailed here so console access is via the command line rather than virt-manager.

A point to note about macvtap interfaces. They do not allow guest to hypervisor communication. Guest to guest is fine as is guest to rest of LAN. If you need to communicate with a guest from the hypervisor or vice versa you will need to create bridged interfaces and use those.

Configuring a RAID 5 array using software RAID

Here's how to create a 3 disk RAID 5 array using software RAID on Linux.
To make it slightly more interesting I'm going to create it initially using only 2 disks and then add the afterwards. Why? Because I can.

I'm doing this on my MicroServer. The RAID array will hold the virtual guests and the ISO storage pool.

First partition the disks and make sure they partitions are aligned:

# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) align-check optimal 1
1 aligned
(parted) p
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary raid

Tip - here's how to check the disks are ready to go

mdadm -E /dev/sd[bc]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)

Tip - if these aren't brand new disks check they haven't got any md superblocks already present. If they have zero the superblock (see later in post)

# mdadm -E /dev/sd[bc]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

 
Here's the magic. How to create the array, note the use of the 'missing' parameter for the 3rd disk.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Tip - how to check the array status

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdb1[0]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 25 19:54:54 2015
Raid Level : raid5
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Mon May 25 19:54:54 2015
State : active, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : einstein.at.home:0 (local to host einstein.at.home)
UUID : cc117a2a:439506c1:429d86cf:35514c71
Events : 0

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 0 0 4 removed

Finally save the configuration

mdadm --detail --scan --verbose >> /etc/mdadm.conf

So now let's add the missing disk
First clone the partition table from one of the existing disks

sfdisk -d /dev/sdb | sfdisk /dev/sdd --force

(For completeness zero the superblock as shown above, although not technically necessary on a new disk of course)
Now add the disk to the array:

mdadm --add /dev/md0 /dev/sdd1

The array will now resilver. This will takes hours. You can check /proc/mdstat for progress.
Don't forget to update mdadm.conf as shown above.

Tip - You can speed up resilvering by increasing these kernel parameters

echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 16384 > /sys/block/md0/md/stripe_cache_size

 Finally if you want to destroy an array

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --zero-superblock /dev/sdb
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd

Hypervisor Build on MicroServer

Here's the build details of the hypervisor on my MicroServer.

Making use of the internal USB socket to install Centos 6.6 on an 8GB USB stick and using all the hard drives as guest storage. There are 3 hard drives which will be configured as RAID 5 using Linux software RAID.

Use the lovely ILO to virtual mount and boot the Centos 6.6 ISO and the remote console to do the install. A big advantage over my first MicroServer.

Tip - quick way to erase a USB stick in Windows - use diskpart

DISKPART> list disk
DISKPART> select disk 2
DISKPART> clean

Perform a minimal install to the USB stick.
Tip - to avoid having to update after the install, use an install URL and add an additional repository for the updates. It'll all be done in one go. Here's the ones I use:
Install URL: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/os/x86_64/
Additional repo: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/updates/x86_64/

Install additional packages:
xauth (for X window virt-manager to work)
ntp (time sync)
parted (disk partitioning as fdisk is deprecated and doesn't like new large disks)
acpid (so you can do a graceful shutdown with the power button)
hp-ams (HP's agentless management service, get from the HP website. Integrates into ILO)

Configure and enable firewall and allow only ssh inbound (/etc/sysconfig/iptables)

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Configure and enable ntp
Create a user id and block root access via ssh

Install Virtualisation

yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"
yum install dejavu-lgc-sans-fonts

Allow 'paul' to run virt-manager and manage the local qemu
Create /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

                [Remote libvirt SSH access]

       Identity=unix-group:paul

       Action=org.libvirt.unix.manage

       ResultAny=yes

       ResultInactive=yes

       ResultActive=yes

 

See the next post for details of how to configure the RAID 5 disks and configure KVM to use them.

Using the virtual serial port with KVM

So you've installed some linux virtual guests on your shiney new KVM hypervisor, virt-manager is great for managing them but needs X11. Given the guests are linux wouldn't it be nice to be able to get to the console using CLI? Here's how...

For RHEL 6 guests:

Add the following lines to /etc/grub.conf below the splashimage line

serial --unit=0
terminal --timeout=5 serial console

Then append console=ttyS0 to your kernel line

 kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_rhel6-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_rhel6/lv_swap crashkernel=auto rd_LVM_LV=vg_rhel6/lv_root rd_NO_DM rhgb quiet console=ttyS0

For RHEL 7 guests:

Add the following lines to /etc/default/grub

GRUB_TERMINAL="serial"GRUB_SERIAL_COMMAND="serial"

Amend the GRUB_CMDLINE_LINUX to include console=ttyS0

GRUB_CMDLINE_LINUX="rd.lvm.lv=centos_tos7/swap rd.lvm.lv=centos_tos7/root rhgb quiet console=ttyS0"

Now regenerate the grub configuration

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the guest and you're done.

Access your lovely new text console using virsh console <guest name>

Using winbind to authenticate to an active directory - Part 3

Having successfully set up my AD domain and got RHEL 6 clients authenticating to it, I'll now describe how to do the same thing for RHEL 5 clients.

You can read about the AD setup in Part 1 and the RHEL 6 configuration in Part 2.

I have performed a customized RHEL 5 installation, removing Desktop Environments, Applications, Printer Servers and reset the run level to 3.

yum install samba3x-winbind (you don't need samba for authentication)

Run authconfig-tui --enablemkhomedir to configure winbind, set the following options:
user information: use winbind
authentication: winbind
winbind domain: MY
security model: ads
winbind ads realm: MY.DOM
winbind domain controllers: dc.my.dom
template shell: /bin/bash

Don't click join domain at this stage.
When the wizard completes it will attempt to start winbind and fail. We haven't finished configuring yet so this is fine.
Remove the existing configuration databases, rm -f /var/lib/samba/*

Modify /etc/samba/smb.conf with the following options. This will map all AD users & groups into the range 100000-199999.

workgroup = MY
password server = dc1.my.dom
realm =MY.DOM
security = ads
idmap config * : backend = tdb
idmap config * : range = 200000-299999
idmap config my:backend  = rid
idmap config my:range    = 100000 - 199999
idmap config my:base_rid = 0
idmap uid = 100000-299999
idmap gid = 100000-299999

You can also set these useful options:

template shell = /bin/bash
winbind use default domain = true
winbind enum users = yes  (let's you use getent passwd to see AD users)
winbind enum groups = yes  (let's you use getent group to see AD groups)
winbind offline logon = true  (cached users can log in when AD is unavailable. Very useful)

Now rerun authconfig-tui --enablemkhomedir and click join domain. Supply the AD domain password when prompted and the client will join the domain successfully. You will be able to see the client in the Windows ADUC tool

It should now start up properly:
chkconfig winbind on ; service winbind start


I'm not setting up centralised home directories so we'll use oddjobd to automatically create home directories when an AD user logs on for the first time to a client.

We'll set the home directories up in /home/MY and make them private. RHEL 6 uses oddjobd for this. RHEL 5 doesn't.

mkdir /home/MY
chmod 711 /home/MY 
Modify /etc/pam.d/system-auth-ac to fix home dir permissions 
    session     optional      pam_mkhomedir.so umask=0077

Now restrict ssh access to users in the grp-linux-servers AD group.

Edit /etc/security/pam_winbind.conf
    require_membership_of = grp-linux-servers

Finally restrict root access via sudo to those in the grp-root-access AD group

Edit /etc/sudoers
%grp-root-access        ALL=(ALL)       NOPASSWD: ALL

Some useful commands to check things are working:

wbinfo -u / wbinfo -g will pull a user and group list from the AD
wbinfo -D my / wbinfo --dc-info=my will put details of AD and associated DCs
getent passwd / getent group will show the client's passwd & group list including AD entries.

In Part 4 I'll describe how to achieve this on a Solaris 10 client. This is the most complex to set up as there is no out the box integration and it also requires additional freeware packages.

Using winbind to authenticate to an active directory - Part 2

Here I'll describe how to configure a RHEL 6 installation to authenticate using winbind to the AD my.dom set up in Part 1. RHEL 6 is the easiest client to configure as it has samba and winbind integration built into the OS.

I'll assume a basic server installation has been performed. You may need to add additional packages if you've gone with a minimal install.

yum install winbind (you don't need samba for authentication)

Run system-config-authentication or authconfig-tui to configure winbind, set the following options:

user account database: winbind
winbind domain: MY
security model: ads
winbind ads domain: MY.DOM
winbind domain controllers: dc.my.dom
template shell: /bin/bash
Advanced option: create home directories on first logon

Don't click join domain at this stage.
When the wizard completes it will attempt to start winbind and fail. We haven't finished configuring yet so this is fine.
Remove the existing configuration databases, rm -f /var/lib/samba/*

The key thing when integrating into a domain is to make sure the Unix UID / GID are consistent across all unix hosts. Winbind provides several mapping strategies for doing this. I'm using idmap_rid as it's simple to setup, doesn't require an external database and performs an algorithmic mapping from Windows SIDs to UID / GIDs

Modify /etc/samba/smb.conf with the following options. This will map all AD users & groups into the range 100000-199999.

workgroup = MY
password server = dc1.my.dom
realm =MY.DOM
security = ads
idmap config * : backend = tdb
idmap config * : range = 200000-299999
idmap config my:backend  = rid
idmap config my:range    = 100000 - 199999
idmap config my:base_rid = 0
idmap uid = 100000-299999
idmap gid = 100000-299999

You can also set these useful options:

template shell = /bin/bash
winbind use default domain = true
winbind enum users = yes  (let's you use getent passwd to see AD users)
winbind enum groups = yes  (let's you use getent group to see AD groups)
winbind offline logon = true  (cached users can log in when AD is unavailable. Very useful)

Now rerun the authentication gui / tui and click join domain. Supply the AD domain password when prompted and the client will join the domain successfully. You will be able to see the client in the Windows ADUC tool

It should now start up properly:
chkconfig winbind on ; service winbind start

I'm not setting up centralised home directories so we'll use oddjobd to automatically create home directories when an AD user logs on for the first time to a client.

We'll set the home directories up in /home/MY and make them private:

mkdir /home/MY
chmod 711 /home/MY
Edit /etc/oddjobd.conf.d/oddjobd-mkhomedir.conf
Change 2 umasks to -u 0077 (from 0002) to make home dirs private
service oddjobd restart

Now restrict ssh access to users in the grp-linux-servers AD group.

Edit /etc/security/pam_winbind.conf
    require_membership_of = grp-linux-servers

Finally restrict root access via sudo to those in the grp-root-access AD group

Edit /etc/sudoers
%grp-root-access        ALL=(ALL)       NOPASSWD: ALL

Some useful commands to check things are working:

wbinfo -u / wbinfo -g will pull a user and group list from the AD
wbinfo -D my / wbinfo --dc-info=my will put details of AD and associated DCs
getent passwd / getent group will show the client's passwd & group list including AD entries.

In Part 3 I'll describe how to do the same thing for a RHEL 5 client. The principals are the same but RHEL 5 isn't so tightly integrated with windbind out the box so we have to do a little bit of extra work.

Sudo : The basics

Basic structure of ACL entries for the sudoers file
Always use visudo to edit as it does syntax checking for you

Command aliases:
 Cmnd_Alias DIAGS = /usr/sbin/tcpdump,/usr/sbin/ethtool

Network / host aliases:
Host_Alias LAN = 192.168.0.0/255.255.255.0

You can then allow a user to access these commands:

This will allow Paul to run tcpdump & ethtool from anywhere without reentering his password:
paul    ALL = NOPASSWD:DIAGS

Or you can restrict it to certain subnets:
paul    LAN = NOPASSWD:DIAGS

Omit the NOPASSWD parameter if you want Paul to reenter his password everytime he uses sudo.

If you want paul to be able to become root and do anything:
paul    ALL=(ALL)     NOPASSWD:ALL

Replacing mirrored disks under Linux software RAID

So you've got a Linux server using software RAID with disk mirroring configured and when it's healthy it looks something like this:
# cat /proc/mdstat 
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [linear] 
md3 : active raid1 sdb3[1] sda3[0]
      949995648 blocks [2/2] [UU]
      
md2 : active raid1 sdb2[1] sda2[0]
      8514368 blocks [2/2] [UU]
      
md1 : active raid1 sdb1[1] sda1[0]
      8602688 blocks [2/2] [UU]
      
md4 : active raid1 sdb4[1] sda4[0]
      9638912 blocks [2/2] [UU]

A disk goes pop, the system is still running but you need to replace the dead disk. After physically replacing the disk here's how to add it back to the raid configuration.

So we see that /dev/sda has died.

# cat /proc/mdstat 

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [linear] 

md3 : active raid1 sdb3[1]

      949995648 blocks [2/1] [_U]

      

md2 : active raid1 sdb2[1]

      8514368 blocks [2/1] [_U]

      

md1 : active raid1 sdb1[1]

      8602688 blocks [2/1] [_U]

      

md4 : active raid1 sdb4[1]

      9638912 blocks [2/1] [_U]

First copy the partition table from /dev/sdb onto /dev/sda:

# sfdisk -d /dev/sdb > /tmp/partitiontable

# sfdisk /dev/sda < /tmp/partitiontable

# sfdisk -l /dev/sda ; sfdisk -l /dev/sdb  (They should now be identical)

Now add the partitions on /dev/sda back into the raid devices:

# mdadm -a /dev/md1 /dev/sda1

# mdadm -a /dev/md2 /dev/sda2

# mdadm -a /dev/md3 /dev/sda3

# mdadm -a /dev/md4 /dev/sda4

This will probably take some time so watch its' progress with:

# watch -n 5 cat /proc/mdstat

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [linear]

md3 : active raid1 sda3[2] sdb3[1]

      949995648 blocks [2/1] [_U]

      [>....................]  recovery =  2.3% (21904000/949995648) finish=241.8min speed=63952K/sec

md2 : active raid1 sda2[0] sdb2[1]

      8514368 blocks [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]

      8602688 blocks [2/2] [UU]

md4 : active raid1 sda4[2] sdb4[1]

      9638912 blocks [2/1] [_U]

        resync=DELAYED

Packet capture with tcpdump

Solaris had snoop, Linux has tcpdump. Here's a couple of handy examples:

Interactive capture of traffic on interface eth0:

• tcpdump -i eth0

Capture traffic to a file, output can be read in Wireshark:

• tcpdump -w output.pcap -i eth0

Filter to a specific source or destination or both:

• tcpdump -i eth0 src 192.168.100.1
• tcpdump -i eth0 dst 192.168.100.1
• tcpdump -i eth0 src 192.168.100.1 or dst 192.168.100.1

Further filter by port:

• tcpdump -i eth0 src 192.168.100.1 or dst 192.168.100.1 and port 22

Bonded interfaces in RHEL

Here's how to create a bonded interface in RHEL 6

This bond is working in failover mode with VLAN tagging over interfaces eth0 & eth1:


ifcfg-bond0

DEVICE=bond0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="mode=1 miimon=100"

ifcfg-bond0.x  (where x is vlan number)

DEVICE=bond0.x
IPADDR=y.y.y.y
NETMASK=y.y.y.y
GATEWAY=y.y.y.y
USERCTL=no
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
IPV6INIT=no

ifcfg-ethx (where x is 0 or 1)

DEVICE=ethx
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Note that creating bonding.conf in /etc/modprobe.d is no longer required in the latest RHEL 6 releases.

Restart network services and the bond will be created. Make sure all previous non-bonded interfaces are destroyed before restarting network services otherwise they will interfere with the bond creation. A reboot is the easiest way to sort this out.

To check the status of a bonded interface:

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth1

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth1

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 1

Permanent HW addr: 00:9c:02:24:7c:dc

Slave queue ID: 0

Slave Interface: eth0

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: 00:9c:02:24:7c:d8

Slave queue ID: 0

To force service to a specific interface:

 ifenslave --change-active bond0 eth0

To remove an interface from a bond

 ifenslave -d bond0 eth0

To add an interface back into a bond

 ifenslave bond0 eth0

Linux Disk Benchmarking

A really quick way to see the write performance of your disk.....
Write a 512MB file


dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync ; rm test

And it tells you how fast it did it

512+0 records in

512+0 records out

536870912 bytes (537 MB) copied, 2.74433 s, 196 MB/s

The fdatasync option forces dd to do a sync to ensure data is really written to disk before it exits which should give a more realistic figure.