Concatenating video files

For video files that come in part, use ffmpeg to merge them into 1 file

Eg: part1.mp4, part2.mp4, part3.mp4

Create a concat file - list.txt

file 'part1.mp4'
file 'part2.mp4'
file 'part3.mp4'

Use ffmpeg to merge into full.mp4

ffmpeg -f contact -safe 0 -i list.txt -c copy full.mp4

If you have subtitles create an additional concat file - subts.txt

file 'part1.mp4.srt'
file 'part2.mp4.srt'
file 'part3.mp4.srt'

Then use ffmpeg to create the full file

ffmpeg -f concat -safe 0 -i list.txt -f concat -safe 0 -i subs.txt -map 0 -map 1 -c copy -metadata:s:s language=eng -metadata:s:s title=English full.mp4

The title= seems to be necessary when doing subtitles and it names the subtitle track.

MFA with Google Authenticator

 Set server up so user can log in with an ssh key or Google authenticator (won't need both). 

If a user doesn't have key access setup they'll have to use Google Authenticator, they won't be able to log in with a password

The user still needs to type the password to log into cockpit

/etc/ssh/sshd_config

ChallengeResponseAuthentication yes

# Uncomment this to force google authenticator AND a public key

#AuthenticationMethods publickey,keyboard-interactive

/etc/pam.d/cockpit

# google authenticator for two-factor

auth  required  pam_google_authenticator.so secret=/home/${USER}/.ssh/.google_authenticator

/etc/pam.d/sshd

#Commented out to not allow passwords

#auth       substack     password-auth

# Use Google authenticator for login

auth       required     pam_google_authenticator.so nullok secret=/home/${USER}/.ssh/.google_authenticator

To setup a user with Google Authenticator make sure to use the -s option

 google-authenticator -s /home/test/.ssh/.google_authenticator

Expand a logical volume used inside a VM

This expands a LV used as a virtual disk inside a virtual machine.

First expand the LV on the hypervisor

# lvextend -L500g /dev/vg_guests/lv_myvideo

# lvs|grep myvideo

  lv_myvideo      vg_guests     -wi-ao---- 500.00g

Reboot the hypervisor so all disks are rescanned. There's probably a way of doing this without a reboot but I couldn't figure out how to get that to work;

Check the VM can see the new disk size

# fdisk -l /dev/vdc

Disk /dev/vdc: 500 GiB, 536870912000 bytes, 1048576000 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x60117149

Device     Boot Start       End   Sectors  Size Id Type

/dev/vdc1        2048 838860799 838858752  400G 83 Linux

Before continuing, stop any processes using the disk and unmount it.

The partition is still 400G so, using fdisk, delete the primary partition and recreate accepting all defaults, except...

Do not remove the existing disk signature

Do you want to remove the signature? [Y]es/[N]o: N

If you accidentally do remove it, repairing the disk should get it back.

# xfs_repair -L /dev/vdc1

Verify the new size

# fdisk -l /dev/vdc1

Disk /dev/vdc1: 500 GiB, 536869863424 bytes, 1048573952 sectors

Finally remount and grow the file system.

# xfs_growfs /dev/vdc1

Replacing A Failed Disk in a software Raid 1 array

 2 Disk array, /dev/sda & /dev/sdc where /dev/sdc has failed.

• Mark disk as failed 
• # mdadm --manage /dev/md0 --fail /dev/sdc1

• Remove disk from array  
• # mdadm --manage /dev/md0 --remove /dev/sdc1

• Remove physical disk

• Copy partition table from working disk 
• # sfdisk -d /dev/sda | sfdisk /dev/sdc

• Add disk to mirror 
• # mdadm --manage /dev/md0 --add /dev/sdc1

• Verify & check on rebuild status
• # mdadm --detail /dev/md0
• # cat /proc/mdstat

Non-root access to system logs

 Access to some system logs in /var/log is restricted to root only. 

Here's a handy way to grant access read-only access to all system logs without all that mucking around with sudo and file permissions.

• Install bindfs from the EPEL repo.
• Create a group (eg: logs) and add all users to it who need logfile access
• Create a mountpoint for the read-only view of /var/log (eg: /log)
• Create a bind mount in /etc/fstab
• /var/log  /log  fuse.bindfs     ro,force-group=logs,perms=g+rD  0 0
• Mount /log

System logs can now be viewed in /log by any user who is a member of the logs group.

This works because bindfs forces group read access on all files in the bind mount  

Updating my SSL cert

I described how to create SSL certs using acme.sh & Let's Encrypt in a previous post.

I've subsequently updated the cert to use ECC crypto and now want to update it so that the root domain is also included in my wildcard cert.

First remove the current cert:

acme.sh --remove --ecc  -d *.durrant.me.uk

Now issue the new one:

acme.sh --issue --keylength ec-256 -d durrant.me.uk -d *.durrant.me.uk --dns dns_cf

And finally install it:

acme.sh --install-cert --ecc -d durrant.me.uk  --key-file /etc/pki/tls/private/durrant.me.uk.key --cert-file /etc/pki/tls/certs/durrant.me.uk.cer --reloadcmd "sudo systemctl restart httpd.service"

A few notes.

• To use ECC , it's as simple as specifying --keylength ec-256 when requesting it. Make sure to always specify --ecc in issue & install operations to tell acme.sh to use the ECC one otherwise it will default to RSA if you have that cert too.
• For the installation to work make sure the user running acme.sh has sudo privilege to restart the web server and permissions to write to the cert & key file. (chown acme does it for me)

NordVPN access from RHEL 8

We can use OpenVPN to get access to NordVPN endpoints, EPEL has an NMCLI plugin to do this.
RHEL 8 doesn't yet have a dedicated EPEL repo so we can temporarily use the RHEL 7 repo.

• yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
• yum install NetworkManager-openvpn

Pull down all the NordVPN endpoint OpenVPN templates

• wget https://downloads.nordcdn.com/configs/archives/servers/ovpn.zip

Pick the one(s) you want to connect with and import it into NetworkManager

• nmcli connection import type openvpn file uk1089.nordvpn.com.udp.ovpn

It needs modifying to include your NordVPN username & password

• nmcli connection modify uk1089.nordvpn.com.udp +vpn.data "username=user@email.com,password-flags=0" vpn.secrets password="MyPassword"

With SELinux enabled you'll need to restore some SELinux contexts on the directory nmcli stores the certs in

• restorecon -R /root/.cert

To start / stop a connection:

• nmcli connection up uk1089.nordvpn.com.udp
• nmcli connection down uk1089.nordvpn.com.udp

To show available connections:

• nmcli connection show

To delete a connection:

• nmcli connection delete uk1089.nordvpn.com.udp

If you want a user other than root to be able to initiate VPN connections, create a polkit file:

/etc/polkit-1/localauthority/50-local.d/50-nmcli-remote-access.pkla

[Remote nmcli SSH access]Identity=unix-user:paulAction=org.freedesktop.NetworkManager.network-controlResultAny=yesResultInactive=yesResultActive=yes

Raspberry Pi - Useful stuff

Some useful config details to get the Pi up and running, this is based on Raspbian with the minimal installation image on version 9.
(Make sure to use a powerful enough PSU otherwise you will get undervoltage warnings in logs and the red LED will flicker)

To update to latest version:

• apt-get update
• apt-get dist-upgrade

Use raspi-config to configure this:

• The Edimax EW-7800Un wireless card is automatically detected. Option 2 & 2 to setup the SSID & password
• Check localization options are correct with option 4, including WiFi region.
• Enable ssh with option 5 & 2 (This can also be done by putting the SD card in another machine, mounting the linux partition and creating the empty file /boot/ssh)

To configure a static IP on wireless, add this to /etc/dhcpcd.conf

• static ip_address=192.168.0.108/24
• static routers=192.168.0.1
• static domain_name_servers=192.168.0.1

The red power LED can be turned off:

• echo 0 > /sys/class/leds/led1/brightness

The green activity LED can also be turned off:

• echo none > /sys/class/leds/led0/trigger

Might as well turn the HDMI output off (saves a few mA)

• tvservice -o

You can disable bluetooth and wifi on the newer models:

• Add dtoverlay=pi3-disable-bt to /boot/config.txt
• Add dtoverlay=pi3-disable-wifi to /boot/config.txt

That's a basic headless wireless system, it's now good to go.

Install rng-tools (fixes startup issues with dnscrypt-proxy, cloudflared)

• apt-get install rng-tools
• Edit /etc/default/rng-tools to specify /dev/hwrng device

User session messages in the system log on Centos 7

Here's how to get rid of those annoying systemd messages that flood /var/log/messages on Centos 7.

Filter them out with an rsyslog filter script in /etc/rsyslog.d -

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

Then restart rsyslog

systemctl restart rsyslog

Shamelessly stolen from Red Hat here

Useful commands to check SSL certs

Here's some useful stuff to examine SSL certs.
I've pulled some of this from here

Check a key

• openssl rsa -in key.file -check
• openssl ec -in key.file (For an EC cert)

Check a cert

• openssl x509 -in cert.file -text -noout

 Test an SSL connection

• openssl s_client -connect some.site.com:443
• openssl s_client -connect some.site.com:443 -servername some.site.com
  (If website uses SNI)

Using acme.sh to automate SSL certs

I'm using https://acme.sh/ to automate the creation and deployment of SSL certs from https://letsencrypt.org/ on my websites. I prefer this method to using certbot as it doesn't require any additional packages to be installed on the server as it is all done in script.

Some notes on how to create the certs and have them load automatically into Apache.
I'm not detailing how to install acme as that's straightforward and covered on the acme website.

I have the acme script installed on the web server as root as it makes copying the certs / keys to the appropriate directories easier.

I use the dns-01 challenge mechanism to issue certs as Cloudflare has an API that supports it.
This is a great way to do it as it means you don't need to copy content to the web servers to authenticate the cert issuing request.

Need the following in ~/.acme.sh/account.conf for it to work:

SAVED_CF_Key='<API key>'
SAVED_CF_Email='<email address>'

Issue the cert

acme.sh --issue --dns dns_cf -d foobar.durrant.me.uk

Add the cert / key locations to Apache
Note that the cer file contains both the site cert and the intermediate cert. Apache understands if you point both parameters at the same file which saves having to maintain two separate files.

SSLCertificateFile /etc/pki/tls/certs/foobar.durrant.me.uk.cer
SSLCertificateKeyFile /etc/pki/tls/private/foobar.durrant.me.uk.key
SSLCertificateChainFile /etc/pki/tls/certs/foobar.durrant.me.uk.cer

Install the certs

acme.sh --install-cert -d foobar.durrant.me.uk --key-file /etc/pki/tls/private/foobar.durrant.me.uk.key --fullchain-file /etc/pki/tls/certs/foobar.durrant.me.uk.cer --reloadcmd "systemctl restart httpd.service"

Set the permissions on the key & cert

chmod 600 /etc/pki/tls/private/foobar.durrant.me.uk.key
chmod 600 /etc/pki/tls/certs/foobar.durrant.me.uk.cer

 Certs will automatically be renewed and reinstalled every 2 months.

Centos 7 - systemd messages flooding logs

The systemd process floods /var/log/messages with notifications whenever a user logs in or a cron job runs.

Mar 27 03:40:01 einstein systemd: Created slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Starting user-0.slice.
Mar 27 03:40:01 einstein systemd: Started Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Starting Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Removed slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Stopping user-0.slice.


Given sar runs every 10 minutes there is a lot of crud in the messages file.

You can create a filter for rsyslogd to tell it to junk the messages:

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Removed slice user-" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

systemctl restart rsyslog.service 


I take no credit for this, thanks to this article on the excellent Redhat Knowledgebase.
(I slightly adapted the solution to include the Removed & Stopping messages too) 

Moving up to Centos 7

So I finally upgraded my entire setup to Centos 7 (7.3 to be precise).
Here are some useful notes

By default I do a minimal installation and add what I need. A few extra packages I always add:
epel-release, acpid, net-tools, bash-completion, bash-completion-extras, xauth

ntpd is replaced with chrony. /etc/chrony.conf is the config file and use chronyc sourcestats to see what's going on (equivalent of ntpq --peers)

systemd replaces init.d startup, here are some useful commands. (Because you've installed bash-completion you can hit tab to figure out the service names)

• systemctl start/stop/status <service> (equiv of service start/stop/status)
• systemctl enable/disable <service> (equiv of chkconfig off/on)
• systemctl daemon-reload (if you edit services will reload configs)
• /usr/lib/systemd/system is the dir containing the service definitions
• systemctl list-units
• systemctl list-unit-files (show state of all services, including those that have failed)

firewalld replaces iptables as the default firewall, here's some helpful stuff:

• By default the public profile is active, this is fine add necessary ports to this
• For neatness, if an app needs multiple ports create a service definition and let that through
• New services are created in /etc/firewalld/services
• firewall-cmd --list-all (show current config)
• firewall-cmd --list-services (show only services allowed through firewall)
• firewall-cmd --info-service <service> (show ports in a service)
• firewall-cmd --permanent --add-service=<new service> (add service to config)
• systemctl restart firewalld.service

New default disk format is xfs. Be aware xfs partitions can be grown but not shrunk but the metadata takes less space so are more efficient.

You no longer need to specify -cu options to fdisk, they are now default.

Finally a couple of notes just for the hypervisor:

Install the HP stuff; hp-ams, hp-health & hponcfg

Configure the libvirt-guests service to ensure guests startup & shutdown when the hypervisor does. Ensure the ON_SHUTDOWN=shutdown option is set otherwise guests will suspend which takes an age as it suspends to the USB stick.

rngd is broken and won't start but is easily fixable. Thanks to this website for showing me how.
In essence, the daemon needs "-r /dev/urandom" to tell it to use the correct random device.

When creating your raid configuration in mdadm.conf add MAILADDR <email> as an option to allow mdadm daemon to email you when there is a disk problem. This will also stop the mdmonitor service from throwing a warning on startup.

To import the virtual machines it's better to create them as brand new guests pointing at the original lvm disks rather than importing the xml configs. This will ensure QEMU configures them optimally for Centos 7. The new virt-manager is improved and makes this easy.

Backing Up My Hypervisor - the return

Following on from my previous post on this my hypervisor has been upgraded to Centos 7.3.

REAR is still working well for the backups, here's a few optimisations worth noting....

My new /etc/rear/local.conf:

TMPDIR="/backup/tmp"
export TMPDIR
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR="/backup/host"
BACKUP=NETFS
BACKUP_URL=file:///backup/host
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

The TMPDIR makes REAR use my backup disk rather than /tmp on the very slow USB stick for creating the backup before moving it to /backup/host/einstein

OUTPUT_URL & ISO_DIR also stop REAR creating the rescue ISO on the USB stick and then copying it over. It's created direct into /backup/host

Recovery is as before. Don't be put off by these warnings, it will still work.

initramfs creation for Kernel 3.10.0-514.10.2.el7.x86_64 failed

You also wont need to recreate the ISO mount point or modify fstab. Everything just works.

Regarding recovery to a larger disk. A simpler solution is to just edit the disk size line in /var/lib/rear/layout/disklayout.conf with the size of the new disk. This will make REAR think the disk matches the original and it will restore without further modification. It will only use the original partition table so the extra space is wasted but for these purposes that's good enough.

disk /dev/sdc 7864320000 msdos

(A 16GB stick would be 16018046976)

Backing Up My Hypervisor

So following on from my last post on Backing Up My Virtual Guests here's a long overdue post on how I'm backing up my hypervisor.

The hypervisor is KVM running on a USB stick with Centos 6.8 and I'm making use of Relax and Recover, which the good folks at the upstream vendor have chosen to add into version 6.8.

Setup is straight forward, install the rear rpm and edit the configuration file /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///backup
BACKUP=NETFS
BACKUP_URL=file:///backup
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

So this will create a backup of the OS installed on the USB stick to my backup drive in /backup/einstein. It won't touch the RAID volume with the LVMs containing my guests but their configuration will be saved as part of backup so they will be available again on restore.
The exclude options stop it backing up some temporary areas to save space and also the isos directory as that's on the RAID volume.

The output will be an  ISO (/backup/einstein/rear-einstein.iso) which can be used to boot the machine for restoration and a large tar ball in the same directory containing the files.

And that's it.
I additionally removed the daily cron job in /etc/cron.d/rear and replaced with the following to perform a backup twice a month as this is sufficient for my needs and matches what I do for my guests.

30 3 2,16 * *   /usr/sbin/rear mkbackup > /dev/null 2>&1

So how do I restore then?

The USB stick has gone pop and if I've put another same sized one in then restoration is easy.

Boot off the ISO, select the Recover einstein.at.home option, mount up the backup drive and run the recover command.

# mkdir /backup

# mount /dev/sdd1 /backup

# rear recover

Once completed (and it will take quite a while!!) recreate the isos mount point (mkdir /mnt/local/isos) and remove any erroneous blank lines that may have got inserted into /etc/fstab (I don't know why this happens but it will generate warnings during boot if you don't fix).

Reboot and bingo you have a working system again.

I used a larger USB stick, it wont recover?

Yes it will, you just need to do a little bit more work. Rear can't find an identical disk to recover to so asks whether you want to map the recovery to another disk. If you pick option 6 (do not map disks) and then option 2 (edit disk layout) you can tell it to force partition the new stick the same way as the old one by uncommenting these lines.

• disk /dev/sde 7864320000 msdos
• part /dev/sde 524288000 1048576 primary boot /dev/sde1
• part /dev/sde 7338983424 525336576 primary lvm /dev/sde2
• lvmdev /dev/vg_einstein /dev/sde2 EC1aq1-z2Sv-8ARK-IrvJ-cGVv-eLFp-JvwXLe 14333952
• lvmgrp /dev/vg_einstein 4096 1749 7163904
• lvmvol /dev/vg_einstein lv_root 1562 12795904
• lvmvol /dev/vg_einstein lv_swap 187 1531904
• fs /dev/mapper/vg_einstein-lv_root / ext4 uuid=c2c66e88-641a-484f-b574-3e0de6170d4b label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16353 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• fs /dev/sde1 /boot ext4 uuid=c5e1b99c-e039-4f76-8947-67ec8e728f6a label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• swap /dev/mapper/vg_einstein-lv_swap uuid=e4e02117-935e-43d3-a7fc-fae005581f68 label=

Now you could get clever here and amend the partition sizes to make use of the extra space but in my situation there isn't any point!!

Next select option 5 (continue recovery) twice and the recovery should then continue as above.

Backing Up Virtual Guests

My virtual guests use LVM for disk storage so we can use the snapshot feature to take live backups.

As these are disk image backups they are large but they are quick - a 10gb disk backup took 5 mins - as I have a huge dedicated backup disk for storing them this isn't a problem.
A note about the backup disk, I've formatted it as ext4 without LVM for maximum compatibility. Should the hardware fail I can pop it into another machine, mount up and go.

Here's an example:

First take a snapshot of the logical volume containing the virtual disk:

lvcreate -s --size=1G -n lv_dns_snap /dev/vg_guests/lv_dns

As the snapshot holds disk updates it needs to be large enough to hold all changes during the lifetime of the snapshot. 1GB is plenty for me. The lvs command will show you how much of the space the snapshot is consuming if you want to tune this.

Next take a copy of the snapshot with dd

dd if=/dev/vg_guests/lv_dns_snap of=/backup/dns-backup.dd bs=1M

Experiment with the block size parameter to see what gives best results for you - 1M worked well for me.

We're done so remove the snapshot

lvremove -f /dev/vg_guests/lv_dns_snap

Whilst we're at it we should also copy the guest xml config file from /etc/libvirt/qemu

Should you need to restore the backup, just shut the guest down and copy the disk image back

dd if=/backup/dns-backup.dd of=/dev/vg_guests/lv_dns

Installing Madsonic on Centos

Some notes on how I ported my Madsonic install from Windows to Centos.

I've put the OS on one disk and my music will live on the other. Set the 2nd disk up and copy my music across first:

pvcreate /dev/vdb1

vgcreate vg_mymusic /dev/vdb1

lvcreate -n lv_mymusic --extents 100%FREE vg_mymusic

mkfs.ext4 /dev/mapper/vg_mymusic-lv_mymusic

Edit fstab & mount to /mymusic

smbclient -L //media1

mount -t cifs //media1/media /mnt -o user=paul

cp -rf /mnt/* /mymusic

I have Madsonic runing on 4040 & 4443 (sssl) so open the firewall
Set FQDN & hostname in hosts or DNS.

Add a user to run it and make sure it can write to the music directory

useradd madsonic

chmod -R 770 /mymusic

chgrp -R madsonic /mymusic

Install & change options in /etc/sysconfig/madsonic

yum localinstall 20141017_madsonic-5.1.5200.rpm

MADSONIC_ARGS="--https-port=4443"

MADSONIC_USER=madsonic

Log into web interface and configure some options:

• Change admin password
• Set the media folder
• Music /mymusic Index 1(all) Music-Artists Music Enabled
• Scan the media folder
• Update last.fm - artist cover sync & artist summary sync
• Disable the guest user
• I just have these icons available to normal users
• Home, Artist, Playing, Starred, Genre, Random, Settings, Playlists, Playlists Editor
• When creating users make sure you check "User is allowed to use last.fm feature" otherwise they won't see artist details.
• I change the welcome message: 

__Welcome to Paul's Music Library!__
\\ \\
Play at your peril.....

Update the SSL cert by replacing subsonic.keystore in /usr/share/madsonic/madsonic-booter.jar
It's the same file that was used on the Windows version.

It appears that some of my wma files have the album cover art embedded into the audio file(??!!) This confuses ffmpeg when Madsonic attempts to convert them to mp3 to play. This can be fixed by updating the transcode options Madsonic passes to ffmpeg. This forces ffmpeg to transcode the first audio track rather than just the first track (which is the cover art)

• For the audio->mp3 options change -map 0:0 to -map 0:a:0

Tip: If you're having problems playing some files - this is how I fixed my wma file problem

• Turn the logging level up to debug (In general options Logfile logging level) to see what options are being passed to ffmpeg. Log file is /var/madsonic/madsonic.log
• Pass these options manually to ffmpeg with debug logging flag to get extra detail
• Eg: /var/madsonic/transcode/ffmpeg -i myfile.wma -map 0:0 -b:a 128k -v 0 -f mp3 -loglevel debug /tmp/myoutputfile.mp3

Hypervisor Build on MicroServer - Part 2

So I'm assuming you've read the first part and also set up the RAID 5 array

We're going to be using logical volumes & LVM on our RAID 5 array for the storage of the virtual guests

# pvcreate /dev/md0

Physical volume "/dev/md0" successfully created

# vgcreate vg_guests /dev/md0

Volume group "vg_guests" successfully created

Also create a volume to hold the ISOs for building the guests

# lvcreate -L 25G vg_guests -n lv_isos

# mkfs.ext4 /dev/vg_guests/lv_isos

Auto-mount it in /etc/fstab

/dev/mapper/vg_guests-lv_isos /isos ext4 defaults 0 2

Mount and set permissions so 'paul' can upload ISOs directly to it

# mount /isos

# chmod 755 /isos

# chgrp paul /isos

Finally upload some ISOs...

Using virt-manager, connect to the local QEMU

• Remove the virtual network created inside the Virtual Network tab ; we'll be using my home lan for the guests
• Add a storage pool for the ISOs so they are available to the guests for mounting
• Name: ISOs
• Type: dir Filesystem Directory
• Target Path: /isos
• Add a storage pool for the RAID 5 LVM array to hold the virtual guest storage
• Name: Guests
• Type: logical: LVM Volume Group
• Target Path: /dev/vg_guests
• Source Name: vg_guests

So now we're set up, here's a run through of creating a typical guest:

Before getting into virt-manager create a logical volume for the storage, eg:

# lvcreate -L 10G vg_guests -n lv_myguestname

Now from within virt-manager:

• Give it a name
• Use Local install media / Use ISO image / Select an ISO from your ISO storage pool
• Set OS type & version, set applicable RAM & CPU
• Select managed or other existing storage & pick your newly created logical volume from the Guests storage pool.
• You'll notice no networking is available. This is fine just make sure to tick the 'Customize configuration before install' box and this can be added next.
• Add Hardware and select Network -  Host device eth0: macvtap
• Select the newly created NIC and make sure source mode is set to Bridge otherwise the interface will not work.

Now hit 'Begin Installation' and install your lovely new guest.

Once built, if it's a Linux guest, configure the virtual serial port as detailed here so console access is via the command line rather than virt-manager.

A point to note about macvtap interfaces. They do not allow guest to hypervisor communication. Guest to guest is fine as is guest to rest of LAN. If you need to communicate with a guest from the hypervisor or vice versa you will need to create bridged interfaces and use those.

Configuring a RAID 5 array using software RAID

Here's how to create a 3 disk RAID 5 array using software RAID on Linux.
To make it slightly more interesting I'm going to create it initially using only 2 disks and then add the afterwards. Why? Because I can.

I'm doing this on my MicroServer. The RAID array will hold the virtual guests and the ISO storage pool.

First partition the disks and make sure they partitions are aligned:

# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) align-check optimal 1
1 aligned
(parted) p
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary raid

Tip - here's how to check the disks are ready to go

mdadm -E /dev/sd[bc]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)

Tip - if these aren't brand new disks check they haven't got any md superblocks already present. If they have zero the superblock (see later in post)

# mdadm -E /dev/sd[bc]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

 
Here's the magic. How to create the array, note the use of the 'missing' parameter for the 3rd disk.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Tip - how to check the array status

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdb1[0]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 25 19:54:54 2015
Raid Level : raid5
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Mon May 25 19:54:54 2015
State : active, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : einstein.at.home:0 (local to host einstein.at.home)
UUID : cc117a2a:439506c1:429d86cf:35514c71
Events : 0

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 0 0 4 removed

Finally save the configuration

mdadm --detail --scan --verbose >> /etc/mdadm.conf

So now let's add the missing disk
First clone the partition table from one of the existing disks

sfdisk -d /dev/sdb | sfdisk /dev/sdd --force

(For completeness zero the superblock as shown above, although not technically necessary on a new disk of course)
Now add the disk to the array:

mdadm --add /dev/md0 /dev/sdd1

The array will now resilver. This will takes hours. You can check /proc/mdstat for progress.
Don't forget to update mdadm.conf as shown above.

Tip - You can speed up resilvering by increasing these kernel parameters

echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 16384 > /sys/block/md0/md/stripe_cache_size

 Finally if you want to destroy an array

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --zero-superblock /dev/sdb
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd

Hypervisor Build on MicroServer

Here's the build details of the hypervisor on my MicroServer.

Making use of the internal USB socket to install Centos 6.6 on an 8GB USB stick and using all the hard drives as guest storage. There are 3 hard drives which will be configured as RAID 5 using Linux software RAID.

Use the lovely ILO to virtual mount and boot the Centos 6.6 ISO and the remote console to do the install. A big advantage over my first MicroServer.

Tip - quick way to erase a USB stick in Windows - use diskpart

DISKPART> list disk
DISKPART> select disk 2
DISKPART> clean

Perform a minimal install to the USB stick.
Tip - to avoid having to update after the install, use an install URL and add an additional repository for the updates. It'll all be done in one go. Here's the ones I use:
Install URL: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/os/x86_64/
Additional repo: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/updates/x86_64/

Install additional packages:
xauth (for X window virt-manager to work)
ntp (time sync)
parted (disk partitioning as fdisk is deprecated and doesn't like new large disks)
acpid (so you can do a graceful shutdown with the power button)
hp-ams (HP's agentless management service, get from the HP website. Integrates into ILO)

Configure and enable firewall and allow only ssh inbound (/etc/sysconfig/iptables)

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Configure and enable ntp
Create a user id and block root access via ssh

Install Virtualisation

yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"
yum install dejavu-lgc-sans-fonts

Allow 'paul' to run virt-manager and manage the local qemu
Create /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

                [Remote libvirt SSH access]

       Identity=unix-group:paul

       Action=org.libvirt.unix.manage

       ResultAny=yes

       ResultInactive=yes

       ResultActive=yes

 

See the next post for details of how to configure the RAID 5 disks and configure KVM to use them.