Backing Up Virtual Guests

My virtual guests use LVM for disk storage so we can use the snapshot feature to take live backups.

As these are disk image backups they are large but they are quick - a 10gb disk backup took 5 mins - as I have a huge dedicated backup disk for storing them this isn't a problem.
A note about the backup disk, I've formatted it as ext4 without LVM for maximum compatibility. Should the hardware fail I can pop it into another machine, mount up and go.

Here's an example:

First take a snapshot of the logical volume containing the virtual disk:

lvcreate -s --size=1G -n lv_dns_snap /dev/vg_guests/lv_dns

As the snapshot holds disk updates it needs to be large enough to hold all changes during the lifetime of the snapshot. 1GB is plenty for me. The lvs command will show you how much of the space the snapshot is consuming if you want to tune this.

Next take a copy of the snapshot with dd

dd if=/dev/vg_guests/lv_dns_snap of=/backup/dns-backup.dd bs=1M

Experiment with the block size parameter to see what gives best results for you - 1M worked well for me.

We're done so remove the snapshot

lvremove -f /dev/vg_guests/lv_dns_snap

Whilst we're at it we should also copy the guest xml config file from /etc/libvirt/qemu

Should you need to restore the backup, just shut the guest down and copy the disk image back

dd if=/backup/dns-backup.dd of=/dev/vg_guests/lv_dns

Installing Madsonic on Centos

Some notes on how I ported my Madsonic install from Windows to Centos.

I've put the OS on one disk and my music will live on the other. Set the 2nd disk up and copy my music across first:

pvcreate /dev/vdb1

vgcreate vg_mymusic /dev/vdb1

lvcreate -n lv_mymusic --extents 100%FREE vg_mymusic

mkfs.ext4 /dev/mapper/vg_mymusic-lv_mymusic

Edit fstab & mount to /mymusic

smbclient -L //media1

mount -t cifs //media1/media /mnt -o user=paul

cp -rf /mnt/* /mymusic

I have Madsonic runing on 4040 & 4443 (sssl) so open the firewall
Set FQDN & hostname in hosts or DNS.

Add a user to run it and make sure it can write to the music directory

useradd madsonic

chmod -R 770 /mymusic

chgrp -R madsonic /mymusic

Install & change options in /etc/sysconfig/madsonic

yum localinstall 20141017_madsonic-5.1.5200.rpm

MADSONIC_ARGS="--https-port=4443"

MADSONIC_USER=madsonic

Log into web interface and configure some options:

• Change admin password
• Set the media folder
• Music /mymusic Index 1(all) Music-Artists Music Enabled
• Scan the media folder
• Update last.fm - artist cover sync & artist summary sync
• Disable the guest user
• I just have these icons available to normal users
• Home, Artist, Playing, Starred, Genre, Random, Settings, Playlists, Playlists Editor
• When creating users make sure you check "User is allowed to use last.fm feature" otherwise they won't see artist details.
• I change the welcome message: 

__Welcome to Paul's Music Library!__
\\ \\
Play at your peril.....

Update the SSL cert by replacing subsonic.keystore in /usr/share/madsonic/madsonic-booter.jar
It's the same file that was used on the Windows version.

It appears that some of my wma files have the album cover art embedded into the audio file(??!!) This confuses ffmpeg when Madsonic attempts to convert them to mp3 to play. This can be fixed by updating the transcode options Madsonic passes to ffmpeg. This forces ffmpeg to transcode the first audio track rather than just the first track (which is the cover art)

• For the audio->mp3 options change -map 0:0 to -map 0:a:0

Tip: If you're having problems playing some files - this is how I fixed my wma file problem

• Turn the logging level up to debug (In general options Logfile logging level) to see what options are being passed to ffmpeg. Log file is /var/madsonic/madsonic.log
• Pass these options manually to ffmpeg with debug logging flag to get extra detail
• Eg: /var/madsonic/transcode/ffmpeg -i myfile.wma -map 0:0 -b:a 128k -v 0 -f mp3 -loglevel debug /tmp/myoutputfile.mp3

Hypervisor Build on MicroServer - Part 2

So I'm assuming you've read the first part and also set up the RAID 5 array

We're going to be using logical volumes & LVM on our RAID 5 array for the storage of the virtual guests

# pvcreate /dev/md0

Physical volume "/dev/md0" successfully created

# vgcreate vg_guests /dev/md0

Volume group "vg_guests" successfully created

Also create a volume to hold the ISOs for building the guests

# lvcreate -L 25G vg_guests -n lv_isos

# mkfs.ext4 /dev/vg_guests/lv_isos

Auto-mount it in /etc/fstab

/dev/mapper/vg_guests-lv_isos /isos ext4 defaults 0 2

Mount and set permissions so 'paul' can upload ISOs directly to it

# mount /isos

# chmod 755 /isos

# chgrp paul /isos

Finally upload some ISOs...

Using virt-manager, connect to the local QEMU

• Remove the virtual network created inside the Virtual Network tab ; we'll be using my home lan for the guests
• Add a storage pool for the ISOs so they are available to the guests for mounting
• Name: ISOs
• Type: dir Filesystem Directory
• Target Path: /isos
• Add a storage pool for the RAID 5 LVM array to hold the virtual guest storage
• Name: Guests
• Type: logical: LVM Volume Group
• Target Path: /dev/vg_guests
• Source Name: vg_guests

So now we're set up, here's a run through of creating a typical guest:

Before getting into virt-manager create a logical volume for the storage, eg:

# lvcreate -L 10G vg_guests -n lv_myguestname

Now from within virt-manager:

• Give it a name
• Use Local install media / Use ISO image / Select an ISO from your ISO storage pool
• Set OS type & version, set applicable RAM & CPU
• Select managed or other existing storage & pick your newly created logical volume from the Guests storage pool.
• You'll notice no networking is available. This is fine just make sure to tick the 'Customize configuration before install' box and this can be added next.
• Add Hardware and select Network -  Host device eth0: macvtap
• Select the newly created NIC and make sure source mode is set to Bridge otherwise the interface will not work.

Now hit 'Begin Installation' and install your lovely new guest.

Once built, if it's a Linux guest, configure the virtual serial port as detailed here so console access is via the command line rather than virt-manager.

A point to note about macvtap interfaces. They do not allow guest to hypervisor communication. Guest to guest is fine as is guest to rest of LAN. If you need to communicate with a guest from the hypervisor or vice versa you will need to create bridged interfaces and use those.

Configuring a RAID 5 array using software RAID

Here's how to create a 3 disk RAID 5 array using software RAID on Linux.
To make it slightly more interesting I'm going to create it initially using only 2 disks and then add the afterwards. Why? Because I can.

I'm doing this on my MicroServer. The RAID array will hold the virtual guests and the ISO storage pool.

First partition the disks and make sure they partitions are aligned:

# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) align-check optimal 1
1 aligned
(parted) p
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary raid

Tip - here's how to check the disks are ready to go

mdadm -E /dev/sd[bc]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)

Tip - if these aren't brand new disks check they haven't got any md superblocks already present. If they have zero the superblock (see later in post)

# mdadm -E /dev/sd[bc]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

 
Here's the magic. How to create the array, note the use of the 'missing' parameter for the 3rd disk.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Tip - how to check the array status

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdb1[0]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 25 19:54:54 2015
Raid Level : raid5
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Mon May 25 19:54:54 2015
State : active, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : einstein.at.home:0 (local to host einstein.at.home)
UUID : cc117a2a:439506c1:429d86cf:35514c71
Events : 0

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 0 0 4 removed

Finally save the configuration

mdadm --detail --scan --verbose >> /etc/mdadm.conf

So now let's add the missing disk
First clone the partition table from one of the existing disks

sfdisk -d /dev/sdb | sfdisk /dev/sdd --force

(For completeness zero the superblock as shown above, although not technically necessary on a new disk of course)
Now add the disk to the array:

mdadm --add /dev/md0 /dev/sdd1

The array will now resilver. This will takes hours. You can check /proc/mdstat for progress.
Don't forget to update mdadm.conf as shown above.

Tip - You can speed up resilvering by increasing these kernel parameters

echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 16384 > /sys/block/md0/md/stripe_cache_size

 Finally if you want to destroy an array

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --zero-superblock /dev/sdb
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd

Hypervisor Build on MicroServer

Here's the build details of the hypervisor on my MicroServer.

Making use of the internal USB socket to install Centos 6.6 on an 8GB USB stick and using all the hard drives as guest storage. There are 3 hard drives which will be configured as RAID 5 using Linux software RAID.

Use the lovely ILO to virtual mount and boot the Centos 6.6 ISO and the remote console to do the install. A big advantage over my first MicroServer.

Tip - quick way to erase a USB stick in Windows - use diskpart

DISKPART> list disk
DISKPART> select disk 2
DISKPART> clean

Perform a minimal install to the USB stick.
Tip - to avoid having to update after the install, use an install URL and add an additional repository for the updates. It'll all be done in one go. Here's the ones I use:
Install URL: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/os/x86_64/
Additional repo: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/updates/x86_64/

Install additional packages:
xauth (for X window virt-manager to work)
ntp (time sync)
parted (disk partitioning as fdisk is deprecated and doesn't like new large disks)
acpid (so you can do a graceful shutdown with the power button)
hp-ams (HP's agentless management service, get from the HP website. Integrates into ILO)

Configure and enable firewall and allow only ssh inbound (/etc/sysconfig/iptables)

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Configure and enable ntp
Create a user id and block root access via ssh

Install Virtualisation

yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"
yum install dejavu-lgc-sans-fonts

Allow 'paul' to run virt-manager and manage the local qemu
Create /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

                [Remote libvirt SSH access]

       Identity=unix-group:paul

       Action=org.libvirt.unix.manage

       ResultAny=yes

       ResultInactive=yes

       ResultActive=yes

 

See the next post for details of how to configure the RAID 5 disks and configure KVM to use them.

My Virtual Baby Gets A Sibling

You may have seen my earlier post about how I'm finally getting around to doing something with my HP Microserver.

Typical!! No sooner do I start working on it than another great deal comes out for the newer Gen 8 model. Priced at £179 with an £80 cashback promo it was to good a deal to miss. So for the princely sum of £99 the virtual baby now looks like this.

I've transplanted the disks and memory from the old model into this one and will be configuring it up as previously described.

So why did I go for this newer model?

• Faster Intel processor rather than AMD
• Built in HP ILO for remote management.
• Faster SATA controller (6Gb rather than 3Gb albeit only on first 2 disks)
• It looks sexier, gotta love that blue led strip at the bottom
• Because I can!!

There are some upgraded features; USB 3 sockets, 2 NICs, easier access to internals but they're less important to me at the moment.

So did I really need to do this? For what I'm going to be using it for, of course not! But for the price how could I say no.

Using the virtual serial port with KVM

So you've installed some linux virtual guests on your shiney new KVM hypervisor, virt-manager is great for managing them but needs X11. Given the guests are linux wouldn't it be nice to be able to get to the console using CLI? Here's how...

For RHEL 6 guests:

Add the following lines to /etc/grub.conf below the splashimage line

serial --unit=0
terminal --timeout=5 serial console

Then append console=ttyS0 to your kernel line

 kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_rhel6-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_rhel6/lv_swap crashkernel=auto rd_LVM_LV=vg_rhel6/lv_root rd_NO_DM rhgb quiet console=ttyS0

For RHEL 7 guests:

Add the following lines to /etc/default/grub

GRUB_TERMINAL="serial"GRUB_SERIAL_COMMAND="serial"

Amend the GRUB_CMDLINE_LINUX to include console=ttyS0

GRUB_CMDLINE_LINUX="rd.lvm.lv=centos_tos7/swap rd.lvm.lv=centos_tos7/root rhgb quiet console=ttyS0"

Now regenerate the grub configuration

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the guest and you're done.

Access your lovely new text console using virsh console <guest name>

My New Virtual Baby

I bought one of these little HP MicroServers last year on a cashback promo deal. I've spent a few months fiddling around with it and researching how best to set it up. So here's my thoughts and details of my final configuration.

The server (model N54L)  cost £119 after cashback and came with 4GB memory and no disks.
I bought an extra 4GB on EBay (£44) and a couple of  Western Digital 2TB green hard drives (£57 each) from Amazon.

What do I want to do with it?

My intention was to configure it as a virtualisation host and have a guest running my Logitech Media Server, allowing me to get rid of an ageing Windows PC currently doing this. Further guests could be spun up for other functions I may want in the future.

Performance isn't a key factor for this given what I'm using it for and as I intend to run the guests on Linux the AMD processor seems fine. I did want resilient storage to guard against drive failure and a low power consumption given the machine will be on for long periods of time. The small physical footprint and low noise level is a bonus too.

What configuration to go with, how about vSphere?

First thoughts were to go with a VMware vSphere solution. A basic free version is available with limits on the physical hardware and missing lots of enterprise features, not a problem for this though. The great thing with this is it can run as a bare metal hypervisor and as the server has a bootable internal USB socket I installed it to a USB stick. This means all your disk space is available for your guests.

The MicroServer is a supported configuration and, having used vSphere and Workstation before, it was easy to set up. This would have been a great solution had it not been for a few gotchas.

• Didn't like having to use the vSphere client to configure it. It's Windows only and also deprecated and doesn't have all the available functionality in it. VMware want you to run vCenter which is WebGui based (yay!) but would consume precious hardware resources as it would run as an additional guest (boo!). Oh yes, and it costs money!!
• The hypervisor doesn't recognise the internal RAID controller (it's one of those nasty fake ones). Although you can join disks together within vSphere to make a large guest storage pool it can't be made resilient.

 

Hmm, okay - how about something Linux-like?

Being a bit of a RHEL fan I thought how about trying their virtualisation product RHEV, or rather the upstream open source version oVirt. This is based around Linux's kvm virtualisation technology.
Ok so this is a bit trickier to set up but I am familiar with it and am even certified in it (click if you don't believe me)
You can't really do this one on a USB stick so added an additional smaller hard drive I had lying around, installed Centos 6 and put oVirt on top of that. This would leave the other drives for guest storage.

So I quite liked this solution. Based around RHEL / kvm which is great (and totally free) and the administration is done through a web gui. Although I didn't try this bit out I would be able to make use of LVM and Linux RAID to create resilient and extendable storage for the guests. These both play very well with kvm and I have experience from work (and did I mention my awesome certifications haha!)

In the end though I abandoned this solution mainly due to the performance overhead of having to run the oVirt management engine and using an extra disk slot to host the hypervisor. Don't get me wrong oVirt gives you a lot but is really more designed for the enterprise environment. It's what the E stands for in RHEV after all.

And so the winner is?

In the end I've gone with a bit of a hybrid solution that takes the best of both worlds. There will be blog posts to follow on how I set this all up but here's the solution in a nutshell.

• Centos 6 minimal installation onto a USB stick, booted from the internal USB port.
• Add virtualisation packages to provide kvm virtualisation
• Use software raid (mdadm) to create a RAID 5 volume. With only 2 disks I hear you say? Yes, I added the 3rd disk later. Thank you Awesome William Matley for the idea.
• Use LVM on the RAID 5 volume to provision logical volumes to host each guest. These can be easily extended, cloned and backed up.
• Administration via ssh and the virsh CLI or virt-manager for the GUI (via X tunnelling). Remote administration via port forwarding on the router.

 So there you have, well done for reading this far. A pretty good virtualisation solution for not a lot of money even if I do say so myself!!