Hypervisor Build on MicroServer - Part 2

So I'm assuming you've read the first part and also set up the RAID 5 array

We're going to be using logical volumes & LVM on our RAID 5 array for the storage of the virtual guests

# pvcreate /dev/md0

Physical volume "/dev/md0" successfully created

# vgcreate vg_guests /dev/md0

Volume group "vg_guests" successfully created

Also create a volume to hold the ISOs for building the guests

# lvcreate -L 25G vg_guests -n lv_isos

# mkfs.ext4 /dev/vg_guests/lv_isos

Auto-mount it in /etc/fstab

/dev/mapper/vg_guests-lv_isos /isos ext4 defaults 0 2

Mount and set permissions so 'paul' can upload ISOs directly to it

# mount /isos

# chmod 755 /isos

# chgrp paul /isos

Finally upload some ISOs...

Using virt-manager, connect to the local QEMU

• Remove the virtual network created inside the Virtual Network tab ; we'll be using my home lan for the guests
• Add a storage pool for the ISOs so they are available to the guests for mounting
• Name: ISOs
• Type: dir Filesystem Directory
• Target Path: /isos
• Add a storage pool for the RAID 5 LVM array to hold the virtual guest storage
• Name: Guests
• Type: logical: LVM Volume Group
• Target Path: /dev/vg_guests
• Source Name: vg_guests

So now we're set up, here's a run through of creating a typical guest:

Before getting into virt-manager create a logical volume for the storage, eg:

# lvcreate -L 10G vg_guests -n lv_myguestname

Now from within virt-manager:

• Give it a name
• Use Local install media / Use ISO image / Select an ISO from your ISO storage pool
• Set OS type & version, set applicable RAM & CPU
• Select managed or other existing storage & pick your newly created logical volume from the Guests storage pool.
• You'll notice no networking is available. This is fine just make sure to tick the 'Customize configuration before install' box and this can be added next.
• Add Hardware and select Network -  Host device eth0: macvtap
• Select the newly created NIC and make sure source mode is set to Bridge otherwise the interface will not work.

Now hit 'Begin Installation' and install your lovely new guest.

Once built, if it's a Linux guest, configure the virtual serial port as detailed here so console access is via the command line rather than virt-manager.

A point to note about macvtap interfaces. They do not allow guest to hypervisor communication. Guest to guest is fine as is guest to rest of LAN. If you need to communicate with a guest from the hypervisor or vice versa you will need to create bridged interfaces and use those.

Configuring a RAID 5 array using software RAID

Here's how to create a 3 disk RAID 5 array using software RAID on Linux.
To make it slightly more interesting I'm going to create it initially using only 2 disks and then add the afterwards. Why? Because I can.

I'm doing this on my MicroServer. The RAID array will hold the virtual guests and the ISO storage pool.

First partition the disks and make sure they partitions are aligned:

# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) align-check optimal 1
1 aligned
(parted) p
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary raid

Tip - here's how to check the disks are ready to go

mdadm -E /dev/sd[bc]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)

Tip - if these aren't brand new disks check they haven't got any md superblocks already present. If they have zero the superblock (see later in post)

# mdadm -E /dev/sd[bc]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

 
Here's the magic. How to create the array, note the use of the 'missing' parameter for the 3rd disk.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Tip - how to check the array status

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdb1[0]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 25 19:54:54 2015
Raid Level : raid5
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Mon May 25 19:54:54 2015
State : active, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : einstein.at.home:0 (local to host einstein.at.home)
UUID : cc117a2a:439506c1:429d86cf:35514c71
Events : 0

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 0 0 4 removed

Finally save the configuration

mdadm --detail --scan --verbose >> /etc/mdadm.conf

So now let's add the missing disk
First clone the partition table from one of the existing disks

sfdisk -d /dev/sdb | sfdisk /dev/sdd --force

(For completeness zero the superblock as shown above, although not technically necessary on a new disk of course)
Now add the disk to the array:

mdadm --add /dev/md0 /dev/sdd1

The array will now resilver. This will takes hours. You can check /proc/mdstat for progress.
Don't forget to update mdadm.conf as shown above.

Tip - You can speed up resilvering by increasing these kernel parameters

echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 16384 > /sys/block/md0/md/stripe_cache_size

 Finally if you want to destroy an array

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --zero-superblock /dev/sdb
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd

Hypervisor Build on MicroServer

Here's the build details of the hypervisor on my MicroServer.

Making use of the internal USB socket to install Centos 6.6 on an 8GB USB stick and using all the hard drives as guest storage. There are 3 hard drives which will be configured as RAID 5 using Linux software RAID.

Use the lovely ILO to virtual mount and boot the Centos 6.6 ISO and the remote console to do the install. A big advantage over my first MicroServer.

Tip - quick way to erase a USB stick in Windows - use diskpart

DISKPART> list disk
DISKPART> select disk 2
DISKPART> clean

Perform a minimal install to the USB stick.
Tip - to avoid having to update after the install, use an install URL and add an additional repository for the updates. It'll all be done in one go. Here's the ones I use:
Install URL: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/os/x86_64/
Additional repo: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/updates/x86_64/

Install additional packages:
xauth (for X window virt-manager to work)
ntp (time sync)
parted (disk partitioning as fdisk is deprecated and doesn't like new large disks)
acpid (so you can do a graceful shutdown with the power button)
hp-ams (HP's agentless management service, get from the HP website. Integrates into ILO)

Configure and enable firewall and allow only ssh inbound (/etc/sysconfig/iptables)

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Configure and enable ntp
Create a user id and block root access via ssh

Install Virtualisation

yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"
yum install dejavu-lgc-sans-fonts

Allow 'paul' to run virt-manager and manage the local qemu
Create /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

                [Remote libvirt SSH access]

       Identity=unix-group:paul

       Action=org.libvirt.unix.manage

       ResultAny=yes

       ResultInactive=yes

       ResultActive=yes

 

See the next post for details of how to configure the RAID 5 disks and configure KVM to use them.