Non-root access to system logs

 Access to some system logs in /var/log is restricted to root only. 

Here's a handy way to grant access read-only access to all system logs without all that mucking around with sudo and file permissions.

• Install bindfs from the EPEL repo.
• Create a group (eg: logs) and add all users to it who need logfile access
• Create a mountpoint for the read-only view of /var/log (eg: /log)
• Create a bind mount in /etc/fstab
• /var/log  /log  fuse.bindfs     ro,force-group=logs,perms=g+rD  0 0
• Mount /log

System logs can now be viewed in /log by any user who is a member of the logs group.

This works because bindfs forces group read access on all files in the bind mount  

User session messages in the system log on Centos 7

Here's how to get rid of those annoying systemd messages that flood /var/log/messages on Centos 7.

Filter them out with an rsyslog filter script in /etc/rsyslog.d -

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

Then restart rsyslog

systemctl restart rsyslog

Shamelessly stolen from Red Hat here

Useful commands to check SSL certs

Here's some useful stuff to examine SSL certs.
I've pulled some of this from here

Check a key

• openssl rsa -in key.file -check
• openssl ec -in key.file (For an EC cert)

Check a cert

• openssl x509 -in cert.file -text -noout

 Test an SSL connection

• openssl s_client -connect some.site.com:443
• openssl s_client -connect some.site.com:443 -servername some.site.com
  (If website uses SNI)

Centos 7 - systemd messages flooding logs

The systemd process floods /var/log/messages with notifications whenever a user logs in or a cron job runs.

Mar 27 03:40:01 einstein systemd: Created slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Starting user-0.slice.
Mar 27 03:40:01 einstein systemd: Started Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Starting Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Removed slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Stopping user-0.slice.


Given sar runs every 10 minutes there is a lot of crud in the messages file.

You can create a filter for rsyslogd to tell it to junk the messages:

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Removed slice user-" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

systemctl restart rsyslog.service 


I take no credit for this, thanks to this article on the excellent Redhat Knowledgebase.
(I slightly adapted the solution to include the Removed & Stopping messages too) 

Moving up to Centos 7

So I finally upgraded my entire setup to Centos 7 (7.3 to be precise).
Here are some useful notes

By default I do a minimal installation and add what I need. A few extra packages I always add:
epel-release, acpid, net-tools, bash-completion, bash-completion-extras, xauth

ntpd is replaced with chrony. /etc/chrony.conf is the config file and use chronyc sourcestats to see what's going on (equivalent of ntpq --peers)

systemd replaces init.d startup, here are some useful commands. (Because you've installed bash-completion you can hit tab to figure out the service names)

• systemctl start/stop/status <service> (equiv of service start/stop/status)
• systemctl enable/disable <service> (equiv of chkconfig off/on)
• systemctl daemon-reload (if you edit services will reload configs)
• /usr/lib/systemd/system is the dir containing the service definitions
• systemctl list-units
• systemctl list-unit-files (show state of all services, including those that have failed)

firewalld replaces iptables as the default firewall, here's some helpful stuff:

• By default the public profile is active, this is fine add necessary ports to this
• For neatness, if an app needs multiple ports create a service definition and let that through
• New services are created in /etc/firewalld/services
• firewall-cmd --list-all (show current config)
• firewall-cmd --list-services (show only services allowed through firewall)
• firewall-cmd --info-service <service> (show ports in a service)
• firewall-cmd --permanent --add-service=<new service> (add service to config)
• systemctl restart firewalld.service

New default disk format is xfs. Be aware xfs partitions can be grown but not shrunk but the metadata takes less space so are more efficient.

You no longer need to specify -cu options to fdisk, they are now default.

Finally a couple of notes just for the hypervisor:

Install the HP stuff; hp-ams, hp-health & hponcfg

Configure the libvirt-guests service to ensure guests startup & shutdown when the hypervisor does. Ensure the ON_SHUTDOWN=shutdown option is set otherwise guests will suspend which takes an age as it suspends to the USB stick.

rngd is broken and won't start but is easily fixable. Thanks to this website for showing me how.
In essence, the daemon needs "-r /dev/urandom" to tell it to use the correct random device.

When creating your raid configuration in mdadm.conf add MAILADDR <email> as an option to allow mdadm daemon to email you when there is a disk problem. This will also stop the mdmonitor service from throwing a warning on startup.

To import the virtual machines it's better to create them as brand new guests pointing at the original lvm disks rather than importing the xml configs. This will ensure QEMU configures them optimally for Centos 7. The new virt-manager is improved and makes this easy.

Backing Up My Hypervisor - the return

Following on from my previous post on this my hypervisor has been upgraded to Centos 7.3.

REAR is still working well for the backups, here's a few optimisations worth noting....

My new /etc/rear/local.conf:

TMPDIR="/backup/tmp"
export TMPDIR
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR="/backup/host"
BACKUP=NETFS
BACKUP_URL=file:///backup/host
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

The TMPDIR makes REAR use my backup disk rather than /tmp on the very slow USB stick for creating the backup before moving it to /backup/host/einstein

OUTPUT_URL & ISO_DIR also stop REAR creating the rescue ISO on the USB stick and then copying it over. It's created direct into /backup/host

Recovery is as before. Don't be put off by these warnings, it will still work.

initramfs creation for Kernel 3.10.0-514.10.2.el7.x86_64 failed

You also wont need to recreate the ISO mount point or modify fstab. Everything just works.

Regarding recovery to a larger disk. A simpler solution is to just edit the disk size line in /var/lib/rear/layout/disklayout.conf with the size of the new disk. This will make REAR think the disk matches the original and it will restore without further modification. It will only use the original partition table so the extra space is wasted but for these purposes that's good enough.

disk /dev/sdc 7864320000 msdos

(A 16GB stick would be 16018046976)

Backing Up My Hypervisor

So following on from my last post on Backing Up My Virtual Guests here's a long overdue post on how I'm backing up my hypervisor.

The hypervisor is KVM running on a USB stick with Centos 6.8 and I'm making use of Relax and Recover, which the good folks at the upstream vendor have chosen to add into version 6.8.

Setup is straight forward, install the rear rpm and edit the configuration file /etc/rear/local.conf

OUTPUT=ISO
OUTPUT_URL=file:///backup
BACKUP=NETFS
BACKUP_URL=file:///backup
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

So this will create a backup of the OS installed on the USB stick to my backup drive in /backup/einstein. It won't touch the RAID volume with the LVMs containing my guests but their configuration will be saved as part of backup so they will be available again on restore.
The exclude options stop it backing up some temporary areas to save space and also the isos directory as that's on the RAID volume.

The output will be an  ISO (/backup/einstein/rear-einstein.iso) which can be used to boot the machine for restoration and a large tar ball in the same directory containing the files.

And that's it.
I additionally removed the daily cron job in /etc/cron.d/rear and replaced with the following to perform a backup twice a month as this is sufficient for my needs and matches what I do for my guests.

30 3 2,16 * *   /usr/sbin/rear mkbackup > /dev/null 2>&1

So how do I restore then?

The USB stick has gone pop and if I've put another same sized one in then restoration is easy.

Boot off the ISO, select the Recover einstein.at.home option, mount up the backup drive and run the recover command.

# mkdir /backup

# mount /dev/sdd1 /backup

# rear recover

Once completed (and it will take quite a while!!) recreate the isos mount point (mkdir /mnt/local/isos) and remove any erroneous blank lines that may have got inserted into /etc/fstab (I don't know why this happens but it will generate warnings during boot if you don't fix).

Reboot and bingo you have a working system again.

I used a larger USB stick, it wont recover?

Yes it will, you just need to do a little bit more work. Rear can't find an identical disk to recover to so asks whether you want to map the recovery to another disk. If you pick option 6 (do not map disks) and then option 2 (edit disk layout) you can tell it to force partition the new stick the same way as the old one by uncommenting these lines.

• disk /dev/sde 7864320000 msdos
• part /dev/sde 524288000 1048576 primary boot /dev/sde1
• part /dev/sde 7338983424 525336576 primary lvm /dev/sde2
• lvmdev /dev/vg_einstein /dev/sde2 EC1aq1-z2Sv-8ARK-IrvJ-cGVv-eLFp-JvwXLe 14333952
• lvmgrp /dev/vg_einstein 4096 1749 7163904
• lvmvol /dev/vg_einstein lv_root 1562 12795904
• lvmvol /dev/vg_einstein lv_swap 187 1531904
• fs /dev/mapper/vg_einstein-lv_root / ext4 uuid=c2c66e88-641a-484f-b574-3e0de6170d4b label= blocksize=4096 reserved_blocks=4% max_mounts=-1 check_interval=0d bytes_per_inode=16353 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• fs /dev/sde1 /boot ext4 uuid=c5e1b99c-e039-4f76-8947-67ec8e728f6a label= blocksize=1024 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=4095 default_mount_options=user_xattr,acl options=rw,relatime,barrier=1,data=ordered
• swap /dev/mapper/vg_einstein-lv_swap uuid=e4e02117-935e-43d3-a7fc-fae005581f68 label=

Now you could get clever here and amend the partition sizes to make use of the extra space but in my situation there isn't any point!!

Next select option 5 (continue recovery) twice and the recovery should then continue as above.

Backing Up Virtual Guests

My virtual guests use LVM for disk storage so we can use the snapshot feature to take live backups.

As these are disk image backups they are large but they are quick - a 10gb disk backup took 5 mins - as I have a huge dedicated backup disk for storing them this isn't a problem.
A note about the backup disk, I've formatted it as ext4 without LVM for maximum compatibility. Should the hardware fail I can pop it into another machine, mount up and go.

Here's an example:

First take a snapshot of the logical volume containing the virtual disk:

lvcreate -s --size=1G -n lv_dns_snap /dev/vg_guests/lv_dns

As the snapshot holds disk updates it needs to be large enough to hold all changes during the lifetime of the snapshot. 1GB is plenty for me. The lvs command will show you how much of the space the snapshot is consuming if you want to tune this.

Next take a copy of the snapshot with dd

dd if=/dev/vg_guests/lv_dns_snap of=/backup/dns-backup.dd bs=1M

Experiment with the block size parameter to see what gives best results for you - 1M worked well for me.

We're done so remove the snapshot

lvremove -f /dev/vg_guests/lv_dns_snap

Whilst we're at it we should also copy the guest xml config file from /etc/libvirt/qemu

Should you need to restore the backup, just shut the guest down and copy the disk image back

dd if=/backup/dns-backup.dd of=/dev/vg_guests/lv_dns

Installing Madsonic on Centos

Some notes on how I ported my Madsonic install from Windows to Centos.

I've put the OS on one disk and my music will live on the other. Set the 2nd disk up and copy my music across first:

pvcreate /dev/vdb1

vgcreate vg_mymusic /dev/vdb1

lvcreate -n lv_mymusic --extents 100%FREE vg_mymusic

mkfs.ext4 /dev/mapper/vg_mymusic-lv_mymusic

Edit fstab & mount to /mymusic

smbclient -L //media1

mount -t cifs //media1/media /mnt -o user=paul

cp -rf /mnt/* /mymusic

I have Madsonic runing on 4040 & 4443 (sssl) so open the firewall
Set FQDN & hostname in hosts or DNS.

Add a user to run it and make sure it can write to the music directory

useradd madsonic

chmod -R 770 /mymusic

chgrp -R madsonic /mymusic

Install & change options in /etc/sysconfig/madsonic

yum localinstall 20141017_madsonic-5.1.5200.rpm

MADSONIC_ARGS="--https-port=4443"

MADSONIC_USER=madsonic

Log into web interface and configure some options:

• Change admin password
• Set the media folder
• Music /mymusic Index 1(all) Music-Artists Music Enabled
• Scan the media folder
• Update last.fm - artist cover sync & artist summary sync
• Disable the guest user
• I just have these icons available to normal users
• Home, Artist, Playing, Starred, Genre, Random, Settings, Playlists, Playlists Editor
• When creating users make sure you check "User is allowed to use last.fm feature" otherwise they won't see artist details.
• I change the welcome message: 

__Welcome to Paul's Music Library!__
\\ \\
Play at your peril.....

Update the SSL cert by replacing subsonic.keystore in /usr/share/madsonic/madsonic-booter.jar
It's the same file that was used on the Windows version.

It appears that some of my wma files have the album cover art embedded into the audio file(??!!) This confuses ffmpeg when Madsonic attempts to convert them to mp3 to play. This can be fixed by updating the transcode options Madsonic passes to ffmpeg. This forces ffmpeg to transcode the first audio track rather than just the first track (which is the cover art)

• For the audio->mp3 options change -map 0:0 to -map 0:a:0

Tip: If you're having problems playing some files - this is how I fixed my wma file problem

• Turn the logging level up to debug (In general options Logfile logging level) to see what options are being passed to ffmpeg. Log file is /var/madsonic/madsonic.log
• Pass these options manually to ffmpeg with debug logging flag to get extra detail
• Eg: /var/madsonic/transcode/ffmpeg -i myfile.wma -map 0:0 -b:a 128k -v 0 -f mp3 -loglevel debug /tmp/myoutputfile.mp3

Hypervisor Build on MicroServer - Part 2

So I'm assuming you've read the first part and also set up the RAID 5 array

We're going to be using logical volumes & LVM on our RAID 5 array for the storage of the virtual guests

# pvcreate /dev/md0

Physical volume "/dev/md0" successfully created

# vgcreate vg_guests /dev/md0

Volume group "vg_guests" successfully created

Also create a volume to hold the ISOs for building the guests

# lvcreate -L 25G vg_guests -n lv_isos

# mkfs.ext4 /dev/vg_guests/lv_isos

Auto-mount it in /etc/fstab

/dev/mapper/vg_guests-lv_isos /isos ext4 defaults 0 2

Mount and set permissions so 'paul' can upload ISOs directly to it

# mount /isos

# chmod 755 /isos

# chgrp paul /isos

Finally upload some ISOs...

Using virt-manager, connect to the local QEMU

• Remove the virtual network created inside the Virtual Network tab ; we'll be using my home lan for the guests
• Add a storage pool for the ISOs so they are available to the guests for mounting
• Name: ISOs
• Type: dir Filesystem Directory
• Target Path: /isos
• Add a storage pool for the RAID 5 LVM array to hold the virtual guest storage
• Name: Guests
• Type: logical: LVM Volume Group
• Target Path: /dev/vg_guests
• Source Name: vg_guests

So now we're set up, here's a run through of creating a typical guest:

Before getting into virt-manager create a logical volume for the storage, eg:

# lvcreate -L 10G vg_guests -n lv_myguestname

Now from within virt-manager:

• Give it a name
• Use Local install media / Use ISO image / Select an ISO from your ISO storage pool
• Set OS type & version, set applicable RAM & CPU
• Select managed or other existing storage & pick your newly created logical volume from the Guests storage pool.
• You'll notice no networking is available. This is fine just make sure to tick the 'Customize configuration before install' box and this can be added next.
• Add Hardware and select Network -  Host device eth0: macvtap
• Select the newly created NIC and make sure source mode is set to Bridge otherwise the interface will not work.

Now hit 'Begin Installation' and install your lovely new guest.

Once built, if it's a Linux guest, configure the virtual serial port as detailed here so console access is via the command line rather than virt-manager.

A point to note about macvtap interfaces. They do not allow guest to hypervisor communication. Guest to guest is fine as is guest to rest of LAN. If you need to communicate with a guest from the hypervisor or vice versa you will need to create bridged interfaces and use those.

Configuring a RAID 5 array using software RAID

Here's how to create a 3 disk RAID 5 array using software RAID on Linux.
To make it slightly more interesting I'm going to create it initially using only 2 disks and then add the afterwards. Why? Because I can.

I'm doing this on my MicroServer. The RAID array will hold the virtual guests and the ISO storage pool.

First partition the disks and make sure they partitions are aligned:

# parted /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart primary ext4 0% 100%
(parted) set 1 raid on
(parted) align-check optimal 1
1 aligned
(parted) p
Model: ATA WDC WD20EZRX-00D (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 2000GB 2000GB primary raid

Tip - here's how to check the disks are ready to go

mdadm -E /dev/sd[bc]
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907026944 sectors at 2048 (type fd)

Tip - if these aren't brand new disks check they haven't got any md superblocks already present. If they have zero the superblock (see later in post)

# mdadm -E /dev/sd[bc]1
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdc1.

 
Here's the magic. How to create the array, note the use of the 'missing' parameter for the 3rd disk.

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 missing
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Tip - how to check the array status

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[1] sdb1[0]
3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 25 19:54:54 2015
Raid Level : raid5
Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Mon May 25 19:54:54 2015
State : active, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : einstein.at.home:0 (local to host einstein.at.home)
UUID : cc117a2a:439506c1:429d86cf:35514c71
Events : 0

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
4 0 0 4 removed

Finally save the configuration

mdadm --detail --scan --verbose >> /etc/mdadm.conf

So now let's add the missing disk
First clone the partition table from one of the existing disks

sfdisk -d /dev/sdb | sfdisk /dev/sdd --force

(For completeness zero the superblock as shown above, although not technically necessary on a new disk of course)
Now add the disk to the array:

mdadm --add /dev/md0 /dev/sdd1

The array will now resilver. This will takes hours. You can check /proc/mdstat for progress.
Don't forget to update mdadm.conf as shown above.

Tip - You can speed up resilvering by increasing these kernel parameters

echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 16384 > /sys/block/md0/md/stripe_cache_size

 Finally if you want to destroy an array

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --zero-superblock /dev/sdb
# mdadm --zero-superblock /dev/sdc
# mdadm --zero-superblock /dev/sdd

Hypervisor Build on MicroServer

Here's the build details of the hypervisor on my MicroServer.

Making use of the internal USB socket to install Centos 6.6 on an 8GB USB stick and using all the hard drives as guest storage. There are 3 hard drives which will be configured as RAID 5 using Linux software RAID.

Use the lovely ILO to virtual mount and boot the Centos 6.6 ISO and the remote console to do the install. A big advantage over my first MicroServer.

Tip - quick way to erase a USB stick in Windows - use diskpart

DISKPART> list disk
DISKPART> select disk 2
DISKPART> clean

Perform a minimal install to the USB stick.
Tip - to avoid having to update after the install, use an install URL and add an additional repository for the updates. It'll all be done in one go. Here's the ones I use:
Install URL: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/os/x86_64/
Additional repo: http://mirrors.ukfast.co.uk/sites/ftp.centos.org/6.6/updates/x86_64/

Install additional packages:
xauth (for X window virt-manager to work)
ntp (time sync)
parted (disk partitioning as fdisk is deprecated and doesn't like new large disks)
acpid (so you can do a graceful shutdown with the power button)
hp-ams (HP's agentless management service, get from the HP website. Integrates into ILO)

Configure and enable firewall and allow only ssh inbound (/etc/sysconfig/iptables)

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Configure and enable ntp
Create a user id and block root access via ssh

Install Virtualisation

yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"
yum install dejavu-lgc-sans-fonts

Allow 'paul' to run virt-manager and manage the local qemu
Create /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

                [Remote libvirt SSH access]

       Identity=unix-group:paul

       Action=org.libvirt.unix.manage

       ResultAny=yes

       ResultInactive=yes

       ResultActive=yes

 

See the next post for details of how to configure the RAID 5 disks and configure KVM to use them.

Using the virtual serial port with KVM

So you've installed some linux virtual guests on your shiney new KVM hypervisor, virt-manager is great for managing them but needs X11. Given the guests are linux wouldn't it be nice to be able to get to the console using CLI? Here's how...

For RHEL 6 guests:

Add the following lines to /etc/grub.conf below the splashimage line

serial --unit=0
terminal --timeout=5 serial console

Then append console=ttyS0 to your kernel line

 kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_rhel6-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_rhel6/lv_swap crashkernel=auto rd_LVM_LV=vg_rhel6/lv_root rd_NO_DM rhgb quiet console=ttyS0

For RHEL 7 guests:

Add the following lines to /etc/default/grub

GRUB_TERMINAL="serial"GRUB_SERIAL_COMMAND="serial"

Amend the GRUB_CMDLINE_LINUX to include console=ttyS0

GRUB_CMDLINE_LINUX="rd.lvm.lv=centos_tos7/swap rd.lvm.lv=centos_tos7/root rhgb quiet console=ttyS0"

Now regenerate the grub configuration

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the guest and you're done.

Access your lovely new text console using virsh console <guest name>

My New Virtual Baby

I bought one of these little HP MicroServers last year on a cashback promo deal. I've spent a few months fiddling around with it and researching how best to set it up. So here's my thoughts and details of my final configuration.

The server (model N54L)  cost £119 after cashback and came with 4GB memory and no disks.
I bought an extra 4GB on EBay (£44) and a couple of  Western Digital 2TB green hard drives (£57 each) from Amazon.

What do I want to do with it?

My intention was to configure it as a virtualisation host and have a guest running my Logitech Media Server, allowing me to get rid of an ageing Windows PC currently doing this. Further guests could be spun up for other functions I may want in the future.

Performance isn't a key factor for this given what I'm using it for and as I intend to run the guests on Linux the AMD processor seems fine. I did want resilient storage to guard against drive failure and a low power consumption given the machine will be on for long periods of time. The small physical footprint and low noise level is a bonus too.

What configuration to go with, how about vSphere?

First thoughts were to go with a VMware vSphere solution. A basic free version is available with limits on the physical hardware and missing lots of enterprise features, not a problem for this though. The great thing with this is it can run as a bare metal hypervisor and as the server has a bootable internal USB socket I installed it to a USB stick. This means all your disk space is available for your guests.

The MicroServer is a supported configuration and, having used vSphere and Workstation before, it was easy to set up. This would have been a great solution had it not been for a few gotchas.

• Didn't like having to use the vSphere client to configure it. It's Windows only and also deprecated and doesn't have all the available functionality in it. VMware want you to run vCenter which is WebGui based (yay!) but would consume precious hardware resources as it would run as an additional guest (boo!). Oh yes, and it costs money!!
• The hypervisor doesn't recognise the internal RAID controller (it's one of those nasty fake ones). Although you can join disks together within vSphere to make a large guest storage pool it can't be made resilient.

 

Hmm, okay - how about something Linux-like?

Being a bit of a RHEL fan I thought how about trying their virtualisation product RHEV, or rather the upstream open source version oVirt. This is based around Linux's kvm virtualisation technology.
Ok so this is a bit trickier to set up but I am familiar with it and am even certified in it (click if you don't believe me)
You can't really do this one on a USB stick so added an additional smaller hard drive I had lying around, installed Centos 6 and put oVirt on top of that. This would leave the other drives for guest storage.

So I quite liked this solution. Based around RHEL / kvm which is great (and totally free) and the administration is done through a web gui. Although I didn't try this bit out I would be able to make use of LVM and Linux RAID to create resilient and extendable storage for the guests. These both play very well with kvm and I have experience from work (and did I mention my awesome certifications haha!)

In the end though I abandoned this solution mainly due to the performance overhead of having to run the oVirt management engine and using an extra disk slot to host the hypervisor. Don't get me wrong oVirt gives you a lot but is really more designed for the enterprise environment. It's what the E stands for in RHEV after all.

And so the winner is?

In the end I've gone with a bit of a hybrid solution that takes the best of both worlds. There will be blog posts to follow on how I set this all up but here's the solution in a nutshell.

• Centos 6 minimal installation onto a USB stick, booted from the internal USB port.
• Add virtualisation packages to provide kvm virtualisation
• Use software raid (mdadm) to create a RAID 5 volume. With only 2 disks I hear you say? Yes, I added the 3rd disk later. Thank you Awesome William Matley for the idea.
• Use LVM on the RAID 5 volume to provision logical volumes to host each guest. These can be easily extended, cloned and backed up.
• Administration via ssh and the virsh CLI or virt-manager for the GUI (via X tunnelling). Remote administration via port forwarding on the router.

 So there you have, well done for reading this far. A pretty good virtualisation solution for not a lot of money even if I do say so myself!!