Using acme.sh to automate SSL certs

I'm using https://acme.sh/ to automate the creation and deployment of SSL certs from https://letsencrypt.org/ on my websites. I prefer this method to using certbot as it doesn't require any additional packages to be installed on the server as it is all done in script.

Some notes on how to create the certs and have them load automatically into Apache.
I'm not detailing how to install acme as that's straightforward and covered on the acme website.

I have the acme script installed on the web server as root as it makes copying the certs / keys to the appropriate directories easier.

I use the dns-01 challenge mechanism to issue certs as Cloudflare has an API that supports it.
This is a great way to do it as it means you don't need to copy content to the web servers to authenticate the cert issuing request.

Need the following in ~/.acme.sh/account.conf for it to work:

SAVED_CF_Key='<API key>'
SAVED_CF_Email='<email address>'

Issue the cert

acme.sh --issue --dns dns_cf -d foobar.durrant.me.uk

Add the cert / key locations to Apache
Note that the cer file contains both the site cert and the intermediate cert. Apache understands if you point both parameters at the same file which saves having to maintain two separate files.

SSLCertificateFile /etc/pki/tls/certs/foobar.durrant.me.uk.cer
SSLCertificateKeyFile /etc/pki/tls/private/foobar.durrant.me.uk.key
SSLCertificateChainFile /etc/pki/tls/certs/foobar.durrant.me.uk.cer

Install the certs

acme.sh --install-cert -d foobar.durrant.me.uk --key-file /etc/pki/tls/private/foobar.durrant.me.uk.key --fullchain-file /etc/pki/tls/certs/foobar.durrant.me.uk.cer --reloadcmd "systemctl restart httpd.service"

Set the permissions on the key & cert

chmod 600 /etc/pki/tls/private/foobar.durrant.me.uk.key
chmod 600 /etc/pki/tls/certs/foobar.durrant.me.uk.cer

 Certs will automatically be renewed and reinstalled every 2 months.

Centos 7 - systemd messages flooding logs

The systemd process floods /var/log/messages with notifications whenever a user logs in or a cron job runs.

Mar 27 03:40:01 einstein systemd: Created slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Starting user-0.slice.
Mar 27 03:40:01 einstein systemd: Started Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Starting Session 1317 of user root.
Mar 27 03:40:01 einstein systemd: Removed slice user-0.slice.
Mar 27 03:40:01 einstein systemd: Stopping user-0.slice.


Given sar runs every 10 minutes there is a lot of crud in the messages file.

You can create a filter for rsyslogd to tell it to junk the messages:

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Removed slice user-" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

systemctl restart rsyslog.service 


I take no credit for this, thanks to this article on the excellent Redhat Knowledgebase.
(I slightly adapted the solution to include the Removed & Stopping messages too) 

Moving up to Centos 7

So I finally upgraded my entire setup to Centos 7 (7.3 to be precise).
Here are some useful notes

By default I do a minimal installation and add what I need. A few extra packages I always add:
epel-release, acpid, net-tools, bash-completion, bash-completion-extras, xauth

ntpd is replaced with chrony. /etc/chrony.conf is the config file and use chronyc sourcestats to see what's going on (equivalent of ntpq --peers)

systemd replaces init.d startup, here are some useful commands. (Because you've installed bash-completion you can hit tab to figure out the service names)

• systemctl start/stop/status <service> (equiv of service start/stop/status)
• systemctl enable/disable <service> (equiv of chkconfig off/on)
• systemctl daemon-reload (if you edit services will reload configs)
• /usr/lib/systemd/system is the dir containing the service definitions
• systemctl list-units
• systemctl list-unit-files (show state of all services, including those that have failed)

firewalld replaces iptables as the default firewall, here's some helpful stuff:

• By default the public profile is active, this is fine add necessary ports to this
• For neatness, if an app needs multiple ports create a service definition and let that through
• New services are created in /etc/firewalld/services
• firewall-cmd --list-all (show current config)
• firewall-cmd --list-services (show only services allowed through firewall)
• firewall-cmd --info-service <service> (show ports in a service)
• firewall-cmd --permanent --add-service=<new service> (add service to config)
• systemctl restart firewalld.service

New default disk format is xfs. Be aware xfs partitions can be grown but not shrunk but the metadata takes less space so are more efficient.

You no longer need to specify -cu options to fdisk, they are now default.

Finally a couple of notes just for the hypervisor:

Install the HP stuff; hp-ams, hp-health & hponcfg

Configure the libvirt-guests service to ensure guests startup & shutdown when the hypervisor does. Ensure the ON_SHUTDOWN=shutdown option is set otherwise guests will suspend which takes an age as it suspends to the USB stick.

rngd is broken and won't start but is easily fixable. Thanks to this website for showing me how.
In essence, the daemon needs "-r /dev/urandom" to tell it to use the correct random device.

When creating your raid configuration in mdadm.conf add MAILADDR <email> as an option to allow mdadm daemon to email you when there is a disk problem. This will also stop the mdmonitor service from throwing a warning on startup.

To import the virtual machines it's better to create them as brand new guests pointing at the original lvm disks rather than importing the xml configs. This will ensure QEMU configures them optimally for Centos 7. The new virt-manager is improved and makes this easy.

Backing Up My Hypervisor - the return

Following on from my previous post on this my hypervisor has been upgraded to Centos 7.3.

REAR is still working well for the backups, here's a few optimisations worth noting....

My new /etc/rear/local.conf:

TMPDIR="/backup/tmp"
export TMPDIR
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR="/backup/host"
BACKUP=NETFS
BACKUP_URL=file:///backup/host
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
EXCLUDE_RECREATE=( "${EXCLUDE_RECREATE[@]}" "fs:/isos" )

The TMPDIR makes REAR use my backup disk rather than /tmp on the very slow USB stick for creating the backup before moving it to /backup/host/einstein

OUTPUT_URL & ISO_DIR also stop REAR creating the rescue ISO on the USB stick and then copying it over. It's created direct into /backup/host

Recovery is as before. Don't be put off by these warnings, it will still work.

initramfs creation for Kernel 3.10.0-514.10.2.el7.x86_64 failed

You also wont need to recreate the ISO mount point or modify fstab. Everything just works.

Regarding recovery to a larger disk. A simpler solution is to just edit the disk size line in /var/lib/rear/layout/disklayout.conf with the size of the new disk. This will make REAR think the disk matches the original and it will restore without further modification. It will only use the original partition table so the extra space is wasted but for these purposes that's good enough.

disk /dev/sdc 7864320000 msdos

(A 16GB stick would be 16018046976)