DHCP Failover on RHEL 7

As always, i am not the authority on this subject; however, I have successfully added “failover” to our existing DHCP server in which the OS had been replaced several times while simply copying the dhcpd.conf over each time.

Configuring a failover DHCP is essentially not difficult. However, if you are in an “Enterprise” or “Corporate” environment (i.e. multiple subnets), then your router will require an additional “ip-helper” for each subnet. You or your network engineer will need to perform this task for the following system to work. In our case we simply added a secondary ip helper-address <IP> to each subnet (VLAN) in our hardware router.

Prerequisites: 0) properly configured router. 1) dhcpd running and configured properly. 2) EPEL repo installed. 3) ssh-key passwordless logins configured between the two DHCP servers. 4) Time is synchronized on servers (via ntpd or vm-tools’ options)

I reviewed the following sources for this process:
http://blog.whatgeek.com.pt/2012/03/dhcp-failover-load-balancing-and-synchronization-centos-6/
https://kb.isc.org/article/AA-00502/0/A-Basic-Guide-to-Configuring-DHCP-Failover.html
https://www.howtoforge.com/how-to-set-up-dhcp-failover-on-centos5.1
http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/
http://linux.die.net/man/5/incrontab
http://www.lithodyne.net/docs/dhcp/dhcp-5.html

The first link above had the best idea of creating include files for the configuration. This allowed me to automate copying the dhcpd.conf file to the secondary server upon any changes.

Obviously, your IP scheme will be much different, please adjust accordingly. Also, this write-up may in-fact not apply to all configurations out there – You may consider this post just another resource for your research.

Let’s begin…

In addition to our existing RHEL 7 server running dhcpd, I have configured a second machine running the same. For now, the secondary dhcpd service is stopped.

In my case, I edited the primary server’s /etc/dhcp/dhcpd.conf to contain

include "/etc/dhcp/dhcpd.failover";

and to contain at least one pool declaration. In my case, because i was still testing things, in an existing subnet I commented out the existing range statement and added the pool statement just below with the same range and the required failover statement:

 subnet 10.20.0.0 netmask 255.255.0.0 {
    option broadcast-address 10.20.255.255;
    option routers 10.20.1.1;
    #range 10.20.20.1 10.20.22.254;
    pool {
        range 10.20.20.1 10.20.22.254;
        failover peer "dhcpfailover";
        }
    }

Again, note that at least one pool is required. I learned the hard way that without it, the dhcpd service will not start, leaving my network without a server for several minutes. If you are unsure where to put the include statement, just put it after your initial options and just before you first subnet.

You can either add a pool statement to each of your subnets at this point, or just do one for now for testing purposes. Each pool requires a failover peer ... statement for failover to actually work.

You may test your dhcp.conf file with the command dhcp -t -cf /etc/dhcp/dhcp.conf.

At this point, you can copy your primary /etc/dhcp/dhcpd.conf to your secondary server. We will ultimately script a mirroring process.  Just to re-iterate, this /etc/dhcp/dhcpd.conf contains include "/etc/dhcp/dhcpd.failover"; and one pool. This .conf file is copied identically to the secondary dhcp server.

Now, one of the most important parts is for the contents of the include files.  Each dhcp server will have a differing /etc/dhcp/dhcpd.failover file.

Create your primary server’s /etc/dhcp/dhcpd.failover to contain

# Failover specific configurations
failover peer "dhcpfailover" {
primary;
address 10.10.0.100;
port 647;
peer address 10.10.0.101;
peer port 647;
max-response-delay 60;
max-unacked-updates 10;
mclt 600;
split 128; #128 is balanced; use 255 if primary is 100% responsible until failure.
load balance max seconds 3;
}

and the secondary server’s /etc/dhcp/dhcpd.failover to contain

# Failover specific configurations
failover peer "dhcpfailover" {
secondary;
address 10.10.0.101;
port 647;
peer address 10.10.0.100;
peer port 647;
max-response-delay 60;
max-unacked-updates 10;
load balance max seconds 3;
}

obviously, where my primary DHCP is 10.10.0.100 and my secondary DHCP server IP is 10.10.0.101 ; Change yours accordingly.

You will also have to open the firewall to TCP port 647 on each server. In my case I chose to allow only from the specified IP sources.

At this point, you may start your secondary server’s dhcpd service with systemctl start dhcpd. If it starts properly without error, then it is safe to restart your primary server’s dhcpd service with systemctl restart dhcpd. You should test that it’s running properly at this point, and if not fix it promptly or reverse your changes and review and try again. You may also use commands such as journalctl -xn30 or systemctl -n30 status dhcpd to locate faults. You should also enable the dhcp service for auto-start with systemctl enable dhcpd.

In this system, your DHCP changes should only be applied to the primary server. The secondary server only exists for failover purposes.

Changes to the primary DHCP server do not by-default mirror to the secondary server, so we will automate this. For this process, we'll use root ssh; I have chosen to allow root ssh via ssh-key so that it can be automated with scripts. Of course I have firewalled my ssh ports to allow only certain certain IP ranges.

** Before proceeding, please note: a comment by "ZsZs" recommended replacing my incrond usage with a systemd built-in feature. I concur but have NOT yet tried it. Please refer to https://wiki.archlinux.org/index.php/rsync#Automated_backup_with_systemd_and_inotify for a better alternative. ...continuing...

I chose to utilize incrond to simplify this mirroring process. incrond utilizes the inotify-tools to watch a file (or directory) for changes to execute a specified command. This tool is not in the default RHEL repositories. To install it you will need the “Extra Packages for Enterprise Linux” (EPEL) which is quite easy to install.
For RHEL 7, installation is as follows:

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -i epel-release-latest-7.noarch.rpm

Afterward, install and enable incrond as follows:

yum -y install inotify-tools incron
systemctl enable incrond

First, let’s write a script to copy dhcp.conf to the secondary server and restart it’s service. Create a file /root/scripts/update-failover-server.sh to contain: (due to potential issues, use full command paths)

#!/bin/bash
/usr/bin/scp /etc/dhcp/dhcpd.conf root@10.10.0.101:/etc/dhcp/dhcpd.conf
/usr/bin/ssh root@10.10.0.101 '/usr/bin/systemctl restart dhcpd'
/usr/bin/systemctl restart incrond #CRITICAL ISSUE; one-time trigger and subsequent fail work-around

and be sure to mark it executable (chmod +x). Again, these are my IP’s, yours will vary. Most importantly, note that I have already enabled passwordless login between the servers with ssh-keys. This automation will NOT work without such. You may in fact want to to test your script’s success by running it manually first.

We can now configure a “watch” for any changes to the dhcpd.conf file. Use the command EDITOR=nano incrontab -e to edit the incron-file with syntax FILE TRIGGERLIST COMMAND [OPTION] (refer to the links referenced above):

/etc/dhcp/dhcpd.conf IN_MODIFY,IN_ATTRIB,IN_CREATE /root/scripts/update-failover-server.sh

Here, I’m trying to cover any modification to dhcpd.conf. Editors vs. Webmin modify the file differently, so this should cover both instances.

We can now start the incrond services with the command systemctl start incrond.

At this point, both servers should be running and able to serve IP addresses. You should verify such.

Now, you may test that any changes to your primary dhcpd.conf propagate to the secondary server. Go ahead and modify your primary /etc/dhcp/dhcpd.conf by your preferred method and analyze what happens.

As you find that everything is a success, you may add pool statements to each subnet while moving the range statements within the pool.

---
As Always, Good Luck! 

Please comment or tip me or use any/all of my affiliate links; Thank YOU!

You can thank me with bitcoin.    

SEO:
DHCP Failover
DHCP Failover Linux
DHCPd Failover
DHCPd Failover Linux
DHCP Server Failover
DHCP Server Failover Linux

95% Written with StackEdit.

vCenter Server Appliance (vCSA) 5.5 SSL certificate replacement and root password reset the hard and/or easy way. (or The vCSA is Dead, Long Live the vCSA)

It all started because my SSL certificate was expired and with recent updates to Chrome and Firefox we could not login. I found this perfect article (http://www.virtuallyghetto.com/2013/04/automating-ssl-certificate-regeneration.html) to regenerate the certificate via commandline, but i was locked out of the root account. I thought, as many do, that i had forgotten the root password, but it turns out that with v5.5, root passwords expire in 90 days and if you don't set an SMTP server and email address, you will never be notified. Furthermore after some time the account will be locked.

On the vCSA console you have an option to Login. I don't really know why, but somehow after failing some logins or semi-resetting the account, i found that if i mistyped the password 3 times, i would actually get the root prompt. Now if you cannot get the root prompt, luckily you can use this excellent method (http://www.virtuallyghetto.com/2013/09/how-to-recover-vcsa-55-from-expired.html) to get to the files also. If you do use the link's method, please note that you will need to mount/remount with read-write access which is not mentioned. hint: mount -o remount,rw /mnt

So in the following commands, you will see [/mnt]. What i mean by this is that if you use the link's method, then you need /mnt/path ; however, if were able to get the vCSA's root prompt, then simply use the /path (excluding /mnt)

Of course you should bakup any files you plan to change!

We will use vi because it's the editor built into the vCSA. Remember in vi you can i or a to insert or append to get into typing mode. Afterward, <esc>wq! to force save & quit or <esc>q! to abort and quit.

Firstly, i found that the password policy required a new, previously unused, password that had to meet complexity requirements. This sucked tremendously considering i really really wanted to re-use the password. If you wish to do the same try this, if not skip it.

change root password policy with vi [/mnt]/etc/pam.d/common-password

change

password        requisite       pam_cracklib.so dcredit=-1 ucredit=-1 lcredit=-1 ocredit=-1 minlen=8 difok=4 retry=3
password        required        pam_pwhistory.so enforce_for_root remember=4 retry=3

to

password        requisite       pam_cracklib.so dcredit=-1 ucredit=-1 lcredit=-1 ocredit=-1 minlen=8 difok=0 retry=0
password        required        pam_pwhistory.so enforce_for_root remember=0 retry=3

The above should set that the password does not need to have to be significantly different from prior passwords and not to spam the prompt requesting retries.

An alternative way might be to purge the password history file with rm [/mnt]/etc/security/opasswd which of course i did.

If you can get to the vCSA's root prompt, then you can change the root password with passwd

However, if you cannot, then potentially you can reuse your old password by reinstantiating the password hash from a backup shadow.* file. Revisit the above mentioned article (http://www.virtuallyghetto.com/2013/09/how-to-recover-vcsa-55-from-expired.html) as it explains the shadowfile.

Thusly, I was successful with restoring the default password of vmware by finding that [/mnt]/etc/shadow.UPDATE contained its hash. I copied the hash (installation specific i'm sure) into the existing [/mnt]/etc/shadow's root line and made sure the 5th field was nothing (::). it looked like such:

root:$2y$10$Gye6636Oxy/2yK01.7MW0ud8pSE90cEYr92kLSwDvJmULjmTPnu0O:16581:0:90:7:::

Once I had accomplished all this, i rebooted and was able to login with vmware. I reset my password with passwd at the root prompt. It complained it was too simple a password, but accepted it none-the-less.

Note: Through all this, I found that the actual client login Administrator@vSphere.local had never changed changed, so don't expect it to be the new one you just changed.

Futhermore, I followed this very fitting article (http://www.virtuallyghetto.com/2013/09/administrator-password-expiration-in.html) to completely disable root password expiration with chage -M -1 -E -1 root.

If you use VMWare Update manager (VUM), you may need to remmediate it's connection with http://kb.vmware.com/kb/2048210.

In addition to the resources linked above, the following were referenced during my adventure-less adventure.

• (http://www.cyberciti.biz/faq/rhel-fedora-centos-linux-password-quality-control/)
• (http://www.vladan.fr/how-to-change-the-default-password-policies-in-vsphere-5-5/)
• (http://www.vladan.fr/how-to-unlock-the-vmware-vcsa-root-password/)
• (http://www.itworld.com/article/2726217/endpoint-protection/how-to-enforce-password-complexity-on-linux.html)

My other recent posts also relate to this issue due to Mozilla, Google and Microsoft weak SSL deprecation policies.

• http://steronius.blogspot.com/2015/09/vcenter-chrome-45-errsslweakserverephem.html
• http://steronius.blogspot.com/2015/08/https-ssl-cipher-remediation-for.html

---
As Always, Good Luck!

Please comment or tip me or use any/all of my affiliate links; Thank YOU!

You can thank me with bitcoin.    

VMWare vCenter vSphere Web Client Chrome 45 ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY

VMWare vCenter vSphere Web Client + Chrome 45
Server has a weak ephemeral Diffie-Hellman public key
ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY
a.k.a Forward Secrecy

I patched this together rather quickly, but i think it's all here!  Today we had issue with Chrome 45 failing to connect to VMWare 5.1 vCenter vSphere Web Server (vSphere Web Client).  This is how I fixed it. (Also seems to work in Firefox.)

create new self-signed certificate (in linux):

openssl genrsa 2048 > rui.key
openssl req -new -key rui.key > rui.csr
#openssl x509 -in rui.cer -out rui.crt
openssl x509 -req -days 1825 -in rui.csr -signkey rui.key -out rui.crt
openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx

you must use testpassword if you retain the default tomcat keystorePass setting in the xml mentioned below.

backup and replace files in C:\Program Files\VMware\Infrastructure\vSphereWebClient\DMServer\config\ssl\ with the files just created.  *** my other vCenter did not have this folder. It was C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\ instead. (upgrade vs. fresh install?)

edit C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\config\tomcat-server.xml

was

        <Connector port="9443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="500" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" ciphers="SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_256_CBC_SHA, TLS_DH_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_128_CBC_SHA, TLS_DH_DSS_WITH_AES_128_CBC_SHA" keystoreFile="C:\ProgramData\vmware\vSphere Web Client\ssl\rui.pfx" keystorePass="testpassword" keystoreType="PKCS12"></Connector>

changed to (removed RC4 and DHE-only ciphers)

        <Connector port="9443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="500" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" ciphers="TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_256_CBC_SHA, TLS_DH_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DH_RSA_WITH_AES_128_CBC_SHA, TLS_DH_DSS_WITH_AES_128_CBC_SHA" keystoreFile="C:\ProgramData\vmware\vSphere Web Client\ssl\rui.pfx" keystorePass="testpassword" keystoreType="PKCS12"></Connector>

restart both vspherewebclientsvc and vctomcat services. Be patient, it will take a few minutes for the services to be ready to serve the web-pages correctly.

resources used:
http://www.vkernel.ro/blog/replacing-vmware-vcenter-server-default-self-signed-certificate
https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl
https://www.chromium.org/administrators/err_ssl_weak_server_ephemeral_dh_key

My other recent post (https SSL cipher remediation for webservers 2015) is also related due to Mozilla, Google and Microsoft weak SSL deprecation policies.

---
As Always, Good Luck! 
Please comment or tip me or use any/all of my affiliate links; Thank YOU!
If I saved your ass, you can thank me with bitcoin.