First, we start by understanding, what is a server cluster? A server cluster is a collection of servers that provide similar network services. They share an IP address/FQDN and the client hosts communicate with that IP/FQDN to obtain network services. The part to notice here is that multiple servers are used for auto redundancy and in cases, load balancing. Which simply means, even if one of the servers are down, the other servers would take over and the client will get the services from the backup servers. The whole process is automatic and with proper configuration, seamless. The client wouldn't even know that it's using a backup server.
For example, let us assume that server 1 and server 2 have IP address of IP1 and IP2 respectively. Both of them are configured as web servers. We have an IP address IP3 that can be resolved to www.testdomain.com. So, to create an auto redundant cluster, we configure the cluster in such manner that the servers share the virtual IP of IP3 between them so if anyone queries www.testdomain.com, he would resolve to IP3 and the request would be forwarded to the server cluster. Even if server1 is down, server 2 would be using the virtual IP of IP3 and would be able to provide services for www.testdomain.com.
Cluster configuration comes in many forms, and the complexity may differ from design to design. For example, the servers in the cluster may be configured to use an external common storage. like SAN or NAS. Again, there may be cases where each server has it's own storage and the configurations are identical on both servers.
We have 2 servers
- primary.example.inv (IP 192.168.1.1)
- secondary.example.inv (IP 192.168.1.2)
- Virtual IP: 192.168.1.100
- A FQDN services.example.inv which can be resolved to IP 192.168.1.100
First, we have to keep in mind that the hostnames are really important. We have to set the hostnames properly in the following locations-
NOTE: If there is any confusion about how to set the hostnames, please consult an earlier post
Then, we have to setup heartbeat in the server. CentOS 6 still does have heartbeat RPMs in their repositories, so yum install will not work just yet. After googling for a while, I found heartbeat here
Here's how we can download the heartbeat RPMs and setup some dependencies.
Great! We have installed heartbeat to our server :)
Next, it's time to configure heartbeat. All the heartbeat config files are stored in /etc/ha.d. We have to modify 3 files only. However, since the configuration files are not present yet, we will have to copy sample configuration files from /usr. Here's how it's done-
Time to edit to configuration files:
Different deadtime and initdead may be used for experimenting. For example, I tried with deadtime of only 10 seconds and initdead of only 20 seconds for testing. In case of live servers, the real environment would decide which values are appropriate.
In this file, we define the virtual IP address to be used by heartbeat.
NOTE: It should be kept in mind that we do NOT need to assign the IP to any interface. Heartbeat would manage requests coming towards the virtual IP i.e. 192.168.1.4.
First, we would set hostname and then setup heartbeat in the secondary server. The setup process is identical to the primary server.
It is very important that the configuration files of primary and secondary servers are identical. However, minor tuning may be necessary in some cases.
As we need identical configuration files, we would be copying the directory /etc/ha.d from primary server to the secondary server.
Time to edit to configuration files:
Since we are using unicast instead of broadcast, we have to specify which way heartbeat periodic packets should be sent
Because both of the servers have separate storage, we have to keep in mind that both servers are running independent instances of samba. But since we want to configure the second server as backup, we have to make sure that both samba servers have identical user database. This post can be consulted to see how samba users can be migrated.
First, we have to make sure that /etc/passwd, /etc/group and /etc/shadow are identical. I was unable to come up with an automatic method that could sync only the human users ( UID 500 and onwards). So, synchronizing these files must be done manually, which is described in the samba user migration post.
Then, we synchronize samba files in both machines
Now we are almost ready.
NOTE: It should be kept in mind that heartbeat would deal with starting/stopping any managed service. Any heartbeat managed service should NEVER be manually started/stopped because it would create problems in normal cluster operation. This is the reason why these services should also not be run during startup process.
It is always a good idea to analyze the log files. Heartbeat logs are stored in /var/log/ha-log. Useful information can also be found in /var/log/messages. The, /var/log/messages file should show which server is acting as the domain controller. So, the first thing to do is, to analyze the log files.
Then, since both the servers have the same user database, testing can be done by logging into the domain from windows clients. If a domain user can log into the domain using the primary server, the same user should also be able to login to the domain if the primary server is powered down (or heartbeat in primary server is stopped).
- I have SELinux disabled.
- In this configuration, heartbeat uses port UDP 694 and firewall must not block this port.
- The virtual IP is managed by heartbeat and does not need to be assigned to any device.
- The heartbeat managed services are started/stopped by heartbeat. They should not be run/stopped manually.
- The heartbeat managed services should not be automatically run at system startup.