Samba Domain Controller Cluster Using Heartbeat and CentOS 6

In this segment, we would be creating a Samba Server Cluster using Heartbeat and CentOS 6.

First, we start by understanding, what is a server cluster? A server cluster is a collection of servers that provide similar network services. They share an IP address/FQDN and the client hosts communicate with that IP/FQDN to obtain network services. The part to notice here is that multiple servers are used for auto redundancy and in cases, load balancing. Which simply means, even if one of the servers are down, the other servers would take over and the client will get the services from the backup servers. The whole process is automatic and with proper configuration, seamless. The client wouldn't even know that it's using a backup server.

For example, let us assume that server 1 and server 2 have IP address of IP1 and IP2 respectively. Both of them are configured as web servers. We have an IP address IP3 that can be resolved to www.testdomain.com. So, to create an auto redundant cluster, we configure the cluster in such manner that the servers share the virtual IP of IP3 between them so if anyone queries www.testdomain.com, he would resolve to IP3 and the request would be forwarded to the server cluster. Even if server1 is down, server 2 would be using the virtual IP of IP3 and would be able to provide services for www.testdomain.com.

Cluster configuration comes in many forms, and the complexity may differ from design to design. For example, the servers in the cluster may be configured to use an external common storage. like SAN or NAS. Again, there may be cases where each server has it's own storage and the configurations are identical on both servers.

Scenario
We have 2 servers

• primary.example.inv (IP 192.168.1.1)
• secondary.example.inv (IP 192.168.1.2)
• Virtual IP: 192.168.1.100
• A FQDN services.example.inv which can be resolved to IP 192.168.1.100 

primary.example.inv is already configured as Primary Domain Controller for the domain example.inv.  Please consult the earlier post on how to configure a Samba Primary Domain Controller. We would be configuring secondary.example.inv as the backup domain controller.

 primary.example.inv Configuration
First, we have to keep in mind that the hostnames are really important. We have to set the hostnames properly in the following locations-

[root@primary ~]# vim /etc/sysconfig/network [root@primary ~]# vim /etc/sysconfig/hosts  

NOTE: If there is any confusion about how to set the hostnames, please consult an earlier post


Then, we have to setup heartbeat in the server. CentOS 6 still does have heartbeat RPMs in their repositories, so  yum install will not work just yet. After googling for a while, I found heartbeat here

heartbeat RPM 
heartbeat-libs RPM

Here's how we can download the heartbeat RPMs and setup some dependencies.


[root@primary ~]# wget http://dl.fedoraproject.org/pub/epel/6/i386/heartbeat-3.0.4-1.el6.i686.rpm [root@primary ~]# wget http://dl.fedoraproject.org/pub/epel/6/i386/heartbeat-libs-3.0.4-1.el6.i686.rpm [root@primary ~]# yum install PyXML cluster-glue resource-agents [root@primary ~]# rpm -ivh heartbeat-*  

Great! We have installed heartbeat to our server :)

Next, it's time to configure heartbeat. All the heartbeat config files are stored in /etc/ha.d. We have to modify 3 files only. However, since the configuration files are not present yet, we will have to copy sample configuration files from /usr. Here's how it's done-


[root@primary ~]# cd /usr/share/doc/heartbeat-3.0.4/ [root@primary ~]# cp authkeys /etc/ha.d/ [root@primary ~]# cp haresources /etc/ha.d/ [root@primary ~]# cp ha.cf /etc/ha.d/  

Time to edit to configuration files:

[root@primary ~]# chmod 600 /etc/ha.d/authkeys [root@primary ~]# vim /etc/ha.d/authkeys ### just add the following line ### auth 2 2 sha1 secretkey ### since SHA1 is considered the stongest, we would be using SHA encryption. The phrase "secreykey" will be shared with the other servers in the cluster###  



[root@primary ~]# vim /etc/ha.d/ha.cf ### modify the following lines ### logfile /var/log/ha-log keepalive 2 deadtime 30 initdead 120 udpport 694 ucast eth0 192.168.1.2 ## heartbeat supports broadcast, multicast and unicast. since the cluster contains only 2 servers, we provide the IP address of the other server here. auto_failback on node primary.example.inv node secondary.example.inv ## we list the node names stating the actual primary server first.  

Different deadtime and initdead may be used for experimenting. For example, I tried with deadtime of only 10 seconds and initdead of only 20 seconds for testing. In case of live servers, the real environment would decide which values are appropriate.

[root@primary ~]# vim /etc/ha.d/haresources services.example.inv IPaddr::192.168.1.100 smb nmb  

In this file, we define the virtual IP address to be used by heartbeat.

NOTE: It should be kept in mind that we do NOT need to assign the IP to any interface. Heartbeat would manage requests coming towards the virtual IP i.e. 192.168.1.4.


secondary.example.inv Configuration
First, we would set hostname and then setup heartbeat in the secondary server. The setup process is identical to the primary server.
It is very important that the configuration files of primary and secondary servers are identical. However, minor tuning may be necessary in some cases.
As we need identical configuration files, we would be copying the directory /etc/ha.d from primary server to the secondary server.

Time to edit to configuration files:

[root@secondary ~]# scp -r 192.168.1.1:/etc/ha.d /etc/ [root@secondary ~]# chmod 600 /etc/ha.d/authkeys  

Since we are using unicast instead of broadcast, we have to specify which way heartbeat periodic packets should be sent

[root@seconday ~]# vim /etc/ha.d/ha.cf ucast eth0 192.168.1.1  


Samba Configuration
Because both of the servers have separate storage, we have to keep in mind that both servers are running independent instances of samba. But since we want to configure the second server as backup, we have to make sure that both samba servers have identical user database. This post can be consulted to see how samba users can be migrated.

First, we have to make sure that /etc/passwd, /etc/group and /etc/shadow are identical. I was unable to come up with an automatic method that could sync only the human users ( UID 500 and onwards). So, synchronizing these files must be done manually, which is described in the samba user migration post.

Then, we synchronize samba files in both machines

[root@primary ~]# yum install rsync [root@primary ~]# rsync -av -e ssh /etc/samba/* 192.168.1.2:/etc/samba/ [root@primary ~]# rsync -av -e ssh /var/lib/samba/* 192.168.1.2:/var/lib/samba/  

Now we are almost ready.


Running Heartbeat

Primary Server:
[root@primary ~]# chkconfig smb off; chkconfig nmb off; chkconfig heartbeat on [root@primary ~]# service heartbeat start  

Secondary Server:
[root@secondary ~]# chkconfig smb off; chkconfig nmb off; chkconfig heartbeat on [root@secondary ~]# service heartbeat start  

NOTE: It should be kept in mind that heartbeat would deal with starting/stopping any managed service. Any heartbeat managed service should NEVER be manually started/stopped because it would create problems in normal cluster operation. This is the reason why these services should also not be run during startup process.

Testing
It is always a good idea to analyze the log files. Heartbeat logs are stored in /var/log/ha-log. Useful information can also be found in /var/log/messages. The, /var/log/messages file should show which server is acting as the domain controller. So, the first thing to do is, to analyze the log files.

Then, since both the servers have the same user database, testing can be done by logging into the domain from windows clients. If a domain user can log into the domain using the primary server, the same user should also be able to login to the domain if the primary server is powered down (or heartbeat in primary server is stopped).

Troubleshooting

1. I have SELinux disabled.
2. In this configuration, heartbeat uses port UDP 694 and firewall must not block this port.
3. The virtual IP is managed by heartbeat and does not need to be assigned to any device.
4. The heartbeat managed services are started/stopped by heartbeat. They should not be run/stopped manually.
5. The heartbeat managed services should not be automatically run at system startup.

Hope it helps :)