In this case I want to create a redhat cluster using iscsi disks and RHCS.
The goal is to have a service which depend on IP and disk to failover between 2 nodes
Following IP adresses where used:
iscshi-host: 192.168.10.31
cluster node 1: 192.168.10.32
cluster node 2: 192.168.10.33
First I setup the iscsi host
I setup 1vcpu, 385mb ram and a root disk of 8G using free vmware server
Install a standard centos node (in my case ver 5.5 64 bit)
during install choose to install gnome, server and servergui packages
After install I disabled firewall and SElinux
when the system is ready do the following to share a 512mb file as an iscsi disk to the local network
for data and a 25mb file used for quorum in the cluster
1. install the needed software
# yum install iscsi-initiator-utils
# yum install openssl openssl-devel
# yum install scsi-target-utils
2. Enable and start iscsi target software
# chkconfig tgtd on
# service tgtd start
3. Create the 2 files to be shared as iscsi disks
# mkdir /iscsishares
# dd if=/dev/zero of=/iscsishares/LUN0 bs=1M count=512
# dd if=/dev/zero of=/iscsishares/LUN1 bs=1M count=25
4. Create the targets
# tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2010-11.iscsihost:storage.disk1.sys1.xyz
# tgtadm --lld iscsi --op new --mode target --tid 2 -T iqn.2010-11.iscsihost:storage.disk2.sys1.xyz
5. Add your LUN to the target
# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /iscsishares/LUN0
# tgtadm --lld iscsi --op new --mode logicalunit --tid 2 --lun 1 -b /iscsishares/LUN1
6. Make LUN available from all initiators
# tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
# tgtadm --lld iscsi --op bind --mode target --tid 2 -I ALL
7. check it works by issuing
# tgtadm --lld iscsi --op show --mode target
8. make it sticky by inserting the three lines in /etc/rc.local
tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2010-11.iscsihost:storage.disk1.sys1.xyz
tgtadm --lld iscsi --op new --mode target --tid 2 -T iqn.2010-11.iscsihost:storage.disk2.sys1.xyz
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /iscsishares/LUN0
tgtadm --lld iscsi --op new --mode logicalunit --tid 2 --lun 1 -b /iscsishares/LUN1
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
tgtadm --lld iscsi --op bind --mode target --tid 2 -I ALL
Install both clusternodes - make sure you choose following packages: gnome, server, servergui, clustering, storage clustering
My test servers have this footprint: 1vcpu, 385mb ram and a root disk of 8G
Check that ALL 3 servers are using ntp and that they agree on time/date
Install and configure iscsi initiator software on both clusternodes
# yum install iscsi-initiator-utils
# service iscsi start
# service iscsid start
Discover iscsi devices (you should see your 2 devices):
# iscsiadm --mode discovery --type sendtargets --portal 192.168.10.31
import the iscsi devices
# iscsiadm -m node -T iqn.2010-11.iscsihost:storage.disk1.sys1.xyz -p 192.168.10.31 -l
# iscsiadm -m node -T iqn.2010-11.iscsihost:storage.disk2.sys1.xyz -p 192.168.10.31 -l
When that succeeds make sure it happens automatically.
# iscsiadm -m node -T iqn.2010-11.iscsihost:storage.disk1.sys1.xyz -p 192.168.10.31 -o update -n node.startup -v automatic
# iscsiadm -m node -T iqn.2010-11.iscsihost:storage.disk2.sys1.xyz -p 192.168.10.31 -o update -n node.startup -v automatic
You will be able to see your new devices in /proc/partitions
Now make sure that both clusternodes have themselves and each other in /etc/hosts eg:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.10.32 cluster1
192.168.10.33 cluster2
::1 localhost6.localdomain6 localhost6
Now you are ready to play around with cluster software ... finally
Start by creating a basic /etc/cluster/cluster.conf (needed for the daemons to start)
I have attached a real basic one with a cluster called skau-cluster and with both my clusternodes.
this file must be EXACTLY the same on both clusternodes
<?xml version="1.0" ?>
<cluster config_version="3" name="skau-cluster">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="cluster1" nodeid="1" votes="1">
<fence/>
</clusternode>
<clusternode name="cluster2" nodeid="2" votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
Start the necessary cluster daemons
# chkconfig cman on
# chkconfig rgmanager on
# chkconfig clvmd on
# chkconfig qdiskd on
# service cman start
# service rgmanager start
# service clvmd start
# service qdiskd start
What we now have are 3 devices on each cluster node - sda which is the / device, sdb 512MB for data and sdc
25mb for quorum
Lets start out by setting up quorum
Now the reason you need quorum is to maintain a host as cluster master - it works by using votes, in my setup
both nodes have 1 vote, in order to get vote majority they race to get ownership of the quorum device which also
holds a vote.
quorum devices are only necessary in two node clusters.
Read more
first initialize the quorumdisk for qdisk use (only do this on 1 node)
# mkqdisk -c /dev/sdc -l qdisk
Check on both nodes that you see the quorum device
# mkqdisk -L
Because our cluster daemon cman depends on a quorum device, we need to make sure that quorum daemon starts before cman, I don't know why redhat has'nt done this themselves, but simply change order on both nodes like this:
# mv /etc/rc3.d/S22qdiskd /etc/rc3.d/S20qdiskd
Now make your /etc/cluster/cluster.conf reflect the usage of quorum device
<?xml version="1.0"?>
<cluster alias="skau-cluster" config_version="6" name="skau-cluster">
<quorumd interval="3" tko="23" label="qdisk" votes="3"/>
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="cluster1" nodeid="1" votes="1">
<fence/>
</clusternode>
<clusternode name="cluster2" nodeid="2" votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
<fencedevice agent="fence_scsi" name="qdisk"/>
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
Check that you have 2 cluster members:
[root@cluster1 ~]# clustat
Cluster Status for skau-cluster @ Wed Nov 17 05:54:23 2010
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
cluster1 1 Online, Local
cluster2 2 Online
/dev/disk/by-id/scsi-1IET_00020001 0 Online, Quorum Disk
From now on you can use the system-config-cluster tool - It is a really easytouse frontend for /etc/cluster/cluster.conf
file
I will do my work in the GUI, but I will also show the XML generated.
