HP Serviceguard

HP Serviceguard was the very first solution for high availability clustering software. Serviceguard uses resource virtualization to ensure that critical applications, databases, or services are continually up and running. Users typically don’t care which server they run their application on—they just want to make sure that application is running. Serviceguard hides the details of which server an application is actually running on at any given time, which means it can move applications between servers on the fly.

 

What is a Serviceguard cluster?

Any host system running in a Serviceguard cluster is called an active node. Under normal conditions, a fully operating Serviceguard cluster monitors the health of the cluster’s components on all its active nodes.Most Serviceguard packages are failover packages. When you configure a failover package, you specify which active node will be the primary node where the package will start, and one or more other nodes, called adoptive nodes, that can also run the package. 

Creating a Serviceguard  cluster

Before you configure the cluster, you will need to create a volume group on the shared disks, so that Service Guard can create a “Cluster Lock Disk”.

 

On testnode1

 # pvcreate /dev/rdsk/c0t4d0

# pvcreate /dev/rdsk/c1t4d0

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgcreate vg01 /dev/dsk/c0t4d0 /dev/dsk/c1t4d0

# mkdir /etc/cmcluster/maps

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

 On testnode2

 # mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# mkdir /etc/cmcluster/maps

# rcp testnode1:/etc/cmcluster/maps/vg01 /etc/cmcluster/maps/vg01

# vgimport –s –m /etc/cmcluster/maps/vg01 vg01

# vgchange –a y vg01

# vgchange –a n vg01

You can now go ahead and create the cluster.

# cd /etc/cmcluster

# cmquerycl –v –C cmclconfig.ascii –n testnode1 -n testnode2

After creating this file, you will need to edit it to suit your cluster. You will need to

alter the following lines:

 # vi cmclconfig.ascii

 CLUSTER_NAME simeon1

NODE_NAME testnode1

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.101

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.92

NETWORK_INTERFACE lan2

NODE_NAME testnode2

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.102

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.93

NETWORK_INTERFACE lan2

HEARTBEAT_INTERVAL 2000000

NODE_TIMEOUT 5000000

MAX_CONFIGURED_PACKAGES 10

 To check that there are no mistakes in the file, use the check command.

 # cmcheckconf –C /etc/cmcluster/cmclconfig.ascii

 To apply the configuration file, usr the apply command.

 # cmapplyconf –C /etc/cmcluster/cmclconfig.ascii

 You are now ready to startup the cluster.

On testnode1

# vgchange –a n vg01

# cmruncl –v

# cmviewcl –v

# vgchange –c y vg01

On Both nodes

# netstat -in

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

On testnode1

# cmhaltcl –v

# vgchange –a n vg01

Creating a basic package

Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary.

Creating or modifying a package requires the following broad steps:

1. Generate the package ASCII file

2. Edit the package ASCII file

3. Generate the package control script

4. Edit the package control script

5. Distribute the control script to the cluster nodes

6. Apply the package ASCII file

Once the cluster is up and running, we can create a package to run in it. This package will run an “xclock” on a client PC, and mount a file system, using the IP Address 164.39.8.101 The first step is to make a logical volume.

On node1

# vgchange –a e vg01

# lvcreate –L 20 vg01

# lvextend –m 1 /dev/vg01/lvol1

# newfs –F vxfs /dev/vg01/rlvol1

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

# vgchange –a n vg01

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

On Node 2

# vgexport vg01

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgimport –s –m /etc/cmlcuster/maps/vg01 vg01

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

Now we can make the package.

On Node1

# cd /etc/cmcluster

# mkdir pkg_1

# cd pkg_1

# cmmakepkg –p pkg_1.conf

The file pkg_1.conf will need to be edited to suit.

# vi pkg_1.conf

PACKAGE_NAME pkg_1

NODE_NAME testnode1

NODE_NAME testnode2

RUN_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

RUN_SCRIPT_TIMEOUT NO_TIMEOUT

HALT_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

HALT_SCRIPT_TIMEOUT NO_TIMEOUT

SERVICE_NAME pkg_1_service

SUBNET 164.39.8.0

Add the IP address that will be used for the package in the hosts file.

# vi /etc/hosts

164.39.8.101 pkg_1 xclock

# cmmakepkg –s pkg_1.cntl

The file pkg_1.cntl will need to be edited to suit

# vi pkg_1.cntl

VG[0]=”vg01″

LV[0]=”/dev/vg01/lvol1″; FS[0]=”/pkg_1″; FS_MOUNT_OPT[0]=”"

IP[0]=”164.39.8.101″

SUBNET[0]=”164.39.8.0″

SERVICE_NAME[0]=”pkg_1_service”

SERVICE_CMD[0/etc/cmcluster/pkg_1/service_pkg_1"

#SERVICE_RESTART[0]=”"

function customer_defined_run_cmds

{

# ADD customer defined run commands.

if [ $(hostname) = "testnode1" ]

then COLOUR=”blue”

else COLOUR=”red”

fi

/usr/bin/X11/xclock -bg “$COLOUR” -update 1 –display \

164.39.11.232:0.0 &

test_return 51

}

function customer_defined_halt_cmds
{

# ADD customer defined halt commands.

if

ps -ef |grep -v grep |grep -q “xclock”

then

kill -9 $(ps -ef |grep -v grep |grep “xclock”|cut -c10-14)

fi

test_return 52

}

# vi pkg_1_service

#!/usr/bin/sh

#This is a script to monitor that xclock is running

while true

do

if

ps -ef |grep -v grep |grep -q “xclock”

then

sleep 5

else

echo “Package xclock failed at $(date) from node \

$(hostname) > /etc/cmcluster/pkg_1/pkg_1.cntl.log”

exit 99

fi

done

# chmod u+x pkg_1_servce

# cd /etc/cmcluster

# rcp –r testnode1:/etc/cmcluster/pkg_1

 On node 1

# cmcheckconf -P pkg_1.conf

# cmapplyconf –P pkg_1.conf

# cmmodpkg –e –n testnode1 -n testnode2 pkg_1

# cmviewcl –v –p pkg_1

# cmhaltpkg pkg_1

# cmviewcl –v|more

# tail /etc/cmcluster/pkg_1/pkg_1.log

# tail /var/adm/syslog/syslog.log

Starting and stopping a Serviceguard Package

Normally, when a cluster starts up, the packages configured as part of the cluster will start up on their configured nodes. You may need to start a package manually after it has been halted manually. You can do this either in Serviceguard Manager or on the Serviceguard command line.

If you want to stop a package, use the following commands

# cmviewcl –v –p <package name>

# cmahalt <package name>

# cmviewcl –v –p <package name>

You halt a Serviceguard package when you wish to bring the package out of use but wish the node to continue in operation. You can halt a package using Serviceguard Manager or on the Serviceguard command line. Halting a package has a different effect than halting the node. When you halt the node, its failover packages may switch to adoptive nodes.

If you want to start a package, then use the following commands

# cmviewcl –v –p <package name>

# cmmodpkg –e <package name>

OR

# cmrunpkg –n <host name> <package name>

# cmmodpkg –e <package name>

Deleting a Serviceguard Package

To delete a package, use the following commands

 # cmdeleteconf -p <package name>

Stopping and starting the Serviceguard Cluster

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

To stop the cluster, use the following commands

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

 

To stop the cluster, use the following commands

# cmiewcl –v

# cmhaltcl

OR

# cmhaltcl –f

To start the cluster: -

# cmruncl –v

# cmviewcl –v

Automatically Restarting the Serviceguard Cluster

You can configure your cluster to automatically restart after an event, such as a long-term power failure, which brought down all nodes in the cluster. This is done by setting AUTOSTART_CMCLD to 1 in the

/etc/rc.config.d/cmcluster file.

Stopping and starting a node in the Serviceguard Cluster

Starting a node will not cause any active packages to be moved to the new node. However, if a package is DOWN, has its switching enabled, and is able to run on the new node, that package will automatically run there.

To stop a node in the cluster

# cmhaltnode –n <node name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=0

To start a node in the cluster.

# cmrunnode –n <name name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

Deleting a cluster

When there are no packages left in the cluster, and you want to delete the  cluster, use the following commands.

# vgchange –c n vg01

# cmhaltcl –f

# cmdeleteconf –c <cluster_name>

# vgchange –a y vg01

# lvremove /dev/vg01/lvol1

# vgremove vg01

# rm –r /dev/vg01

What does High Availability mean?

In terms of computers HA means a system design protocol which ensures the availability of a resource in a given amount of time or any terms of measurement.

HA-Clustering and Failover

This is a computer cluster implemented for making a resource available all the time or reducing the downtime of the resource. For example, an application server serving the user requests continuously and in come point of time it stopped responding. In that time the users waiting for a response from the server should not get disappointed.

Here comes the clustering technology, the cluster configuration in the network transfers or redirects the user requests to the backup server. This process is called Failover.

The Clustering Software

The clustering software plays main role in the successful functioning of the HA-Clusters. Everyone have a question of who maintains the information and transactions processed by the First Machine.

Yes, the clustering software does that for us. It will copy the last good known configuration from the machine which failed to complete its duty to the machine which is ready to serve for us.

Heart Beat of Clustering

HA clusters use a private Heart-Beat network. This network is responsible for monitoring the health status of all the nodes in the cluster.

Split-Brain in Cluster network

The term “Split-Brain” technically explains the following scenario. When all the private network link goes down but the individual cluster nodes are still running, there is a possibility of confusion. Each node in the cluster may think that other node is down and will supply duplicate data. So this is a tedious configure the network to avoid split-brains.

Cluster Node Configurations

 These are the available clustering node configurations

 1. Active/Active – Traffic is directed to another existing node or balanced across all other nodes

2. Active/Passive – A complete new node is served when primary node fails

3.  N+1 – Serving a complete new node in the place of primary node and the new node    in the cluster must be capable of handling any services which the primary node handled with/without any special additional software.

4. N+M – Here more than one dedicated nodes are served for handling failovers.
This   may require high cost and maintenance.

5. N-to-1 – Here the backup node is serving temporarily until the primary node is brought back online. Then the running services are transferred to primary node.

6. N-to-N — A combination of Active/Active and N+M clusters, N to N clusters.

Requirements for a High Availability Clustering Environment

 

The HA-Clustering is widely used technology in Data centers and Industries where adaptive infrastructure is required. This requires high investment hence this is called as costly technology.

Below given are the minimum requirements for setting up a HA Clustered Environment.

1. You must be having suitable advanced application which is capable to support HA-C. Not all the applications can run in High Availability Environment.
2. The application should be capable of running its own multiple instances.
3. The application you are using in HA-C must have command line utility. i.e ability to control the app through command line interface by executing some commands.
4. The application should have the capability of using Storage Area Network (SAN).

The application must be capable of storing the data in a SAN, so that even a crash occurs the other back-up node can use the saved-state of information.

Applications of HA Clustering Technology

So, where is this clustering more useful and applied ? HA Clustering is used for

1. Critical databases

2. File sharing networks

3. Business Applications

4. E-Commerce Websites

Some famous HA-Clustering products

Vendor	Product Name	Supported Platforms
Hewlett-Packard	HP ServiceGuard	HP-UX, Linux
IBM	High Availability Cluster	AIX, Linux
Open –Source	Linux-HA	Linux
Microsoft	Microsoft Cluster Server (MSCS)	Windows server 2008
Red Hat	Red Hat Cluster Suite	Linux

My next update in HA-Clustering would be about configuring the HA clusters in live environment.

Wait for a couple days….

Regards,
R.Gopinath