HP Serviceguard

HP Serviceguard was the very first solution for high availability clustering software. Serviceguard uses resource virtualization to ensure that critical applications, databases, or services are continually up and running. Users typically don’t care which server they run their application on—they just want to make sure that application is running. Serviceguard hides the details of which server an application is actually running on at any given time, which means it can move applications between servers on the fly.

 

What is a Serviceguard cluster?

Any host system running in a Serviceguard cluster is called an active node. Under normal conditions, a fully operating Serviceguard cluster monitors the health of the cluster’s components on all its active nodes.Most Serviceguard packages are failover packages. When you configure a failover package, you specify which active node will be the primary node where the package will start, and one or more other nodes, called adoptive nodes, that can also run the package. 

Creating a Serviceguard  cluster

Before you configure the cluster, you will need to create a volume group on the shared disks, so that Service Guard can create a “Cluster Lock Disk”.

 

On testnode1

 # pvcreate /dev/rdsk/c0t4d0

# pvcreate /dev/rdsk/c1t4d0

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgcreate vg01 /dev/dsk/c0t4d0 /dev/dsk/c1t4d0

# mkdir /etc/cmcluster/maps

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

 On testnode2

 # mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# mkdir /etc/cmcluster/maps

# rcp testnode1:/etc/cmcluster/maps/vg01 /etc/cmcluster/maps/vg01

# vgimport –s –m /etc/cmcluster/maps/vg01 vg01

# vgchange –a y vg01

# vgchange –a n vg01

You can now go ahead and create the cluster.

# cd /etc/cmcluster

# cmquerycl –v –C cmclconfig.ascii –n testnode1 -n testnode2

After creating this file, you will need to edit it to suit your cluster. You will need to

alter the following lines:

 # vi cmclconfig.ascii

 CLUSTER_NAME simeon1

NODE_NAME testnode1

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.101

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.92

NETWORK_INTERFACE lan2

NODE_NAME testnode2

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.102

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.93

NETWORK_INTERFACE lan2

HEARTBEAT_INTERVAL 2000000

NODE_TIMEOUT 5000000

MAX_CONFIGURED_PACKAGES 10

 To check that there are no mistakes in the file, use the check command.

 # cmcheckconf –C /etc/cmcluster/cmclconfig.ascii

 To apply the configuration file, usr the apply command.

 # cmapplyconf –C /etc/cmcluster/cmclconfig.ascii

 You are now ready to startup the cluster.

On testnode1

# vgchange –a n vg01

# cmruncl –v

# cmviewcl –v

# vgchange –c y vg01

On Both nodes

# netstat -in

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

On testnode1

# cmhaltcl –v

# vgchange –a n vg01

Creating a basic package

Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary.

Creating or modifying a package requires the following broad steps:

1. Generate the package ASCII file

2. Edit the package ASCII file

3. Generate the package control script

4. Edit the package control script

5. Distribute the control script to the cluster nodes

6. Apply the package ASCII file

Once the cluster is up and running, we can create a package to run in it. This package will run an “xclock” on a client PC, and mount a file system, using the IP Address 164.39.8.101 The first step is to make a logical volume.

On node1

# vgchange –a e vg01

# lvcreate –L 20 vg01

# lvextend –m 1 /dev/vg01/lvol1

# newfs –F vxfs /dev/vg01/rlvol1

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

# vgchange –a n vg01

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

On Node 2

# vgexport vg01

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgimport –s –m /etc/cmlcuster/maps/vg01 vg01

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

Now we can make the package.

On Node1

# cd /etc/cmcluster

# mkdir pkg_1

# cd pkg_1

# cmmakepkg –p pkg_1.conf

The file pkg_1.conf will need to be edited to suit.

# vi pkg_1.conf

PACKAGE_NAME pkg_1

NODE_NAME testnode1

NODE_NAME testnode2

RUN_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

RUN_SCRIPT_TIMEOUT NO_TIMEOUT

HALT_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

HALT_SCRIPT_TIMEOUT NO_TIMEOUT

SERVICE_NAME pkg_1_service

SUBNET 164.39.8.0

Add the IP address that will be used for the package in the hosts file.

# vi /etc/hosts

164.39.8.101 pkg_1 xclock

# cmmakepkg –s pkg_1.cntl

The file pkg_1.cntl will need to be edited to suit

# vi pkg_1.cntl

VG[0]=”vg01″

LV[0]=”/dev/vg01/lvol1″; FS[0]=”/pkg_1″; FS_MOUNT_OPT[0]=”"

IP[0]=”164.39.8.101″

SUBNET[0]=”164.39.8.0″

SERVICE_NAME[0]=”pkg_1_service”

SERVICE_CMD[0/etc/cmcluster/pkg_1/service_pkg_1"

#SERVICE_RESTART[0]=”"

function customer_defined_run_cmds

{

# ADD customer defined run commands.

if [ $(hostname) = "testnode1" ]

then COLOUR=”blue”

else COLOUR=”red”

fi

/usr/bin/X11/xclock -bg “$COLOUR” -update 1 –display \

164.39.11.232:0.0 &

test_return 51

}

function customer_defined_halt_cmds
{

# ADD customer defined halt commands.

if

ps -ef |grep -v grep |grep -q “xclock”

then

kill -9 $(ps -ef |grep -v grep |grep “xclock”|cut -c10-14)

fi

test_return 52

}

# vi pkg_1_service

#!/usr/bin/sh

#This is a script to monitor that xclock is running

while true

do

if

ps -ef |grep -v grep |grep -q “xclock”

then

sleep 5

else

echo “Package xclock failed at $(date) from node \

$(hostname) > /etc/cmcluster/pkg_1/pkg_1.cntl.log”

exit 99

fi

done

# chmod u+x pkg_1_servce

# cd /etc/cmcluster

# rcp –r testnode1:/etc/cmcluster/pkg_1

 On node 1

# cmcheckconf -P pkg_1.conf

# cmapplyconf –P pkg_1.conf

# cmmodpkg –e –n testnode1 -n testnode2 pkg_1

# cmviewcl –v –p pkg_1

# cmhaltpkg pkg_1

# cmviewcl –v|more

# tail /etc/cmcluster/pkg_1/pkg_1.log

# tail /var/adm/syslog/syslog.log

Starting and stopping a Serviceguard Package

Normally, when a cluster starts up, the packages configured as part of the cluster will start up on their configured nodes. You may need to start a package manually after it has been halted manually. You can do this either in Serviceguard Manager or on the Serviceguard command line.

If you want to stop a package, use the following commands

# cmviewcl –v –p <package name>

# cmahalt <package name>

# cmviewcl –v –p <package name>

You halt a Serviceguard package when you wish to bring the package out of use but wish the node to continue in operation. You can halt a package using Serviceguard Manager or on the Serviceguard command line. Halting a package has a different effect than halting the node. When you halt the node, its failover packages may switch to adoptive nodes.

If you want to start a package, then use the following commands

# cmviewcl –v –p <package name>

# cmmodpkg –e <package name>

OR

# cmrunpkg –n <host name> <package name>

# cmmodpkg –e <package name>

Deleting a Serviceguard Package

To delete a package, use the following commands

 # cmdeleteconf -p <package name>

Stopping and starting the Serviceguard Cluster

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

To stop the cluster, use the following commands

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

 

To stop the cluster, use the following commands

# cmiewcl –v

# cmhaltcl

OR

# cmhaltcl –f

To start the cluster: -

# cmruncl –v

# cmviewcl –v

Automatically Restarting the Serviceguard Cluster

You can configure your cluster to automatically restart after an event, such as a long-term power failure, which brought down all nodes in the cluster. This is done by setting AUTOSTART_CMCLD to 1 in the

/etc/rc.config.d/cmcluster file.

Stopping and starting a node in the Serviceguard Cluster

Starting a node will not cause any active packages to be moved to the new node. However, if a package is DOWN, has its switching enabled, and is able to run on the new node, that package will automatically run there.

To stop a node in the cluster

# cmhaltnode –n <node name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=0

To start a node in the cluster.

# cmrunnode –n <name name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

Deleting a cluster

When there are no packages left in the cluster, and you want to delete the  cluster, use the following commands.

# vgchange –c n vg01

# cmhaltcl –f

# cmdeleteconf –c <cluster_name>

# vgchange –a y vg01

# lvremove /dev/vg01/lvol1

# vgremove vg01

# rm –r /dev/vg01