HP Serviceguard was the very first solution for high availability clustering software. Serviceguard uses resource virtualization to ensure that critical applications, databases, or services are continually up and running. Users typically don’t care which server they run their application on—they just want to make sure that application is running. Serviceguard hides the details of which server an application is actually running on at any given time, which means it can move applications between servers on the fly.

What is a Serviceguard cluster?

Any host system running in a Serviceguard cluster is called an active node. Under normal conditions, a fully operating Serviceguard cluster monitors the health of the cluster’s components on all its active nodes.Most Serviceguard packages are failover packages. When you configure a failover package, you specify which active node will be the primary node where the package will start, and one or more other nodes, called adoptive nodes, that can also run the package. 

Creating a Serviceguard  cluster

Before you configure the cluster, you will need to create a volume group on the shared disks, so that Service Guard can create a “Cluster Lock Disk”.

On testnode1

 # pvcreate /dev/rdsk/c0t4d0

# pvcreate /dev/rdsk/c1t4d0

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgcreate vg01 /dev/dsk/c0t4d0 /dev/dsk/c1t4d0

# mkdir /etc/cmcluster/maps

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

 On testnode2

 # mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# mkdir /etc/cmcluster/maps

# rcp testnode1:/etc/cmcluster/maps/vg01 /etc/cmcluster/maps/vg01

# vgimport –s –m /etc/cmcluster/maps/vg01 vg01

# vgchange –a y vg01

# vgchange –a n vg01

You can now go ahead and create the cluster.

# cd /etc/cmcluster

# cmquerycl –v –C cmclconfig.ascii –n testnode1 -n testnode2

After creating this file, you will need to edit it to suit your cluster. You will need to

alter the following lines:

 # vi cmclconfig.ascii

 CLUSTER_NAME simeon1

NODE_NAME testnode1

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.101

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.92

NETWORK_INTERFACE lan2

NODE_NAME testnode2

NETWORK_INTERFACE lan0

HEARTBEAT_IP 192.168.100.102

NETWORK_INTERFACE lan1

HEARTBEAT_IP 164.39.8.93

NETWORK_INTERFACE lan2

HEARTBEAT_INTERVAL 2000000

NODE_TIMEOUT 5000000

MAX_CONFIGURED_PACKAGES 10

 To check that there are no mistakes in the file, use the check command.

 # cmcheckconf –C /etc/cmcluster/cmclconfig.ascii

 To apply the configuration file, usr the apply command.

 # cmapplyconf –C /etc/cmcluster/cmclconfig.ascii

 You are now ready to startup the cluster.

On testnode1

# vgchange –a n vg01

# cmruncl –v

# cmviewcl –v

# vgchange –c y vg01

On Both nodes

# netstat -in

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

On testnode1

# cmhaltcl –v

# vgchange –a n vg01

Creating a basic package

Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary.

Creating or modifying a package requires the following broad steps:

1. Generate the package ASCII file

2. Edit the package ASCII file

3. Generate the package control script

4. Edit the package control script

5. Distribute the control script to the cluster nodes

6. Apply the package ASCII file

Once the cluster is up and running, we can create a package to run in it. This package will run an “xclock” on a client PC, and mount a file system, using the IP Address 164.39.8.101 The first step is to make a logical volume.

On node1

# vgchange –a e vg01

# lvcreate –L 20 vg01

# lvextend –m 1 /dev/vg01/lvol1

# newfs –F vxfs /dev/vg01/rlvol1

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

# vgchange –a n vg01

# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01

On Node 2

# vgexport vg01

# mkdir /dev/vg01

# mknod /dev/vg01/group c 64 0×010000

# vgimport –s –m /etc/cmlcuster/maps/vg01 vg01

# mkdir /pkg_1

# vi /etc/fstab

#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG

Now we can make the package.

On Node1

# cd /etc/cmcluster

# mkdir pkg_1

# cd pkg_1

# cmmakepkg –p pkg_1.conf

The file pkg_1.conf will need to be edited to suit.

# vi pkg_1.conf

PACKAGE_NAME pkg_1

NODE_NAME testnode1

NODE_NAME testnode2

RUN_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

RUN_SCRIPT_TIMEOUT NO_TIMEOUT

HALT_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl

HALT_SCRIPT_TIMEOUT NO_TIMEOUT

SERVICE_NAME pkg_1_service

SUBNET 164.39.8.0

Add the IP address that will be used for the package in the hosts file.

# vi /etc/hosts

164.39.8.101 pkg_1 xclock

# cmmakepkg –s pkg_1.cntl

The file pkg_1.cntl will need to be edited to suit

# vi pkg_1.cntl

VG[0]=”vg01″

LV[0]=”/dev/vg01/lvol1″; FS[0]=”/pkg_1″; FS_MOUNT_OPT[0]=”"

IP[0]=”164.39.8.101″

SUBNET[0]=”164.39.8.0″

SERVICE_NAME[0]=”pkg_1_service”

SERVICE_CMD[0/etc/cmcluster/pkg_1/service_pkg_1"

#SERVICE_RESTART[0]=”"

function customer_defined_run_cmds

{

# ADD customer defined run commands.

if [ $(hostname) = "testnode1" ]

then COLOUR=”blue”

else COLOUR=”red”

fi

/usr/bin/X11/xclock -bg “$COLOUR” -update 1 –display \

164.39.11.232:0.0 &

test_return 51

}

function customer_defined_halt_cmds
{

# ADD customer defined halt commands.

if

ps -ef |grep -v grep |grep -q “xclock”

then

kill -9 $(ps -ef |grep -v grep |grep “xclock”|cut -c10-14)

fi

test_return 52

}

# vi pkg_1_service

#!/usr/bin/sh

#This is a script to monitor that xclock is running

while true

do

if

ps -ef |grep -v grep |grep -q “xclock”

then

sleep 5

else

echo “Package xclock failed at $(date) from node \

$(hostname) > /etc/cmcluster/pkg_1/pkg_1.cntl.log”

exit 99

fi

done

# chmod u+x pkg_1_servce

# cd /etc/cmcluster

# rcp –r testnode1:/etc/cmcluster/pkg_1

 On node 1

# cmcheckconf -P pkg_1.conf

# cmapplyconf –P pkg_1.conf

# cmmodpkg –e –n testnode1 -n testnode2 pkg_1

# cmviewcl –v –p pkg_1

# cmhaltpkg pkg_1

# cmviewcl –v|more

# tail /etc/cmcluster/pkg_1/pkg_1.log

# tail /var/adm/syslog/syslog.log

Starting and stopping a Serviceguard Package

Normally, when a cluster starts up, the packages configured as part of the cluster will start up on their configured nodes. You may need to start a package manually after it has been halted manually. You can do this either in Serviceguard Manager or on the Serviceguard command line.

If you want to stop a package, use the following commands

# cmviewcl –v –p <package name>

# cmahalt <package name>

# cmviewcl –v –p <package name>

You halt a Serviceguard package when you wish to bring the package out of use but wish the node to continue in operation. You can halt a package using Serviceguard Manager or on the Serviceguard command line. Halting a package has a different effect than halting the node. When you halt the node, its failover packages may switch to adoptive nodes.

If you want to start a package, then use the following commands

# cmviewcl –v –p <package name>

# cmmodpkg –e <package name>

OR

# cmrunpkg –n <host name> <package name>

# cmmodpkg –e <package name>

Deleting a Serviceguard Package

To delete a package, use the following commands

 # cmdeleteconf -p <package name>

Stopping and starting the Serviceguard Cluster

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

To stop the cluster, use the following commands

The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.

To stop the cluster, use the following commands

# cmiewcl –v

# cmhaltcl

OR

# cmhaltcl –f

To start the cluster: -

# cmruncl –v

# cmviewcl –v

Automatically Restarting the Serviceguard Cluster

You can configure your cluster to automatically restart after an event, such as a long-term power failure, which brought down all nodes in the cluster. This is done by setting AUTOSTART_CMCLD to 1 in the

/etc/rc.config.d/cmcluster file.

Stopping and starting a node in the Serviceguard Cluster

Starting a node will not cause any active packages to be moved to the new node. However, if a package is DOWN, has its switching enabled, and is able to run on the new node, that package will automatically run there.

To stop a node in the cluster

# cmhaltnode –n <node name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=0

To start a node in the cluster.

# cmrunnode –n <name name>

# cmviewcl –v

# vi /etc/rc.config.d/cmcluster

AUTOSTART_CMCLD=1

Deleting a cluster

When there are no packages left in the cluster, and you want to delete the  cluster, use the following commands.

# vgchange –c n vg01

# cmhaltcl –f

# cmdeleteconf –c <cluster_name>

# vgchange –a y vg01

# lvremove /dev/vg01/lvol1

# vgremove vg01

# rm –r /dev/vg01

In terms of computers HA means a system design protocol which ensures the availability of a resource in a given amount of time or any terms of measurement.

HA-Clustering and Failover

This is a computer cluster implemented for making a resource available all the time or reducing the downtime of the resource. For example, an application server serving the user requests continuously and in come point of time it stopped responding. In that time the users waiting for a response from the server should not get disappointed.

Here comes the clustering technology, the cluster configuration in the network transfers or redirects the user requests to the backup server. This process is called Failover.

High-availability-clustering

The Clustering Software

The clustering software plays main role in the successful functioning of the HA-Clusters. Everyone have a question of who maintains the information and transactions processed by the First Machine.

Yes, the clustering software does that for us. It will copy the last good known configuration from the machine which failed to complete its duty to the machine which is ready to serve for us.

Heart Beat of Clustering

HA clusters use a private Heart-Beat network. This network is responsible for monitoring the health status of all the nodes in the cluster.

HA-Clustering-Heart Beat

Split-Brain in Cluster network

The term “Split-Brain” technically explains the following scenario. When all the private network link goes down but the individual cluster nodes are still running, there is a possibility of confusion. Each node in the cluster may think that other node is down and will supply duplicate data. So this is a tedious configure the network to avoid split-brains.

Cluster Node Configurations

 These are the available clustering node configurations

 1. Active/Active – Traffic is directed to another existing node or balanced across all other nodes

2. Active/Passive – A complete new node is served when primary node fails

3.  N+1 – Serving a complete new node in the place of primary node and the new node    in the cluster must be capable of handling any services which the primary node handled with/without any special additional software.

4. N+M – Here more than one dedicated nodes are served for handling failovers.
This   may require high cost and maintenance.

5. N-to-1 – Here the backup node is serving temporarily until the primary node is brought back online. Then the running services are transferred to primary node.

6. N-to-N — A combination of Active/Active and N+M clusters, N to N clusters.

Requirements for a High Availability Clustering Environment

The HA-Clustering is widely used technology in Data centers and Industries where adaptive infrastructure is required. This requires high investment hence this is called as costly technology.

Below given are the minimum requirements for setting up a HA Clustered Environment.

1. You must be having suitable advanced application which is capable to support HA-C. Not all the applications can run in High Availability Environment.
2. The application should be capable of running its own multiple instances.
3. The application you are using in HA-C must have command line utility. i.e ability to control the app through command line interface by executing some commands.
4. The application should have the capability of using Storage Area Network (SAN).

The application must be capable of storing the data in a SAN, so that even a crash occurs the other back-up node can use the saved-state of information.

Applications of HA Clustering Technology

So, where is this clustering more useful and applied ? HA Clustering is used for

1. Critical databases

2. File sharing networks

3. Business Applications

4. E-Commerce Websites

Some famous HA-Clustering products

My next update in HA-Clustering would be about configuring the HA clusters in live environment.

Wait for a couple days….

Regards,
R.Gopinath

HP-UX LVM is used to manage disk space with special features like mirroring, high availability, stripping, etc. Physical volumes can be organized into a logical volume groups with equal size extent.

Each volume group can have one or more physical volumes. A system can have more than single volume groups. A volume group can be moved from a system to another instead of physical disk.

A volume group can be divided into virtual disks, called logical volumes. A logical volume can span a number of physical volumes or represent only a portion of one physical volume. Disks can be added into logical volumes with different sizes.

The following commands are used to create, display and manipulate the LVM:

1. pvcreate

Creates physical volume to use in logical volume group.

Syntax:  pvcreate -f /dev/rdsk/c2tod1

2. pvdisplay

Displays information about physical volumes in LVM volume group.

Syntax:  pvdisplay /dev/dsk/c2tod1

3. pvmove

Move allocated physical extents frm one LVM physcal volume to another.

Syntax:  pvmove /dev/dsk/c1t0d0 PVG0

4. vgcreate

Creates LVM volume group.

Syntax:  Create a directory for volume group. 

mkdir /dev/vg00 

vgcreate -s 2 /dev/vg00 /dev/dsk/c1t0d0

5. vgdisplay

Display information about LVM volume groups.

Syntax:  vgdisplay -v /dev/vg00

6. vgexport

Export an LVM volume group and its associated logical volumes.

Syntax:  vgexport -v -f vg00.outfile /dev/vg00

7. vgimport

Import an LVM volume group onto the system.

Syntax: vgimport -v /dev/vg00 /dev/dsk/c0t1d0

8. vgextend

Extend an LVM volume group by adding physical volumes.

Syntax:  vgextend /dev/vg00 /dev/dsk/c0t1d0

9. vgreduce

Remove physical volumes from an LVM volume group.

Syntax:  vgreduce /dev/vg00 /dev/dsk/c0t1d0

10. vgremove

Remove LVM logical volume group from system.

Syntax:  vgremove /dev/vg00 /dev/dsk/c0t1d0

11. vgscan

Scans physcal volumes for LVM volume group.

Syntax: vgscan

12. lvcreate

Creates logical volume in LVM volume group.

Syntax: 

lvcreate -L 100 /dev/vg03

13.  lvdisplay

Displays information about logical volumes .

lvdisplay /dev/vg00

14. lvextend

Increase space,mirrors for logical volumes.

lvextend -L 400 /dev/vg03

15.  lvreduce

Reduce number of physical extents, mirrors.

lvreduce -L 50 /dev/vg03

16. lvremove

Remove one or more logical volumes from system.

lvremove -f /dev/vg03

17. lvlnboot

Prepare LVM logical volume to be root, boot, primary swap, or dump volume.

lvlnboot -r /dev/vg00

18. lvrmboot

Remove logical volume link to root, primary swap, or dump volume.

lvrmboot -r /dev/vg00

Regards,
R.Gopinath

vPars Is a Virtual Partitions product that enables you to run multiple instances of HP-UX simultaneously on one hard partition by dividing that hard partition further into virtual partitions.

Each virtual partition is assigned its own subset of hardware, runs a separate instance of HP-UX, and hosts its own set of applications. Because each instance of HP-UX is isolated from all other instances, vPars provides applicationand Operating System (OS) fault isolation.

Each instance of HP-UX can have different patches and a different kernel .

vPar-Representation

Key Terms of vPar

 

Complex:

A complex is the entire partitionable server, including both cabinets, all cells, I/O chassis, cables, and power and utility components.

Cabinet

A cabinet is the Superdome hardware “box”, which contains the cells, Guardian Service Processor (GSP), internal I/O chassis, I/O fans, cabinet fans, and power supplies. A complex has up to two cabinets.

Hard partition

A hard partition is any isolated hardware environment, such as an nPartition within a Superdome complex or an entire rp7400/N4000 server.

npartition

An nPartition is a subset of a complex that divides the complex into groups of cell boards where each group operates independently of other groups. An nPartition can run a single instance of HP-UX or be further divided into virtual partitions.

Virtual partition

A virtual partition is a software partition of a hard partition that contains an instance of

HP-UX. Though a hard partition can contain multiple virtual partitions, a virtual partition cannot span a hard partition boundary.

vPars Monitor and Database

vPars 

For each hard partition, the vPars Monitor manages the assignment of hardware resources to virtual partitions, boots virtual partitions and their kernels, and emulates certain firmware calls. By emulating these specific calls, vPars creates the illusion to each HP-UX instance that it is running on a standalone server, consisting of the hardware that has been assigned to it.

Once a virtual partition is launched, the Monitor transfers ownership of the hardware to the virtual partition. At that point the Monitor is not involved in accessing I/O hardware, physical memory, or process to processor cycles: the individual HP-UX instances have complete ownership of their respective hardware resources. This allows each partition to run at full speed.

vPars Partition Database

At the heart of the vPars Monitor is the partition database. The partition database contains partition configuration information. Using the partition database, the Monitor tracks which virtual partitions exist and what hardware resources and partition attributes are associated with each partition.

When the Monitor boots , it reads a copy of the partition database from a file on the same disk from which the Monitor /stand/vpmon is booted. The default file is /stand/vpdb. Then, the Monitor creates a master copy of the vPars partition database in the memory reserved by the Monitor.

The operating system of each virtual partition also keeps a local copy of the partition database in a file, by default /stand/vpdb, on its local boot disk

 Boot Sequence

This section describes the boot differences in a vPars system relative to a non-vPars system.

Boot Sequence: Quick Reference

On a server without vPars, a simplified boot sequence is:

 

 

 

vPar-boot-sequence

 

 

Adding vPars adds the Monitor layer, so now hpux (for Integrity, hpux.efi) loads the Monitor.

Then the Monitor boots the kernels of the virtual partitions. The boot sequence becomes

 

 

 

vPar-boot