Encountered a shutdown issue with 11gR2 Clusterware on Redhad 5.4

We encountered a shutdown issue with 11gR2 Clusterware on Redhad 5.4

We encountered a shutdown issue with 11gR2 Clusterware and Redhad 5.4. The services would start fine, but the shutdown script never appeared to run before the shutdown of the OCFS2. This results in a not clean shutdown of the instance on the node.

The Solution

To solve this problem we have done to actions:

During the start stanza of the script, put in a command: touch /var/lock/subsys/ohasd Change the K19ohasd to K18ohasd in the /etc/rc?.d

1. The change in the /etc/init.d/ohasd

case "$1" in
  start)
    start
    touch /var/lock/subsys/ohasd
    ;;
  stop)
    stop
    rm /var/lock/subsys/ohasd
    ;;
esac

2. Change the shutdown sequence

OCFS2

rc0.d/K19ocfs2
rc1.d/K19ocfs2
rc6.d/K19ocfs2

rc2.d/S25ocfs2
rc3.d/S25ocfs2
rc4.d/S25ocfs2
rc5.d/S25ocfs2

OHASD

OHASD - (was K19)

rc0.d/K18ohasd
rc1.d/K18ohasd
rc2.d/K18ohasd
rc4.d/K18ohasd
rc6.d/K18ohasd

rc3.d/S96ohasd
rc5.d/S96ohasd

The Clusterware Startup Sequence overview

Short summary of the startup sequence: INIT spawns init.ohasd (with respawn) which in turn starts the OHASD process (Oracle High Availability Services Daemon). This daemon spawns 4 processes.

Level 1: OHASD Spawns:

cssdagent - Agent responsible for spawning CSSD. orarootagent - Agent responsible for managing all root owned ohasd resources. oraagent - Agent responsible for managing all oracle owned ohasd resources. cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).

Level 2: OHASD rootagent spawns:

CRSD - Primary daemon responsible for managing cluster resources. CTSSD - Cluster Time Synchronization Services Daemon Diskmon ACFS (ASM Cluster File System) Drivers

Level 2: OHASD oraagent spawns:

MDNSD - Used for DNS lookup GIPCD - Used for inter-process and inter-node communication GPNPD - Grid Plug and Play Profile Daemon EVMD - Event Monitor Daemon ASM - Resource for monitoring ASM instances

Level 3: CRSD spawns:

orarootagent - Agent responsible for managing all root owned crsd resources. oraagent - Agent responsible for managing all oracle owned crsd resources.

Level 4: CRSD rootagent spawns:

Network resource - To monitor the public network SCAN VIP(s) - Single Client Access Name Virtual IPs Node VIPs - One per node ACFS Registery - For mounting ASM Cluster File System GNS VIP (optional) - VIP for GNS

Level 4: CRSD oraagent spawns:

ASM Resouce - ASM Instance(s) resource Diskgroup - Used for managing/monitoring ASM diskgroups. DB Resource - Used for monitoring and managing the DB and instances SCAN Listener - Listener for single client access name, listening on SCAN VIP Listener - Node listener listening on the Node VIP Services - Used for monitoring and managing services ONS - Oracle Notification Service eONS - Enhanced Oracle Notification Service GSD - For 9i backward compatibility GNS (optional) - Grid Naming Service - Performs name resolution

The Clusterware Startup Sequence image