Systems Down Macro

The systems_down_alarms_macros.cfg is a script macro for reporting when systems go down or off the network and, optionally, when they come back up.

///////////////////////////////////////////////////////////////////////////////
//
// systems_down_alarms_macros.cfg
//
///////////////////////////////////////////////////////////////////////////////
 
systems_down(SYS)
	init
		status =piktstatus
		level =piktlevel
		task "Report systems down or off the network"
		input proc "(SYS)"
		dat $host 1
		keys $host

	begin
                // initialize the mission-critical systems list
		=initmisscrit
		if $alert() =~ "DownSystems|DownServers|DownClients"
			set #interactive = #true()
		else
			set #interactive = #false()
		fi

	rule	// determine if host is mission-critical
		=setmisscrit

	rule
		=bypass_server_reboots

	rule	// if high-level alert, bypass non mission-critical systems
		if =highlevelalert
			if ! #misscrit
				set $state = %state
				next
			fi
		fi

	rule	// if low-level alert, bypass mission-critical systems
		if ! =highlevelalert
			if #misscrit
				set $state = %state
				next
			fi
		fi

#ifdef debug
	rule
		output $host
#elsedef
	rule
		if #interactive
			output $host
			output =newline
		fi
#endifdef

	rule	// initialize messages
		set $dnmsg = "$host is down, or off the network (ping failure)"
		set $upmsg = "$host is (back) up"

	rule	// initially, assume system is up
		set $state = "+"

	rule	// ping the host
		// do the initial poll quickly
		if =pingfail($host, 1, 1)
			// if the first ping failed, try again with more
			// retries and longer timeouts
			if =pingfail($host, 3, 5)
				set $state = "-"
			fi
		fi

	rule
		if $state eq "+"
#ifdef verbose
			// for high-level alerts, report if systems back up
			if =highlevelalert
				// if state was "-", is now "+", report change
				if #defined(%state) && $state ne %state
					output mail $upmsg
				fi
			fi
#endifdef
			next
		fi

	// only down hosts after this point

	rule	// report downages for interactive scripts
		if #interactive
			output $dnmsg
			output =newline
			next
		fi

	// non-interactive scripts after this point

	rule	// page, but only periodically, if highest-level alert
		if =highestlevelalert
			=hourly(=page($dnmsg, =allpager, =always), )
		fi

	rule	// for all systems, always report new downages
		if $state ne %state
			output mail $dnmsg
			next
		fi

	rule	// for missioncritical systems, report continuing downages,
		// but only periodically
		if #misscrit
			=every_four_hours(output mail $dnmsg, )
			if =highestlevelalert
				=hourly(=output_other_mail(SYSDOWN, 'PIKT SysDown', =sysadmins, $dnmsg), )
			fi
			next
		fi

	end
		quit
 
///////////////////////////////////////////////////////////////////////////////

You might invoke the =systems_down() macro in your alarms.cfg file thusly:

///////////////////////////////////////////////////////////////////////////////
//
// downage_alarms.cfg
//
///////////////////////////////////////////////////////////////////////////////
 
#if piktmaster
 
SysDown
        =systems_down(=piktc -L -H down)

#endif

///////////////////////////////////////////////////////////////////////////////

#if piktmaster | piktmistress

ACDown

        =systems_down(=piktc -L +H ac)

#endif

///////////////////////////////////////////////////////////////////////////////

#if piktmaster | piktmistress

PowerDown

        =systems_down(=piktc -L +H power)

#endif
 
///////////////////////////////////////////////////////////////////////////////

where 'down' is a host group of known down systems (specified in down_systems.cfg), 'ac' is a host group of networked air-conditioning systems, and 'power' is a host group of networked power supply systems.

Since monitoring air conditioning and power unit downages is so vital, we run these scripts on both the piktmaster system as well as the so-called 'piktmistress', an alias (specified in systems.cfg for a system that backs up the piktmaster for certain crucial functions.

In the SysDown macro invocation, the macro argument '=piktc -L -H down' is a call to piktc to list all systems except for known down systems.  Similarly, the macro arguments in ACDown and PowerDown has piktc list the air conditioning and power systems respectively.

Output from this script might look like, for example:

URGENT:
    SysDown
       Report systems down or off the network

       oslo is down, or off the network (ping failure)
       manchester is down, or off the network (ping failure)
       kiev is (back) up

For more examples, see Samples.