Developing Alarm Scripts

[posted 1999/09/03]

What follows is a short narrative of how I go about developing alarm scripts.  Some of what follows may or may not be obvious.

A while ago, we received the following alert from one of our Solaris systems:

                                PIKT ALERT
                         Wed Sep  1 08:30:13 1999
                                 cologne

CRITICAL:
    DfChkCritical
        Detect filesystem near-full situations

        Filesystem /var on /dev/dsk/c0t0d0s3 is 96% full, 2282 Kb left

        43563   /var/crash
        4671    /var/sadm
        4113    /var/pikt
        444     /var/adm
        131     /var/snmp
        85      /var/cron
        46      /var/spool
        30      /var/dmi
        22      /var/log
        14      /var/yp

That 43 MB in /var/crash came as a surprise.  Out of curiosity, I then issued the following command to check out our other Solaris systems:

# piktc -xv +C "=dusk /var/crash" +H solaris -H downsys

/var/crash was empty on most systems, held 20-40 MB on several other systems, and exceeded 300 MB on one system!

This called for development of another alarm:

///////////////////////////////////////////////////////////////////////////////

#if solaris

RemoveCrashFilesNotice  // before implementing this alarm, old /var/crash
                        // files had accumulated to as much as 300 MB on some
                        // of our systems

        init
                status active
                level notice
                task "Remove old system crash files"
                input proc "=find /var/crash -type f -exec =ll {} \\; 2>/dev/null"
                =lldata

        rule    // rm if more than one week old
                if #fileage($mon,$date,$time) > 7
                        exec "=rm $name"
                endif

#endif  // solaris

///////////////////////////////////////////////////////////////////////////////

To test this, I first added it to the Test alert (in alerts.cfg on the piktmaster, vienna) and installed Test on cologne:

vienna# piktc -iv +A Test +H cologne

Then, on cologne, I ran it manually:

cologne# pikt +A Test

(Note that, when developing or debugging an alarm script, you can also edit the .alt files directly on the clients.)

After verifying that it worked properly (by "ls -l /var/crash/cologne"), I then (on the piktmaster) deleted the temporary Test alert from cologne:

vienna# piktc -tv +A Test +H cologne

processing cologne...
disabling alert(s)...
Test disabled
deleting file(s)...
Test.alt deleted
deleting file(s)...
Test.hst deleted
deleting file(s)...
Test.log deleted

Then, in alerts.cfg (on vienna), I moved RemoveCrashFilesNotice from the Test alert to the Notice alert, taking care to put the appropriate #if wrapper around it:

Notice

...

        alarms

...

#if solaris
                        RemoveCrashFilesNotice
#endif

Finally, I installed the amended Notice alert on all solaris systems:

vienna# piktc -iv +A Notice +H solaris -H downsys

So, with RemoveCrashFilesNotice now in place, that's one less reason for /var to fill up.

[Here is a follow-up to the previous installment.]

Yesterday I walked through the process of creating a new alarm, RemoveCrashFilesNotice.  Today, we received a flurry of alerts messages like so:

                                PIKT ALERT
                         Thu Sep  2 09:55:50 1999
                                  minsk

DEBUG:
    PiktNoticeLogChk
        Detect pikt log errors

        /usr/bin/find: cannot open /var/crash: No such file or directory
        Sep  2 03:52:20 WARNING: in scan(), RemoveCrashFilesNotice, no input data

This sort of debug message is standard.  Some adjustments are in order.

First, we have to modify the alarm script by adding a "2>/dev/null" to the end of the input proc statement.

///////////////////////////////////////////////////////////////////////////////

#if solaris

RemoveCrashFilesNotice  // before implementing this alarm, old /var/crash
                        // files had accumulated to as much as 300 MB on some
                        // of our systems

        init
                status active
                level notice
                task "Remove old system crash files"
                input proc "=find /var/crash -type f -exec =ll {} \\; 2>/dev/null"
                =lldata

        rule    // rm if more than one week old
                if #fileage($mon,$date,$time) > 7
                        exec "=rm $name"
                endif

#endif  // solaris

///////////////////////////////////////////////////////////////////////////////

This will suppress the

        /usr/bin/find: cannot open /var/crash: No such file or directory

log entry.

Next, we go to PiktNoticeLogChk in alarms.cfg and add "RemoveCrashFilesNotice" to the list of alarms where empty input is permissible:

        rule    // a check against a badly formed input command
                // that results in no input
                if    $inline =~ "WARNING:.+no input data"
                   && $inline !~ "SysMsgScanNotice|
                                          NumberedPacctFileNotice|
                                          NumberedSyslogFileNotice|
                                          TmpChkNotice|
                                          DfChkNotice|MailChkNotice|
                                          MailFileChkNotice|
                                          SpoolChkDateNotice|MailArcChkNotice|
                                          MajordomoArcChkNotice|
                                          RemoveCrashFilesNotice"
                        output mail $inline
                        next
                endif

(It should be clear that a reinstall with

   piktc -iv +A Notice Debug +H all

is required after these edits.)

I contend that with these two adjustments, we shouldn't see any more debug messages like the one above.

Again, this is standard operating procedure for me.

For more examples, see Developer's Notes.