Script Development and Testing
(NOTE: Some of the techniques shown or described on this page--marked in purple--require new features in the latest official PIKT 1.19.0 release (pikt-current.tar.gz) that are unavailable in any previous version.)
When developing Pikt scripts, the usual procedure is to write the init section with the appropriate input statement, then perhaps add a rule to display the script input. After registering the script in alerts.cfg, you would then install the script, run it interactively, add some rules, reinstall, rerun, add some more rules (or maybe revise or remove existing rules), reinstall, rerun, and so on until you have successfully accomplished the script task.
So, for example, if you were developing a script to report high system load averages, you might begin with the following addition to alarms.cfg (or one of its #include files):
///////////////////////////////////////////////////////////////////////////////
LoadAverage
init
status =piktstatus
level =piktlevel
task "Report perilously high system load averages"
input proc "=uptime"
rule
output $inlin
///////////////////////////////////////////////////////////////////////////////
You would register the script in alerts.cfg; we recommend in a special Test alerts group:
///////////////////////////////////////////////////////////////////////////////
Test
timing =piktnever
mailcmd "=mailx -s 'PIKT Alert on =pikthostname: Test' =pikt-test"
status testing
level debug
alarms
LoadAverage
///////////////////////////////////////////////////////////////////////////////
Next you would install the Test alert, with the newly registered LoadAverage script, using a piktc command like:
# piktc -iv +A Test +H piktmaster processing hamburg... installing file(s)... Test.alt installedThen run the script interactively (on the current system, the piktmaster) with:
# pikt +A Test 9:12am up 6 days 1:36, 18 users, load average: 0.00, 0.07, 0.11We'll have more to say about the script development and testing procedure as we go along (for example, enclosing the 'output $inlin' statement within '#ifdef debug ... #endifdef' would be more appropriate), but recycling through the above edit-install-test sequence (with just the one initial edit to alerts.cfg) is basically how you should proceed.
If you were writing a script to analyze log files, you probably would want your input statement to look something like
#ifdef debug
input file "/var/log/messages"
#elsedef
input logfile "/var/log/messages"
#endifdef
'input file' invariably begins input at the beginning of the file, while 'input logfile' begins input with the latest new material. By using 'input logfile, you might see different input each time you run the script, or worse, there may be no new log file entries, hence no script input at all. During the development process, you probably want to be looking at the same input each time you run the script, so initially use 'input file', not 'input logfile'.
As you go along, you will want to thoroughly test your setup. You don't want to install a buggy Pikt script to report security break-ins, then be hit with hundreds of pages going off at once because of some stupid programming error. Or have some wayward script accidentally delete /etc/passwd or some other important system file. Following are some tips and techniques you might use to avoid making some embarrassing and maybe even disastrous mistakes.
A standard debugging technique is to insert diagnostic code, maybe simply to output (to stdout) the current input line, as in
#ifdef debug
rule
output $inlin
#endifdef
or, for example, to show some parsed ps input (in a script 'input proc' statement):
#ifdef debug
rule
output "$text(#cpu,1) $text(#mem,1) $prc"
#endifdef
Note that you would ordinarily use 'output mail' or 'output log' or some other variant in your monitoring scripts. When developing and testing Pikt scripts, however, although you might want to test report mailing and message logging, typically you will run your scripts interactively at the command line and view script output on-screen. Hence the use of simple 'output'.
Here is a suggested debug stanza from defines.cfg:
///////////////////////////////////////////////////////////////////////////////
debug
#if test
TRUE
#else
FALSE // by default, don't run any debug code
#endif
///////////////////////////////////////////////////////////////////////////////
When debugging, you might want to see verbose script output. For this purpose, you might also use the verbose define:
///////////////////////////////////////////////////////////////////////////////
verbose
#if new
TRUE // usually set this to FALSE; but set this to TRUE
// when debugging; or occasionally set this to TRUE
// to get a fuller report of all that PIKT is
// doing silently, behind-the-scenes
#else
FALSE
#endif
#setdef verbose = FALSE // general override for now
///////////////////////////////////////////////////////////////////////////////
Note how, by means of the #setdef directive, we set the verbose define to FALSE. We can set verbose to TRUE by changing the line to '#setdef verbose = TRUE', or (in the case of new systems) by commenting out the #setdef directive, or by using 'piktc ... +D verbose' when installing or configuring PIKT.
Here is a typical use of the verbose directive:
rule // report size change
if #defined(%size)
&& $size ne %size
=output_alarm_log("=piktfileproblems_log"
"$name has changed size to $size,
was %size")
#ifdef verbose
output mail "$name has changed size to $size,
was %size"
#endifdef
set #changed = #true()
endif
If a file size changes, in all cases we log this, but we only send alert e-mail if in verbose mode.
When testing and debugging, we probably don't want PIKT to exec any actions that might actually change anything. We would use the doexec define for this purpose.
Suppose we are developing a script, RunawayCPUProcs, to report "runaway" CPU processes (that is, using 100% or nearly 100% of the system CPU). The script is written (and maybe some other PIKT components are in place, such as objects.cfg pieces), and now it is time to install and try out our new code. The best way to proceed is to run the new script interactively within a special Test alert, defined in alerts.cfg for example as:
///////////////////////////////////////////////////////////////////////////////
#ifdef pikttest
//# if piktmaster
Test // use this for testing newly developed alarm scripts;
// install with:
// 'piktc -ivT +D test +A Test +H ...'
// or maybe:
// 'piktc -ivT +D test debug verbose -D page doexec +A Test +H ...'
// after testing, remove all traces of the Test alert
// with:
// 'piktc -tvT +A Test +H ...'
timing 5 * * * * 0
// timing =piktnever
nicecmd "=nice -n 19"
mailcmd "=mailx -a 'From: piktadmin' -s 'PIKT Alert on =pikthostname: Test'
=pikt-test"
status testing
level debug
alarms
// NightlyUpdatesLogScan
RunawayCPUProcs
//# endif // piktmaster
#endifdef // pikttest
///////////////////////////////////////////////////////////////////////////////
pikttest is a special, built-in PIKT define for configuration-wide testing. Unless we set pikttest to TRUE at the piktc command line, as in
# piktc -ivT +A Test +H piktmasterby default pikttest is set to FALSE, and the Test stanza in alerts.cfg would be ignored.
In the Test alert comments, we refer to another test define with '+D test'. The test define would appear in defines.cfg as:
test FALSE // set to TRUE when we want to run PIKT in test
// mode (e.g., after a major program revision),
// or when we are testing a new alarm/script
Unlike pikttest, which is set configuration-wide using the '-T' option with piktc, the conventional test define is something you can turn on and off at will (for example, by means of a '#setdef test = TRUE' or '#setdef test = FALSE' anywhere within your PIKT configuration).
Note the alert timing. In the example, we have set the Test script to run every 5 minutes. To be on the safe side, to ensure that this Test script is never accidentally scheduled to be run by the piktd daemon, we could instead uncomment and use the 'timing =piktnever' line.
Although we have explicitly set 'status testing', this will happen by default when we use the '-T' option with piktc. In 'status testing' mode, command strings are reported and/or logged rather than executed. This is a surefire way to prevent accidental and/or disastrous command execs.
So, we install the Test alert on the piktmaster using the piktc command shown above, or perhaps by explicitly activating or deactivating some defines at the command line:
# piktc -ivT +D debug verbose -D page doexec +A Test +H piktmasterWe would then run the Test alert interactively on the piktmaster using the command
# pikt -T +A TestWe might then see, on-screen, any script output, including any special output we have activated by setting the debug and verbose (and perhaps other) defines to TRUE. The script should run safely with no risk of our inadvertently setting off a page rm'ing a system file or exec'ing some other command possibly harmful to the system.
If we encounter a bug in our RunawayCPUProcs script, we fix it (in alarms.cfg, or one of its #include files), then do a piktc reinstall, then redo the interactive pikt command. We follow this revise-reinstall-rerun sequence again (and again) until the script is bug-free and we are confident that everything is just right.
BUT, running RunawayCPUProcs interactively on the piktmaster is one thing; having piktd run this on maybe dozens of systems around our organization is another. We can extend our testing of the RunawayCPUProcs script by adding it to a regular, non-Test alerts.cfg stanza in the following fashion, for example:
///////////////////////////////////////////////////////////////////////////////
SysAdminsUrgent=pikttest // stuff deserving nearly immediate attention
#if missioncritical
timing
15 * * * 1-5 // mon-fri
15 */2 * * 0,6 // sat-sun
#else
timing
15 6-18 * * 1-5 // mon-fri
#endif
nicecmd "=nice -n 19"
mailcmd "=mailx -s 'PIKT Alert on =pikthostname: Urgent' =sysadmins-urgent"
lpcmd "=lp =piktprinter"
#ifndef pikttest
status active
level urgent
#elsedef
status testing
level debug
#endifdef
alarms // stuff of interest to the sysadmins
#ifndef pikttest
[... fully tested and bug-free scripts ...]
#elsedef // pikttest
RunawayCPUProcs
[... other scripts being tested ...]
#endifdef // pikttest
///////////////////////////////////////////////////////////////////////////////
=pikttest references a special, built-in PIKT macro designed to work in tandem with the pikttest define. The effect of appending =pikttest to the alert stanza name will become clear in just a moment.
We have inserted RunawayCPUProcs after the #elsedef in the SysAdminsUrgent alert. That is, only include RunawayCPUProcs if we install SysAdminsUrgent in pikttest testing mode.
In macros.cfg, we have defined a special =alerts macro in this way:
alerts
#ifndef pikttest
Emergency
SysAdminsUrgent
CodersUrgent
SysAdminsCritical
...
#elsedef
Emergency=pikttest
SysAdminsUrgent=pikttest
CodersUrgent=pikttest
SysAdminsCritical=pikttest
...
#endifdef
Now, if we want to install our production-ready scripts and alerts, we would use the command
# piktc -iv +A =alerts -H down sick processing genoa... installing file(s)... Emergency.alt installed SysAdminsUrgent.alt installed CodersUrgent.alt installed SysAdminsCritical.alt installed ... ...Or, if we want to install our test scripts and alerts, we would use the command
# piktc -ivT +A =alerts -H down sick processing genoa... installing file(s)... EmergencyTest.alt installed SysAdminsUrgentTest.alt installed CodersUrgentTest.alt installed SysAdminsCriticalTest.alt installed ... ...Note the addition of the '-T' (test) option in the second piktc example. Note also that the =pikttest macro has resolved to the 'Test' string appended to each alert name.
We usually install both production-ready and test alerts and scripts using the combined command:
# piktc -iv +A =alerts -H down sick; piktc -ivT +A =alerts -H down sickWe would schedule both using
# piktc -ev +A =alerts -H down sick; piktc -evT +A =alerts -H down sick(Of course, remember to do a 'piktc -rv -H down sick' to restart piktd after any alert rescheduling. Restarting piktd has it reread any new alert schedulings in piktd.conf.)
The net effect of all this is that, in addition to all the other alerts, we would install two different urgent sysadmins alerts: SysAdminsUrgent, and SysAdminsUrgentTest. The RunawayCPUProcs script would appear only SysAdminsUrgentTest, because that's how we specified the '#ifndef pikttest ... #elsedef ... #endifdef' in the SysAdminsUrgent=pikttest alerts.cfg stanza.
When developing, testing, and debugging scripts, we don't want to bother fellow staff members with verbose output or maybe embarrass ourselves by our coding mistakes. To shield our fellow staff members from test alert e-mails and other debugging output, we could configure our pikt_mail_macros.cfg #include file in this way:
///////////////////////////////////////////////////////////////////////////////
//
// pikt mail macros - pikt mail routing
//
///////////////////////////////////////////////////////////////////////////////
// piktadmin
byrd byrd\
piktadmin =byrd // pikt head honcho
///////////////////////////////////////////////////////////////////////////////
// everyone else
#ifdef pikttest
dowland =piktnullchar
boyce =piktnullchar
telemann =piktnullchar
tartini =piktnullchar
josquin =piktnullchar
#elsedef
dowland dowland\
telemann telemann\
tartini tartini\
josquin desprez.gmail\
boyce
# if missioncritical
boyce\
# else
=piktnullchar
# endif
#endifdef
///////////////////////////////////////////////////////////////////////////////
// mail groups
sysadmins =dowland =boyce =piktadmin =piktnullchar
coders =telemann =tartini =josquin =piktnullchar
///////////////////////////////////////////////////////////////////////////////
// the various pikt- macros, the addresses used in the alerts.cfg mailcmd
pikt-emergency =sysadmins =coders
sysadmins-urgent =sysadmins
...
///////////////////////////////////////////////////////////////////////////////
With pikttest set to TRUE, because we used the '-T' option with piktc, all e-mail macros for fellow staff members are set to the =piktnullchar, that is a blank.
In pikttest mode, the =sysadmins macro resolves to simply
byrd\so only the piktadmin receives alert e-mail; while in production, non-pikttest mode, the =sysadmins macro resolves to
dowland\ boyce\ byrd\(on mission-critical systems; on non mission-critical systems, =sysadmins would resolve to 'dowland\ byrd\'--that is, leaving out boyce).
By these clever macro tricks, and the complicated interplay of the pikttest define, the =pikttest macro, and the appropriate use of piktc commands, we have kept our testing alert e-mails "private"--FYI, for the piktadmin to see only.
In fact, even in non-test mode, we might have use for private messaging. For example, after major PIKT reconfigurations (creating new alert groups, moving scripts around from one alert group to another, trying out some new macros or defines or some other pervasive new PIKT technique, etc.), we might want to temporarily shut down all "public" PIKT messaging and route alert e-mails for a time to the piktadmin only. If we specify another define in defines.cfg:
private FALSE // set to TRUE if we want to keep all PIKT messaging
// private, sending only to the piktadmin, for example,
// when testing or making major configuration changes
in our pikt_mail_macros.cfg #include file, we might expand our pikt_mail_macros.cfg #include file to look something like this:
...
///////////////////////////////////////////////////////////////////////////////
// everyone else
#ifdef pikttest
dowland =piktnullchar
boyce =piktnullchar
telemann =piktnullchar
tartini =piktnullchar
josquin =piktnullchar
#elsifdef private
dowland =piktnullchar
boyce =piktnullchar
telemann =piktnullchar
tartini =piktnullchar
josquin =piktnullchar
#elsedef
# if new | sidelined | sick | down
dowland =piktnullchar
boyce =piktnullchar
telemann =piktnullchar
tartini =piktnullchar
josquin =piktnullchar
# else
dowland dowland\
telemann telemann\
tartini tartini\
josquin desprez.gmail\
boyce
# if missioncritical
boyce\
# else
=piktnullchar
# endif
# endif // new | sidelined | sick | down
#endifdef
///////////////////////////////////////////////////////////////////////////////
...
We would take PIKT report messaging private by re-enabling all alerts with
# piktc -ev +D private +A =alerts -H down sick; piktc -evT +D private +A =alerts -H down sickfollowed by a site-wide piktd restart ('piktc -rv -H down sick'). We could of course return to public messaging at some later time by re-enabling with 'piktc -ev -D private ...' (or instead of specifying '-D private' at the piktc command line just going with a 'private FALSE' default setting in defines.cfg).
If you have followed this rather complex discussion thus far, you can see that, depending on the system and how we have specified our PIKT defines, if
- we are in pikttest mode
- we are in "private" messaging mode
- for new, sidelined, "sick" or "down" systems
After we have experience running RunawayCPUProcs in pikttest mode and are confident in taking it public, we can move it within the SysAdminsUrgent=pikttest alerts.cfg stanza from test environment to production environment:
alarms // stuff of interest to the sysadmins
#ifndef pikttest
[... fully tested and bug-free scripts ...]
RunawayCPUProcs
#elsedef // pikttest
[... other scripts being tested ...]
#endifdef // pikttest
After making this change, remember to reinstall all alerts using a command such as (note the addition of the '-b' backup option):
# piktc -ivb +A =alerts -H down sick; piktc -ivbT +A =alerts -H down sickNow, RunawayCPUProcs output will be e-mailed publicly. And, we should add, any command execs will take effect since the script is no longer running in pikttest mode.
If you don't quite understand yet how this all works, don't be too troubled. These are advanced PIKT topics for the seasoned, experienced PIKT guru. Among all of these advanced techniques, employ what you understand at a level and pace you are comfortable with. Know, too, that there are other ways to do your script development and testing, debugging, alert messaging, etc. With PIKT, you have tremendous power and flexibility to achieve all sorts of cleverness, and do things your own way in a fashion that makes the most sense to you.
 prev page
|
1st page |
next page 
|
