Process Counts Example
Case Study 4: ProcCountsChk
Recently, we have faced a crisis where a bug in the current version of our Web-based e-mail client has the imapd, under occasional and mysterious circumstances, spawning instances of itself every second or so. For a handful of users, we are seeing occasional "imapd storms" with per-user imapd counts reaching into the dozens, hundreds, and sometimes even thousands! At about the same time, but for different reasons, we began seeing "ypserv storms". Not only do these storms risk losing user mail files, they also imperil the entire system. Listing 7 is a Pikt script we have put into operation to deal with these sorts of problems.
Listing 7: ProcCountsChk
ProcCountsChk
init
status active
level emergency
task "Report unusually high counts of per-user procs."
// note: a defunct process might show an empty comm field
// below, so we pipe the ps output through the awk filter
input proc "=ps -eo user,comm | =behead(1) | =awk 'NF==2' | \
=sort | =uniq -c"
dat #count 1
dat $user 2
dat $proc 3
begin // read in process and threshold data from objects file
if #fopen(PROCCOUNTS, "=proccounts_obj", "r") != #err()
while #read(PROCCOUNTS) > 0
if #split($rdlin) == 5
set #lgcnt[$1] = #val($2) // log thresholds
set #alcnt[$1] = #val($3) // alert thresholds
set #pgcnt[$1] = #val($4) // page thresholds
set #klcnt[$1] = #val($5) // kill thresholds
// else send an error message?
fi
endwhile
do #fclose(PROCCOUNTS)
else
output mail "Can't open =proccounts_obj for reading!"
quit
fi
rule
foreach #keys($pr, #lgcnt)
if $proc =~~ "$pr$" // '=~~', not 'eq', so that '\\*'
// works as a default
if #lgcnt[$pr] && #count >= #lgcnt[$pr]
// for gathering diagnostic stats
output log "=proccounts_log" $inline
fi
if #alcnt[$pr] && #count >= #alcnt[$pr]
output mail $inline
if $proc eq "imapd" // special case
=archive_mail_file($user, #true())
fi
fi
if #pgcnt[$pr] && #count >= #pgcnt[$pr]
exec wait "echo '=pikthostname: $inlin' | \
=mailx -s '=pikthostname: $inlin' \
=pagesysadmins"
pause 5
fi
if #klcnt[$pr] && #count >= #klcnt[$pr]
=kill_user_proc($proc, $user, #true())
fi
next // next input line
fi
endforeach
The input proc statement yields input like
34 root /usr/lib/sendmail
404 chico imapd
1 zeppo imapd
In the begin section, we read data in from the ProcCounts.obj file (see Listing 8).
Listing 8: ProcCounts
ProcCounts
// 0 signifies take no action; 1 signifies always take action
// proc log alert page kill
# if moscow
imapd 10 100 1000 100
# endif
# if mailserver
sendmail 50 100 200 200
# else
sendmail 5 10 20 40
# endif
# if nisserver
ypserv 2 3 3 0
# endif
crack 1 1 1 1
sniffit 1 1 1 1
// ...
// wild card should be last in ProcCounts list
\\* 10 20 40 0
In the script's only rule, we check to see if the actual per-user process count exceeds the thresholds we set in the begin section, also if the threshold is non-zero.
Instead of 'foreach #keys($pr, #lgcnt)', we could have used 'for $pr in #keys(#lgcnt)'. These accomplish the same purpose but with somewhat different syntax. Variety of expression and keyword synonyms are typical of Pikt. Did you notice the use of 'if ... endif' in Case Studies 2 and 3 as opposed to 'if ... fi' in the current case study? Another example: elif, elsif, elseif are synonymous, and all achieve identical effect.
If the #lgcnt[] threshold is non-zero and if the process count exceeds the #lgcnt[] threshold, we log some diagnostic statistics for post-mortem analysis. If the process count exceeds #alcnt[], we send alert mail reporting that fact. In the case of imapd only, we also backup the user's mail file by means of the =archive_mail_file() macro (not shown).
If #count exceeds #pgcnt[], we send a short alert message to =pagesysadmins, a macro that resolves to the sysadmins' pager numbers.
Finally, if #count exceeds #klcnt[], we kill off the user processes by means of the =kill_user_proc() macro (see Listing 9).
Listing 9: kill_user_proc()
kill_user_proc(P, U, M)
// kill off all instances of a given process for a given user
// (P) is the process name (e.g., $proc, or "imapd")
// (U) is the user (e.g., $user, or "root")
// (M) is whether or not to output mail (e.g., #true())
set #killcount = 1 // initialize
while #killcount > 0
set #killcount = 0
do #popen(KILL, "=ps -eo pid,user,comm", "r")
while #read(KILL) > 0
if #split($readline) != 3
cont
fi
if $2 eq (U)
&& $3 eq (P)
#ifdef debug
output log "=proccounts_log" "$1, $2, $3"
output log "=proccounts_log" "(P), (U), $text((M))"
#endifdef
exec wait "=kill -9 $1"
set #killcount += 1
fi
endwhile
do #pclose(KILL)
endwhile
if (M)
output mail "killed all (U) (P) processes"
fi
Here is a sample alert message:
PIKT ALERT
Tue Apr 25 00:17:02 2000
moscow
URGENT:
ProcCountsChk
Report unusually high counts \
of per-user procs.
404 chico imapd
saved user mail file as
/var/mail/arc/chico.956639822
killed all chico imapd \
processes
(We still don't have an understanding of these problems, much less fixes, but at least we are not losing any more user e-mail, and our mail server is coping.)
Before leaving ProcCountsChk, note that by defining all count thresholds to 1 across the board, we can guard against users running "dangerous" or "forbidden" programs such as Crack or Sniffit.
 prev page |
1st page | next page  |
