Process Counts Example

Case Study 4: ProcCountsChk

Recently, we have faced a crisis where a bug in the current version of our Web-based e-mail client has the imapd, under occasional and mysterious circumstances, spawning instances of itself every second or so.  For a handful of users, we are seeing occasional "imapd storms" with per-user imapd counts reaching into the dozens, hundreds, and sometimes even thousands! At about the same time, but for different reasons, we began seeing "ypserv storms".  Not only do these storms risk losing user mail files, they also imperil the entire system.  Listing 7 is a Pikt script we have put into operation to deal with these sorts of problems.
 

Listing 7: ProcCountsChk

ProcCountsChk

    init
        status active
        level emergency
        task "Report unusually high counts of per-user procs."
        // note: a defunct process might show an empty comm field
        // below, so we pipe the ps output through the awk filter
        input proc "=ps -eo user,comm | =behead(1) | =awk 'NF==2' | \
                    =sort | =uniq -c"
        dat #count 1
        dat $user  2
        dat $proc  3

    begin   // read in process and threshold data from objects file
        if #fopen(PROCCOUNTS, "=proccounts_obj", "r") != #err()
            while #read(PROCCOUNTS) > 0
                if #split($rdlin) == 5
                   set #lgcnt[$1] = #val($2)    // log   thresholds
                   set #alcnt[$1] = #val($3)    // alert thresholds
                   set #pgcnt[$1] = #val($4)    // page  thresholds
                   set #klcnt[$1] = #val($5)    // kill  thresholds
             // else send an error message?
                fi
            endwhile
            do #fclose(PROCCOUNTS)
        else
            output mail "Can't open =proccounts_obj for reading!"
            quit
        fi

    rule
        foreach #keys($pr, #lgcnt)
            if $proc =~~ "$pr$"   // '=~~', not 'eq', so that '\\*'
                                  // works as a default
                if #lgcnt[$pr] && #count >= #lgcnt[$pr]
                    // for gathering diagnostic stats
                    output log "=proccounts_log" $inline
                fi
                if #alcnt[$pr] && #count >= #alcnt[$pr]
                    output mail $inline
                    if $proc eq "imapd"  // special case
                        =archive_mail_file($user, #true())
                    fi
                fi
                if #pgcnt[$pr] && #count >= #pgcnt[$pr]
                    exec wait "echo '=pikthostname: $inlin' | \
                               =mailx -s '=pikthostname: $inlin' \
                               =pagesysadmins"
                    pause 5
                fi
                if #klcnt[$pr] && #count >= #klcnt[$pr]
                    =kill_user_proc($proc, $user, #true())
                fi
                next    // next input line
            fi
        endforeach

The input proc statement yields input like

   34  root /usr/lib/sendmail
  404  chico imapd
    1  zeppo imapd

In the begin section, we read data in from the ProcCounts.obj file (see Listing 8).
 

Listing 8: ProcCounts

ProcCounts

// 0 signifies take no action; 1 signifies always take action

//      proc            log         alert       page        kill

#    if moscow
        imapd           10          100         1000        100
#    endif
#    if mailserver
        sendmail        50          100         200         200
#    else
        sendmail        5           10          20          40
#    endif
#    if nisserver
        ypserv          2           3           3           0
#    endif
        crack           1           1           1           1
        sniffit         1           1           1           1
//      ...
// wild card should be last in ProcCounts list
        \\*             10          20          40          0

In the script's only rule, we check to see if the actual per-user process count exceeds the thresholds we set in the begin section, also if the threshold is non-zero.

Instead of 'foreach #keys($pr, #lgcnt)', we could have used 'for $pr in #keys(#lgcnt)'.  These accomplish the same purpose but with somewhat different syntax.  Variety of expression and keyword synonyms are typical of Pikt.  Did you notice the use of 'if ... endif' in Case Studies 2 and 3 as opposed to 'if ... fi' in the current case study? Another example: elif, elsif, elseif are synonymous, and all achieve identical effect.

If the #lgcnt[] threshold is non-zero and if the process count exceeds the #lgcnt[] threshold, we log some diagnostic statistics for post-mortem analysis.  If the process count exceeds #alcnt[], we send alert mail reporting that fact.  In the case of imapd only, we also backup the user's mail file by means of the =archive_mail_file() macro (not shown).

If #count exceeds #pgcnt[], we send a short alert message to =pagesysadmins, a macro that resolves to the sysadmins' pager numbers.

Finally, if #count exceeds #klcnt[], we kill off the user processes by means of the =kill_user_proc() macro (see Listing 9).
 

Listing 9: kill_user_proc()

kill_user_proc(P, U, M)
    // kill off all instances of a given process for a given user
    // (P) is the process name (e.g., $proc, or "imapd")
    // (U) is the user (e.g., $user, or "root")
    // (M) is whether or not to output mail (e.g., #true())
    set #killcount = 1    // initialize
    while #killcount > 0
        set #killcount = 0
        do #popen(KILL, "=ps -eo pid,user,comm", "r")
        while #read(KILL) > 0
            if #split($readline) != 3
                cont
            fi
            if    $2 eq (U)
               && $3 eq (P)
#ifdef debug
                output log "=proccounts_log" "$1, $2, $3"
                output log "=proccounts_log" "(P), (U), $text((M))"
#endifdef
                exec wait "=kill -9 $1"
                set #killcount += 1
            fi
        endwhile
        do #pclose(KILL)
    endwhile
    if (M)
        output mail "killed all (U) (P) processes"
    fi

Here is a sample alert message:

              PIKT ALERT
       Tue Apr 25 00:17:02 2000
                moscow

URGENT:
  ProcCountsChk
    Report unusually high counts \
      of per-user procs.

    404  chico imapd
    saved user mail file as
      /var/mail/arc/chico.956639822
    killed all chico imapd \
      processes

(We still don't have an understanding of these problems, much less fixes, but at least we are not losing any more user e-mail, and our mail server is coping.)

Before leaving ProcCountsChk, note that by defining all count thresholds to 1 across the board, we can guard against users running "dangerous" or "forbidden" programs such as Crack or Sniffit.

prev page

1st page

next page