Swap File Utilization

In this example, we report excessive swap file utilization, exhaustion of system virtual memory.

The SwapExhausted script might send an alert message like the following:

                                PIKT ALERT
                         Wed Dec 12 21:13:21 2001
                                  kiev2

EMERGENCY:
    SwapExhausted
        Report when the system swapfile is exhausted

        swap utilization is 100%:

        swapfile             dev  swaplo blocks   free
        /dev/dsk/c0t1d0s1   32,9       8 1025112      0

        swap                  662992  655952    7040    99%    /tmp

        655652  /tmp/SAS_worka0000268D
        4       /tmp/screens

        contents of /tmp:

        total 624
        drwx------   2 droulser perf         473 Dec 12 17:41 SAS_worka0000268D
        drwxr-xr-x   2 root     other         69 Dec 12 08:14 screens
        -rw-rw-r--   1 root     sys       301828 Dec 12 19:32 ups_data

        last pid: 11924;  load averages:  0.27,  0.07,  0.05    21:13:24
        34 processes:  33 sleeping, 1 on cpu

        Memory: 224M real, 4320K free, 501M swap in use


          PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
        11924 root       1   0    0 1580K 1004K cpu0    0:01  1.27% top
            1 root       1  58    0  708K  164K sleep   0:08  0.09% init
         3524 droulser   4  51    0   24M   20M run     2:10 19.27% sas
        ...

In the SwapExhausted Pikt script following, note how we go beyond reporting just the fact that system swap space is exhausted.  We also provide diagnostic information to suggest possible causes.

#if solaris

SwapExhausted           // note: we should seriously consider having this
                        // script auto-installing auxiliary swap files, at
                        // least in some situations; we might then have
                        // a system startup script auto-delete these auxiliary
                        // swap files on system reboot

        init
                status =piktstatus
                level =piktlevel
                task "Report when the system swapfile is exhausted"
                input proc "=swap -l | =behead(1)"
                dat $swapfile 1
                dat #blocks   $-1
                dat #free     $
                // keys $swapfile

        begin
                set #blksum = 0
                set #fresum = 0
                set #hr = #hour()

        rule
                set $line[#innum()] = $inlin
                set #blksum += #blocks
                set #fresum += #free

        end
                set #use = (#blksum-#fresum)/#blksum
                if #use >= 100%
                        output mail "swap utilization is
                                     $text(100*#use,0)%:=newline"
                        output mail "swapfile           dev  swaplo blocks   free"
                        for #i=1 #i<=#innum() #i+=1
                                output mail $line[#i]
                        endfor
                        output mail =newline
                        output mail $command("=dfk /tmp | =behead(1)")
                        =dutop(10, /tmp)
                        output mail "contents of /tmp:=newline"
                        =outputproc(mail, "=ll /tmp")
                        output mail =newline
                        =toptop(20)
                endif

#endif  // solaris

This is just one program example.  You could add rules, or write new scripts, for example to:  report when system virtual memory runs low, create additional swap files on the fly, log memory usage over time, log and report per-process or per-user memory usage, etc.

For more examples, see Samples.