Enhancing the Command Line

(NOTE: Some of the techniques shown or described on this page--marked in purple--require new features in the latest official PIKT 1.19.0 release (pikt-current.tar.gz) that are unavailable in any previous version.)

'piktc -x +C ...' is a fast, intelligent, macro-aware alternative for ssh'ing commands to run on remote systems.

In its simplest form, piktc remote command execution involves sending a single command to a single remote host, as in this command from the piktmaster hamburg to the PIKT slave xyzserver5:

# piktc -xv +C "w" +H xyzserver5

processing xyzserver5...
executing command(s)...
 02:11:44 up 72 days,  7:58, 37 users,  load average: 0.03, 0.06, 0.01
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
lully    pts/6     14Mar07 53days  0.01s  0.01s /bin/bash
schuetz  pts/10    Wed07   18:30m  0.01s  0.01s ssh root
root     pts/11    07May07  7days  0.02s  0.02s -bash
schuetz  pts/7     30Mar07 19:18m  1:46   0.81s abcmodel.prod
root     pts/2     04May07  9days  0.19s  0.02s vi /etc/xyzserver/node.conf
"w" executed

The '-x' option says to execute, '-v' specifies verbose mode, '+C "w"' indicates the command string, and '+H xyzserver5' indicates the remote host.

Without the -v option, the output would be

 02:11:44 up 72 days,  7:58, 37 users,  load average: 0.03, 0.06, 0.01
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
lully    pts/6     14Mar07 53days  0.01s  0.01s /bin/bash
schuetz  pts/10    Wed07   18:30m  0.01s  0.01s ssh root
root     pts/11    07May07  7days  0.02s  0.02s -bash
schuetz  pts/7     30Mar07 19:18m  1:46   0.81s abcmodel.prod
root     pts/2     04May07  9days  0.19s  0.02s vi /etc/xyzserver/node.conf

You can send commands to more than one remote host, as in:

# piktc -x +C "uptime" +H stockholm prague venice
 05:25:04 up 2 days, 11:36,  3 users,  load average: 1.00, 1.00, 1.00
 05:25:04 up 2 days, 11:44,  1 user,  load average: 0.00, 0.00, 0.00
 05:25:05 up 2 days, 11:57,  1 user,  load average: 0.07, 0.03, 0.00

But note there is no guarantee that the output order will match the host order you specify at the command line; also that with a long list of hosts you would not easily connect the hostname to the command output. So, you might add verbose mode, or, if you prefer, you could add a second and third command (hostname, echo) to the command string, also omit the -v option, as in

# piktc -x +C "hostname; uptime; echo" +H stockholm prague venice
venice
 05:27:47 up 2 days, 11:38,  3 users,  load average: 1.00, 1.00, 1.00

stockholm
 05:27:47 up 2 days, 11:46,  1 user,  load average: 0.00, 0.00, 0.00

prague
 05:27:47 up 2 days, 11:59,  1 user,  load average: 0.05, 0.03, 0.00

You can, if you prefer, break your commands into separate quoted strings, as in

# piktc -x +C "hostname" "uptime" "echo" +H stockholm prague venice

You can specify single systems, as in '+H xyzserver5', or you can specify several systems, as in '+H stockholm prague venice'. You can also specify system groups (defined in systems.cfg), as in this command to check some directory ownerships and permissions on the DNS servers:

# piktc -x +C "hostname; ls -ld /var/bind; echo" +H dnsserver

What if you want to run this command, but excluding dusseldorf, which is down for maintenance (or some other reason)? (If you try sending a piktc command to a down system, the piktc command will hang for a while, resuming operation again only after a ~10 second timeout.) For this purpose, piktc allows some rudimentary host set operations, as in

# piktc -x +C "hostname; ls -ld /var/bind; echo" +H dnsserver -H dusseldorf

Although you can specify down systems at the command line, it is usually better to define the set of down systems in systems.cfg as:

down    // set '-H down' for these
        members
#verbatim

where the down_systems.cfg file might look something like:

hotspare
jdkacme3
dusseldorf
tirana

It is also usual to specify a set of so-called "sick" systems--that is, systems that are up but have issues that prevent normal piktmaster to PIKT slave communications (such as RPC or other network difficulties, or maybe temporarily unresolved piktc-to-piktc_svc authentication problems):

sick    // set '-H sick' for these
        members
#verbatim

Then, in normal piktc operations, you would usually exclude the "down" and "sick" systems by appending '-H down sick' to all piktc commands, as in:

# piktc -x +C "hostname; ls -ld /var/bind; echo" +H dnsserver -H down sick

That way, we can avoid wasting time attempting to send commands to the down and sick systems.

You can send commands to all systems, as in this command to check the time and date everywhere:

# piktc -x +C "hostname; date" +H all -H down sick

Here, 'all' is a built-in, automatic hostgroup specifying all PIKT-defined systems (i.e., registered in systems.cfg).

You could also try this:

# piktc -x +C "hostname; date" -H down sick

Without specifying a +H and only specifying a -H, there is an implicit '+H all'.

You can add and subtract host sets at the piktc command line as much as you please, for example this command to restart the syslogd:

# piktc -x +C "/etc/init.d/syslog restart" +H dbserver pvsserver jdsserver
                                           -H murmansk madrid down sick

If one or more of your remote commands involves double quotes, you can wrap all of the +C commands using single quotes, as in

# piktc -x +C 'hostname; egrep "root|wheel" /etc/group; echo' +H brest livorno

You can employ awk, sed, sort, uniq, egrep and other Unix filters in your +C command strings, as in this command to poll all systems' crontabs to determine timing frequencies for the nightly-updates command:

# piktc -x +C "crontab -l | egrep nightly-updates |
                    egrep -v '^#' | awk '{print \$1 \" \" \$2}'"
                -H downsys | sort | uniq -c | sort

      1 02 20
      1 03 20
      1 04 20
      ...
      2 34 20
      2 58 20
      3 48 20
     10 26 20

Or feed piktc command output to Unix filters, as in this command to poll the AcmeViewStats.log (contains CPU usage info) across all client systems and calculate the AcmeView app's mean CPU usage:

root@# piktc -x +C "=cat =logdir/AcmeViewStats.log 2>/dev/null"
                     +H client -H down sick |
            egrep -iv "no such file" | awk '{print $NF}' |
            awk '{t+=$1; n+=1}; END {print t; print n; print t/n}'
6.52683
598
0.0109144

(That is, 6.52683 seconds total CPU time over 598 logfile entries, averaging to 0.0109144, or 0.7 seconds of CPU time for every 60 seconds of real time.)

In your +C command strings, be careful in some situations to use backslashes to escape single or double quotes.

Your piktc '+C' command strings can reference PIKT macros. To see, for example, which of your (up) systems are currently running the portmapper, you might issue the command

# piktc -xv +C "=psall | =egrep 'portmap|rpcbind'" -H down sick

where =psall is defined in the piktmaster macros.cfg file as

psall
#if linux
                =ps aux
#elsif solaris | hpux
                =ps -ef
#elsif sunos
                =ps -aux
...
#endif

=ps and =egrep would be defined with the appropriate command path for each operating system, for example, for SuSE Linux:

...
egrep           /bin/egrep
...
ps              /bin/ps
...

As another example, suppose we need to check /tmp usage across all of our (up) systems. We might run the command

# piktc -x +C "=hostname; =dfk /tmp" -H down sick
kiev0
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/sd0e             993006      17  893689     0%    /tmp
kiev
Filesystem            kbytes    used   avail capacity  Mounted on
swap                  769096     296  768800     1%    /tmp
ottawa
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda8              4150700    128044   3811808   4% /tmp
...

The =dfk macro automatically resolves, per the operating system on the target machine, to the (path and) command options giving us the desired consistent display (for example, to "/bin/df", "/bin/df -k", or "/bin/bdf").

Using macros in 'piktc -x +C' commands is extremely handy at sites running more than one operating system or operating system variant, for through the magic of PIKT macros the identical macro command string works the appropriate way everywhere.

You can also specify macros, perhaps with arguments, representing complex command sequences, then reference those macros in your piktc commands, for example:

# piktc -x +C "=startup(=mprod)" +H server -H down sick

piktc remote command execution is quite fast. The slowest part is the initial (and constant in duration) piktc preprocessing overhead. The actual command executions run very fast. Depending on your system and network speeds, you could typically poll or execute commands on 50 or so systems in a minute or less.

Despite this speediness, what if you want to issue some time-consuming or especially urgent command(s) in rapid succession on multiple hosts without waiting for the command(s) to finish on one host before proceeding to the next? Instead of '-x' (execute with wait), you could use '-X' (capital X, execute with no wait), as in this command to shut down all servers (perhaps because your air conditioning has failed and the machine room temperatures are reaching dangerously high levels):

# piktc -X "init 0" +H server -H down sick

(Note that if you use the '-X' option, you probably don't want to use the -v (verbose) option.)

What if you want to launch commands on remote systems, but with a pause between systems? You could use the -p option, as in:

# piktc -xp20v +C "/etc/init.d/acme-pvs restart" +H pvsserver -H down sick

which would run the PVS restart command on all pvsservers (minus any down or sick ones), but with a 20-second pause between systems.

You may also combine file installs with command executions in a single piktc operation, as in:

# piktc -ixv +F =aliases +C "=newaliases" +H vienna ...

processing vienna...
installing file(s)...
/etc/aliases installed
executing command(s)...
"=newaliases" executed

...

'-x' (also '-X') may be combined with many other piktc options, such as '-B' (for restoring from backup), '-e' (for enabling alerts), '-t' (deleting files), and so on. Note that, when combined with other piktc operations in this way, commands are always exec'ed after the other operation. (So, for example, 'newaliases' would always be exec'ed after the aliases file install.)

Using the 'piktc -L' (list) feature, you can maintain custom system lists, for example for referencing within other programs, as in this Perl statement

 = `piktc -L +H hpux -H down`;

You need not employ 'piktc -x +C' to remotely execute commands; you can still use ssh if you prefer, but use the 'piktc -L' to assist ssh, as in a command loop you might use to upgrade Solaris PIKT binaries:

# for sys in `piktc -L +H solaris -H piktmaster down sick`
> do
> echo $sys
> ssh $sys "/pikt/lib/programs/svcstart.pl -k;
            cp /pikt/bin/pikt* /pikt/bin/bak;
            cp /usr/local/pikt/bin/solaris/pikt* /pikt/bin;
            /pikt/lib/programs/svcstart.pl -r"
> done

It must be pointed out, though, that by using ssh instead of piktc, you do without several benefits, chief among which are PIKT macros in your command strings.

There is a way, however, to seamlessly combine ssh with PIKT remote command execution, and that's by using the piktc command execution substitute, piktx. Unlike piktc, piktx is ssh-based, and will run on any system (piktc is designed to run only on the piktmaster), if that system has certain basic, essential PIKT config files (e.g., a local copy of systems.cfg), also password-less ssh access to remote systems.

There's much more still that could be said about PIKT remote command execution, but this is enough for now.

It deserves emphasis: PIKT has much to offer, in its totality perhaps too much for your limited needs. In that case, just pick and choose among its many functionalities. We do have to say, though, that piktc (and piktx) remote command execution by itself is perhaps reason enough to justify using PIKT. You will be amazed how much time and effort this one PIKT feature alone might save you!

1st page