root
292 lines
11 KiB
Plaintext
Executable File
292 lines
11 KiB
Plaintext
Executable File
Genders
|
|
|
|
Jim Garlick
|
|
|
|
High Performance Computing Systems Group
|
|
Livermore Computing
|
|
Lawrence Livermore National Laboratory
|
|
|
|
June 19, 2000
|
|
|
|
(Revised by Albert Chu - May 2003, April 2004)
|
|
(Revised by Ben Casses - July 2015)
|
|
|
|
------------------------------------------------------------------------
|
|
NOTE: This document is part of the "Genders" software package,
|
|
approved by LLNL for public release. Please see the DISCLAIMER file
|
|
distributed with the package for restrictions.
|
|
------------------------------------------------------------------------
|
|
|
|
Introduction
|
|
|
|
Genders is a simple concept that has been extended to address configuration
|
|
management issues that arise on MPP clusters[1]. The basic concept is that a
|
|
genders file containing a list of node names and their attributes is
|
|
replicated on each node in a cluster. By representing the configuration of
|
|
an MPP in this file, scripts and rdist Distfiles that might otherwise
|
|
contain hard-coded node lists, can be generalized to work with different
|
|
configurations of the same cluster, and even with multiple clusters.
|
|
|
|
The framework has proven to be useful on clusters ranging in size from the
|
|
ASCI Blue-Pacific SST system at 1464 nodes, to collections of two or three
|
|
workstations.
|
|
|
|
Basic Operation
|
|
|
|
This section describes the formats of the genders file and the nodeattr
|
|
command. The genders file, briefly described in the introduction, contains
|
|
a list of node names and attributes. The genders file is typically found
|
|
in /etc (on Linux when installed from the genders RPM) or /admin/etc (on
|
|
other systems).
|
|
|
|
Genders Format
|
|
|
|
Each line of the genders file may have one of the following formats. See
|
|
the section "Host Ranges" below for most information on host ranges.
|
|
|
|
nodename attr[=value],attr[=value],...
|
|
nodename1,nodename2,... attr[=value],attr[=value],...
|
|
nodenames[A-B] attr[=value],attr[=value],...
|
|
|
|
The nodename(s) is the shortened[2] hostname of a node. This is followed by
|
|
any number of spaces or tabs, and then the comma-separated list of attributes,
|
|
each of which can optionally have a value. The substitution string "%n" can
|
|
be used in an attribute value to represent the nodename. Nodenames can be
|
|
listed on multiple lines, so a node's attributes can be specified on multiple
|
|
lines. However, no single node may have duplicate attributes.
|
|
|
|
There must be no spaces embedded in the attribute list and there is
|
|
no provision for continuation lines. Commas and equal characters are special
|
|
and cannot appear in attribute names or their values. Comments are prefixed
|
|
with the hash chracter (#) and can appear anywhere in the file. The active
|
|
genders file is typically found at /etc/genders or /admin/etc/genders.
|
|
|
|
Here is an example genders file from a small 16-node linux cluster:
|
|
|
|
# slc cluster genders file
|
|
slci,slcj,slc[0-15] eth2=e%n,cluster=slc,all
|
|
slci passwdhost
|
|
slci,slcj management
|
|
slc[1-15] compute
|
|
|
|
Host Ranges
|
|
|
|
As noted in section above, the genders database accepts ranges of
|
|
nodenames in the general form: prefix[n-m,l-k,...], where n < m and l <
|
|
k, etc., as an alternative to explicit lists of nodenames.
|
|
|
|
This range syntax is meant only as a convenience on clusters with a
|
|
prefixNN naming convention and specification of ranges should not be
|
|
considered necessary -- the list foo1,foo9 could be specified as such,
|
|
or by the range foo[1,9].
|
|
|
|
Some examples of range usage follow:
|
|
|
|
foo01,foo02,foo03,foo04,foo05: foo[01-05]
|
|
foo3,foo7,foo9,foo11: foo[3,7,9-11]
|
|
fooi,fooj,foo0,foo1,foo2: fooi,fooj,foo[0-2]
|
|
|
|
Host ranges with suffixes, i.e. foo[0-2]x, are generally supported, but behavior
|
|
may differ.
|
|
|
|
Nodeattr Usage
|
|
|
|
A program called nodeattr is used to query data in the genders file. Because
|
|
the genders file is replicated on all nodes in a cluster, this query is a
|
|
local operation, not dependent on the network. Nodeattr is invoked as
|
|
follows:
|
|
|
|
nodeattr [-f genders] [-q | -c | -n | -s] [-r] attr[=val]
|
|
nodeattr [-f genders] [-v] [node] attr[=val]
|
|
nodeattr [-f genders] -l [node]
|
|
nodeattr [-f genders] -k
|
|
|
|
The -f option specifies a genders file path other than the default.
|
|
|
|
The -q, -c, -n, or -s options followed by an attribute name cause a list of
|
|
nodes having the specified attribute to be printed on stdout, formatted
|
|
according to the option specified: -q (default) is host range, -c is
|
|
comma-separated, -n is newline separated, and -s is space separated.
|
|
|
|
If none of the formatting options are specified, nodeattr returns a zero
|
|
value if the local node has the specified attribute, else nonzero. The -v
|
|
option causes any value associated with the attribute to go to stdout. If a
|
|
node name is specified before the attribute, it is queried instead of the
|
|
local node.
|
|
|
|
The -l option is used to generate a list of attributes for a particular
|
|
node. If no node is listed, all attributes appearing in the genders file
|
|
will be listed.
|
|
|
|
The -k option checks the genders file for proper formatting. Information
|
|
about improper formatting will be output to standard error.
|
|
|
|
Nodeattr can usually be found in /usr/bin (Linux RPM) or /admin/scripts
|
|
(other systems).
|
|
|
|
Programming with Genders
|
|
|
|
Three different APIs have been developed that allow programmers to parse
|
|
and query the genders file. "libgenders" is a C API which can be linked
|
|
by including genders.h and linking the library libgenders. "Libgenders"
|
|
is a Perl module developed with Perl extensions. "Genders" is another
|
|
Perl modules, but offers a more traditional Perl API than "Libgenders."
|
|
The reader can learn more about these libraries by reading the manpages
|
|
libgenders(3), Libgenders(3), and Genders(3).
|
|
|
|
Example: Rc Script
|
|
|
|
The nodeattr command can enable a single script to behave differently
|
|
depending on the role or gender of the node it is executing on, simplifying
|
|
configuration management of the MPP. For example, all nodes in an MPP
|
|
could have the same rc.local file:
|
|
|
|
#!/bin/sh
|
|
if nodeattr raid; then
|
|
initialize_raid
|
|
fi
|
|
if nodeattr login;
|
|
then sshd
|
|
fi
|
|
...
|
|
|
|
If the node has "raid" in its attribute list, it would execute
|
|
initialize_raid; if it had "login" in its attribute list, it would start the
|
|
secure shell daemon. This technique is simpler and less prone to failure
|
|
than adding code to test for "clues" to a nodes function, such as the
|
|
existence of a RAID device in /dev or a different set of installed products
|
|
on a login node. Further, changing the role of a node does not require
|
|
a laborious audit of scripts; one simply changes the genders file and
|
|
all genders-enabled scripts change their behavior.
|
|
|
|
Attribute values might be used to store node configuration information such
|
|
as a node's default IP route. For example, the node may have an attribute:
|
|
|
|
iprouter=big-rtr.llnl.gov
|
|
|
|
and rc script could extract this value and pass it to the route command:
|
|
|
|
route add net default `nodeattr -v iprouter`
|
|
|
|
Example: Host Lists
|
|
|
|
Host lists are commonly needed on MPP's to set up parallel jobs or to run
|
|
system administration commands on particular sets of nodes. In the following
|
|
example, nodeattr generates a newline-separated list of nodes with the login
|
|
attribute to pass to the fping parallel ping command:
|
|
|
|
fping `nodeattr -c login`
|
|
|
|
Batch system configurations typically need lists of nodes that belong to
|
|
batch queues. These might be generated by assigning nodes belonging to a
|
|
particular queue an attribute in the genders file and then using nodeattr to
|
|
generate the batch configuration. For example, an NQS manager host might get
|
|
its configuration as follows, assuming the nqsmgr and nqscompute attributes
|
|
are appropriately assigned in the genders file:
|
|
|
|
#!/bin/sh
|
|
|
|
#
|
|
# Configure queues on the nqsmgr node.
|
|
#
|
|
if nodeattr nqsmgr; then
|
|
|
|
# All nodes with the nqscompute attribute will be B queue destinations.
|
|
# Make a comma-separated list of B@host destinations.
|
|
for host in `nodeattr -n nqscompute`; do
|
|
CPU_QUEUES=${CPU_QUEUES}B@${host},
|
|
done
|
|
CPU_QUEUES=`echo $CPU_QUEUES|sed 's/,$//'`
|
|
|
|
echo Adding manager queues
|
|
qmgr << EOT
|
|
set log_file /var/spool/nqs/log
|
|
set shell_strategy fixed = (/bin/sh)
|
|
set default destination_retry wait 1
|
|
create pipe_queue X priority = 10 \
|
|
server = (/usr/lib/nqs/pipeclient) \
|
|
destination = ($CPU_QUEUES)
|
|
run_limit = 1
|
|
set default batch_request queue X
|
|
enable queue X
|
|
start queue X
|
|
exit
|
|
EOT
|
|
fi
|
|
|
|
#
|
|
# Configure queues on the nqscompute nodes.
|
|
#
|
|
if nodeattr nqscompute; then
|
|
echo Adding compute queues
|
|
qmgr << EOT2
|
|
set log_file /var/spool/nqs/log
|
|
set shell_strategy fixed = (/bin/sh)
|
|
create batch_queue A priority=10
|
|
create pipe_queue B priority=10 \
|
|
server=(/usr/lib/nqs/pipeclient) \
|
|
destination=(A) run_limit=1
|
|
enable queue A
|
|
enable queue B
|
|
start queue A
|
|
start queue B
|
|
exit
|
|
EOT2
|
|
fi
|
|
exit 0
|
|
|
|
Example: nqs_mknmap
|
|
|
|
The following script uses the Genders library to get a list of all nodes
|
|
in the cluster. It then adds them to the NMAP database used by NQS:
|
|
|
|
#!/usr/bin/perl
|
|
#
|
|
# $Id: TUTORIAL,v 1.9 2004-11-13 19:58:09 achu Exp $
|
|
# $Source: /g/g0/achu/temp/genders-cvsbackup-full/genders/TUTORIAL,v $
|
|
#
|
|
|
|
use Genders;
|
|
|
|
$prog = "nqs_mknmap";
|
|
$path_nmapmgr = "/usr/bin/nmapmgr";
|
|
$path_nmapdir = "/etc/nmap";
|
|
|
|
if (opendir(DIR, $path_nmapdir)) {
|
|
foreach $file (grep(!/^\.\.?$/, readdir(DIR))) {
|
|
unlink("$path_nmapdir/$file");
|
|
}
|
|
closedir(DIR);
|
|
}
|
|
if (! -d $path_nmapdir) {
|
|
mkdir($path_nmapdir, 0755);
|
|
}
|
|
|
|
if (open(PIPE, "|$path_nmapmgr")) {
|
|
printf PIPE ("create\n");
|
|
$i = 0;
|
|
$obj = Genders->new();
|
|
foreach $node ($obj->getnodes()) {
|
|
printf PIPE ("add mid %d %s\n", $i, $node);
|
|
$i++;
|
|
}
|
|
printf PIPE ("exit\n");
|
|
close(PIPE);
|
|
} else {
|
|
printf STDERR ("%s: could not exec $path_nmap_mgr\n", $prog);
|
|
exit(1);
|
|
}
|
|
exit(0);
|
|
|
|
------------------------------------------------------------------------
|
|
[1] Here a cluster is a collection of nodes that are administered as a unit,
|
|
where each node runs its own UNIX operating system image.
|
|
|
|
[2] This should be the hostname as reported by the hostname -s command.
|
|
Names for other network interfaces on the same node do not appear in the
|
|
genders file.
|
|
|
|
------------------------------------------------------------------------
|
|
$Id: TUTORIAL,v 1.9 2004-11-13 19:58:09 achu Exp $
|
|
$Source: /g/g0/achu/temp/genders-cvsbackup-full/genders/TUTORIAL,v $
|