mpi-selector(1) User Commands mpi-selector(1)
NAME
mpi-selector - A simple site-wide/per-user MPI selection tool for clus-
ters with more than one MPI implementation installed.
SYNOPSIS
Commands for end users
mpi-selector [options] --list
mpi-selector [options] --set <name>
mpi-selector [options] --unset
mpi-selector [options] --query
mpi-selector [options] --version
Commands for MPI implementations
mpi-selector [options] --register <name> --source-dir <dir>
mpi-selector [options] --unregister <name>
DESCRIPTION
The mpi-selector command is a simplistic tool to select one of multiple
MPI implementations. mpi-selector allows system administrators to set
a site-wide default MPI implementation while also allowing users to set
their own default MPI implementation (thereby overriding the system-
wide default).
Note that both the site-wide and per-user defaults are independent from
each other; a system administrator may choose not to have a site-wide
default while a user may choose to have a personal default -- and vice
versa.
The system is effected by having system-wide shell startup files that
looks first at the user’s MPI preferences. If found, the MPI implemen-
tation indicated by the user’s preferences is setup in the current
environment. If not found, look for a site-wide default. If found,
the MPI implementation indicated in by the site-wide default is setup
in the current environment. If not found, exit silently.
End Users / System Administrators
The mpi-selector command provides four main actions:
* List which MPI implementations are available
* Set a default (either on a per-user or site-wide basis)
* Unset a default (either on a per-user or site-wide basis)
* Query what the current default is
A common scenario is that a system administrator sets a site-wide
default for a supported MPI implementation that most users will use.
Power users then change their per-user defaults to use a different MPI
implementation.
Another common scenario is for power users to frequently use mpi-selec-
tor to swap back and forth between multiple different MPI implementa-
tions.
NOTE: The mpi-selector command only changes the defaults for new
shells. Specifically, after you invoke the mpi-selector command to
change the default MPI implementation, this change does not take effect
until you start a new shell. This is intentional. See the "KNOWN LIM-
ITATIONS" section, below.
MPI Implementations
MPI implementations register themselves with mpi-selector when they are
installed and unregister themselves when they are uninstalled. Each
MPI installation provides two files that setup the environment for
itself:
* mpivars.sh: File sourceable by Bourne-like shells (sh, bash, etc.)
* mpivars.csh: File sourceable by C-like shells (csh, tcsh, etc.)
These files are expected to be in a single directory and "registered"
with mpi-selector using the --register and --source-dir options. mpi-
selector will copy these files to its own internal store; it is safe to
remove the originals after the mpi-selector registration completes suc-
cessfully.
The <name> argument to --register must be simplistic -- it cannot con-
tain any shell special characters (not even if they are escaped), nor
can it contain spaces. The intent is to provide simple names that
users can type without escaping or quoting. Names not conforming to
these rules will be rejected and the registration will fail.
When an MPI implementation is uninstalled, it should unregister with
mpi-selector via the --unregister option.
OPTIONS
--list: List which MPI implementations are available
--no: Assume "no" to any interactive questions asked.
--query: See what the current default is. If specified with no
options, whichevery default has precedence -- if any -- will be shown.
If specified with --user, only show the per-user default (if there is
one). If specified with --system, only show the site-wide default (if
there is one).
--register: Register a new MPI implementation. Must be combined with
the --source-dir option.
--set <name>: Set the default MPI implementation. May be combined with
--system or --user (--user is the default and does not need to be spec-
ified).
--source-dir: Specify the location where mpivars.* files should be
copied from. Only meaningful when used with the --register option.
--system: When used with --set or --unset, specifies to work with the
site-wide default (vs. the per-user default). When used with --query,
it specifies to specifically query the site-wide default.
--unregister: Unregister an MPI implementation.
--user: When used with --set or --unset, specifies to work with the
per-user default (vs. the site-wide default). When used with --query,
it specifies to specifically query the per-user default.
--unset: Unset the default MPI implementation. May be combined with
--system or --user (--user is the default and does not need to be
explicitly specified).
--verbose: Be verbose.
--version: Return the version of mpi-selector.
--yes: Assume "yes" to any interactive questions asked.
EXAMPLES
Examples for End Users / System Administrators
The four main actions that system administrators and end users invoke
are: listing which MPI implementations are available, setting a
default, unsetting a default, and querying what the current default is.
Listing which MPI implementations are available
The --list option to the mpi-selector command shows a simple list of
which MPI implementations are available:
shell$ mpi-selector --list
mympi-1.2.3
mympi-4.5.6
othermpi-7.8.9
shell$
Setting a default
By default, MPI selections are performed on a per-user basis with the
--set option, using a name from the list of available MPI implementa-
tions (which can be obtained via the --list command):
shell$ mpi-selector --set mympi-4.5.6
shell$
Note that the default takes effect in the next shell that is started;
it does NOT take effect in the current shell!
If a default MPI is already set, setting a new default will cause an
interactive confirmation prompt. This interactive prompt can be
avoided by using the --yes option, which assumes a "yes" answer to all
questions:
shell$ mpi-selector --set mympi-4.5.6
shell$ mpi-selector --set mympi-4.5.6 --yes
shell$
If the --system option is used, the site-wide default is modified
instead of the per-user default. Since this option typically reqires
writing files into protected areas, root access may be required.
shell# mpi-selector --set mympi-4.5.6 --system
shell#
Unsetting a default
Unset the current default with the --unset option:
shell$ mpi-selector --unset
shell$
Similar to --set, the --system option can be used to unset the site-
wide default
shell# mpi-selector --unset --system
shell#
Querying what the current default is
The --query option can be used to see what the current MPI
implementation is (more specifically, what the MPI implementation will
be for the next shell that is started). It indicates both which MPI is
the default and at what level the default was set (per-user vs.
site-wide):
shell$ mpi-selector --set mympi-1.2.3
shell$ mpi-selector --query
default:mympi-1.2.3
level:user
shell$
Note that if there is no per-user default, the system default will be
shown:
shell# mpi-selector --set othermpi-7.8.9 --system
shell$ mpi-selector --unset
shell$ mpi-selector --query
default:othermpi-7.8.9
level:system
shell$ mpi-selector --set mympi-1.2.3
shell$ mpi-selector --query
default:mympi-1.2.3
level:user
shell$
If there is no per-user default and no site-wide default, --query will
return silently:
shell$ mpi-selector --query
shell$
Examples for MPI Implementations
Registering and unregistering typically writes files into protected
areas, and therefore usually requires root access.
If there are no MPI implementations registered, --list will return
silently:
shell# mpi-selector --list
shell#
An MPI with mpivars.sh and mpivars.csh in /opt/mympi/bin can be regis-
tered as follows:
shell# mpi-selector --register mympi-1.2.3 \
--source-dir /opt/mympi/bin
shell# mpi-selector --list
mympi-1.2.3
shell#
Note that re-registering the same <name> will cause an interactive con-
firmation prompt; the --yes option can be supplied to assume "yes" to
all questions asked:
shell# mpi-selector --list
mympi-1.2.3
shell# mpi-selector --register mympi-1.2.3 \
--source-dir /opt/mympi/bin --yes
mympi-1.2.3 is already registered.
Overwriting previously registered files.
shell# mpi-selector --list
mympi-1.2.3
shell#
Unregistering is also simple:
shell# mpi-selector --list
mympi-1.2.3
shell# mpi-selector --unregister mympi-1.2.3
shell# mpi-selector --list
shell#
Registering and Unregistering in RPMs
Registering and unregistering via RPM is unfortunately more complicated
than it needs to be because of the following issues:
1. Although RPM obeys dependency ordering of "rpm -i a b c". That is,
c will be installed before a if a requires c.
Regardless, RPM’s must know a) that the mpi-selector command is
installed, and b) be able to find it in its path.
2. RPM does not obey dependency ordering of "rpm -e a b c". That is,
c may be uninstalled before a, even if a requires c.
Hence, the mpi-select command may disappear before an RPM using the
mpi-select command in a scriptlet is uninstalled.
3. "Updating" RPMs will first uninstall the old RPM and then
re-install the new one.
Additionally, the staged installations (such as the OFED installer)
require telling the mpi-selector command additional information so that
various internal data files can be found.
In general, MPI installations via RPMs should register during the %post
scriptlet and unregister during the %preun scriptlet (not during the
%postun scriptlet!).
If RPMs "require" the mpi-selector RPM, they can be assured that the
mpi-selector command will exist and be installed properly, but they
still need to be able to find mpi-selector in their PATH. Hence, if
mpi-selector is not installed into a default PATH location, the %post
scriptlet won’t be able to find it, and the registration call will
fail. The simplest workaround (at least for the moment) is to set the
PATH to where mpi-selector is installed before installing any RPMs that
use it.
With that in mind, here is a possible %post scriptlet for OFED-
installed RPMS:
mpi-selector --register <name> --source-dir <source_dir> \
--yes --silent
Note the following:
1. The --yes option forces an overwrite if, for some reason, a
previous MPI of the name name is already registered.
2. The --silent option makes mpi-selector run silently, since RPMs
are supposed to install with no output.
Here is a possible %preun scriptlet for OFED-installed RPMs:
mpi-selector --unregister <name> --yes ││ \
/bin/true > /dev/null 2> /dev/null
Note the following:
1. We use %preun instead of %postun because of RPM’s upgrade behavior.
2. Since RPM does not honor dependencies when uninstalling, it is
possible that mpi-selector is no longer installed, and therefore
the command may fail. However, since mpi-selector is no longer
installed, we don’t care that it failed (i.e., there’s nothing to
unregister from), so just redirect all output to /dev/null and
ensure that the return code from the overall command is "true" (RPM
will abort if any individual scriptlet command fails).
KNOWN LIMITATIONS
The main known limitation of mpi-selector is that it only affects
future shells -- running it does not affect the current shell. After
you run mpi-selector to set a new default MPI (regardless of whether it
is a system-level or user-specific default), that default will not take
effect until you start a new shell -- even though --query will report
the new default.
This behavior is because mpi-selector defaults are only read during
shell startup. It was an intentional design decision -- mpi-selector
is intended to be a simplistic tool, and an all-encompassing solution.
Other solutions for modifying the current environment exist, such as
the Environment Modules package (<http://modules.sourceforge.net/>) and
SoftEnv from Argonne National Laboratory (and probably others). Using
these tools, you can immediately change the environment of the current
shell (to include switching to use a different MPI implementation). As
such, these already-existing, mature tools are better suited for such
usage patterns; mpi-selector is not intended to replace them.
For rsh/ssh-based parallel environments, switching defaults frequently
should be done with care. Specifically, rsh/ssh-based launchers may
depends on a common environment across all nodes (e.g., to find helper
executables and/or libraries for a specific MPI). Consider the follow-
ing example:
shell$ mpi-selector --set mympi-1.2.3
shell$ mpicc my_mpi_app.c -o my_mpi_app
shell$ mpirun -np 32 --hostfile myhosts my_mpi_app
While my_mpi_app is starting, it may be dangerous to switch the mpi-
selector default (perhaps in a different window) because the rsh and/or
ssh commands currently executing may be relying on finding the same MPI
implementation on all nodes. Changing the default while the applica-
tion is launching may cause a different MPI implementation to be found
on some nodes, thereby causing undefined behavior.
FILES
$HOME/.mpi-selector Location of per-user default selection
/etc/sysconfig/mpi-selector Location of site-wide default selection.
/var/mpi-selector/data/data Directory containing registered MPI shell
startup files.
AUTHOR
Written by Jeff Squyres.
REPORTING BUGS
Send bug reports to the OpenFabrics general mailing list (see
<http://www.openfabrics.org/>). This is a high-volume mailing list, so
be sure to put "mpi-selector" in the subject to ensure that it is not
missed.
COPYRIGHT
Copyright (c) 2007 Cisco Systems, Inc. All rights reserved.
This is free software; see the source for copying conditions. There is
NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
1.0.0 March 10, 2008 mpi-selector(1)
