Myrinet received Solaris-Ready certification in July 2001.
and Sun HPC ClusterTools Software
for Solaris
Myrinet received Solaris-Ready certification in July 2001. |
Myrinet
GM-1 Software and Sun HPC ClusterTools Software for Solaris |
To download Myrinet software,
you need a login and password.
If you don't already have these, please
request them.
GM Binary | GM Source | ClusterTools Binary | ClusterTools Source | Performance
The current GM software release for Solaris on UltraSPARC hosts is gm-1.6.4_Solaris. Compared to the previous GM 1.6 software, this gm-1.6.4_Solaris release provides significant performance improvements in both latency and bandwidth.
Starting with gm-1.6_Solaris, the following important features have been included:
For UltraSPARC/Solaris users, Myricom also supports a loadable protocol module for the Sun HPC ClusterTools software. The loadable protocol modules (PM) are used by the MPI implementation of Sun HPC ClusterTools to carry the traffic between processes. In order to exploit the low latency and high data rates of Myrinet, Myricom developed the Myrinet protocol module directly over GM.
The Solaris GM software and the ClusterTools software are distributed both in source and in binary form. This GM distribution will run under Solaris 7 and higher on UltraSPARC processors. GM 1.6.4 software for Solaris on x86 is not supported. The ClusterTools software has been tested with Sun HPC ClusterTools 4.0 and 5.0 software.
The standard (default) GM distribution supports 8 GM Ports. 16-port GM Binaries are provided for customers with special requirements. Refer to the FAQ for consequences of using these 16-port GM Binaries.
Abbreviated Binary Installation Instructions
Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions could also be answered in the FAQ.
Current cautions and common problems:
GM installation is performed in three easy steps:
1. Unpacking and loading the GM driver.
gunzip -c gm-1.6.4_Solaris64-sun4u-SunOS-5.8-8port-heart-beat-on.tar.gz | tar xvf -
cd gm-1.6.4_Solaris-sun4u-SunOS-5.8
Pick an installation directory path <GM_install_path>. It is usually best for <GM_install_path> to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using <GM_install_path> on all machines that are to share the installation. <GM_install_path> must be an absolute path. That is, it must start with "/". However, <GM_install_path> may contain symbolic links.
To install GM you must first run
./GM_INSTALL <GM_install_path>
to install the GM shared components. You may omit <GM_install_path> to install in /opt/gm/.
Next, you must run
su root
<GM_install_path>/sbin/gm_install_drivers
on each machine to install the drivers on that machine.
If you wish for the driver to auto-load an boot, you must create appropriate links in the /etc/rcN directories to the /etc/init.d/gm and /etc/init.d/myri scripts. Alternatively, you may start and stop the drivers manually using
/etc/init.d/gm start
/etc/init.d/gm stop
or
/etc/init.d/gm restart
to start, stop, or restart the driver, respectively.
Note:
2. Running the GM Mapper.
After installing and starting the driver on all the nodes in your cluster, you need to map your network to configure it and make it usable.
To run the mapper, execute
su root
<GM_install_path>/sbin/mapper <GM_install_path>/etc/gm/map_once.args
There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details.
3. Testing the GM Installation.
A variety of test scripts are available in bin/ to test your GM installation. We recommend the five tests described in tests/README.
Congratulations! You have successfully installed GM.
| NIC | |||||
|---|---|---|---|---|---|
| Operating System/ Processor |
PCI64B and PCI64C (Lanai 9) |
PCI64 and PCI64A (Lanai 7) |
PCI32C (1MB) (Lanai 4) |
PCI32B (512KB) (Lanai 4) |
PCI32A (256KB) (Lanai 4) |
| Solaris 7, 8, 9 for UltraSPARC |
GM-1.6.4_Solaris | GM-1.6.4_Solaris | GM-1.4 | GM-1.2.3 | GM-1.2.3 |
| Solaris 6 for UltraSPARC |
GM-1.6_Solaris | GM-1.6_Solaris | GM-1.4 | GM-1.2.3 | GM-1.2.3 |
The source release is prebuilt with new-features and heart-beat enabled.
Abbreviated Source Installation Instructions
To compile GM on Solaris UltraSPARC platforms, you will need the Sun C++ compiler and GNU make. Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions could also be answered in the FAQ.
Current cautions and common problems:
GM installation is performed in four easy steps:
1. Configuring and compiling the GM driver
gunzip -c gm-1.6.4_Solaris.tar.gz | tar xvf -
cd gm-1.6.4_Solaris
autoconf
autoheader
./configure
Add the --disable-64b configure option if you are building a 32-bit driver on a machine running a 64-bit kernel, or --enable-64b if you are building a 64-bit driver on a machine running a 32-bit kernel. Without these options, a driver matching the running kernel on the compilation machine will be built.
Execute
make depend
make
to build a GM binary distribution in the binary/ subdirectory.
Note:
./configure --disable-pci
2. Installing the GM driver
Pick an installation directory path <GM_install_path>. It is usually best for <GM_install_path> to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using <GM_install_path> on all machines that are to share the installation. <GM_install_path> must be an absolute path; it must start with "/". However, <GM_install_path> may contain symbolic links.
To install GM you must first run
cd binary
./GM_INSTALL <GM_install_path>
to install the GM shared components. You may omit <GM_install_path> to install in /opt/gm/.
Next, you must run
su root
<GM_install_path>/sbin/gm_install_drivers
on each machine to install the drivers on that machine.
If you wish for the driver to auto-load an boot, you must create appropriate links in the /etc/rcN directories to the /etc/init.d/gm and /etc/init.d/myri scripts. Alternatively, you may start and stop the drivers manually using
/etc/init.d/gm start
/etc/init.d/gm stop
or
/etc/init.d/gm restart
to start, stop, or restart the driver, respectively.
Note:
3. Running the GM Mapper.
After installing the driver on all the nodes in your cluster, you need to map your network to configure it and make it usable.
To run the mapper, execute
su root
<GM_install_path>/sbin/mapper <GM_install_path>/etc/gm/map_once.args
There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details.
4. Testing the GM Installation.
A variety of test scripts are available in bin/ to test your GM installation. We recommend the five tests described in tests/README.
Congratulations! You have successfully installed GM.
Refer to the Software and Documentation Page for details of GM-based software (MPICH-GM, PVM on GM). If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-solaris and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.
The Sun HPC ClusterTools software implements MPI communication between processes using loadable protocol modules (PM) across TCP networks, shared memory interfaces (SHM), and remote shared memory (RSM). In order to exploit the low latency and high data rates of Myrinet, Myricom developed the Myrinet protocol module directly over GM. The Myrinet PM is very efficient, both in using GM's OS-bypass communication, and in avoiding memory copying whenever possible.
| NIC | |||||
|---|---|---|---|---|---|
| Operating System/ Processor |
PCI64B and PCI64C (Lanai 9) |
PCI64 and PCI64A (Lanai 7) |
PCI32C (1MB) (Lanai 4) |
PCI32B (512KB) (Lanai 4) |
PCI32A (256KB) (Lanai 4) |
| Solaris 8 and 9 for UltraSPARC |
ClusterTools | ClusterTools | -- | -- | -- |
Here is a summary of the CHANGES between ClusterTools releases.
Validation
The Myrinet Protocol Module Validation Guide for Sun HPC ClusterTools 4.0 (Adobe Acrobat version) is available and contains a detailed discussion of the installation instructions below, as well as testing and troubleshooting. The Myrinet Protocol Module (PM) has been tested with Sun HPC ClusterTools 4.0 and 5.0 software.
Abbreviated Binary Installation Instructions
The installation of Myrinet PM for Sun HPC ClusterTools involves the following 8 steps:
1. Install Sun HPC ClusterTools.
Install Sun HPC ClusterTools into the directory <CT_install_path> (default is /opt/SUNWhpc).
2. Install the Myrinet PM libraries.
gunzip -c CT-Myrinet-PM-1.2-no-mt.tar.gz | tar xvf -
su root
cp myr*.so* <CT_install_path>/lib
cp sparcv9/myr*.so* <CT_install_path>/lib/sparcv9
3. Set up the GM hostnames.
For each node of the cluster, plumb myri0 and set its IP address by the command
su root
ifconfig myri0 plumb x.x.x.x up
Then run ip2hostname
to obtain the hostname corresponding to myri0's IP address.
su root
ip2hostname x.x.x.x (myri0's IP address)
and set the node's GM name to this hostname.
su root <GM_install_path>/bin/gm_setname -h hostname
Rerun the GM mapper to make the change globally known to all nodes. This should be done whenever a new node is added to the network, the network topology has changed, or a node's GM name is changed.
su root
<GM_install_path>/sbin/mapper <GM_install_path>/etc/gm/map_once.args
4. Construct the configure file <CT_install_path>/conf/gm.conf.
An example of a <CT_install_path>/conf/gm.conf file for a cluster consisting of two nodes is given as follows:
-----------------------------------------
2
# node_name board_num port_num port_ids
u81-t 1 4 4 5 6 7
u82-t 1 4 4 5 6 7
-----------------------------------------
In this file, the first line specifies the total number of nodes in the cluster. Starting on the second line, the first column specifies the node's GM name; the second column specifies the number of Myrinet NICs installed on this node; the third column specifies the number of GM ports available for CT on each Myrinet NIC; from the fourth column on are the GM port ids that can be used on each Myrinet NIC.
Set the environment variable MPI_MYR_CONF to <CT_install_path>/conf/gm.conf file.
setenv MPI_MYR_CONF <CT_install_path>/conf/gm.conf
If no MPI_MYR_CONF is set, /opt/SUNWhpc/conf/gm.conf will be used as default.
5. Modify <CT_install_path>/conf/hpc.conf.
On each node of the cluster, modify <CT_install_path>/conf/hpc.conf to provide Myrinet PM with the proper rank.
------------------------------------------------------------ ... # List the available Protocol Modules # PMODULE LIBRARY Begin PMODULES shm () rsm () myr () tcp () End PMODULES # SHM settings # NAME RANK Begin PM=shm shm 5 End PM # RSM settings # NAME RANK AVAIL Begin PM=rsm wrsm 20 1 End PM # MYR settings # NAME RANK Begin PM=myr myr 30 End PM ... # TCP settings # NAME RANK MTU STRIPE LATENCY BANDWIDTH Begin PM=tcp midn 0 16384 0 20 150 idn 10 16384 0 20 150 ... myri 161 4096 0 20 150 ... ------------------------------------------------------------
6. Start the daemon (requires root privileges).
On the master node,
su root
/etc/init.d/sunhpc.cre_master start
/etc/init.d/sunhpc.cre_node start
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node start
7. Initialize the "all" partition (if necessary).
Initialize the "all" partition (requires root privileges) when mpinfo -N does not have all nodes in "all" partition, using
su root
<CT_install_path>/etc/part_initialize
================================================================================ example: mpinfo -N NAME UP PARTITION OS OSREL NCPU FMEM FSWP LOAD1 LOAD5 LOAD15 u81 y all SunOS 5.8 4 3901 3910 0.02 0.04 0.02 u82 y - SunOS 5.8 4 3873 3873 3.05 2.48 1.42 ================================================================================
8. Rerun the daemon (if necessary).
If the <CT_install_path>/conf/hpc.conf is modified (eg. rank changed, NIC added/removed, PM added/removed, etc.), or the status of any NIC in <CT_install_path>/conf/hpc.conf is changed (eg. NIC up/down, etc.), the daemon should be rerun.
Stop the daemon (requires root privileges).
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node stop
On the master node,
su root
/etc/init.d/sunhpc.cre_node stop
/etc/init.d/sunhpc.cre_master stop
Restart the daemon.
Repeat Step 6.
Congratulations! You have successfully installed Myrinet PM for Sun HPC ClusterTools.
If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-clustertools, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.
| NIC | |||||
|---|---|---|---|---|---|
| Operating System/ Processor |
PCI64B and PCI64C (Lanai 9) |
PCI64 and PCI64A (Lanai 7) |
PCI32C (1MB) (Lanai 4) |
PCI32B (512KB) (Lanai 4) |
PCI32A (256KB) (Lanai 4) |
| Solaris 8 and 9 for UltraSPARC |
ClusterTools | ClusterTools | -- | -- | -- |
Here is a summary of the CHANGES between ClusterTools releases.
Validation
The Myrinet Protocol Module Validation Guide for Sun HPC ClusterTools 4.0 (Adobe Acrobat version) is available and contains a detailed discussion of the installation instructions below, as well as testing and troubleshooting. The Myrinet Protocol Module (PM) has been tested with Sun HPC ClusterTools 4.0 and 5.0 software.
Abbreviated Source Installation Instructions
The installation of Myrinet PM for Sun ClusterTools involves the following 8 steps:
1. Install Sun HPC ClusterTools.
Install Sun HPC ClusterTools into the directory <CT_install_path> (default is /opt/SUNWhpc).
2. Install the Myrinet PM libraries.
gunzip -c CT-Myrinet-PM-1.2-no-mt-src.tar.gz | tar xvf -
cd CT-4.0/mpi/engine/myr
For a normal build:
make lib32/libmt32/lib64/libmt64
su root
cp ../../solaris/release/lib/myr* <CT_install_path>/lib/.
cp ../../solaris/release/lib/sparcv9/myr* <CT_install_path>/lib/sparcv9/.
For a build with debug turned on:
make libg32/libmtg32/libg64/libmtg64
su root
cp ../../solaris/debug/lib/myr* <CT_install_path>/lib/.
cp ../../solaris/debug/lib/sparcv9/myr* <CT_install_path>/lib/sparcv9/.
3. Set up the GM hostnames.
For each node of the cluster, plumb myri0 and set its IP address by the command
su root
ifconfig myri0 plumb x.x.x.x up
Then run ip2hostname
to obtain the hostname corresponding to myri0's IP address.
su root
ip2hostname x.x.x.x (myri0's IP address)
and set the node's GM name to this hostname.
su root <GM_install_path>/bin/gm_setname -h hostname
Rerun the GM mapper to make the change globally known to all nodes. This should be done whenever a new node is added to the network, the network topology has changed, or a node's GM name is changed.
su root
<GM_install_path>/sbin/mapper <GM_install_path>/etc/gm/map_once.args
4. Construct the configure file <CT_install_path>/conf/gm.conf.
An example of a <CT_install_path>/conf/gm.conf file for a cluster consisting of two nodes is given as follows:
-----------------------------------------
2
# node_name board_num port_num port_ids
u81-t 1 4 4 5 6 7
u82-t 1 4 4 5 6 7
-----------------------------------------
In this file, the first line specifies the total number of nodes in the cluster. Starting on the second line, the first column specifies the node's GM name; the second column specifies the number of Myrinet NICs installed on this node; the third column specifies the number of GM ports available for CT on each Myrinet NIC; from the fourth column on are the GM port ids that can be used on each Myrient NIC.
Set the environment variable MPI_MYR_CONF to <CT_install_path>/conf/gm.conf file.
setenv MPI_MYR_CONF <CT_install_path>/conf/gm.conf
If no MPI_MYR_CONF is set, /opt/SUNWhpc/conf/gm.conf will be used as default.
5. Modify <CT_install_path>/conf/hpc.conf.
On each node of the cluster, modify <CT_install_path>/conf/hpc.conf to provide Myrinet PM with the proper rank.
------------------------------------------------------------ ... # List the available Protocol Modules # PMODULE LIBRARY Begin PMODULES shm () rsm () myr () tcp () End PMODULES # SHM settings # NAME RANK Begin PM=shm shm 5 End PM # RSM settings # NAME RANK AVAIL Begin PM=rsm wrsm 20 1 End PM # MYR settings # NAME RANK Begin PM=myr myr 30 End PM ... # TCP settings # NAME RANK MTU STRIPE LATENCY BANDWIDTH Begin PM=tcp midn 0 16384 0 20 150 idn 10 16384 0 20 150 ... myri 161 4096 0 20 150 ... ------------------------------------------------------------
6. Start the daemon (requires root privileges).
On the master node,
su root
/etc/init.d/sunhpc.cre_master start
/etc/init.d/sunhpc.cre_node start
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node start
7. Initialize the "all" partition (if necessary).
Initialize the "all" partition (requires root privileges) when mpinfo -N does not have all nodes in "all" partition, using
su root
<CT_install_path>/etc/part_initialize
================================================================================ example: mpinfo -N NAME UP PARTITION OS OSREL NCPU FMEM FSWP LOAD1 LOAD5 LOAD15 u81 y all SunOS 5.8 4 3901 3910 0.02 0.04 0.02 u82 y - SunOS 5.8 4 3873 3873 3.05 2.48 1.42 ================================================================================
8. Rerun the daemon (if necessary).
If the <CT_install_path>/conf/hpc.conf is modified (eg. rank changed, NIC added/removed, PM added/removed, etc.), or the status of any NIC in <CT_install_path>/conf/hpc.conf is changed (eg. NIC up/down, etc.), the daemon should be rerun.
Stop the daemon (requires root privileges).
On the non-master nodes,
su root
/etc/init.d/sunhpc.cre_node stop
On the master node,
su root
/etc/init.d/sunhpc.cre_node stop
/etc/init.d/sunhpc.cre_master stop
Restart the daemon.
Repeat Step 6.
Congratulations! You have successfully installed Myrinet PM for Sun HPC ClusterTools.
If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-clustertools, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.
Last updated: 14 June 2006