Myrinet GM-2 Software for FreeBSD

Myrinet GM-2 Software
for FreeBSD

FreeBSD is a continuation of the "Berkeley Standard Distribution" UNIX. It is robust, it is free, and it is supported on IA32 and Alpha platforms.

The Myrinet GM-2 software for FreeBSD is maintained by Andrew Gallatin, a Myricom employee and member of the Myricom software team who is also a volunteer contributor to the FreeBSD project.

To download Myrinet software, you need a login and password.
If you don't already have these, please request them.

Binary | Source

The FreeBSD GM-2 software is distributed in both source and binary form. The binary software is available for FreeBSD 4.1.1-RELEASE through 4.7-RELEASE and 4.9-RELEASE on the IA32 platform. Note that 4.8-RELEASE requires a patch for proper operation. Refer to the README-FreeBSD for details. The source code can be used on all 4.x versions of FreeBSD on both IA32 and Alpha.

For FreeBSD v5.x and v6.x and AMD64 support, as well as support for PCIXE and PCIXF Myrinet NICs, we recommend the use of the MX driver/firmware instead of GM-2.

Recommended-version GM-2 Binary Downloads for FreeBSD
NIC

Operating System / Architecture PCIXD
(Lanai XP) PCI64B and PCI64C
(Lanai 9) PCI64 and PCI64A
(Lanai 7)

FreeBSD v4.1.1 through
FreeBSD v4.7 and FreeBSD v4.9 for IA32 GM-2.0.11_FreeBSD GM-2.0.11_FreeBSD GM-2.0.11_FreeBSD

Recommended-version GM-2 **Binary** Downloads for FreeBSD
	NIC
Operating System / Architecture	PCIXD (Lanai XP)	PCI64B and PCI64C (Lanai 9)	PCI64 and PCI64A (Lanai 7)
FreeBSD v4.1.1 through FreeBSD v4.7 and FreeBSD v4.9 for IA32	GM-2.0.11_FreeBSD	GM-2.0.11_FreeBSD	GM-2.0.11_FreeBSD

Abbreviated Binary Installation Instructions

Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions could also be answered in the FAQ.

Current cautions and common problems:

Do not use the configure flag --enable-directcopy with GM 2.0.11. This flag is not a valid option to GM 2.0.11. It will be re-enabled in a future release.

If the host is rebooted, you must reload the GM driver.

GM-2 does not interoperate with GM-1.

If you will be running MPI applications over GM-2, you must use MPICH-GM 1.2.5..10 or later.

GM installation is performed in four easy steps:

1. Unpacking and loading the GM driver.

        tar zxvf gm-2.0.11_FreeBSD-x86-FreeBSD-4.1.1-RELEASE-16ports.tar.gz

Select an installation directory path <install_path>. It is usually best for <install_path> to be the path to an NFS directory available on all machines that are to share this GM installation. The directory must be accessible using <install_path> on all machines that are to share the installation. <install_path> must be an absolute path; it must start with /. However, <install_path> may contain symbolic links.

        cd gm-2.0.11_FreeBSD-x86-FreeBSD-4.1.1-RELEASE/
        ./GM_INSTALL <install_path>

If you omit <install_path>, the driver will be installed in the default directory, /opt/gm/.

Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /usr/local/etc/rc.d/gm start

on all of the machines in your cluster.

Details of these installation scripts can be found in the README-linux. When the GM module is loaded, a mapper daemon called gm_mapper is started for each Myrinet NIC contained in the machine. The PIDs of the running gm_mappers are stored in /var/run/gm_mapper/pid.<board_id>.

Important notes:

Stopping the mapper while GM is running is not supported. The gm_mapper should be left running at all times, and it will not interfere with the performance of jobs running over Myrinet.
The installation scripts do not configure the IP device. If you wish to run IP over GM/Myrinet (ethernet emulation), you must configure the device. Refer to step 3. However, if you wish to stop the GM driver, IP over Myrinet will be shut down as part of the gm stop process.
Running /usr/local/etc/rc.d/gm stop kills all of the gm_mapper daemons, shuts down IP over Myrinet, and unloads the gm module.

If you wish for the driver to auto-load at boot, you must make the appropriate symlink in the /usr/local/etc/rc.d directory:

        su root
        ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

Alternatively, you may start and stop the drivers manually using

 
        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

 
        su root
        /usr/local/etc/rc.d/gm restart

to start, stop, or restart the driver, respectively.

For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section of the README-FreeBSD.

3. Enabling IP over Myrinet (Ethernet emulation) (OPTIONAL)

If you wish to run IP over Myrinet (ethernet emulation), the FreeBSD command to enable IP over GM is as follows:

   /sbin/ifconfig myri0 <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

3. Testing the GM Installation.

Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests.

Check the LEDs on each switch port and NIC port
Run gm_board_info on one host
Run gm_debug to test the PCI bandwidth
Run gm_allsize to test the links in the network
Run gm_stress to test the network

Each of these steps is detailed in the Troubleshooting section of the FAQ.

The test scripts (gm_board_info, gm_debug, gm_allsize, gm_stress) are available in <install_path>/bin in your GM installation. A README describing each of these tests can be found in <install_path>/bin/README.

Congratulations! You have successfully installed GM.

Recommended-version GM Source Downloads for FreeBSD
NIC

Operating System / Architecture PCIXD
(Lanai XP) PCI64B and PCI64C
(Lanai 9) PCI64 and PCI64A
(Lanai 7)

FreeBSD v4.x
IA32, Alpha GM-2.0.11_FreeBSD GM-2.0.11_FreeBSD GM-2.0.11_FreeBSD

Recommended-version GM **Source** Downloads for FreeBSD
	NIC
Operating System / Architecture	PCIXD (Lanai XP)	PCI64B and PCI64C (Lanai 9)	PCI64 and PCI64A (Lanai 7)
FreeBSD v4.x IA32, Alpha	GM-2.0.11_FreeBSD	GM-2.0.11_FreeBSD	GM-2.0.11_FreeBSD

Abbreviated Source Installation Instructions

To compile GM on FreeBSD platforms, you will need GNU make which is available from the /usr/ports/devel/gmake port or the gmake package. Please be sure to read the README files for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions could also be answered in the FAQ.

Current cautions and common problems:

Do not use the configure flag --enable-directcopy with GM 2.0.11. This flag is not a valid option to GM 2.0.11. It will be re-enabled in a future release.

If the host is rebooted, you must reload the GM driver.

GM-2 does not interoperate with GM-1.

If you will be running MPI applications over GM-2, you must use MPICH-GM 1.2.5..10 or later.

GM installation is performed in four easy steps:

1. Configuring, compiling, and loading the GM driver

First, install GNU make from the /usr/ports/devel/gmake port or the gmake package.

        gunzip gm-2.0.11_MacOSX_FreeBSD.tar.gz | tar xvf -
        cd {GM_HOME} 
        ./configure
        gmake

By default, we assume that your FreeBSD kernel build directory is located in /sys/compile/GENERIC. If your FreeBSD kernel build directory is not located in /sys/compile/GENERIC, you must configure with the following option:

        ./configure --with-freebsd-kernel=<freebsd-source-dir>

where <freebsd-source-dir> specifies the directory for the FreeBSD kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options.

If you are running the binary "GENERIC" kernel, you must install the kernel sources, configure a "GENERIC" kernel:

         # cd /sys/<arch>/conf; config -g GENERIC

and run make depend in the GENERIC kernel build directory (/sys/compile/GENERIC)

         # cd /sys/compile/GENERIC; make depend

2. Installing the GM driver.

        cd binary
        ./GM_INSTALL <install_path>

If you omit <install_path>, the driver will be installed in the default directory, /opt/gm/.

Next, you must run

        su root
        <install_path>/sbin/gm_install_drivers
        /usr/local/etc/rc.d/gm start

on all of the machines in your cluster.

Important notes:

Stopping the mapper while GM is running is not supported. The gm_mapper should be left running at all times, and it will not interfere with the performance of jobs running over Myrinet.
The installation scripts do not configure the IP device. If you wish to run IP over GM/Myrinet (ethernet emulation), you must configure the device. Refer to step 3. However, if you wish to stop the GM driver, IP over Myrinet will be shut down as part of the gm stop process.
Running /usr/local/etc/rc.d/gm stop kills all of the gm_mapper daemons, shuts down IP over Myrinet, and unloads the gm module.

If you wish for the driver to auto-load at boot, you must make the appropriate symlink in the /usr/local/etc/rc.d directory:

        su root
        ln -s /usr/local/etc/rc.d/gm /usr/local/etc/rc.d/gm.sh

Alternatively, you may start and stop the drivers manually using

 
        su root
        /usr/local/etc/rc.d/gm start
        /usr/local/etc/rc.d/gm stop

 
        su root
        /usr/local/etc/rc.d/gm restart

to start, stop, or restart the driver, respectively.

For directions on how to uninstall the GM driver, refer to the "Miscellaneous" section of the README-FreeBSD.

3. Testing the GM Installation.

Once the GM software has been properly installed on all of the hosts in your cluster, you are ready to validate your Myrinet installation by performing the following sequence of tests.

Check the LEDs on each switch port and NIC port
Run gm_board_info on one host
Run gm_debug to test the PCI bandwidth
Run gm_allsize to test the links in the network
Run gm_stress to test the network

Each of these steps is detailed in the Troubleshooting section of the FAQ.

Congratulations! You have successfully installed GM.

Refer to the Software and Documentation Page for details of GM-based software (MPICH-GM, VI-GM, PVM-GM, Sockets-GM). Currently only MPICH-GM, PVM-GM, and Sockets-GM can be used with GM-2. All other middleware requires GM-1. If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README-FreeBSD and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.

Last updated: 16 May 2006