************************************************************************ * Myricom GM networking software and documentation * * Copyright (c) 2001 by Myricom, Inc. * * All rights reserved. See the file `COPYING' for copyright notice. * ************************************************************************ README-FreeBSD for gm-1.5 Supported platforms: v4.x for ia32 and alpha. Supported interfaces: LANai4 with 1MB or 512K (PCI32 only, SBus is not supported by FreeBSD), LANai7 (PCI64, PCI64A), and LANai9 (PCI64B, PCI64C) (If you have LANai4 with 256K, you will need to upgrade your interface, or use a previous version of GM (gm-1.2.3). For installation instructions of an earlier GM version please refer to the respective README and README- files.) Table of Contents: ----------------- I. GM Binary Installation a. Unpacking and loading the GM driver b. Running the GM Mapper c. Testing the GM installation II. GM Source Installation a. Configuring, compiling, and loading the GM driver b. Running the GM Mapper c. Testing the GM installation III. Verifying the GM performance IV. Running IP over GM V. Improving IP Performance VI. Fork() and System() Support VII. Caveats a. Loading/unloading the GM driver b. Difficulties to unload the GM module? ************************************************************************ If difficulties are encountered, please consult the FAQ http://www.myri.com/scs/GM_FAQ.html and all technical support questions should be directed to help@myri.com. ************************************************************************ ========================== I. GM Binary Installation ========================== GM Binary installation is supported for FreeBSD/x86 versions 4.1.1-RELEASE through 4.5-RELEASE. If you're running an older version of FreeBSD (4.0 through 4.1-RELEASE), or if you are running FreeBSD on an alpha, you must build the driver from source. Please skip to section II.( GM Source Installation Instructions). Current cautions and common problems: * If the host is rebooted, you must reload the GM driver (and rerun the GM Mapper). * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). GM installation is performed in the following three steps. 1. Unpacking and loading the GM driver. -------------------------------------- tar zxvf gm-1.5.1_FreeBSD-x86-FreeBSD-4.1.1-RELEASE-new-features-8ports.tar.gz | tar xvf - cd gm-1.5.1_FreeBSD-x86-FreeBSD-4.1.1-RELEASE su root ./GM_INSTALL 2. Running the GM Mapper ------------------------ cd binary/sbin/ su root ./mapper map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation ------------------------------ A variety of test scripts are available in {GM_HOME}/binary/bin to test your GM installation. We recommend the five tests described in tests/README. =========================== II. GM Source Installation =========================== Current cautions and common problems: * If the host is rebooted, you must reload the GM driver (and rerun the GM Mapper). * The GM Mapper must be run before any communication over Myrinet can occur. * In most cases, the GM Mapper should not be run continuously. It is intrusive and can interfere with network traffic intensive jobs (e.g, MPI applications). GM installation is performed in the following three steps. 1. Configure, compile, and load the GM driver: --------------------------------------------- Install GNU make from the /usr/ports/devel/gmake port or the gmake package. gunzip -c gm-1.5_FreeBSD.tar.gz | tar xvf - cd {GM_HOME} ./configure gmake cd binary su root ./GM_INSTALL By default, we assume that your FreeBSD kernel build directory is located in /sys/compile/GENERIC. If your FreeBSD kernel build directory is not located in /sys/compile/GENERIC, you must configure with the following option: ./configure --with-freebsd-kernel= where specifies the directory for the FreeBSD kernel source. The kernel header files MUST match the running kernel exactly: not only should they both be from the same version, but they should also contain the same kernel configuration options. If you are running the binary "GENERIC" kernel, you must install the kernel sources, configure a "GENERIC" kernel: # cd /sys//conf; config -g GENERIC and run "make depend" in the GENERIC kernel build directory (/sys/compile/GENERIC) # cd /sys/compile/GENERIC; make depend By default, we also assume that you have LANai9 or LANai7 interfaces. If you have LANai4 (with 1MB of memory), you will need to configure with: ./configure --disable-new-features If you have LANai4 with 512K, you will need to configure with: ./configure --disable-new-features --with-min-supported-sram=256 and you will only have 4 GM ports available instead of 8. (As previously noted, if you have LANai4 with 256K, you cannot install gm-1.5. You will need to upgrade your interface or use a previous version of GM (gm-1.2.3). Note: If you have a mixture of hosts with LANai4 and LANai7 (or LANai9) interfaces that need to talk to each other, you must configure with --disable-new-features on all of the hosts. For a complete listing of all options to configure, type: ./configure --help Note: Do not use the configure flag --enable-directcopy. This flag is not a valid option to GM 1.5. It will be re-enabled in a future release. The GM_INSTALL script will unload any existing GM device driver, load the current device driver and create /dev/gm device i-nodes. It does not configure the IP device, nor does it set up any scripts to load the GM driver at boot time. During the GM_INSTALL phase, GM prints messages to the kernel log (dmesg). If the running kernel and the kernel header used for compilation are mismatched, GM will print a warning message to the kernel log. Please be sure to read the {GM_HOME}/README and the {GM_HOME}/README-FreeBSD for further details of operating-system-specific caveats. 2. Running the GM Mapper ------------------------ cd binary/sbin/ su root ./mapper map_once.args There are 3 ways in which the GM Mapper can be run. Refer to section 2 'Running the Mapper' of the README-linux for details. 3. Testing the GM Installation ------------------------------ A variety of test scripts are available in {GM_HOME}/binary/bin to test your GM installation. We recommend the five tests described in tests/README. ================================= III. Verifying the GM Performance ================================= We recommend the following test to verify the GM performance. View the results of the hardware benchmark test of the PCI bus with the DMA engine of the Myrinet adapter. cd {GM_HOME}/binary/bin ./gm_debug --no-counters Note: The output of this command gives the maximum sustained bandwidth that can be obtained from the PCI bus. Refer to the section entitled "GM Performance" in the {GM_HOME}/README for complete details on expected GM performance. ====================== IV. Running IP over GM ====================== The FreeBSD command to enable IP over GM is as follows: /sbin/ifconfig myri0 up where you must replace 'myri0' with the appropriate name (myri1, myr2, etc.) if you have more than one Myrinet interface per host. For more information, please refer to the FAQ (http://www.myri.com/scs/GM_FAQ.html). =========================== V. Improving IP performance =========================== IP tuning consists of the following: sudo sysctl -w net.inet.tcp.sendspace=65536 sudo sysctl -w net.inet.tcp.recvspace=65536 Some additional gain can sometimes be seen by increasing these values to 262144. If one increases these values to 262144, one must also increase kern.ipc.maxsockbuf to something larger, such as 524288. If you would like to obtain the maximum performance out of IP, you can increase the size of your mbuf clusters to PAGE_SIZE by changing MCLSHIFT in sys/{i386,alpha}/include/param.h to 12 on i386 and 13 on alpha. (eg, PAGE_SIZE == 1 << MCLSHIFT). This requires a kernel rebuild and a GM rebuild. ================================ VI. Fork() and System() Support ================================ By default, GM supports the function vfork(). As the function popen() calls vfork(), it is also safe to use at any time with GM. However, GM does not support fork() when a GM port is open, as it is incompatible with some GM OS manipulations. Please contact help@myri.com for further details. ============= VII. Caveats ============= ----------------------------------- a. Loading/unloading the GM driver ----------------------------------- To load the driver, run 'kldload binary/sbin/gm.ko' To unload the driver, run 'kldunload -n gm.ko' ----------------------------------------- b. Difficulties to unload the GM module? ----------------------------------------- >>Let's say a user left a mapper running and forgot about it. She then >>attempts to unload the module, and receives the following message: % sudo kldunload gm kldunload: can't unload file: Device busy >> At this point, this message will appear on the system console: WARNING: GM: 1 refs to minor unit 8. module cannot be unloaded until all ports are closed >> Now the user has enough information to track down the process & >> kill it. The easiest way to do this is to use fstat (like lsof, >> but in base system) to look for processes with /dev/gm* open: % fstat | grep gm root mapper 601 3 / 8484 crw------- gm8 rw % sudo kill -TERM 601 % sudo kldunload gm %