Copyright Myricom 2003-2009, Markus Fischer fischer@myri.com REQUIREMENTS The Sockets over Myrinet Module (AF_MYRI) implemenation require that you have the kernel sources installed in order to compile the software package. There is *no* requirement to rebuild your operating system or patch your distribution. Sockets over Myrinet Module will nicely fit into your environment without any system specific changes. The Sockets-GM Module requires GM-2.0.14 or GM-2.1.4 or higher. The Sockets-MX Module requires MX 1.1.1 or higher. It requires the configure --enable-kernel-lib option when compiling MX. MX needs to be loaded before loading Sockets-MX into the kernel. INSTALLATION Enabling Sockets over Myrinet: Establishing Sockets over Myrinet functionality is achieved by inserting the af_myri module into the OS. This is typically done using insmod under Linux. Starting with Version 1.5, the AF_MYRI module requires an IP interface as an argument to specify which network is set under SDP. (e.g: insmod /opt/myri/af_myri.o sdp_interface=myri0) Mapping Socket Communication of applications to use Sockets over Myrinet: The goal is to allow the module af_myri.o to map TCP/IP Traffic to Myrinet. This should be achieved for already precompiled applications. The following options exist for achieving this mapping: 1) Modification to the syscall_table which will remap AF_INET traffic to the AF_MYRI (27) protocol family. As a result any application will be mapped to Sockets over Myrinet If only specific applications should be mapped to AF_MYRI, then the second option is more feasible 2) Provide a small shared library which will remap AF_INET traffic when calling socket() to the AF_MYRI protocol family: e.g.: export LD_PRELOAD=/opt/sdp-myri/libgmsocks.so.1.0 If you want any application to be run under the new sockets family, then you should add the LD_PRELOAD configuration at a system/distribution specific location. e.g: /etc/bash.bashrc This will enable remote shells to be run under the new protocol family as well (e.g: ssh remote-host ls will be run under the new protocol) FUNCTIONALITY The AF_MYRI desgin is implementing a new socket protocol family. Any application which has been mapped to AF_MYRI will trap into the Kernel, but experience no additional TCP/IP protocol overhead. Depending on the message size RDMA or buffer semantics will be used to transfer data. One might wonder what happens if application are run in a heterogeneous network environment. As such you could consider to have a distributed application within and outside Myrinet. In this case the new socket protocol will use the traditional AF_INET protocol to achieve full transparency. An architectural overview is depicted in the following: __________________________________ | User Mode Socket Application | ===============|================== Libc | User ========================================= Socket Layer (AF Families) Kernel | | AF_INET | | AF_MYRI | <--------------| TCP/IP | GM / MX | | | | |--------| | | Ethernet Network Myrinet By enabling the fallback mechanism to use AF_INET over TCP/IP it is also possible to let system services work with user mode applications. For example if a remote system has an sshd running and the local hosts contacts via ssh under the new socket protocol, then ssh will still work since it will fall back to AF_INET. However, the newly created process on the remote host will be able to talk directly using the new socket protocol, without overhead. CONFIGURATION GM: A default GM installation is required. An easy way to have the GM hostnames be suitable for Sockets-GM, an /etc/gm/hostname. (e.g: /etc/gm/hostname.0) file containing the AF_INET IP address will set the name automatically. You need the IP addresses, followed by a colon and board id. Example: user@host:~$ more /etc/gm/hostname.0 192.131.130.83:0 user@host:~$ more /etc/gm/hostname.1 192.131.130.83:1 Note: if you create a new hostname entry GM needs to be restarted. MX: When using myri0 with assinged IP addresses as the SDP Interface (default Interface), then Sockets-MX will not require any additional configuration. Otherwise (e.g. sdp_interface=eth0) the MX hostnames must match the IP address of the chosen interface. You can set the MX hostname using mx_hostname under the bin directory e.g.: sudo /opt/mx/sbin/mx_start_stop restart sudo /opt/mx/bin/mx_hostname -n $1 sudo /opt/mx/bin/mx_hostname -c If the MX hostname and the diagnosed sdp_ip_addr do not match, the Sockets-MX module will fail to load to avoid unnecessary configuration issues. Please consult dmesg, or tail /var/log/messages for further information For your convenience, the Sockets-MX_MODULE directory contains a script which takes a hostfile as input: ./verify_hostnames. This hostfile should contain the names of the nodes which the script will configure for AF_MYRI use. It assumes that MX has been loaded. MODULE PARAMETERS AF_MYRI offers the following parameters to be set during module load time: sdp_interface={eth0, myri0} - specify which interface to use for SDP sdp_num_board={1,2} - specify the number of Myrinet Boards to use sdp_zero_copy={0,1} - turn on/off zero copy, [0] sdp_zero_copy_zize={1000 - n} - specify the message size in bytes when to switch to the zero copy protocol [1000000] COMPILING A common makefile is provided for Linux 2.4.x and Linux 2.6.x Configuration is performed as module params. Run autoheader; autoconf; configure; make LOADING AF_MYRI An example for a Sockets-MX installation using myri0 and board instance #0 sudo ./af_myri.ko sdp_interface=myri0 sdp_board_id=0 dmesg should show something similar to: Sockets-MX 1.5 - (C) 2004-2009 Myricom Inc SDP IP: 10.0.0.2 SDP Interface: myri0 SDP Port #: 3 NET: Registered protocol family 27 EXAMPLES The examples directory contains small examples to get familiar with Sockets. over Myrinet. There are additional Readme files which help. Otherwise please do not hesitate to contact help@myri.com