Copyright Myricom 2003-2004, Markus Fischer fischer@myri.com

REQUIREMENTS

  The Sockets-GM Module implemenation require that you have
  the kernel sources installed in order to compile Sockets-GM.

  There is *no* requirement to rebuild your operating system or
  patch your distribution.
  Sockets-GM Module will nicely fit into your environment without
  any system specific changes.

  Sockets-GM Module requires GM-2.0.12 or GM-2.1.2 or higher.

INSTALLATION

  Enabling Sockets-GM_MODULE:

  Establishing Sockets-GM functionality is achieved by 
  inserting the module into the OS.
  This is typically done using insmod under Linux.

  The module requires an IP address as an argument for identification.
  (e.g: insmod /opt/sdp-myri/af_myri.o sdp_ip_addr=192.168.1.112)
  The AF_INET IP address should be taken (eth0).

  Mapping Socket Communication of applications to use Sockets-GM

  The goal is to allow the module af_myri.o to map TCP/IP Traffic onto GM.
  This should be achieved for already precompiled applications.
  The following options exist for achieving this mapping:

   1) Modification to the syscall_table which will remap AF_INET traffic to
      the Sockets-GM protocol family.
      As a result any application will be mapped to Sockets-GM.
      If only specific applications should be mapped to Sockets-GM,
      then the second option is more feasible

   2) Provide a small shared library which will remap AF_INET traffic when
      calling socket() to the Sockets-GM protocol family:

      e.g.: export LD_PRELOAD=/opt/sdp-myri/libgmsocks.so.1.0

      If you want any application to be run under the new sockets
      family, then you should add the LD_PRELOAD configuration at a
      system/distribution specific location.
      e.g: /etc/bash.bashrc
      
      This will enable remote shells to be run under the new protocol
      family as well (e.g: ssh remote-host ls  will be run under the
      new protocol)

      
FUNCTIONALITY

  The Sockets-GM_MODULE approach is implementing a new socket protocol family.
  Any application which has been mapped to Sockets-GM will trap into the
  Kernel, but experience no additional protocol overhead. The Myrinet network
  which guarantees error free data delivery is a key for achieving a reliable
  protocol for socket applications.

  Depending on the message size RDMA or buffer semantics will be used to
  transfer data.

  One might wonder what happens if application are run in a heterogeneous
  network environment. As such you could consider to have a distributed
  application within and outside Myrinet.
  In this case the new socket protocol will use the traditional AF_INET
  protocol to achieve full transparency.

  An architectural overview is depicted in the following:

           __________________________________
          |   User Mode Socket Application   |
           ===============|==================
                         Libc
                          |                        User
          =========================================
	          Socket Layer (AF Families)       Kernel
                    |                |
            AF_INET |                | AF_MYRINET
                    | <--------------|
                 TCP/IP     |       GM
                    |       |        |
                    |       |--------|
                    |                |
            Ethernet Network       Myrinet


    By enabling the fallback mechanism to use AF_INET over TCP/IP it
    is also possible to let system services work with user mode
    applications. For example if a remote system has an sshd running
    and the local hosts contacts via ssh under the new socket protocol,
    then ssh will still work since it will fall back to AF_INET.
    However, the newly created process on the remote host will be
    able to talk directly using the new socket protocol, without overhead.

CONFIGURATION

   GM: A default GM installation is required. An easy way to have the GM
       hostnames be suitable for Sockets-GM is to create the file
       /etc/gm/hostname.<boardid> (e.g: /etc/gm/hostname.0)
       containing the AF_INET IP address.
       When GM is started it will set the name automatically.
       Channelbonding is supported and the complete GM hostname must
       be the IP addresses, followed by a colon and the board id.
       Example:
       user@host:~$ more /etc/gm/hostname.0
       192.131.130.83:0
       user@host:~$ more /etc/gm/hostname.1
       192.131.130.83:1
      
       Note: if you create a new hostname entry GM needs to be restarted.
	     on the participating nodes (/etc/init.d/gm restart) 

MODULE PARAMETERS

   AF_MYRI offers the following parameters to be set during module load time:

   sdp_num_board={1,2}  - specify the number of Myrinet Boards to use (Board bonding), [1]
   sdp_board_one={1,2,...} - specify which board number to use as board 1, [0] 

   sdp_board_two={1,2,...} - specify which board number to use as board 2, [1]

   sdp_zero_copy={0,1} - turn on zero copy, [0]

   sdp_zero_copy_zize={1000 - n} - specify the message size in bytes when to switch to the zero copy protocol [1000000]

COMPILING

   A common makefile is provided for Linux 2.4.x and Linux 2.6.x
   Configuration is performed as module params.

LOADING AF_MYRI
   An example for a fully features Sockets-GM installation
   sudo ./af_myri.ko sdp_num_board=2 sdp_board_one=2 sdp_board_two=4 sdp_ip_addr=192.168.1.111 sdp_zero_copy=1 sdp_zero_copy_len=500000

   This command allows to use two boards for sending, receiving. The board 
   numbers are 2 and 4 [gm_board_info can be called to get information], zero
   cop is enabled and will start for data transfers with 500000 Bytes.
EXAMPLES

The examples directory contains small examples to get familiar with Sockets-GM.
There are additional Readme files which help.