Myri-10G
10-Gigabit Ethernet
Windows CIFS, iSCSI and AOE (ATAoE) Initiator Performance Measurements

We report three different kinds of storage performance measurements for a NAS/SAN solution for Windows: CIFS, iSCSI and AOE. While iSCSI is using TCP/IP sockets, ATAoE uses Layer-2 and is therefore not routable. For the setup Myri-10G-PCIE-8A-C NICs were connected point-to-point (switchless). Tests were between paris of dual socket, quad-core Intel Nehalem CPUs. The Myri10GE driver was using 9000 Byte (jumbo) frames.

Iometer was used for generating traffic and measuring performance for CIFS, iSCSI and AOE connected hosts.


Initiator Configuration

The Initiator was running Windows Server 2008 (x64). Myri10GE for Windows (NDIS 6.x) software was whql certified with Version 1.0.2 (x64). Microsofts's iSCSI Initiator and an Open Source AOE Initiator for Windows were used to connect to the target hosts. The target disk becomes visible as a (local) disk to which Iometer can write and read from. Transactions are then forwarded over the network to the target which performs the actions.

Target Configuration

For CIFS measurements we used a Windows Server 2008 which exported (shared) a RAMDISK which was then mounted by a Windows client. For iSCSI and AOE targets we used a Ubuntu Linux server which provided both the iscsi and aoe target software (at the time of the measurement no AOE target is available for Windows). As of the time of the measurement, the latest Linux 2.6.28 kernels from kernel.org were used. For testing with software header digests and data digests, Cryptographic API under Cryptographic options and CRC32c CRC algorithm in the kernel were enabled. They are the kernel options CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C respectively.

On the Target, a LUN with a nullio device with 10000 sectors was used.

Performance measurements as reported with Iometer

CIFS
W2k8 <-> W2k8 ; CIFS

Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms) cpu load
        4k      rd    35807                    139               3.53     17.77
        4k      wr    54385                    212               2.34     12.82

        8k      rd    48175                    376               2.64     14.88
        8k      wr    57595                    449               1.02     12.84

       16k      rd    39096                    614               3.86      9.05
       16k      wr    55811                    872               2.29     15.66

       32k      rd    24102                    673               7.06      9.26
       32k      wr    33994                   1059               3.71     13.74

       64k      rd     6326                    395              19.70      6.41
       64k      rd    17825                   1114               7.17     10.19 
W2K8 <-> W2K8, iSCSI (no header, data digest)
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms) cpu load
									
        4k      rd    90642                    354                1.44    10.12
        4k      wr   110875                    435                1.16    11.34
											
        8k      rd    60290                    487                2.11     8.42
        8k      wr    88585                    692                1.44     8.8

       16k      rd    38096                    594                3.36     8.05
       16k      wr    50289                    785                2.54     6.84

       32k      rd    21102                    663                6.06     7.26
       32k      wr    27162                    848                4.71     4.74

       64k      rd    11417                    713               11.21     6.41
       64k      rd    13647                    853               10.42     3.54

W2k8 <-> Ubuntu ; iSCSI (no digest) [iSCSI Enterprise Target]
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms) cpu load
																						
        4k      rd    41943                    163                3.74     5.46
        4k      wr    48462                    189                3.12     4.86

        8k      rd    40036                    312                3.19     7.87
        8k      wr    48618                    379                2.63     6.18

       16k      rd    20747                    324                6.17     5.48
       16k      wr    40565                    633                3.15     5.45

       32k      rd    16948                    612                7.59     5.85
       32k      wr    31226                    979                4.08     5.33

       64k      rd    15690                    980                8.16     5.94
       64k      rd    12260                    766               10.49     2.84
																						
Config on Ubuntu:

  maxRecvDataSegmentLen=262144
  MaxBurstLength=16776192
  Maxxmitdatasegment=262144
  maxoutstandingr2t=8
  InitialR2T=No
  ImmediateData=Yes
																						
a) MaxRecvDataSegmentLength - Sets the maximum data segment length that can be received. This value should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB.
Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.

b) MaxBurstLength - Sets the maximum amount of either unsolicited or solicited data the initiator may send in a single burst. Any amount of data exceeding this value must be explicitly solicited by the target. This value should be set to multiples of PAGE_SIZE. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 262144.

c) MaxXmitDataSegmentLength - Sets the maximum data segment length that can be sent. This value actually used is the minimum of MaxXmitDataSegmentLength and the MaxRecvDataSegmentLength announced by the initiator. It should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.
																						
d) DataDigest  - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's data segment will be protected by a CRC32C checksum. The default is "None". Note that data digests are not supported during discovery sessions.

e) MaxOutstandingR2T  - Controls the maximum number of data transfers the target may request at once, each of up to MaxBurstLength bytes. The default is 1.
																						
f) InitialR2T  - If set to "Yes" (default), the initiator has to wait for the target to solicit SCSI data before sending it. Setting it to "No" allows the initiator to send a burst of FirstBurstLength bytes unsolicited right after and/or (depending on the setting of ImmediateData together with the command. Thus setting it to "No" may improve performance.
																						
g) ImmediateData  - This allows the initiator to append unsolicited data to a command. To achieve better performance, this should be set to "Yes". The default is "No".

h) DataPDUInOrder  - It tells initiator if data has to be sent in order. Default is "Yes", which is also recommended.
																						
i) DataSequencerInOrder  - It tells initiator if data has to be sent in order. Default is	"Yes", which is also recommended.
																						
j) HeaderDigest  - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's header segments will be protected by a CRC32C checksum. The default is "None".
Note that header digests are not supported during discovery sessions.

k) Wthreads - The iSCSI target employs several threads to perform the actual block I/O to the device. Depending on your hardware and your (expected) workload, the number of these threads may be carefully adjusted. The default value of 8 should be sufficient for most purposes.


Earlier tests were between pairs of single-core dual-processor 2.6GHz Opterons with the Tyan S2895 motherboard.
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms)
					 
        4k      rd   110673                    432       0.01ms
                wr    89372                    347       1.46ms

        16k     rd    32833                    512       3.99ms
                wr    34240                    535       3.80ms

        32k     rd    18467                    577       6.81ms
                wr    18651                    582       6.82ms

        64k     rd    10015                    624      12.30ms
                wr    10344                    648      12.52ms

ATAoE
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms)

        8k      rd    57490                    449       0.31
                wr    44496                    345       0.40

        16k     rd    34443                    536       0.52
                wr    25740                    403       0.69

        32k     rd    17456                    545       1.04
                wr    13782                    430       1.37
											
        64k     rd     8361                    521       2.11
                wr     6568                    410       2.75

iSCSI (no header, data digest)
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms)

        8k      rd    41806                    326       0.43
                wr    48898                    380       0.37

        16k     rd    24864                    389       0.73
                wr    30759                    481       0.52

        32k     rd    15496                    484       1.17
                wr    17532                    548       1.04
						
        64k     rd     8768                    544       2.02
                wr    11232                    702       1.63

iSCSI, crc32
Block Size  Method     IOPS  (Throughput) MBytes/s  avg resp time (ms)

        8k      rd    25079                    195       0.71
                wr    38600                    301       0.46

        16k     rd    13887                    217       1.29
                wr    19992                    313       0.89

        32k     rd     8493                    265       2.12
                wr    11961                    373       1.50

        64k     rd     5039                    314       3.57
                wr     6326                    395       2.85

Myricom banner
Last updated: 01 November 2009