Performance of PVFS2-MX over MX-10G

Performance measurements are presented for PVFS2-MX over MX-10G using the bmi_pingpong benchmark from the PVFS2 Test Suite.

The environment for these tests consists of two dual-core, dual-socket 2.2 GHz Tyan Thunder K8WE S2895 machines with a 10G-PCIE-8A-C NIC in each machine. The two machines were connected point-to-point (switchless). Each machine has 8 GB of memory and was running Linux 2.6.11.12, MX-10G 1.2.0i, and PVFS2 2.6.1.

Latency

The following graph shows small message latency of BMI-MX using MX-10G compared to BMI-TCP using Myri-10G Ethernet. The message sizes range from 64 bytes to 4 KB and represent the size of typical metadata messages. BMI-MX is 65-80% faster than BMI-TCP. Latency is measured in microseconds (µs, shown as usecs in the graphs).

Throughput

The following graph shows the throughput of a single message sent between two processes with BMI-MX using MX-10G compared to BMI-TCP using Myri-10G Ethernet. The message sizes range from 64 bytes to 4 MB. Most IO will be between 4 KB and 4 MB. PVFS2 will fragment larger messages into 4 MB messages as needed. BMI-MX is 30-50% faster than BMI-TCP for larger messages. Throughput is measured in MB/s.

Latency for Expected and Unexpected Messages

This graph shows small message latency of BMI-MX for expected and unexpected messages on MX-10G. PVFS2 uses unexpected messages to send or retrieve metadata as well as to initiate bulk IO transfers. The message sizes range from 64 bytes to 4 KB and represent the size of typical metadata messages. Latency is measured in microseconds (µs, shown as usecs in the graphs).

Last updated: 21 May 2007