There are five HDFS micro-benchmarks: (1) Sequential Write Latency (SWL), (2) Sequential Read Latency (SRL), (3) Random Read Latency (RRL), (4) Sequential Write Throughput (SWT), and (5) Sequential Read Throughput (SRT). For detailed configuration and setup, please refer to our userguide.

HDFS Sequential Write and Read Latency

Sequential Write Latency (SWL)

swl

Sequential Read Latency (SRL)

srl

Experimental Testbed: Each node of our testbed has two 4-core 2.53 GHz Intel Xeon E5630 (Westmere) processors and 24 GB main memory. The nodes support 16x PCI Express Gen2 interfaces and are equipped with Mellanox ConnectX QDR HCAs with PCI Express Gen2 interfaces. The operating system used was RedHat Enterprise Linux Server release 6.1 (Santiago).

These experiments are run on 4 DataNodes. Each DataNode has 12GB of RAM disk, 300GB OCZ VeloDrive PCIe SSD, and 1TB HDD. HDFS block size is kept to 128 MB. The NameNode runs in a different node of the Hadoop cluster and the benchmark is run in the NameNode. The RDMA-IB design improves the job execution time of SWL by up to 45.7% over IPoIB (32Gbps) and 54.8% over 10GigE. For SRL, it provides a maximum of 52.7% improvement over IPoIB (32Gbps) and 54% over 10GigE.


HDFS Sequential Write and Read Throughput

Sequential Write Throughput (SWT)

swt

Sequential Read Throughput (SRT)

srt

Experimental Testbed: Each node of our testbed has two 4-core 2.53 GHz Intel Xeon E5630 (Westmere) processors and 24 GB main memory. The nodes support 16x PCI Express Gen2 interfaces and are equipped with Mellanox ConnectX QDR HCAs with PCI Express Gen2 interfaces. The operating system used was RedHat Enterprise Linux Server release 6.1 (Santiago).

These experiments are performed on 4 DataNodes for 40GB data size. The number of clients is varied from 4 to 16. The NameNode runs in a different node of the Hadoop cluster and the benchmark runs in the NameNode. The RDMA-IB design improves the throughput of SWT by up to 108% over IPoIB (32Gbps) and 151% over 10GigE. For SRT, it provides a maximum of 58% improvement over IPoIB (32Gbps) and 102% over 10GigE.


HDFS Random Read Latency

Random Read Latency (RRL)

rrl

Experimental Testbed: Each node of our testbed has two 4-core 2.53 GHz Intel Xeon E5630 (Westmere) processors and 24 GB main memory. The nodes support 16x PCI Express Gen2 interfaces and are equipped with Mellanox ConnectX QDR HCAs with PCI Express Gen2 interfaces. The operating system used was RedHat Enterprise Linux Server release 6.1 (Santiago).

These experiments are performed on 4 DataNodes. Each DataNode has 12GB of RAM disk, 300GB OCZ VeloDrive PCIe SSD, and 1TB HDD. HDFS block size is kept to 128 MB. The NameNode runs in a different node of the Hadoop cluster and the benchmark runs in the NameNode. The RDMA-IB design reduces the execution time of RRL by up to 64.8% over IPoIB (32Gbps) and 69.5% over 10GigE.