RandomWriter Execution Time
TeraGen Execution Time
Experimental Testbed: Each node of our testbed has two 4-core 2.53 GHz Intel Xeon E5630 (Westmere) processors and 24 GB main memory. The nodes support 16x PCI Express Gen2 interfaces and are equipped with Mellanox ConnectX QDR HCAs with PCI Express Gen2 interfaces. The operating system used was RedHat Enterprise Linux Server release 6.4 (Santiago).
These experiments are performed in 8 DataNodes with a total of 32 maps. Each DataNode has a single 1TB HDD. HDFS block size is kept to 256 MB. Each TaskTracker launches 4 concurrent maps. The NameNode runs in a different node of the Hadoop cluster and the benchmark is run in the NameNode.
The RDMA-IB design improves the job execution time of RandomWriter by 15% - 16% and TeraGen by 7% - 12% compared to IPoIB (32Gbps). Compared to 10GigE, the improvement is 21% - 37% in RandomWriter and 3% - 19% in TeraGen.
RandomWriter Execution Time on SSD
TeraGen Execution Time on SSD
Experimental Testbed: Each node of our testbed has two 4-core 2.53 GHz Intel Xeon E5630 (Westmere) processors and 24 GB main memory. The nodes support 16x PCI Express Gen2 interfaces and are equipped with Mellanox ConnectX QDR HCAs with PCI Express Gen2 interfaces. The operating system used was RedHat Enterprise Linux Server release 6.4 (Santiago).
These experiments are performed in 4 DataNodes with a total of 16 maps. Each DataNode has a single 300GB OCZ VeloDrive PCIe SSD. HDFS block size is kept to 256 MB. Each TaskTracker launches 4 concurrent maps. The NameNode runs in a different node of the Hadoop cluster and the benchmark is run in the NameNode.
The RDMA-IB design improves the job execution time of RandomWriter by 11% - 12% and TeraGen by 16% - 18% compared to IPoIB (32Gbps). Compared to 10GigE, the improvement is 14% - 21% in RandomWriter and 21% - 25% in TeraGen.
RandomWriter Execution Time
TeraGen Execution Time
Experimental Testbed: Each node of our testbed is dual-socket containing Intel Sandy Bridge (E5-2680) dual octa-core processors running at 2.70GHz. Each node has 32GB of main memory, a SE10P (B0-KNC) co-processor, and a Mellanox IB FDR MT4099 HCA. The host processors run CentOS release 6.3 (Final)
The RandomWriter experiments are performed in 32 DataNodes with a total of 128 maps. Each DataNode has a single 80GB HDD. HDFS block size is kept to 256 MB. Each TaskTracker launches 4 concurrent maps. The NameNode runs in a different node of the Hadoop cluster and the benchmark is run in the NameNode. The RDMA-IB design improves the job execution time of RandomWriter by 12% - 14%.
The TeraGen experiments are performed in 16 DataNodes with a total of 64 maps. The RDMA-IB design improves the job execution time of TeraGen by 17% - 20% compared to IPoIB (56Gbps).
