RDMA for Apache Hadoop Change Log ---------------------- This file briefly describes the changes to the RDMA for Apache Hadoop software package. The logs are arranged in the "most recent first" order. Release 0.9.9 - 2014-03-31 NEW FEATURES - On-demand connection setup - HDFS over native InfiniBand and RoCE - Parallel replication support - MapReduce over native InfiniBand and RoCE - Advanced optimization in overlapping - map, shuffle, and merge - shuffle, merge, and reduce - RPC over native InfiniBand and RoCE - Intelligent buffer allocation and adjustment for serialization BUG FIXES - Fix an issue in framework startup - Fix for configurable port selection - MapReduce over native InfiniBand and RoCE - Fix for progress calculation in Reduce tasks - HDFS over native InfiniBand and RoCE - Fix an issue in DataNode initialization Release 0.9.8 - 2013-11-08 NEW FEATURES - Based on Apache Hadoop 1.2.1 - High performance design with native InfiniBand and RoCE support at the verbs level for HDFS, MapReduce, and RPC components - Compliant with Apache Hadoop 1.2.1 APIs and applications - Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB) - HDFS over native InfiniBand and RoCE - RDMA-based write - RDMA-based replication - Tuned packet size - MapReduce over native InfiniBand and RoCE - RDMA-based shuffle - Prefetching and caching of map outputs - In-memory merge - Overlapping of merge and reduce - RPC over native InfiniBand and RoCE - JVM-bypassed buffer management - On-demand InfiniBand connection setup - RDMA or send/recv based adaptive communication - Tested with - Mellanox InfiniBand adapters (DDR, QDR and FDR) - RoCE support with Mellanox adapters - Various multi-core platforms - Different file systems with disks and SSDs BUG FIXES - MapReduce over InfiniBand - Reducing the waiting time for the prefetch cache read - Enabling Reduce tasks to randomize the shuffle pattern - HDFS over InfiniBand - Fix an issue in processing multiple blocks concurrently in a single task Release 0.9.1 - 2013-08-09 BUG FIXES - MapReduce over InfiniBand - Fix an issue in handling re-initiated tasks for a failed task scenario - Fix an issue in resolving hostnames inside ReduceTask - RPC over InfiniBand - Fix an issue in processing messages inside RPC server - HDFS over InfiniBand - Improved debug messages and code cleanup Release 0.9.0 - 2013-06-13 NEW FEATURES - Based on Apache Hadoop 0.20.2 - High performance design with native InfiniBand support at the verbs level for HDFS, MapReduce, and RPC components - Compliant with Apache Hadoop 0.20.2 APIs and applications - Easily configurable for both native InfiniBand and the traditional sockets based support (Ethernet and InfiniBand with IPoIB) - HDFS over native InfiniBand - RDMA-based write - RDMA-based replication - Tuned packet size - MapReduce over native InfiniBand - RDMA-based shuffle - Prefetching and caching of map outputs - In-memory merge - Overlapping of merge and reduce - RPC over native InfiniBand - JVM-bypassed buffer management - On-demand InfiniBand connection setup - RDMA or send/recv based adaptive communication - Tested with - Mellanox InfiniBand adapters (DDR, QDR, and FDR) - Various multi-core platforms - Different file systems with disks and SSDs