RDMA-Apache-HBase 0.9.1 Features

  • Based on Apache HBase 1.1.2
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Apache HBase
  • Compliant with Apache HBase 1.1.2 APIs and applications
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup
  • Tested with
    • Mellanox InfiniBand adapters (DDR, QDR, FDR, and EDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms

RDMA-Apache-Spark 0.9.3 Features

New features and enhancements compared to 0.9.1 release are marked as (NEW).

  • Based on Apache Spark 1.5.1
  • (NEW) Built with Apache Hadoop 2.7.3
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Spark
    • RDMA-based data shuffle
    • SEDA-based shuffle architecture
    • (NEW) Support pre-connection, on-demand connection, and connection sharing
    • Non-blocking and chunk-based data transfer
    • Off-JVM-heap buffer management
  • Compliant with Apache Spark 1.5.1 APIs and applications
  • (NEW) RDMA support for Spark SQL
  • (NEW) Integration with HHH in RDMA for Apache Hadoop
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
  • Tested with
    • (NEW) Mellanox InfiniBand adapters (DDR, QDR, FDR, and EDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms
    • RAM Disks, SSDs, and HDDs

RDMA-Apache-Hadoop-2.x 1.1.0 Features

New features and enhancements compared to 1.0.0 release are marked as (NEW).

  • (NEW) Compliant with Apache Hadoop 2.7.3, Hortonworks Data Platform (HDP) 2.5.0.3, and Cloudera Distribution Including Apache Hadoop (CDH) 5.8.2 APIs and applications
  • (NEW) Based on Apache Hadoop 2.7.3
  • High performance design with native InfiniBand and RoCE support at the verbs level for HDFS, MapReduce, and RPC components
  • (NEW) Plugin-based architecture supporting RDMA-based designs for HDFS (HHH, HHH-M, HHH-L, and HHH-L-BB), MapReduce, MapReduce over Lustre and RPC, etc.
    • Plugin for Cloudera Distribution Including Apache Hadoop (CDH) (tested with 5.8.2)
    • Plugin for Apache Hadoop distribution (tested with 2.7.3)
    • Plugin for Hortonworks Data Platform (HDP) (tested with 2.5.0.3)
  • Supports deploying Hadoop with Slurm and PBS in different running modes (HHH, HHH-M, HHH-L, and MapReduce over Lustre)
  • Easily configurable for different running modes (HHH, HHH-M, HHH-L, HHH-L-BB, and MapReduce over Lustre) and different protocols (native InfiniBand, RoCE, and IPoIB)
  • On-demand connection setup
  • HDFS over native InfiniBand and RoCE
    • RDMA-based write
    • RDMA-based replication
    • Parallel replication support
    • Overlapping in different stages of write and replication
    • Enhanced hybrid HDFS design with in-memory and heterogeneous storage (HHH)
      • Supports four modes of operations
        • HHH (default) with I/O operations over RAM disk, SSD, and HDD
        • HHH-M (in-memory) with I/O operations in-memory
        • HHH-L (Lustre-integrated) with I/O operations in local storage and Lustre
        • HHH-L-BB (Burst Buffer) with I/O operations in Memcached-based burst buffer (RDMA-based Memcached) over Lustre
      • Policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, HDD, and Lustre)
        • Greedy and Balanced policies support
        • Automatic policy selection based on available storage types
      • Hybrid replication (in-memory and persistent storage) for HHH default mode
      • Memory replication (in-memory only with lazy persistence) for HHH-M mode
      • Lustre-based fault-tolerance for HHH-L mode
        • No HDFS replication
        • Reduced local storage space usage
  • MapReduce over native InfiniBand and RoCE
    • RDMA-based shuffle
    • Prefetching and caching of map output
    • In-memory merge
    • Advanced optimization in overlapping
      • map, shuffle, and merge
      • shuffle, merge, and reduce
    • Optional disk-assisted shuffle
    • Automatic Locality-aware Shuffle
    • Optimization of in-memory spill for Maps
    • High performance design of MapReduce over Lustre
      • Supports two shuffle approaches
        • Lustre read based shuffle
        • RDMA based shuffle
      • Hybrid shuffle based on both shuffle approaches
        • Configurable distribution support
      • In-memory merge and overlapping of different phases
  • (NEW) Support for priority-based local directory selection in MapReduce Shuffle
  • RPC over native InfiniBand and RoCE
    • JVM-bypassed buffer management
    • RDMA or send/recv based adaptive communication
    • Intelligent buffer allocation and adjustment for serialization
  • Tested with
    • Mellanox InfiniBand adapters (DDR, QDR, FDR, and EDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms
    • RAM Disks, SSDs, HDDs, and Lustre

RDMA-Apache-Hadoop-1.x 0.9.9 Features

  • Based on Apache Hadoop 1.2.1
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Compliant with Apache Hadoop 1.2.1 APIs and applications
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup
  • HDFS over native InfiniBand and RoCE
    • RDMA-based write
    • RDMA-based replication
    • Parallel replication support
  • MapReduce over native InfiniBand and RoCE
    • RDMA-based shuffle
    • Prefetching and caching of map outputs
    • In-memory merge
    • Advanced optimization in overlapping
      • map, shuffle, and merge
      • shuffle, merge, and reduce
  • RPC over native InfiniBand and RoCE
    • JVM-bypassed buffer management
    • RDMA or send/recv based adaptive communication
    • Intelligent buffer allocation and adjustment for serialization
  • Tested with
    • Mellanox InfiniBand adapters (DDR, QDR, and FDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms
    • Different file systems with disks and SSDs

RDMA-Memcached 0.9.5 Features

New features and enhancements compared to 0.9.4 release are marked as (NEW).

  • Memcached server designs based on Memcached 1.4.24
    • Compliant with the new Memcached's core LRU algorithm
  • Memcached client designs based on libMemcached 1.0.18
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Memcached Server and Client
  • High performance design of SSD-assisted hybrid memory
    • Support for enabling and disabling direct I/O for SSD read/write
  • Compliant with libMemcached APIs and applications
  • (NEW) Non-Blocking Libmemcached Set/Get API extensions
    • (NEW) APIs to issue non-blocking set/get requests to the RDMA-based Memcached servers
    • (NEW) APIs to support monitoring the progress of non-blocking requests issued in an asynchronous fashion
    • (NEW) Facilitating overlap of concurrent set/get requests
  • (NEW) Support for burst-buffer mode in Lustre-integrated design of HDFS in RDMA for Apache Hadoop-2.x
  • Support for both RDMA-enhanced and socket-based Memcached clients
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup
  • Tested with
    • (NEW) Native Verbs-level support with Mellanox InfiniBand adapters (DDR, QDR, FDR, and EDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms
    • SATA-SSD, PCIe-SSD, and NVMe-SSD

OSU HiBD-Benchmarks 0.9.2 Features

New features and enhancements compared to 0.9.1 release are marked as (NEW) .

  • Micro-benchmarks for Hadoop Distributed File System (HDFS)
    • Sequential Write Latency (SWL) Benchmark
    • Sequential Read Latency (SRL) Benchmark
    • Random Read Latency (RRL) Benchmark
    • Sequential Write Throughput (SWT) Benchmark
    • Sequential Read Throughput (SRT) Benchmark
    • Support benchmarking
      • Apache Hadoop 1.x and 2.x HDFS
      • Hortonworks Data Platform (HDP) HDFS
      • Cloudera Distribution of Hadoop (CDH) HDFS
  • Micro-benchmarks for Memcached
    • Get Latency Benchmark
    • Set Latency Benchmark
    • Mixed Get/Set Latency Benchmark
    • Non-Blocking API Latency Benchmark
    • Hybrid Memory Latency Benchmark
  • Micro-benchmarks for HBase
    • Get Latency Benchmark
    • Put Latency Benchmark
  • (NEW) Micro-benchmarks for Spark
    • (NEW) GroupBy
    • (NEW) SortBy