MVAPICH2-based Apache Spark (MPI4Spark 0.2)

MPI4Spark is the custom version of the Apache Spark package that exploits high-performance MPI communication on modern HPC clusters that support InfiniBand, Omni-Path, ROCE and HPE Slingshot interconnects for Big Data applications. The MPI communication backend in MPI4Spark uses the MVAPICH2-J Java bindings of MVAPICH2. Major features of MPI4Spark 0.2.0 are given below.

• Based on Apache Spark 3.3.0
• (NEW) Support for the YARN cluster manager
• Compliant with user-level Apache Spark APIs and packages
• High performance design that utilizes MPI-based communication
• Utilizes MPI point-to-point operations
• Relies on MPI Dynamic Process Management (DPM) features for launching executor processes
• (NEW) Relies on the Multiple-Program-Multiple-Data (MPMD) MPI launcher mode for launching executors when using the YARN cluster manager
• Built on top of the MVAPICH2-J Java bindings for MVAPICH2 family of MPI libraries

MVAPICH2-GDR-based Dask Distributed Library (MPI4Dask 0.3)

MPI4Dask is a custom version of the Dask Distributed library that contains high-performance MPI communication backend for Dask. The MPI backend in MPI4Dask uses mpi4py over the MVAPICH2 library and targets modern HPC clusters. Major features of MPI4Dask 0.3 are given below.

• (NEW) Based on Dask Distributed 2022.8.1
• Compliant with user-level Dask APIs and packages
• Support for MPI-based communication for GPU-based Dask applications
• Support for MPI-based communication for CPU-based Dask applications
• Implements point-to-point communication co-routines
• Efficient chunking mechanism implemented for large messages
• Built on top of mpi4py over MVAPICH2, MVAPICH2-X, and MVAPICH2-GDR libraries
• (NEW) Various multi-core platforms
• (NEW) Various benchmarks used by the community
• (NEW) Capable of scaling out to hundreds of workers (processes)

RDMA-based Apache Hadoop 3.x (RDMA-Hadoop-3.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Hadoop 3.x 0.9.1 are given below.

• Based on Apache Hadoop 3.0.0
• Compliant with Apache Hadoop 3.0.0 APIs and applications
• High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS component
• Supports deploying Hadoop with Slurm and PBS in different running modes (HHH and HHH-M)
• Easily configurable for different running modes (HHH and HHH-M) and different protocols (native InfiniBand, RoCE, and IPoIB)
• Enhanced hybrid HDFS design with in-memory and heterogeneous storage (HHH)
  • Supports two modes (default and in-memory) of operations
  • Policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, and HDD)
  • Hybrid replication (in-memory and persistent storage) for HHH default mode
  • Memory replication (in-memory only with lazy persistence) for HHH in-memory mode

RDMA-based Apache Spark (RDMA-Spark)

The RDMA for Apache Spark package is a derivative of Apache Spark. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Spark 0.9.5 are given below. New features and enhancements compared to 0.9.4 release are marked as (NEW).

• Based on Apache Spark 2.1.0
• (NEW) Initial support for POWER architecture
• (NEW) Performance optimization and tuning on OpenPOWER cluster
• High performance design with native InfiniBand and RoCE support at the verbs-level for Spark
  • RDMA-based data shuffle
  • SEDA-based shuffle architecture
  • Support pre-connection, on-demand connection, and connection sharing
  • Non-blocking and chunk-based data transfer
  • Off-JVM-heap buffer management
• RDMA support for Spark SQL
• Integration with HHH in RDMA for Apache Hadoop
• Compliant with Apache Spark 2.1.0 APIs and applications
• Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Apache Kafka (RDMA-Kafka)

The RDMA for Apache Kafka package is a derivative of Apache Kafka. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Kafka 0.9.1 are given below.

• Based on Apache Kafka 1.0.0
• High performance design with native InfiniBand and RoCE support at the verbs-level for Apache Kafka
• Compliant with Apache Kafka 1.0.0 APIs and applications
• Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
• On-demand connection setup
• Support for RDMA Device Selection

RDMA-based Apache Hadoop 2.x (RDMA-Hadoop-2.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Hadoop 2.x 1.3.5 are given below. New features and enhancements compared to 1.3.0 release are marked as (NEW).

• Based on Apache Hadoop 2.8.0
• Compliant with Apache Hadoop 2.8.0 APIs and applications
• (NEW) Support for Containers (Docker and Singularity)
• Initial support for POWER architecture
• Performance optimization and tuning on OpenPOWER cluster
• Hortonworks Data Platform (HDP) 2.5.0.3, and Cloudera Distribution Including Apache Hadoop (CDH) 5.8.2 APIs and applications
• High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
• Plugin-based architecture supporting RDMA-based designs for HDFS (HHH, HHH-M, HHH-L, HHH-L-BB), MapReduce, MapReduce over Lustre and RPC, etc.
  • Plugin for Cloudera Distribution Including Apache Hadoop (CDH) (tested with 5.8.2)
  • Plugin for Apache Hadoop distribution (tested with 2.7.3)
  • Plugin for Hortonworks Data Platform (HDP) (tested with 2.5.0.3)
• Supports deploying Hadoop with Slurm and PBS in different running modes (HHH, HHH-M, HHH-L, and MapReduce over Lustre)
• Enhanced hybrid HDFS design with in-memory and heterogeneous storage (HHH)
  • Supports four modes (default, in-memory, Lustre-integrated, and Lustre-integrated with Burst Buffer) of operations
  • Policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, HDD, and Lustre)
  • Hybrid replication (in-memory and persistent storage) for HHH default mode
  • Memory replication (in-memory only with lazy persistence) for HHH in-memory mode
  • Lustre-based fault-tolerance for HHH Lustre-integrated mode
• Support for priority-based local directory selection in MapReduce Shuffle
• High performance design of MapReduce over Lustre
  • Supports two shuffle approaches (Lustre read and RDMA)
  • Hybrid shuffle based on both shuffle approaches
• Easily configurable for different running modes (HHH, HHH-M, HHH-L, HHH-L-BB, and MapReduce over Lustre) and different protocols (native InfiniBand, RoCE, and IPoIB)

RDMA-based Memcached (RDMA-Memcached)

The RDMA for Memcached/libMemcached package is a derivative of Memcached/libMemcached. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Memcached-based applications. Major features of RDMA for Memcached/libMemcached 0.9.6 are given below. New features and enhancements compared to 0.9.5 release are marked as (NEW).

• (NEW) Based on Memcached 1.5.3
  • (NEW) Compliant with the Memcached’s new item chaining feature in In-Memory mode
  • (NEW) Compliant with the latest Memcached’s LRU maintainer and slab balancer enhancements
• Based on libMemcached 1.0.18
• High performance design with native InfiniBand and RoCE support at the verbs-level for Memcached Server and Client
• High performance design of SSD-assisted hybrid memory
• (NEW) Runtime selection of HCA device for nodes equipped with multiple InfiniBand/RocE HCAs
• (NEW) Enable and disable item chaining through extended server options
• Compliant with libMemcached 1.0.18 APIs and applications
• Non-Blocking Libmemcached Set/Get API extensions
  • APIs to issue non-blocking set/get requests to the RDMA-based Memcached servers
  • APIs to support monitoring the progress of non-blocking requests issued in an asynchronous fashion
  • Facilitating overlap of concurrent set/get requests
• Support for burst-buffer mode in Lustre-integrated design of HDFS in RDMA for Apache Hadoop-2.x
• Support for both RDMA-enhanced and socket-based Memcached clients
• Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)
• On-demand connection setup
• Tested with
  • Native Verbs-level support with Mellanox InfiniBand adapters (QDR, FDR, and EDR)
  • RoCE support with Mellanox adapters
  • Various multi-core platforms
  • SATA-SSD, PCIe-SSD, and NVMe-SSD

RDMA-based Apache HBase (RDMA-HBase)

The RDMA for Apache HBase package is a derivative of Apache HBase. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache HBase 0.9.1 are given below.

• Based on Apache HBase 1.1.2
• High performance design with native InfiniBand and RoCE support at the verbs-level for Apache HBase
• Compliant with Apache HBase 1.1.2 APIs and applications
• Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
• On-demand connection setup

RDMA-based Apache Hadoop 1.x (RDMA-Hadoop-1.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

• Based on Apache Hadoop 1.2.1
• Compliant with Apache Hadoop 1.2.1 APIs and applications
• High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
• Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

OSU HiBD-Benchmarks (OHB)

The OSU HiBD-Benchmarks project aims at developing benchmarks for evaluating Big Data middleware. The current version (0.9.3) of OHB consists of micro-benchmarks for Hadoop Distributed File System (HDFS), Memcached, Yahoo! Cloud Serving Benchmark (YCSB) extension for RDMA-Memcached, HBase and Spark.