Overview

Welcome to the High-Performance Big Data project created by the Network-Based Computing Laboratory of The Ohio State University. The HiBD packages are being used by more than 370 organizations worldwide in 39 countries (Current Users) to accelerate Big Data applications. As of Dec '24, more than 49,750 downloads have taken place from this project's site. The HiBD project contains the following packages:


MVAPICH2-based Apache Spark (MPI4Spark 0.3)

MPI4Spark is the custom version of the Apache Spark package that exploits high-performance MPI communication on modern HPC clusters that support InfiniBand, Omni-Path, ROCE and HPE Slingshot interconnects for Big Data applications. The MPI communication backend in MPI4Spark uses the MVAPICH2-J Java bindings of MVAPICH2. Major features of MPI4Spark 0.3 are given below.

  • Based on Apache Spark 3.3.0
  • Support for the YARN cluster manager
  • Compliant with user-level Apache Spark APIs and packages
  • High performance design that utilizes MPI-based communication
    • Utilizes MPI point-to-point operations
    • (NEW) Enhanced the MPI Dynamic Process Management (DPM) logic for launching executor processes for the standalone cluster manager
    • (NEW) Relies on the Multiple-Program-Multiple-Data (MPMD) launcher mode for both YARN and the Standalone cluster managers
    • (NEW) Supports MVAPICH versions 2.3.7 and 4.0
  • Built on top of the MVAPICH2-J Java bindings for MVAPICH2 family of MPI libraries
A complete set of features and supported platforms can be found here.

MVAPICH2-GDR-based Dask Distributed Library (MPI4Dask 0.3)

MPI4Dask is a custom version of the Dask Distributed library that contains high-performance MPI communication backend for Dask. The MPI backend in MPI4Dask uses mpi4py over the MVAPICH2 library and targets modern HPC clusters. Major features of MPI4Dask 0.3 are given below.

  • (NEW) Based on Dask Distributed 2022.8.1
  • Compliant with user-level Dask APIs and packages
  • Support for MPI-based communication for GPU-based Dask applications
  • Support for MPI-based communication for CPU-based Dask applications
    • Implements point-to-point communication co-routines
    • Efficient chunking mechanism implemented for large messages
  • Built on top of mpi4py over MVAPICH2, MVAPICH2-X, and MVAPICH2-GDR libraries
  • (NEW) Various multi-core platforms
  • (NEW) Various benchmarks used by the community
  • (NEW) Capable of scaling out to hundreds of workers (processes)
A complete set of features and supported platforms can be found here..

RDMA-based Apache Hadoop 3.x (RDMA-Hadoop-3.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Hadoop 3.x 0.9.1 are given below.

  • Based on Apache Hadoop 3.0.0
  • Compliant with Apache Hadoop 3.0.0 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS component
  • Supports deploying Hadoop with Slurm and PBS in different running modes (HHH and HHH-M)
  • Easily configurable for different running modes (HHH and HHH-M) and different protocols (native InfiniBand, RoCE, and IPoIB)
  • Enhanced hybrid HDFS design with in-memory and heterogeneous storage (HHH)
    • Supports two modes (default and in-memory) of operations
    • Policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, and HDD)
    • Hybrid replication (in-memory and persistent storage) for HHH default mode
    • Memory replication (in-memory only with lazy persistence) for HHH in-memory mode
A complete set of features and supported platforms can be found here..

RDMA-based Apache Spark (RDMA-Spark)

The RDMA for Apache Spark package is a derivative of Apache Spark. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Spark 0.9.5 are given below. New features and enhancements compared to 0.9.4 release are marked as (NEW).

  • Based on Apache Spark 2.1.0
  • (NEW) Initial support for POWER architecture
  • (NEW) Performance optimization and tuning on OpenPOWER cluster
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Spark
    • RDMA-based data shuffle
    • SEDA-based shuffle architecture
    • Support pre-connection, on-demand connection, and connection sharing
    • Non-blocking and chunk-based data transfer
    • Off-JVM-heap buffer management
  • RDMA support for Spark SQL
  • Integration with HHH in RDMA for Apache Hadoop
  • Compliant with Apache Spark 2.1.0 APIs and applications
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Apache Kafka (RDMA-Kafka)

The RDMA for Apache Kafka package is a derivative of Apache Kafka. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Kafka 0.9.1 are given below.

  • Based on Apache Kafka 1.0.0
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Apache Kafka
  • Compliant with Apache Kafka 1.0.0 APIs and applications
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup
  • Support for RDMA Device Selection

RDMA-based Apache Hadoop 2.x (RDMA-Hadoop-2.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache Hadoop 2.x 1.3.5 are given below. New features and enhancements compared to 1.3.0 release are marked as (NEW).

  • Based on Apache Hadoop 2.8.0
  • Compliant with Apache Hadoop 2.8.0 APIs and applications
  • (NEW) Support for Containers (Docker and Singularity)
  • Initial support for POWER architecture
  • Performance optimization and tuning on OpenPOWER cluster
  • Hortonworks Data Platform (HDP) 2.5.0.3, and Cloudera Distribution Including Apache Hadoop (CDH) 5.8.2 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Plugin-based architecture supporting RDMA-based designs for HDFS (HHH, HHH-M, HHH-L, HHH-L-BB), MapReduce, MapReduce over Lustre and RPC, etc.
    • Plugin for Cloudera Distribution Including Apache Hadoop (CDH) (tested with 5.8.2)
    • Plugin for Apache Hadoop distribution (tested with 2.7.3)
    • Plugin for Hortonworks Data Platform (HDP) (tested with 2.5.0.3)
  • Supports deploying Hadoop with Slurm and PBS in different running modes (HHH, HHH-M, HHH-L, and MapReduce over Lustre)
  • Enhanced hybrid HDFS design with in-memory and heterogeneous storage (HHH)
    • Supports four modes (default, in-memory, Lustre-integrated, and Lustre-integrated with Burst Buffer) of operations
    • Policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, HDD, and Lustre)
    • Hybrid replication (in-memory and persistent storage) for HHH default mode
    • Memory replication (in-memory only with lazy persistence) for HHH in-memory mode
    • Lustre-based fault-tolerance for HHH Lustre-integrated mode
  • Support for priority-based local directory selection in MapReduce Shuffle
  • High performance design of MapReduce over Lustre
    • Supports two shuffle approaches (Lustre read and RDMA)
    • Hybrid shuffle based on both shuffle approaches
  • Easily configurable for different running modes (HHH, HHH-M, HHH-L, HHH-L-BB, and MapReduce over Lustre) and different protocols (native InfiniBand, RoCE, and IPoIB)
A complete set of features and supported platforms can be found here..

RDMA-based Memcached (RDMA-Memcached)

The RDMA for Memcached/libMemcached package is a derivative of Memcached/libMemcached. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Memcached-based applications. Major features of RDMA for Memcached/libMemcached 0.9.6 are given below. New features and enhancements compared to 0.9.5 release are marked as (NEW).

  • (NEW) Based on Memcached 1.5.3
    • (NEW) Compliant with the Memcached’s new item chaining feature in In-Memory mode
    • (NEW) Compliant with the latest Memcached’s LRU maintainer and slab balancer enhancements
  • Based on libMemcached 1.0.18
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Memcached Server and Client
  • High performance design of SSD-assisted hybrid memory
  • (NEW) Runtime selection of HCA device for nodes equipped with multiple InfiniBand/RocE HCAs
  • (NEW) Enable and disable item chaining through extended server options
  • Compliant with libMemcached 1.0.18 APIs and applications
  • Non-Blocking Libmemcached Set/Get API extensions
    • APIs to issue non-blocking set/get requests to the RDMA-based Memcached servers
    • APIs to support monitoring the progress of non-blocking requests issued in an asynchronous fashion
    • Facilitating overlap of concurrent set/get requests
  • Support for burst-buffer mode in Lustre-integrated design of HDFS in RDMA for Apache Hadoop-2.x
  • Support for both RDMA-enhanced and socket-based Memcached clients
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup
  • Tested with
    • Native Verbs-level support with Mellanox InfiniBand adapters (QDR, FDR, and EDR)
    • RoCE support with Mellanox adapters
    • Various multi-core platforms
    • SATA-SSD, PCIe-SSD, and NVMe-SSD

RDMA-based Apache HBase (RDMA-HBase)

The RDMA for Apache HBase package is a derivative of Apache HBase. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of RDMA for Apache HBase 0.9.1 are given below.

  • Based on Apache HBase 1.1.2
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Apache HBase
  • Compliant with Apache HBase 1.1.2 APIs and applications
  • Easily configurable for native InfiniBand, RoCE, and the traditional sockets based support (Ethernet and InfiniBand with IPoIB)
  • On-demand connection setup

RDMA-based Apache Hadoop 1.x (RDMA-Hadoop-1.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

  • Based on Apache Hadoop 1.2.1
  • Compliant with Apache Hadoop 1.2.1 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

OSU HiBD-Benchmarks (OHB)

The OSU HiBD-Benchmarks project aims at developing benchmarks for evaluating Big Data middleware. The current version (0.9.3) of OHB consists of micro-benchmarks for Hadoop Distributed File System (HDFS), Memcached, Yahoo! Cloud Serving Benchmark (YCSB) extension for RDMA-Memcached, HBase and Spark.

Announcements


Join us for the Upcoming Tutorials at Supercomputing 2024: 1. High-Performance and Smart Networking Technologies for HPC and AI, 2. Principles and Practice of High-Performance Deep Learning/Machine Learning Training and Inference, and 3. Scalable Big Data Processing on High-Performance Computing Systems More Details

MPI4Spark 0.3 (based on Apache Spark 3.3.0) with support for MPI-based communication runtime on high-performance networks (InfiniBand, OPA, ROCE, and Slingshot) to accelerate SPARK-based applications and support for the YARN cluster manager is available. [more]

The 12th Annual MVAPICH User Group (MUG) Conference was held successfully in a hybrid manner on August 19-21, 2024 with more than 220 attendees.

MPI4Dask 0.3 (based on Dask Distributed 2022.8.1) with support for MPI-based communication in Dask for a cluster of CPUs and GPUs, built on top of mpi4py over the MVAPICH2, MVAPICH2-X, and MVAPICH2-GDR library, starting execution of Dask programs using Dask-MPI, compliant with user-level Dask APIs and packages is available. [more]

Partnership and contribution to the NSF-Awarded $20M AI-Institute on Intelligent CyberInfrastructure (ICICLE). Details.

MPI4Dask 0.2 (based on Dask Distributed 2021.01.0) with support for MPI-based communication in Dask for a cluster of CPUs and GPUs, built on top of mpi4py over the MVAPICH2, MVAPICH2-X, and MVAPICH2-GDR library, starting execution of Dask programs using Dask-MPI, compliant with user-level Dask APIs and packages is available. [more]

RDMA-Apache-Hadoop-3.x 0.9.1 (based on Apache Hadoop 3.0.0) with support for high-performance design with native InfiniBand and RoCE support at the verbs level for HDFS component, easily configurable for different running modes (HHH and HHH-M) and different protocols (native InfiniBand, RoCE, and IPoIB), RDMA-based write, RDMA-based replication, policies to efficiently utilize heterogeneous storage devices (RAM Disk, SSD, and HDD), hybrid replication (in-memory and persistent storage) for HHH default mode, and memory replication (in-memory only with lazy persistence) for HHH-M mode is available. [more]

RDMA-Kafka 0.9.1 (based on Apache Kafka 1.0.0) with support for high-performance design with native InfiniBand and RoCE support at the verbs level, compliant with Apache Kafka 1.0.0 APIs and applications, on-demand connection setup, and support for RDMA Device Selection and different protocols (native InfiniBand, RoCE, and IPoIB) is available. [more]

RDMA-Apache-Hadoop-2.x 1.3.5 (based on Apache Hadoop 2.8.0) with support for Containers (Docker and Singularity), OpenPOWER architecture, plugin-based designs for Apache Hadoop 2.7.3, Cloudera Distribution including Apache Hadoop (CDH) 5.8.2, Hortonworks Data Platform (HDP) 2.5.0.3, supporting hybrid HDFS design with in-memory and heterogeneous storage (HHH), Memcached-based burst buffer for MapReduce over Lustre-integrated HDFS, easily configurable for different running modes (HHH, HHH-M, HHH-L, HHH-L-BB, and MapReduce over Lustre) and different protocols (native InfiniBand, RoCE, and IPoIB) is available. [more]

RDMA-Spark 0.9.5 based on Apache Spark 2.1.0, built with Apache Hadoop 2.8.0, initial support for POWER architecture, performance optimization and tuning on OpenPOWER clusters, support for various communication binding policies, support for RDMA device selection, compliant with Spark 2.1.0 APIs and applications, and supporting different JDK (OpenJDK and IBM JDK) protocols (native InfiniBand, RoCE, and IPoIB) is available. [more]

OSU HiBD Benchmarks (OHB) 0.9.3 with support of YCSB Extension for RDMA-Memcached is available.

RDMA-Memcached 0.9.6 (based on Memcached 1.5.3) with native InfiniBand and RoCE support; compliant with new features like chaining, LRU maintainer, and slab balancer; enable/disable chaining through extended server options; high-performance design of SSD-assisted hybrid memory; run-time selection of HCA device for nodes with multiple adapters; and easy configuration for InfiniBand-RDMA, RoCE and sockets are available. [more]

RDMA-HBase 0.9.1 (based on Apache HBase 1.1.2) supporting high-performance design for native InfiniBand and RoCE at the verbs-level, compliant with Apache HBase 1.1.2 APIs and applications, and supporting different protocols (native InfiniBand, RoCE, and IPoIB) is available. [more]

RDMA-based Apache Hadoop 2.X is available on SDSC Comet for XSEDE Users. Visit here for more details.

HiBD in the News