HiBD@SC'14

Multiple events presented by the HiBD team members.

Overview

Welcome to the High-Performance Big Data project created by the Network-Based Computing Laboratory of The Ohio State University. The HiBD project contains the following packages:


RDMA-based Apache Hadoop 2.x (RDMA-Hadoop-2.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

  • Based on Apache Hadoop 2.5.0
  • Compliant with Apache Hadoop 2.5.0 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Apache Hadoop 1.x (RDMA-Hadoop-1.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

  • Based on Apache Hadoop 1.2.1
  • Compliant with Apache Hadoop 1.2.1 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Memcached (RDMA-Memcached)

The RDMA for Memcached/libMemcached package is a derivative of Memcached/libMemcached. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Memcached-based applications. Major features of this package include:

  • Based on Memcached 1.4.20 and libMemcached 1.0.18
  • Compliant with Memcached APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Memcached Server and Client
  • Support for both RDMA-enhanced and socket-based Memcached clients
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

OSU HiBD-Benchmarks (OHB)

The OSU HiBD-Benchmarks project aims at developing benchmarks for evaluating Big Data middleware. The current version (0.7.1) of OHB consists of microbenchmarks for Memcached.

Announcements


(NEW) RDMA-Apache-Hadoop-2.x 0.9.5 (based on Apache Hadoop 2.5.0) with native InfiniBand and RoCE support for HDFS, MapReduce and RPC integrated with other Hadoop components and easy configuration among InfiniBand-RDMA, RoCE and sockets is available. [more]

RDMA-Memcached 0.9.1 (based on Memcached 1.4.20 and libMemcached 1.0.18) with native InfiniBand and RoCE support and easy configuration for InfiniBand-RDMA, RoCE and sockets is available. [more]

OSU HiBD Benchmarks (OHB) 0.7.1 with Memcached Benchmarks (Get, Set and Mixed Get/Set) are available. [more]

RDMA-Apache-Hadoop 0.9.9 (based on Apache Hadoop 1.2.1) with native InfiniBand and RoCE support for HDFS, MapReduce, RPC and integrated with other Hadoop components and easy configuration among InfiniBand-RDMA, RoCE and sockets is available. [more]

Keynote Talk on Accelerating Data Mananagement and Processing on Clusters with RDMA-Enabled Interconnects at ADMS '14 conference. Past Keynote Talk on Big Data/RDMA for Apache Hadoop presented at Big Data Technology Conference.