HiBD@SC'14

Multiple events presented by the HiBD team members.

Overview

Welcome to the High-Performance Big Data project created by the Network-Based Computing Laboratory of The Ohio State University. As of Mar '15, more than 2,550 downloads have taken place from this project's site. The HiBD project contains the following packages:


RDMA-based Apache Hadoop 2.x (RDMA-Hadoop-2.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

  • Based on Apache Hadoop 2.5.0
  • Compliant with Apache Hadoop 2.5.0 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Apache Hadoop 1.x (RDMA-Hadoop-1.x)

The RDMA for Apache Hadoop package is a derivative of Apache Hadoop. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Big Data applications. Major features of this package include:

  • Based on Apache Hadoop 1.2.1
  • Compliant with Apache Hadoop 1.2.1 APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for HDFS, MapReduce, and RPC components
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

RDMA-based Memcached (RDMA-Memcached)

The RDMA for Memcached/libMemcached package is a derivative of Memcached/libMemcached. This package can be used to exploit performance on modern clusters with RDMA-enabled interconnects for Memcached-based applications. Major features of this package include:

  • Based on Memcached 1.4.20 and libMemcached 1.0.18
  • Compliant with Memcached APIs and applications
  • High performance design with native InfiniBand and RoCE support at the verbs-level for Memcached Server and Client
  • Support for both RDMA-enhanced and socket-based Memcached clients
  • Easily configurable for native InfiniBand, RoCE and the traditional sockets-based support (Ethernet and InfiniBand with IPoIB)

OSU HiBD-Benchmarks (OHB)

The OSU HiBD-Benchmarks project aims at developing benchmarks for evaluating Big Data middleware. The current version (0.7.1) of OHB consists of microbenchmarks for Memcached.

Announcements


(NEW) High Performance Big Data Computing (HPBDC) International Workshop is soliciting technical papers. Workshop Details and Paper Submission Instructions.

Tutorial on Big Data, RDMA for Apache Hadoop, Spark and Memcached to be presented at HPCA 2015, ASPLOS 2015, and ISCA 2015 conferences.

RDMA-Apache-Hadoop-2.x 0.9.5 (based on Apache Hadoop 2.5.0) with native InfiniBand and RoCE support for HDFS, MapReduce and RPC integrated with other Hadoop components and easy configuration among InfiniBand-RDMA, RoCE and sockets is available. [more]

RDMA-Memcached 0.9.1 (based on Memcached 1.4.20 and libMemcached 1.0.18) with native InfiniBand and RoCE support and easy configuration for InfiniBand-RDMA, RoCE and sockets is available. [more]

OSU HiBD Benchmarks (OHB) 0.7.1 with Memcached Benchmarks (Get, Set and Mixed Get/Set) are available. [more]