MPI4Spark Changelog ---------------------- This file briefly describes the changes to the MPI4Spark software package. The logs are arranged in the "most recent first" order. MPI4Spark v0.3 - (08/28/2024) * New Features & Enhancements - Enhanced MPI Dynamic Process Management (DPM) logic for launching executor processes for the standalone cluster manager - Relies on the Multiple-Program-Multiple-Data (MPMD) launcher mode for both YARN and the Standalone cluster managers - Supports MVAPICH versions 2.3.7 and 4.0 - Tested with * OSU HiBD-Benchmarks, GroupBy and SortBy * Intel HiBench Suite, Micro Benchmarks, Machine Learning and Graph Workloads MPI4Spark v0.2 - (08/25/2023) * New Features & Enhancements - Support for the YARN Cluster Manager - Executor processes are launched using the Multiple-Program- Multiple-Data (MPMD) MPI launcher mode - Tested with * OSU HiBD-Benchmarks, GroupBy and SortBy * Intel HiBench Suite, Micro Benchmarks, Machine Learning and Graph Workloads MPI4Spark v0.1 (11/11/2022) * New Features & Enhancements - Based on Apache Spark 3.3.0 - Compliant with user-level Apach Spark APIs and packages - High performance design that utilizes MPI-based communication * Utilizes MPI point-to-point operations * Relies on MPI Dynamic Process Management (DPM) features for launching executor processes - Built on top of the MVAPICH2-J Java bindings for MVAPICH2 family of MPI libraries - Tested with * OSU HiBD-Benchmarks, GroupBy and SortBy * Intel HiBench Suite, Micro Benchmarks, Machine Learning and Graph Workloads * Mellanox InfiniBand adapters (EDR and HDR 100G and 200G) * HPC systems with Omni-Path interconnects * Various multi-core platforms