Title:Exoshuffle: Large-Scale Shuffle at the Application Level

Abstract:Shuffle is a key primitive in large-scale data processing applications. The difficulty of large-scale shuffle has inspired a myriad of implementations. While these have greatly improved shuffle performance and reliability over time, it comes at a cost: flexibility. First, each shuffle system is essentially built from scratch, which is a significant developer effort. Second, because each shuffle system is monolithic, they are not flexible to supporting other applications, such as online aggregation of shuffle results.
We show that shuffle can be implemented with high performance and reliability on a general-purpose abstraction for distributed computing: distributed futures. While previous systems have implemented shuffle on top of distributed futures before, we are the first to identify and build the common components necessary to support a large-scale shuffle. We show that it is possible to: (1) express optimizations from previous shuffle systems in a few hundred lines of purely application-level Python code, and (2) achieve interoperability with other data processing applications without modifying the shuffle system. Thus, we present Exoshuffle, an application-level shuffle system that outperforms Spark and achieves 82\% of theoretical performance on a 100TB sort on 100 nodes.