PERFORMANCE MODELING FOR APACHE BEAM

Abstract

Modern stream processing systems handle large amounts of data with near-realtime computational demands. Characterizing the performance of streaming systems is difficult because streaming systems are distributed systems, performance is workload-dependent, and because there are many layers of abstraction and many ways for large systems to go wrong. We contribute to the effort to understand performance by developing a portable benchmark suite for Apache Beam, and by arguing for a set of composable benchmarks that can build larger benchmark suites and performance models. We also develop a general model of a distributed stream processing framework and survey recent work in queueing theory applied to stream processing.