From Stream Processor to a Unified Data Processing System

The Apache Flink community has pushed (and continues to push) the boundary for Stream Processing over the last years, following the understanding that Stream Processing is unifying paradigm to build data processing applications, beyond real-time analytics. The latest major effort in the Flink community is nothing less then re-architecting the API and runtime stack, with the goal to naturally support the spectrum of analytics and data-driven applications, to unify the APIs for batch and streaming (Table API and DataStream API), and to build a streaming runtime that is not only state-of-the-art in stream processing, but also in batch processing performance. In this keynote, we give an overview of the goals and technology behind the above effort, and look at the adoption of Apache Flink for Stream Processing and "beyond streaming" use cases, as well as various efforts in the community to support the growth in users, applications, and ecosystem.