John Liagouris gave a talk at Microsoft Research

John Liagouris gave the following talk at Microsoft Research Redmond, USA: 

Reconfigurable Data Stream Processing

Abstract:

Next-generation streaming systems will not only be scalable and reliable, but also autonomous, flexible, and able to automatically re-configure running applications without downtime. Automatic re-configuration relies heavily on three aspects: (i) accurate profiling to identify computation bottlenecks at runtime, (ii) rigorous performance models to decide the new system configuration that meets the objectives, and (iii) efficient state migration mechanisms to move data around when necessary. After decades of systems research, the state-of-the-art solutions for all these three requirements are still problematic.

In this talk I will present our recent work at ETH Zurich on re-configurable stream processing. The first part of the talk will focus on SnailTrail (NSDI 18), a system for online critical path analysis of distributed streaming dataflows. SnailTrail uses the novel metric of critical participation to generate online performance summaries and provide immediate insights into specific parts of the dataflow computation. The second part of the talk will focus on DS2 (OSDI 18), an auto-scaling controller that leverages online performance metrics along with data flow dependencies to estimate the minimum amount of resources a dataflow needs in order to meet a target throughput. In the third and last part of the talk, I will briefly talk about Megaphone (VLDB 19), a latency-conscious state migration mechanism for streaming dataflows.