Next-Generation AI-Enhanced Stream Processing

Speaker: Dr. Manisha Luthra Agnihotri is the Deputy Head of the German Research Center for Artificial Intelligence (DFKI) in Darmstadt.

MS teams link

Abstract: In this talk, I will outline my vision for next-generation, AI-enhanced data management systems through the lens of learned stream processing. Today’s stream processing platforms demand extensive manual tuning to optimize critical decisions such as query plan selection, operator placement, and parallelism. My vision eliminates these labor-intensive processes by leveraging zero-shot learning to automatically derive optimal configurations, thereby radically enhancing performance and generalisability. A key contribution of my work is a novel learned operator placement optimization provided by a novel cost model that forecasts the execution costs of streaming queries on heterogeneous hardware. Particularly in IoT environments—where diverse hardware and network conditions are the norm—our approach employs graph neural networks to predict query costs accurately, even for unseen placements and query patterns. This approach not only overcomes the generalizability limitations of existing methods but also paves the way for more robust and adaptive cost-based optimizations for stream processing systems. I will also discuss my future research directions, focusing on extending these AI-driven techniques to multi-modal stream processing. This work aims to redefine data management by creating systems that adapt to evolving computational needs for multiple modalities, ultimately setting new standards for understanding data inputs and autonomy in stream processing.

Bio: Manisha Luthra Agnihotri is the Deputy Head of the German Research Center for Artificial Intelligence (DFKI) in Darmstadt and a Research Group Leader at TU Darmstadt. She co-leads the Systems AI for Decision Support group with focus of research on learned system optimizations and multimodal data management. Her work sits at the dynamic intersection of machine learning, data systems, and hardware, with major contributions in learned cost-based optimization and the acceleration of query workloads via GPU and RDMA technologies.

Throughout her academic journey, Manisha has received several prestigious awards, including the German national Best Ph.D. Thesis award from the GI/ITG special interest group on Communication and Distributed Systems (KuVS), the Athena Young Investigator Award, the Anita Borg Faculty Scholarship, the Zeiss Top Dissertation Scholarship, and mentoring and networking accolades from the German Research Foundation (DFG). Her expertise has led her to speak at top-tier institutions such as the University of Toronto, and she has presented her innovative research at premier conferences like SIGMOD, VLDB, ICDE, and EDBT. Manisha also actively contributes to the academic community as a program committee member for major data management conferences, including VLDB, SIGMOD, and EuroSys.