Visual Discovery for Science

Abstract: From social media all the way to satellite images, we are capturing visual data at an unprecedented scale. These images tell a story about our planet. With advances in automatic recognition, we can build a collective understanding of world-scale events as recorded through visual media. In this talk, we will first explore the idea of automatic discovery from visual data in a relatively simpler domain of fashion data from social media images, where powerful recognition models already exist. I will then discuss my work on an annotation-efficient method for building a multimodal vision-language model in the scientific domain of remote sensing where language annotations are sparse. Finally, I will present my recent research on building interpretable models for scientific discovery using such black-box vision language models.

Bio: Utkarsh Mall is a postdoctoral research scientist in Computer Science at Columbia University. His research also focuses on building interpretable, reliable, and data-efficient methods to make novel scientific discoveries from visual data. He has also applied this research in various areas such as agriculture, anthropology, archaeology, urban planning, public health, climate science, etc. Before joining Columbia, he earned his PhD from Cornell University, where he worked on building label-efficient foundation models for scientific domains and leveraging them for unsupervised discoveries. He also co-organizes the CVPR workshop: Computer Vision for Science (CV4Science). His doctoral research was selected for the WACV 2022 and CVPR 2023 Doctoral Consortium.

online: https://teams.microsoft.com/l/meetup-join/19%3a178c8aaec5e040879a9a053743b825cd%40thread.tacv2/1741628666733?context=%7b%22Tid%22%3a%22624d5c4b-45c5-4122-8cd0-44f0f84e945d%22%2c%22Oid%22%3a%222ac13d43-5ced-4b3c-a68a-5b66063407a9%22%7d