Date: Wednesday, April 9th, 2025
9:00 am – 10:00 am Pacific Time
12:00 pm – 1:00 pm Eastern Time
Location: Weekly Seminar, Zoom
Title: Uncertainty Quantification Beyond Calibration
Abstract:
Calibration has emerged as a standard approach to uncertainty quantification, providing valuable insights into model reliability. However, calibration exhibits two key limitations when compared to ideal uncertainty estimates. First, it fails to distinguish between model-based (epistemic) and data-based (aleatoric) uncertainty—a critical limitation when determining whether collecting more data could improve predictions. Second, the uncertainty estimates produced by calibrated models may perform substantially worse than externally trained models specifically designed to predict a model’s error at each point.
This talk will overview two research contributions that address these fundamental limitations. For the first challenge, I will present a new notion called “higher-order calibration” which provides a rigorous theoretical foundation for decomposing a model’s uncertainty, with formal guarantees relating the decomposition to real-world data distributions. For the second challenge, I will demonstrate an equivalence between a model’s level of multicalibration and its competitiveness with externally trained loss predictors, revealing when models can—or cannot—accurately assess their own limitations.
This talk covers joint work with Gustaf Ahdritz, Aravind Gollakota, Parikshit Gopalan, Aayush Karan, and Udi Wieder (https://arxiv.org/abs/2412.18808, https://arxiv.org/abs/2502.20375).
Bio:
Charlotte Peale is a PhD student in Computer Science at Stanford University, advised by Omer Reingold. Her research focuses on algorithmic fairness, learning theory, and uncertainty quantification. She is supported by the Apple Scholars in AI/ML fellowship and completed a research internship at Apple Machine Learning Research, where she developed the work presented in this talk under the mentorship of Parikshit Gopalan.