References

Key Papers on Uncertainty Quantification

Foundational Work

  1. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning (ICML).

  2. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems (NeurIPS).

  3. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. International Conference on Machine Learning (ICML).

Bayesian Deep Learning

  1. Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural networks. International Conference on Machine Learning (ICML).

  2. Graves, A. (2011). Practical variational inference for neural networks. Advances in Neural Information Processing Systems (NeurIPS).

  3. Hernández-Lobato, J. M., & Adams, R. (2015). Probabilistic backpropagation for scalable learning of Bayesian neural networks. International Conference on Machine Learning (ICML).

Ensemble Methods

  1. Dietterich, T. G. (2000). Ensemble methods in machine learning. International Workshop on Multiple Classifier Systems.

  2. Fort, S., Hu, H., & Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757.

Calibration Methods

  1. Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers.

  2. Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

  3. Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. International Conference on Machine Learning (ICML).

Aleatoric and Epistemic Uncertainty

  1. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems (NeurIPS).

  2. Malinin, A., & Gales, M. (2018). Predictive uncertainty estimation via prior networks. Advances in Neural Information Processing Systems (NeurIPS).

Applications and Case Studies

  1. Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017). Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports.

  2. Michelmore, R., Kwiatkowska, M., & Gal, Y. (2018). Evaluating uncertainty quantification in end-to-end autonomous driving control. arXiv preprint arXiv:1811.06817.

Out-of-Distribution Detection

  1. Hendrycks, D., & Gimpel, K. (2017). A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations (ICLR).

  2. Liang, S., Li, Y., & Srikant, R. (2018). Enhancing the reliability of out-of-distribution image detection in neural networks. International Conference on Learning Representations (ICLR).

Surveys and Reviews

  1. Abdar, M., et al. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion.

  2. Gawlikowski, J., et al. (2021). A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342.

  3. Hüllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning.

Software and Tools

Online Resources

Datasets and Benchmarks

  • UCI Machine Learning Repository: Classification and regression datasets

  • Kaggle Uncertainty Quantification Challenges: Various competitions

  • Medical Image Segmentation Datasets: With uncertainty annotations

  • Autonomous Driving Datasets: With rare event annotations