Metrics of Calibration for Probabilistic Predictions

被引:0
|
作者
Arrieta-Ibarra, Imanol [1 ]
Gujral, Paman [1 ]
Tannen, Jonathan [1 ]
Tygert, Mark [1 ]
Xu, Cherie [1 ]
机构
[1] Meta, 1 Facebook Way, Menlo Pk, CA 94025 USA
关键词
reliability diagram; calibration plot; cumulative differences; Kolmogorov-Smirnov; Kuiper;
D O I
暂无
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Many predictions are probabilistic in nature; for example, a prediction could be for precipitation tomorrow, but with only a 30% chance. Given such probabilistic predictions together with the actual outcomes, "reliability diagrams" (also known as "calibration plots") help detect and diagnose statistically significant discrepancies-so-called "miscalibration"-between the predictions and the outcomes. The canonical reliability diagrams are based on histogramming the observed and expected values of the predictions; replacing the hard histogram binning with soft kernel density estimation using smooth convolutional kernels is another common practice. But, which widths of bins or kernels are best? Plots of the cumulative differences between the observed and expected values largely avoid this question, by displaying miscalibration directly as the slopes of secant lines for the graphs. Slope is easy to perceive with quantitative precision, even when the constant offsets of the secant lines are irrelevant; there is no need to bin or perform kernel density estimation. The existing standard metrics of miscalibration each summarize a reliability diagram into a single scalar statistic. The cumulative plots naturally lead to scalar metrics for the deviation of the graph of cumulative differences away from zero; good calibration corresponds to a horizontal, flat graph which deviates little from zero. The cumulative approach is currently unconventional, yet offers many favorable statistical properties, guaranteed via mathematical theory backed by rigorous proofs and illustrative numerical examples. In particular, metrics based on binning or kernel density estimation unavoidably must trade-off statistical confidence for the ability to resolve variations as a function of the predicted probability or vice versa. Widening the bins or kernels averages away random noise while giving up some resolving power. Narrowing the bins or kernels enhances resolving power while not averaging away as much noise. The cumulative methods do not impose such an explicit trade-off. Considering these results, practitioners probably should adopt the cumulative approach as a standard for best practices.
引用
收藏
页数:54
相关论文
共 50 条
  • [31] Calibrated Probabilistic Predictions for Biomedical Applications
    Lambrou, Antonis
    Papadopoulos, Harris
    Gammerman, Alexander
    IEEE 12TH INTERNATIONAL CONFERENCE ON BIOINFORMATICS & BIOENGINEERING, 2012, : 211 - 216
  • [32] Calibration of probabilistic age recognition
    van Leeuwen, David A.
    Basari, Hasan Mohamad
    13TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION 2012 (INTERSPEECH 2012), VOLS 1-3, 2012, : 502 - 505
  • [33] Probabilistic formulation for storm surge predictions
    Contento, Alessandro
    Xu, Hao
    Gardoni, Paolo
    STRUCTURE AND INFRASTRUCTURE ENGINEERING, 2020, 16 (04) : 547 - 566
  • [34] Probabilistic Predictions of Train Delay Evolution
    Spanninger, Thomas
    Buchel, Beda
    Corman, Francesco
    2021 7TH INTERNATIONAL CONFERENCE ON MODELS AND TECHNOLOGIES FOR INTELLIGENT TRANSPORTATION SYSTEMS (MT-ITS), 2021,
  • [35] Probabilistic forecasts, calibration and sharpness
    Gneiting, Tilmann
    Balabdaoui, Fadoua
    Raftery, Adrian E.
    JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 2007, 69 : 243 - 268
  • [36] TCAD calibration for FinFET reliability predictions
    Jha, Vishal
    Hoque, Mazhar
    Manahile, Mihilat
    Rascon, Joseph
    2022 IEEE INTERNATIONAL INTEGRATED RELIABILITY WORKSHOP, IIRW, 2022,
  • [37] Rationalizing predictions by adversarial information calibration
    Sha, Lei
    Camburu, Oana-Maria
    Lukasiewicz, Thomas
    ARTIFICIAL INTELLIGENCE, 2023, 315
  • [38] PROBABILISTIC FORECASTING OF CONSTRUCTION LABOR PRODUCTIVITY METRICS
    Jacobsen, Emil L.
    Teizer, Jochen
    Wandahl, Soren
    Brilakis, Ioannis
    JOURNAL OF INFORMATION TECHNOLOGY IN CONSTRUCTION, 2024, 29 : 58 - 83
  • [39] A Probabilistic Approach to Surgical Tasks and Skill Metrics
    Berniker, Max
    Bhattacharyya, Kiran D.
    Brown, Kristen C.
    Jarc, Anthony
    IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 2022, 69 (07) : 2212 - 2219
  • [40] New Metrics for the Reliability of Approximate and Probabilistic Adders
    Liang, Jinghang
    Han, Jie
    Lombardi, Fabrizio
    IEEE TRANSACTIONS ON COMPUTERS, 2013, 62 (09) : 1760 - 1771