Next Words - Lesson 16 - Calibration, Uncertainty

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 16 - Calibration, Uncertainty

Calibration, reliability diagrams, proper scores, repairs and abstention on held-out text

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 47 pages
  • Quiz, 5 questions
Tags:
calibration uncertainty reliability diagram log loss abstention



What you will learn

  • State the calibration property exactly and separate it from accuracy and sharpness.
  • Draw a reliability diagram, compute expected calibration error, and say what the choice of bins changes.
  • Explain why cross-entropy is a proper scoring rule and accuracy is not.

Courselet Content

1 components
Next Words - Lesson 16 - Calibration, Uncertainty (pdf)
389.63 K

Requirements

  • Cross-entropy, conditional probability

General Overview

Description

Among 1,630 held-out cases given a confidence of 0.5 or more, the mean confidence is 0.601 and the share correct is 0.336. Calibration is a property of a set of predictions and is separate from accuracy and sharpness. Reliability diagrams are drawn on 11,933 predictions, and the expected calibration error ranges from 0.0471 to 0.0785 with the binning alone. Cross-entropy is a proper scoring rule and accuracy is not, since log loss goes from 0.4450 to 5.7091 when every confidence is replaced by 0.999, and accuracy stays at 0.1736. Miscalibration is lowered from 0.0734 to 0.0050 by histogram binning, with accuracy unchanged at 0.1741. Keeping the most confident twentieth of the cases gives an accuracy of 0.519 against 0.174 for answering everything.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 2nd August 2023
  • 2
  • Free
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 7th November 2022
  • 88
  • Free
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/