Next Words - Lesson 13 - Alignment: RLHF, DPO

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 13 - Alignment: RLHF, DPO

Pre-training loss, reward models, the KL optimum, the DPO loss and jailbreaks

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 21 pages
  • Jupyter notebooks, 4 (zip)
  • Quiz, 5 questions
Tags:
RLHF alignment DPO reward model jailbreaks



What you will learn

  • Compute the change in average loss that one incorrect token causes, in nats per token.
  • Compute a Bradley-Terry preference probability and the loss of a reward model from two rewards.
  • Derive the KL-constrained optimal policy and the DPO loss, and show why Z(x) is absent from the loss for two answers to...

Courselet Content

2 components
Next Words - Lesson 13 - Alignment: RLHF, DPO (pdf)
317.2 K

Requirements

  • Conditional probability, logarithms, Python

General Overview

Description

GPT-3 was trained on 300 billion tokens, and in the pre-training loss every token has the same weight. With one incorrect token in 200, the average loss is 0.0145 nats higher. In supervised fine-tuning the prompt is masked, since 50 of 250 tokens, 20 per cent of the gradient, would otherwise be spent on predicting it. Bradley and Terry turn a reward difference into a preference probability, 0.668 for C over B. The KL-constrained optimum is the reference policy times the exponentiated reward, and the reward model is dropped in direct preference optimisation because the difference between two answers to one prompt has no Z(x). The DPO loss of the pair C over B is 0.576. Raters who agree on 80 per cent of pairs, with independent errors, are each right 88.7 per cent. Students work through four notebooks, one on alignment and three workbooks.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 48
  • Free
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 11th September 2024
  • 123
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 6th May 2025
  • 740
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/