Pre-training loss, reward models, the KL optimum, the DPO loss and jailbreaks
|
Next Words - Lesson 13 - Alignment: RLHF, DPO (pdf)
|
317.2 K |
GPT-3 was trained on 300 billion tokens, and in the pre-training loss every token has the same weight. With one incorrect token in 200, the average loss is 0.0145 nats higher. In supervised fine-tuning the prompt is masked, since 50 of 250 tokens, 20 per cent of the gradient, would otherwise be spent on predicting it. Bradley and Terry turn a reward difference into a preference probability, 0.668 for C over B. The KL-constrained optimum is the reference policy times the exponentiated reward, and the reward model is dropped in direct preference optimisation because the difference between two answers to one prompt has no Z(x). The DPO loss of the pair C over B is 0.576. Raters who agree on 80 per cent of pairs, with independent errors, are each right 88.7 per cent. Students work through four notebooks, one on alignment and three workbooks.