Frozen weights, demonstrations, prompt design, written steps, majority votes, prompt cost
|
Next Words - Lesson 14 - In-Context Learning, Prompting (pdf)
|
298.83 K |
Over five random draws of demonstrations, the SST-2 accuracy of GPT-3 is between 86.9 and 95.8 per cent, a range of 8.9 points. In the posterior over tasks the odds are multiplied by 3 with each consistent demonstration, and after one trailing space the top next token of GPT-2 is "urn" where it was "floor". Exact match on NQ is 28.6 with random demonstrations and 41.6 with demonstrations chosen by cosine similarity. LaMDA 137B is right on 14.3 per cent of GSM8K problems with the working before the answer and on 6.1 after it. Accuracy of PaLM 540B grows from 56.5 to 74.4 per cent under a majority vote over 40 chains. The daily bill is $1,000 at 100 tokens against $20,000 at 2,000. Students work through four notebooks, one on in-context learning and three workbooks.