Week 9: multilingual models and scaling laws

Reminders and announcements

Welcome to Week 9! As we near the end of the semester, it's always a difficult time of year. I expect that many of you (like us) are feeling the strain of working hard for so long; you probably have several coursework deadlines coming up, and of course, the weather is colder and darker as well. I hope you can still take a few moments to appreciate the turning of the season and what you've learned so far.

• Solutions to Tutorial 3 will be available by the start of the week.
• Reminder that there are extra help hours this week for the assignment, all in AT 5.04 with Christian:
  • Mon 16 Nov 13:00  
  • Wed 18 Nov 11:00  
  • Fri 20 Nov 15:00
• Assignment submission: To help you submit the assignment material, we created a video that explains how to upload the material via Gradescope. You can find the recording on Learn under the "Assessment" group. Alternatively, you can access it via the following link.    

Overview of the week

This week, Emily and Alessandro will be lecturing as we continue our explorations of LLMs. Specifically, we will discuss how GPT-3 introduced the notion of in-context learning (ICL), a test-time strategy to adapt a model towards a specific task without updating its parameters. This is achieved by prepending a small number of input–output examples to the LLM's context. We will then explore the challenges of creating Large Language Models not just for English, but for many of the languages spoken around the world. Afterwards, we will see how the accuracy of a model across tasks can be predicted reliably by scaling laws that depend on the model size (number of parameters), number of tokens, and number of training steps.

Lectures and readings