Week 8: Transfer learning and large language models

Reminders and announcements

• Tutorial 3 group meetings are this week. This tutorial will tackle issues with web-scale data, dialect and discrimination, and possible mismatches between the preferences of users and those of developers/researchers.
• This week, we also have a guest lecture by Jacqueline Rowe, a PhD student in the Centre for Doctoral Training in Responsible NLP, which is closely related to the tutorial! She'll be talking about bias in Transformer models. The contents of the guest lecture are not directly examinable but may provide further perspectives/examples on our examinable topics.
• We have scheduled extra help hours over the next two weeks to assist you with the assignment. All help hours take place in Appleton Tower 5.07:
  • Mon 9 Nov 13:00    (Andrew)
  • Wed 11 Nov 11:00    (Andrew)
  • Mon 16 Nov 13:00    (Christian)
  • Wed 18 Nov 11:00    (Christian)
  • Fri 20 Nov 15:00    (Christian)

Overview of the Week

This week we will start with a guest lecture on bias in Transformers. Then we'll introduce the concept of large language models (LLMs), i.e., a language model (usually with a Transformer architecture) pre-trained on large amounts of data with unsupervised objectives. This is an instance of the more general framework of transfer learning. In particular, we will study how encoder LLMs (like BERT) can provide contextual representations of words and can be fine-tuned for sequence classification tasks (such as Natural Language Inference). We will also cover other architectures for generative LLMs, including encoder-decoder LLMs (such as T5) and decoder-only LLMs (such as GPT).

Lectures and reading