Welcome to Week 4!
This week we will continue to cover large language models and how to use them for various purposes.
The first tutorial will run this week. Everyone should have been assigned a tutorial group; if you haven't been assigned a group, or you want to change group, please follow the instructions on the Tutorial page. Before attending your tutorial, please prepare answers for the tutorial sheet.
Tutorial 1: Language Models
Tutorials are held in person, see course timetable. will take place in this week, and it deals with neural network language models. Please prepare answers to the following tutorial sheet before attending [pdf]
The solutions for last week's lab are also available now:
Lectures
- Pretrained Language Models [pdf]. Required reading:
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al., NAACL 2019.
Background reading:
- Chapter 9 of Speech and Language Processing, 3rd edition by Jurafsky and Martin provides an alternative presentation of contextualized word embeddings.
- Prompting with LLMs [pdf]. Required reading:
- Sections 1-4 Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, Liu et al. (2021)
Additional reading: Language models are unsupervised multitask learners, Radford et al. (2019)
Decoding with LLMs [pdf]. Required reading:
- The Curious Case of Neural Text Degeneration, Holtzman et al., 2020
- Locally Typical Sampling, Meister et al., 2022
Background reading:
- Neural Machine Translation with Reconstruction, Tu et al., 2017
- Six Challenges for Neural Machine Translation, Koehn et al., 2017