NLU-11: Week 4
Welcome to Week 4!
This week we will continue to cover large language models and how to use them for various purposes. We will cover how to make LLMs generate text properly, and how to use them for parsing, a topic you have learned about in ANLP.
The first tutorial will run this week. Everyone should have been assigned a tutorial group; if you haven't been assigned a group, or you want to change group, please follow the instructions on the Tutorial page. Before attending your tutorial, please prepare answers for the tutorial sheet.
Tutorial 1: Language Models
Tutorials are held in person, see course timetable. will take place in this week, and it deals with neural network language models. Please prepare answers to the following tutorial sheet before attending [pdf]
The solutions for last week's lab are also available now:
Lectures
Lecture 1: Decoding with LLMs [pdf]
Required reading:
- The Curious Case of Neural Text Degeneration, Holtzman et al., 2020 (Sections 1-3, mostly 3)
- Locally Typical Sampling, Meister et al., 2022 (Sections 1-4, mostly with a focus on 4, possibly 3 too)
Lecture 2: Neural parsing [pdf]
Required reading:
Grammar as a Foreign Language, Vinyals et al., NeurIPS 2015. This is the encoder-decoder parsing model introduced in the lecture. (Sections 1-2, skim over results in section 3)
Background reading:
Constituency parsing with a self-attentive encoder, Kitaev and Klein, ACL 2018. This is the transformer-based parsing model introduced in the lecture. (Sections 1-3)
Lecture 3: Unsupervised parsing [pdf]
Required reading:
Unsupervised Parsing via Constituency Tests, Cao et al., EMNLP 2020. (Sections 1, 3-4)