Welcome to Week 5! This week we will continue studying LLMs, and focus on an example application (parsing) and discuss their scaling and issues with security and safety. We also provide the tutorial solutions below and a new lab. Wishing you a productive week of learning!
Please note that Gradescope now accepts submissions of assignment 1 (code and text are separate).
Lab 2: Word Embeddings - in the labs page.
Tutorial solutions
The tutorial solutions are now available on the tutorials page.
Lectures
Neural parsing [pdf] - Required reading:
Grammar as a Foreign Language, Vinyals et al., NeurIPS 2015. This is the encoder-decoder parsing model introduced in the lecture.
Background reading:
Constituency parsing with a self-attentive encoder, Kitaev and Klein, ACL 2018. This is the transformer-based parsing model introduced in the lecture.
- Scaling laws for LLMs [pdf]
Required reading:
- Scaling Laws for Neural Language Models, Kaplan et al. 2020
Training Compute-Optimal Large Language Models, Hoffmann et al. 2022 (Chinchilla scaling laws)
- Safety and security with LLMs [pdf]
Required reading:
- A Watermark for Large Language Models, Kirchenbauer et al., 2023 (sections 1-3)
- Universal and Transferable Adversarial Attacks on Aligned Language Models, Zou et al., 2023 (sections 1-2)
Background reading (optional):
Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information, Shao et al. 2023 (sections 1-3)