Week 4: Word embeddings and neural networks
Reminders and announcements
Welcome to Week 4!
- Tutorial groups this week. Our first group meetings are this week. Please remember to make a good attempt at the problems before you arrive to your meeting. The problems are available under Additional Materials.
- Tutorial groups next week. We will have tutorial group meetings again next week, which again will require advance preparation. The exercises are under Additional Materials.
- Important: If you are in either of the Tuesday tutorial groups (Tue 14:10 or Tue 16:10, starting on 7 Oct), the timetable currently shows these groups meeting in weeks 4, 6, 8, and 11. However, the week 6 meeting is incorrect, and will be happening in week 5! (All tutorial groups meet in weeks 4, 5, 8, and 11.) We have asked the timetabling service repeatedly to fix this, and will keep doing so, but in the meantime, please update your own diary to indicate that you have a tutorial group on 14 Oct, *not* on 21 Oct. We will hold the 14 Oct meeting for these two tutorial groups in a different room (Appleton Tower 5.01). This week's meeting (7 Oct) is listed correctly.
Overview of this week
This week we are starting to shift from more traditional approaches to more recent approches (well, sort of: neural networks are not actually new, they just didn't work well until recently when more powerful computers made it possible to train larger models). We will start by discussing word2vec, one of the earliest word embedding models to make a big impact. It's based on the logistic regression models we learned about last week. Then we'll introduce our first neural network model, the multilayer perceptron (or feedforward neural network) which also builds on logistic regression by adding "hidden layers". We will take a couple of lectures to go over the model itself, how to use it for classification and language modelling, the training mechanism, and some more practical issues with training.
Lectures and reading
| Lecture # | Who? | Slides | Reading |
|---|---|---|---|
| 1 | SG | Dense word embeddings | JM3 5.4-5.5 (*), 5.6-5.10 |
| 2 | SG | Feedforward neural networks | JM3 6.0-6.3.0 (*), 6.4-6.5 See note below. |
| 3 | SG | Training neural networks (slides not available yet) | JM3 6.6.0-6.6.3 (*), 6.6.4-6.6.5 |
Note: Chapter 6 introduces neural networks in a bottom up way (from the pieces to the big picture), and is fairly long. If you are finding it hard to follow the explanation or the maths, or just prefer to see the big picture first, you might want to do this reading after the lecture. In the lecture I am planning to focus more on the big picture first: the relationship of NNs to logistic regression, the limitations of LR that NNs are able to solve, and what allows them do that.
Additional Materials
- Tutorial exercise sheet 1: for discussion in tutorial groups this week (Week 4). Answers will be made available after all groups have met.
- Tutorial exercise sheet 2: for discussion in tutorial groups next week (Week 5).
- As before, please prepare answers in advance.
- You should be able to do all but Q3.4 based on Mon and Wed lecture/reading, Q3.4 relies on Friday lecture.
- Preview of next week's reading. I'm afraid I can't say yet what order the lectures will be in next week, but I do know what the main readings will be.
- Understanding Bias, Part 1 from Machines Gone Wrong by Lim Swee Kiat.
- Blodgett and O'Connor (2017)
- JM3 13.0-13.2 (*), 13.3.1-2, 13.3.3 (*), 13.7 (*)