Week 4: Word embeddings and neural networks
Reminders and announcements
Welcome to Week 4!
- Tutorial groups this week. Our first group meetings are this week. Please remember to make a good attempt at the problems before you arrive to your meeting. The problems are available under Additional Materials.
- Tutorial groups next week. We will have tutorial group meetings again next week, which again will require advance preparation. The exercises are under Additional Materials.
Overview of this week
This week we are starting to shift from more traditional approaches to more recent approaches (well, sort of: neural networks are not actually new, they just didn't work well until recently when more powerful computers made it possible to train larger models). We will start by discussing word2vec, one of the earliest word embedding models to make a big impact. It's based on the logistic regression models we learned about last week. Then we'll introduce our first neural network model, the multilayer perceptron (or feedforward neural network) which also builds on logistic regression by adding "hidden layers". We will take a couple of lectures to go over the model itself, how to use it for classification and language modelling, the training mechanism, and some more practical issues with training.
Lectures and reading
| Lecture # | Who? | Slides | Reading |
|---|---|---|---|
| 1 | EA | Dense word embeddings | JM3 5.4-5.5 (*), 5.6-5.10 |
| 2 | EA | Feedforward neural networks | JM3 6.0-6.3.0 (*), 6.4-6.5 See note below. |
| 3 | SG | Training neural networks | JM3 6.6.0-6.6.3 (*), 6.6.4-6.6.5 |
Note: Chapter 6 introduces neural networks in a bottom up way (from the pieces to the big picture), and is fairly long. If you are finding it hard to follow the explanation or the maths, or just prefer to see the big picture first, you might want to do this reading after the lecture. In the lecture I am planning to focus more on the big picture first: the relationship of NNs to logistic regression, the limitations of LR that NNs are able to solve, and what allows them do that.
Additional Materials
- Tutorial exercise sheet 1: for discussion in tutorial groups this week (Week 4). Answers will be made available after all groups have met.
- Tutorial exercise sheet 2: for discussion in tutorial groups next week (Week 5).
- As before, please prepare answers in advance.
- You should be able to do all but Q3.4 based on Mon and Wed lecture/reading, Q3.4 relies on Friday lecture.