We start by introducing recurrent neural networks, and their application to classification. Then we discuss how to use neural networks to perform language modeling. It turns out it is not as hard (at least conceptually) with what we learned in the previous lectures. Predictions of next words in a language model are effectively classification decisions, so a neural language model is just a sequence of classifiers. We will use this intuition to turn an RNN-based text classifier into an RNN-based language model. We will also look into what information RNN captures in its state and discuss if RNNs can learn syntax and how to test that.
The folder contains slides, required reading and a quiz.
Slides and reading
(The recorded video contains animations which are not visible in pdf)
Recommended reading: Jurafsky and Martin, 3rd edition (online). Chapter 9, but you can skip details of LSTMs as we focus in FNLP on simpler vanilla RNNs, 9.1 - 9.4 and 9.6 would contain relevant material. Note that there is material in the lectures which is not covered in J&M.
Also optionally: study language modeling and seq2seq sections in Lena Voita's NLP course:
Quiz 24: RNN and neural LMs
These questions are designed to test your understanding of the above course content; doing this quiz does not contribute to your overall grade. Some questions require a text answer. You can ask for formative feedback on these from your tutor or on piazza. Other questions are multiple choice or they require a numeric answer: you will get immediate feedback for these. Please don't attempt this quiz until you have acquainted yourself with this lecture and the required reading.
You must be logged onto Learn to do this quiz.