Week 2: N-gram models

Strike information

If you have not seen my Learn announcement about the strike, please read this page.

Reminders and Announcements

Welcome to week 2! A big congratulations to all of you for making it through the first week! It can be a lot to keep track of everything at the start, so here are a few reminders:

• Lab sessions this week!
  • If you are not registered for the course by Monday morning, you won't be assigned to a lab session yet. In that case, just pick one to attend this week. Labs are all in AT 5.05:
    • Monday: 13:10-14:30, 14:40-16:00, or 16:10-17:30
    • Tuesday: 14:40-16:00 or 16:10-17:30
  • The lab materials are available below, under Additional Materials. Before your scheduled lab, please work through Lab 0 from last week, and also check that you can log into a DICE machine (i.e. in one of the Appleton Tower labs).
  • You don't need to find a lab partner in advance, just find a partner once you get to the lab room.
• Response to Intake Form. Please see the bottom of this page for our responses to the intake form.
• TA hours: A reminder that we have drop-in help hours with a TA every Wed from 11:10-12:00  in AT 5.07, if you have a question that's not easily answered online, or just prefer to ask in person!
• If you aren't yet registered, only recently registered, or missed a lot of Week 1:
  • Especially important to do as soon as possible: Watch the lecture 1 video and read the Course Guidance document, so you know how the course is structured.
  • Then, start working through the rest of the Week 1 materials.
  • If you run into problems with either of the above, please post to EdStem. In the meantime, keep working through other material to catch up.

Overview of Week 2

In the lectures this week, we will introduce the notion of a model and how we can train and evaluate one based on data. Specifically, we will focus on n-gram models, a type of language model. We'll introduce these models, and how to train and evaluate them. Although they are not nearly as good as today's large language models, N-gram models are also generative language models! So we'll build up intuitions about how to generate from language models by explaining this using N-gram models. We will also talk about how to avoid overfitting, an issue we'll see again throughout the course.

Lectures and reading

If you have not seen my Learn announcement about the strike, please read this page. Live lectures will not take place, but links to videos and other important information are provided there.

Additional Materials

• Lab 1 is now available.
  • Before your scheduled lab, make sure you've completed Lab 0 from last week, and you know your DICE account/password to log into the computers in the Appleton Tower labs.
    • You should already have a DICE account if you are an Informatics MSc student or as soon as you register for this class.
    • If you do yet not have a DICE account, please make sure to pair up during the lab session with someone who does.
  • The rest of the lab should be done during the lab session, working with a partner. To access the lab, follow the instructions at the link above.
• Optional: Some students mentioned they would like some additional reading to help with the linguistics content. One option is the book mentioned on the Readings page. But if you are looking for something a bit lighter, you could also try Crash Course Linguistics on YouTube. It doesn't match up exactly with what we cover, but if you're interested in getting a quick overview of many different areas of linguistics, it's good for that!

Response to intake form and students' worries

Thank you to everyone who completed our intake form: we received 135 responses. We now know that there are (at least) 37 different languages (and several more dialects) that you speak at home, plus at least another 14 you are familiar with! This means that you'll have a rich source of different language experiences to inform your discussions in tutorial groups and other interactions with each other.

Just over half of respondents have an academic background in computer science (52%), but there are significant numbers of students from linguistics, maths/physics, and engineering backgrounds. So take this opportunity to also engage with your peers who are knowledgeable in different disciplines. In other areas the class also varies a lot, with a few students who've taken NLP classes before, but also nearly a third with no previous technical experience with NLP---though almost everyone said they use generative NLP at least sometimes (with about 45% using it many times a day).

We have also looked over the concerns people expressed in the intake form. While we can't address every concern that was raised, there were a few that came up repeatedly:

• Skills with English and/or writing: Unfortunately, given the size of this course and the many other learning objectives, we can't offer much individual help with these issues, though we will try to provide some general writing guidance in assignment handouts. If you are worried about these issues, please look over the University pages on Academic language courses (mainly for writing) and Academic Skills, which offers resources on academic writing as well as many other topics.
• Background knowledge/skills: Various students were worried about the aspects of the course that they are less familiar with (linguistics, programming, maths).
  • Please remember that only a handful of students in the course have all of this background, and we certainly don't expect any previous experience with NLP, so almost all of you will need to stretch your thinking in some way. We are used to getting a wide variety of students in this class, and although you will need to work hard, it is definitely possible to do well in the end even if you are rather rusty on your maths, or have little experience with programming, or if you don't know anything about linguistics. (If you don't have any of those things, though, it might be reasonable to consider whether this class is right for you.)
  • Your classmates are a great resource, which is one reason why we try to give you opportunities to meet other students, and why we pair you up for the labs and assignments. We can't guarantee that you will get a partner from a different background to yourself, but you'll have a chance to work with different partners, which increases the chances. We also encourage you to find other students to study with, and to ask them or post to the forum when you're confused. 
• Reading: A few students are concerned about the amount of reading, or knowing which reading is necessary. Please note that we do indicate which parts of the reading to prioritize with a *, so you know which sections are most relevant to the day's lecture.
• Past exam papers: A few students also seem to be worried after looking at past exam papers. Please keep in mind:
  • We would not expect you to understand or be able to answer past exam questions during the first week of class!
  • However, it is true that if you are used to being able to predict very closely what questions will be on the exam by looking at previous exams, you will need to find a new strategy. We rarely re-use questions, and we often ask you to use what you've learned and apply it to new situations. We encourage you to practice these skills during the tutorial discussions.
  • Furthermore, while the format and style of questions on the exam will be similar this year to previous years, we made major updates to the course content last year. So there will be a lot of questions from 2024 and earlier that you may not be able to answer even after taking this year's class! We will provide examples of more realistic exam questions for this year during some of the tutorials and nearer the end of the course.
• Working in pairs: While we do strongly encourage students to work together on labs and coursework, this is not compulsory. So if it creates a lot of difficulty for you, you can work alone.