Week 5 – Deep Neural Networks (3): Regularization, Learning Algorithms

https://opencourse.inf.ed.ac.uk/MLP

Week 5

Teaching week commencing: 19 October 2026
Key topics: Deep Neural Networks (3): Regularization, Learning Algorithms

Reading guide

• Goodfellow, Bengio and Courville: Deep Learning — Section 7.12, Section 8.3, and Section 8.5.
• Andrej Karpathy, CS231n notes (Stanford) 
• For keen readers:
  • Sebastian Ruder An overview of gradient descent optimization algorithms
• References:
  • Dropout paper > Srivastava et al, Dropout: a simple way to prevent neural networks from overfitting, JMLR, 15(1), 1929-1958, 2014.
  • Adam paper > Kingma and Ba, Adam: A Method for Stochastic Optimization, ICLR, 2015.

Lecture materials

Scripted recording: Deep Neural Networks (3)

This recording was produced in a previous year. The corresponding lecture content has since changed significantly, so the recording should be treated as supplementary material. References to course logistics are out of date.

Slides: Lecture 05 — Deep Neural Networks (3): Regularization, Learning Algorithms (see below).

Practical materials

Lab 4 — code and notebooks

The Lab 4 branch also includes updated Lab 3 files containing the solutions.

Indicative workload

The original taught-course estimate was 7 hours:

• Lab: 1 hour.
• Recorded lecture: 1 hour.
• Lecture/discussion: 1 hour.
• Independent study: 3 hours.
• Additional work on the PyTorch lab component: 1 hour.

Independent study includes further work on the lab and the suggested reading. The lab exercises also help test your understanding of the material.