Week 2 – Single Layer Networks (2), Stochastic Gradient Descent, Classification
https://opencourse.inf.ed.ac.uk/MLP
Week 2
Teaching week commencing: 28 September 2026
Key topics: Single-layer networks, stochastic gradient descent and classification.
Reading guide
- Nielsen: Neural Networks and Deep Learning — Chapter 1.
- Goodfellow, Bengio and Courville: Deep Learning — Chapter 1 and Sections 4.3, 5.1, 5.7, 5.9, 6.1, 6.2 and 8.1.
Lecture materials
Scripted recording: Single Layer Networks (2) / Classification
This recording was produced in a previous year and provides an alternative or complementary explanation of the material. References to course logistics are out of date.
Slides: Lecture 02 – Single-layer networks (2), stochastic gradient descent and classification (see below).
Practical materials
Indicative workload
The original taught-course estimate was 6 hours:
- Lab: 1 hour.
- Recorded lecture: 1 hour.
- Lecture/discussion: 1 hour.
- Independent study: 3 hours.
Independent study includes further work on the lab and the suggested reading. The lab exercises also help test your understanding of the material.
Selected technical Q&A
Where are the lab solutions?
Solutions to a lab are included in the following lab’s GitHub branch. For example, the Lab 2 branch contains the updated Lab 1 notebook and relevant Python files with solutions.
Do gradients become out of date when network weights are updated?
Gradients are evaluated at the parameter values used for the forward and backward passes. Updating those parameters generally changes the gradients that would be evaluated at the new parameter values.
In a standard training step, all gradients are computed before the parameter update is applied. Backpropagation computes these gradients efficiently, working backwards through the network. This does not mean that each layer’s parameters are updated as soon as its gradients are computed.
Related topics for further reading:
- Internal covariate shift, discussed in Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
- The vanishing-gradient problem and the related exploding-gradient problem.
These topics are covered later in the course.