STRATA
Skip to study
XLVII

IV.06 / AI

Backpropagation and neural learning

How does an error at the output determine the adjustment to a weight several layers earlier?

A source-led introduction1 source references0 related experiments

Start with the question.

Backpropagation computes how a network's loss changes with its parameters by applying the chain rule through its operations. Rumelhart, Hinton, and Williams's 1986 paper gives a historical example of learning internal representations with this procedure. The practical investigation separates a correct gradient calculation from the larger questions of optimization, generalization, and what the learned representation captures.

Check one gradient update by hand.

Take the illustrative loss L(w) = (w − 3)². At w = 0, its derivative is −6. With learning rate 0.1, gradient descent updates w to 0.6; the next update gives 1.08. Compare the loss after each step, then increase the learning rate enough to observe why a valid gradient can still produce unstable optimization.

What would you inspect?

  • Gradient values, learning rate, loss trajectory, and numerical precision.
  • Training loss compared with performance on examples excluded from training.

Take the next step.

Verify the derivative of a tiny composed model numerically, then change one optimization setting and record the resulting behavior.

Read the supporting material.

  1. Learning representations by back-propagating errors

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · A neural-network learning procedure and hidden-representation experiments.

Follow the comparisons.

Read the neighboring questions.