STRATA
Skip to study
XLVII

IV.10 / AI

Transformers and attention

What information can a token receive through an attention head, and at what computational cost?

A source-led introduction1 source references0 related experiments

Start with the question.

The Transformer organizes sequence processing around attention and associated feed-forward operations. Vaswani and colleagues' original paper supplies the architecture and evaluation context. A useful visual explanation follows how token representations become queries, keys, and values, then shows the weighted combination produced by attention. The weights describe an internal computation; explaining a final prediction requires additional analysis.

Calculate a tiny attention mixture.

Use two value vectors, (1,0) and (0,1), with attention weights 0.8 and 0.2. Their weighted output is (0.8,0.2). Change the weights while keeping the values fixed, then change the values while keeping the weights fixed. This separates what is selected from what is carried by the selected representations.

What would you inspect?

  • Attention weights, value vectors, masking rules, and sequence positions.
  • Output changes under controlled edits to tokens or internal representations.

Take the next step.

Trace a two-token attention calculation end to end, including normalization and masking, before comparing its behavior with a full model's output.

Read the supporting material.

  1. Attention Is All You Need

    Ashish Vaswani and coauthors · The Transformer architecture and original sequence-transduction experiments.

Continue through the versioned study records.