STRATA
Skip to study
XLVII

XLVII / Experiments

Workbench

Change a condition, inspect what happens, and keep the result. Six experiments connect an equation with a visible outcome, a numerical record, and questions you can pursue.

Entropy and compression

Which pattern makes a message cheaper to encode?

A message can repeat individual symbols or repeat whole stretches of the same symbol. Those patterns reward different codes. Generate a four-symbol message, then compare a fixed two-bit alphabet, a Huffman code, and a simple run-length format on exactly the same sequence.

The message in its original order

ABCD
Read each row from left to right, then continue on the next row. Colors identify symbols; consecutive blocks reveal ordering that the histogram alone does not show.

Payload bits for the same message

Fixed alphabet1024 bitsHuffman885 bitsRun length3360 bits
Every bar encodes all 512 symbols. The fixed alphabet uses two bits per symbol; each run-length packet uses ten. Codebook and container overhead are excluded.
Symbol entropy
1.699 bits

Per symbol, from the observed histogram.

Huffman payload
1.729 bits

Per symbol; the codebook is excluded.

Run-length payload
6.563 bits

Per symbol; each packet costs 10 bits.

Consecutive runs
336

336 packets cover 512 symbols.

The codebook is built from the observed message. The sequence CSV contains every generated symbol in order.
SymbolCountObserved shareHuffman code
A28355.3%1
B8516.6%00
C7314.3%011
D7113.9%010

Keep a reproducible record.

The JSON includes parameters, model assumptions, sources, and full results. The CSV contains one row per symbol, with its zero-based position.

Method

The calculation follows these rules.

The histogram gives an order-zero Shannon entropy. Huffman coding repeatedly joins the two least frequent branches to build a prefix code. The run-length format stores a two-bit symbol and an eight-bit count; counts run from 1 to 255, and a longer run uses another packet.

H = −Σ pᵢ log₂ pᵢ; Huffman bits = Σ countᵢ × code lengthᵢ; run-length bits = 10 × packets.

Read the result within its assumptions.

  • All payload counts exclude headers, the Huffman codebook, byte padding, and file-container overhead. They are coding comparisons, not sizes of downloadable compressed files.
  • The entropy shown uses individual-symbol frequencies. When neighboring symbols depend on one another, it does not measure the full source entropy rate.
  • An alphabet containing only one observed symbol uses the practical one-bit code 0. Symbols that never occur receive no code.

Take the experiment further.

01

Change the histogram.

Keep repetition at zero and move the probability of A from 25% toward 95%. Record entropy and Huffman bits per symbol at three settings.

02

Change the ordering.

Keep new draws evenly weighted and raise repetition. Explain why a symbol-frequency code and a run-length code respond differently.

03

Count the overhead.

For a 64-symbol message, estimate how many extra codebook bits would erase a small payload saving. State which storage costs your estimate includes.

Read the sources behind the model.

  1. Claude Shannon, A Mathematical Theory of Communication (1948)

    The original paper defines the information measure behind the symbol-entropy calculation.

  2. MIT 18.200, Huffman Codes (2024)

    Lecture notes derive prefix codes, the greedy tree construction, and expected code length.

Keep a direct link to an experiment.