STRATA
Skip to study
XLVII

I.06 / GNOSIS

Shannon information and communication

What did a communication channel have to preserve for a message to survive noise? Follow that engineering question into Shannon’s definition of information.

A source-led introduction1 source references1 related experiments

Start with the question.

Shannon's communication theory makes uncertainty in a source and the capacity of a channel mathematically tractable. Its objects are symbol probabilities, codes, and transmission conditions. This gives the site a precise way to compare predictable and unpredictable sequences. The practical question is how much uncertainty remains before a symbol arrives, and how a coding system turns that structure into reliable communication.

Calculate uncertainty for two binary sources.

A fair binary source has entropy of one bit per symbol. If one outcome occurs with probability 0.9 and the other with 0.1, the binary entropy is about 0.469 bits. Generate short illustrative sequences with these settings, compare their symbol counts, and examine why a short sample may differ noticeably from the generating probabilities.

What would you inspect?

  • Estimated symbol frequencies and the sample length used to estimate them.
  • Average code length compared with the source model's entropy.

Take the next step.

Use the entropy workbench to vary one probability, then record how code length and prediction error respond under the same source assumptions.

Put part of the question to work.

These models isolate a mechanism you can vary. Their assumptions define the part of this topic they can illuminate.

Try entropy and compression.

Change one parameter, compare the numerical output, and export the result. The experiment loads when you open it.

Open its full page

Watch the quantities change.

12 seconds · Original model visualization · Silent

Entropy and compression

Compare symbol frequencies with the ordering that a code can exploit.

Read the explanation and method

The first sweep changes the probability of A while drawing each symbol independently. The other three symbols share the remaining probability.

As one symbol dominates, the observed symbol entropy falls. Huffman codes use short words for frequent symbols; the comparison excludes headers and the codebook.

The second sweep gives all four new draws equal probability and increases repetition. Long runs reward run-length coding, even when a symbol histogram cannot describe the dependence.

Method. The histogram gives an order-zero Shannon entropy. Huffman coding repeatedly joins the two least frequent branches to build a prefix code. The run-length format stores a two-bit symbol and an eight-bit count; counts run from 1 to 255, and a longer run uses another packet.

Read the supporting material.

  1. A Mathematical Theory of Communication

    Nokia Bell Labs · Communication entropy, coding, and noise; does not turn statistical information into a theory of semantic truth.

Continue through the versioned study records.

Follow the comparisons.

B03

Entropy and information

What is being counted in each definition of entropy? Follow the change from messages to physical states and identify the assumptions behind each expression.

Read the neighboring questions.