STRATA
Skip to study
XLVII

IV.16 / AI

Efficient learning and inference

What happens to accuracy, memory use and response time when a model is quantized?

A source-led introduction1 source references0 related experiments

Start with the question.

Efficient inference and learning involve concrete tradeoffs among memory, latency, energy, accuracy, and engineering complexity. GPTQ offers one studied example of post-training weight quantization. The relevant question is whether a change improves the intended workload on the actual hardware, including loading and transfer costs, while preserving the behavior that matters for that workload.

Estimate storage before measuring speed.

A billion weights stored at 16 bits require about two billion bytes for the weights alone. An idealized four-bit representation reduces that component to about half a billion bytes, before scales, metadata, and runtime state. Compare this estimate with measured process memory and latency; a storage reduction need not translate proportionally into faster execution.

What would you inspect?

  • Weight precision, auxiliary metadata, peak memory, batch size, and hardware.
  • Quality changes on representative tasks and latency across input lengths.

Take the next step.

Benchmark one efficiency change against an unchanged baseline with fixed prompts, hardware, and quality criteria, then report the full resource tradeoff.

Read the supporting material.

  1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar and coauthors · Post-training quantization as a memory and computation technique with evaluation tradeoffs.

Continue through the versioned study records.

Follow the comparisons.

Read the neighboring questions.