In one sentence: every hop in a chain of reasoning is one more chance for a quantization error to slip in, so deeper reasoning needs more bits. The paper tries to derive a formula for how many.
Problem:
Quantization makes an LLM smaller by storing each number in it with fewer bits, for example 3 instead of 16. A 3-bit model can often still answer a one-step question like "Who directed Inception?". It fails more often on a question that chains two facts, like "Who is the spouse of the director of Inception?". The paper asks why, and how many more bits longer chains need.
Running example: think of the model's knowledge as a list of N people in which each person points to exactly one other person and each person is pointed to by exactly one, such as Alice → Bob, Bob → Carol, Carol → Dave. A T-hop question says "start at Alice and follow T arrows". Say the 3-bit model gets 10% of the arrows wrong. One hop is then right 90% of the time. Five hops need five right arrows in a row, which happens only about 0.9 × 0.9 × 0.9 × 0.9 × 0.9 ≈ 59% of the time (assuming each arrow is wrong independently of the others).
Goal:
In the arrow example, the question is: how many bits per arrow, on average, are needed to store the list so that a T-hop question is still answered correctly at least half the time? The paper wants one formula that gives this minimum for any number of hops T and list size N. The formula should hold for any compression method, including common quantizers such as GPTQ, as long as two conditions hold: (1) which arrows go wrong does not depend on what the list says, and (2) the number of wrong arrows is fixed in advance rather than random. The paper then wants to carry this per-arrow formula over to the bits per weight of real LLMs, and to check that they behave as predicted.
Method:
Theory. Each extra hop is one more chance to hit a wrong arrow, so the longer the chain, the fewer wrong arrows the list can afford: to stay right at least half the time, up to 50% of arrows may be wrong for 1 hop, but only 13% for 5 hops. Storing a list with fewer mistakes takes more bits, and the paper uses a formula from information theory (rate-distortion theory) to turn "how many mistakes are allowed" into "how many bits are needed". For a list of 4096 people, this gives about 5 bits per arrow for 1-hop questions and nearly 12 for very long chains, a gap of about 2× that the paper calls the "2× separation law".
Experiments. First, the paper trains small neural networks to memorize an arrow list, quantizes them, and checks how quickly their answers break as the chain gets longer. Second, it quantizes four real LLMs to several bit-widths, gives each a 16- or 32-line arrow list in the prompt, and finds the fewest bits at which the model still answers correctly half the time, once for 1-hop and once for 5-hop questions. It then asks whether the gap between those two numbers depends on how "full" the model is. The paper measures this with η: the amount of knowledge stored, divided by the model's total storage space (number of weights × bits per weight at full precision). η = 1 means every bit is used to store knowledge; a small η means lots of spare room, which the paper argues lets the model absorb errors and shrinks the gap.
Results:
What they found. In both the small networks and the four LLMs, longer chains need more bits: the fewest bits that keep 5-hop questions right half the time is about 1.05–1.56× what 1-hop questions need.
Does it match the theory? Mostly not: the theory gets the direction right but overpredicts how many extra bits longer chains need. For 16-line lists it predicts that 5-hop questions need 2.81× the bits of 1-hop questions, yet the measured ratio is only 1.14× for Qwen3, 1.06× for Phi-4-mini, and at most 1.56× in the small networks.
Difference between this paper and the closest baseline:
Closest work: Chakrabarti, Pitassi & Alman, "Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers" (arXiv:2602.02707). It proves that, at every precision level, a one-layer transformer can solve a certain equality-checking task with p bits but not with p − 1 bits. This paper instead asks how the number of bits needed grows with the number of reasoning hops, for any compression method rather than one architecture.