acceptodds
Under review as a conference paper at ICLR 2027

Deriving Scaling Laws for LLM Behavior via Rational Use of Representational Resources

Abstract

Scaling laws forecast language model loss from parameter count and dataset size, but they underdetermine behavior (language models at a given loss can behave arbitrarily differently), and thus cannot model why, or when, behaviors emerge in models. We propose, instead, to rationalize language models not as suboptimal solutions to an objective specified by the data, but as the optimal solutions to an objective jointly defined by the data and the resources available (parameter count, dataset size, etc.), effectively as zero-loss models of a distribution compressed by finite resources. By doing so we connect this approach to the theory of lossy compression. We design a general method to solve for a data-generating process' optimal solutions and show that, in synthetic language modeling tasks, models are generally optimal to within 0.02 nats/token. While these optima are intractable to compute exactly for large corpora, we present a scalable approximate method and analyze in-the-wild language model scientific knowledge as optimal with respect to compressed research taxonomies. Across scales, we are able to predict which facts models acquire, and, for unacquired facts, we predict the evolution of hallucinations with scale, providing empirical support for the theory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scaling laws, pretraining, information bottleneck, rate-distortion theory, emergent abilities, memorization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PARhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper views LLMs under the lens of optimal lossy compression of the data-generating process under a budget. The authors construct a hierarchical rate-distortion model and claim to derive three predictions: power-law loss, abrupt task acquisition, and hallucination. They test this on (1) small transformers trained on a synthetic data, where the optimum is computable, and (2) factual recall in open models, where they fit a temperature that predicts which facts get acquired.

Critical strengths
  • The authors present a sensible framework to tie scaling, emergence and hallucination to one objective and make per-task predictions.
  • Experiment 1 is nicely designed since the optimum is actually computable, and the result that networks conflate the same tasks the optimum does is interesting.
Critical weaknesses
  1. There isn't much "derivation" of the scaling laws. From what I understand, the theory only predicts loss as a function of rate R. The dependence on N -- R in Exp. 1 is fitted from the networks, with no capacity theory connecting N to R. Proposition 1 only covers a single query, and for a single query a network has no capacity constraint to begin with. The constraint only shows up when many queries have to share the same parameters, which is not covered by the proof. So the claim that limited capacity acts like a KL budget is assumed. Also, dataset size and compute are listed as resources in the abstract but are never used or specified.

  2. Evaluation issues:

2(a) Section 4 describes three control libraries (Flat, Order 4, Twins) and held-out tasks, but only Twins appears in the results. Flat and the held-out tasks, which would test Prediction 3 are not reported.

2(b) The description term KL(qk∣∣π)KL(q_k || \pi) is what distinguishes the model from the information bottleneck, but the authors do not carry out any ablation.

2(c) In Exp. 2, models close only 29–54% of the gap between a need-blind baseline and the frontier, i.e., in most families they are closer to random acquisition than to the optimum. This is not the 'optimal' behavior claimed in the abstract.

  1. Missing related work: The router–prototype hierarchy with KL costs is similar to Genewein et al. [1]. Rate-distortion explanations of hallucination are an active line of research [2]. It would be helpful if the authors explain their novel contributions over [1] + Michaud et al. (2023).

References:

[1] Genewein et al. (2015). Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle.

[2] Guo and Li (2026). Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing.

Questions for authors

N/A

Reviewer_RUpWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats a trained LM as the solution to a budgeted rate-distortion problem. It minimizes KL(P⋆∥Q)\mathrm{KL}(P^\star\|Q) subject to a budget KL(Q∥π)≤R\mathrm{KL}(Q\|\pi)\le R above a fixed default distribution π\pi. They extend this to task collections using a hierarchical model where tasks route to shared prototypes. From there, they derive a power law for the loss-rate frontier when task frequencies are zipfian. The theory also predicts plateaus, acquisition ordered by loss removed per nat, and abrupt transitions (prediction 2). There is also "lawful hallucination," where tasks outside the budget get answered with the prototype of their cell (the group of tasks that prototype serves).

Experiment 1 trains small transformers on synthetic Markov tasks arranged in a zipfian tree. They read each networks rate by clustering its output tables with the same solver used to trace the frontier, then compare loss, groupings, and acquisition order against the theory. Experiment 2 looks at factual recall in a larger number of open base models (70M to 33B) on four question families. The "need" for each question comes from citation counts or Wikipedia sitelinks, scaled by per-property factors that aren't part of the fit. They then fit a 4-parameter logistic forecast.

The headline results are networks reported as "generally optimal to within 0.02 nats/token," a log-scale correlation of 0.59 between forecast and observed acquisition sizes, and for founding years, class-and-country cells acquired before their median constituent fact in 16 of the 17 that cross one half.

Critical strengths
  1. The paper gives a correct core theory (Section 3, Appendices A-E), and an optimum that a solver can approximate. Charging each prototype its full description cost yields a separation bound that appears to be new. With the enumerable library in Experiment 1, it is possible to compare a trained network against this optimum on a task-by-task basis.

  2. The most convincing evidence I found is the held-out prediction shown in figure 2 (the large open markers). The rate fitted on other model sizes maps N to R(N), producing a predicted loss that aligns with the measured loss at most held-out sizes.

Critical weaknesses
  1. The optimality claims in Experiment 1 and the Figure 4 caption exceed what the evidence supports. A network's rate is read using the same solver that traces the frontier, specifically as the cheapest grouping of its own tables within 2% of its distortion. This reading method favors the network, so being close to the frontier does not carry much weight on its own. The relative excess over the frontier still grows with size (sec 4.1). No need-blind or wrong-order learner is run through the readout to demonstrate what a non-optimal network looks like. Also reading from Figure 2D, the tree-aware baseline comes within 0.02 nats/token of the frontier at larger rates but dips below it at low rates. The library also ties loss removed per nat to need, meaning the acquisition-order results cannot distinguish between the theory's ordering rule and simple frequency order. In experiment 2, the models close only a median of 29 - 54% of the gap between a need blind store and the frontier, which is difficult to reconcile with the caption for Figure 4 ("Models approach the frontier").

  2. The factual forecast adds little value over a plain popularity regression. Equation 7 is fit and evaluated on the same models and questions. When rearranged, it becomes a pooled logistic regression on log need and log size with a guessing floor, which a plain popularity regression would also match. The theory commits to two constraints that a free regression could relax: one temperature per model shared across families, and content cancelling out (eq 6). The terciles contradict the second constraint, as the acquired fraction at w=λw=\lambda rises from 0.51 all the way up to 0.70 for cheap to expensive facts rather than staying flat (Q4). The first constraint is strained by Figure 5B , where the area to graph temperature ratio is above one for OLMo and below one for Qwen (read off the plot), suggesting a suite effect that the theory does not model. Q1, Q2 and Q3 are what would change my assessment.

Questions for authors
  1. How far are the networks from the frontier, and is the plotted frontier actually the optimum? Providing the excess over the frontier, both absolute and relative, as a function of size with seed confidence intervals would clarify whether "optimal" requires qualification. Additionally, since the solver starts from one cell per task with no restarts reported, performing restarts or annealing down from a single shared cell, or conducting an exact check on a small library, would show whether the curve represents the optimum it is being compared against.

  2. I cannot find results for the Flat and Order 4 control libraries. Are the held-out tasks those plotted in Figure 2E? Could you report results for both control libraries and the held-out tasks, along with grouping agreement for a non-optimal baseline and a seed-to-seed ceiling?

  3. How were the per-property factors chosen, and what happens if they are set to one?

  4. Can the content effect in Section 5.3 be consistent with the theory? Equation 6 charges a fact against its cell's prototype, while equation 3 charges every prototype against one global default. Under equation 3, if the default is near uniform over each family's answers, expensive facts would be acquired at lower need, which matches the ordering reported in Section 5.3. What is the default used in experiment 2, and does the version based on equation 3 fit the tercile curves better?

  5. What does the Zipf result add beyond Michaud et al. (2023), and how does it relate to Cabannes et al. (2024), which shows that the same exponent is minimax optimal for associative memories but is not cited in this paper?

Reviewer_5aanOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models LM behavior as an optimal approximation under an information budget. They do this by penalizing divergence from a default distribution and extending the construction to task routing and shared prototypes. Their validation uses synthetic hierarchical Markov tasks and factual recall across 26 pretrained models and 97 checkpoints.

Critical strengths

The math framework connects several behaviors quite coherently (prototype sharing, acquisition order, and structured errors all arise within one explicit optimization problem). The sythetic experiments also test more than aggregate loss.

Critical weaknesses

The papaer didn't established the link between the chosen information budget and neural capacity; it was assumed in Proposition 1. And given that the synthetic buedget is then inferred from model outputs, frontier proximity needs an independent validation of this choice of measurement.

The factual-recall experiment also didn't clearly establish predictive validation of the proposed optimum. This seems to imply that the syntetic comparison is tautological.

Questions for authors

See the weaknesses above and explain the gap I currently see between your proposed explanation and the strength of your empirical validation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.