acceptodds
Under review as a conference paper at ICLR 2027

Capability–Loss Dissociation in Language Models: A Theory and Empirical Study of Loss Signatures

Abstract

The standard measure of language-model progress is validation loss, but the ability of the validation loss to detect a capability depends on how often the capability is exercised in the evaluation corpus. This dependence is formalized by the loss signature, which is the product of a capability's token share and its per-token information gain. The signature quantifies the change in aggregate validation loss from acquiring a capability, and provides a criterion for when loss is an appropriate measurement tool. We assess this account on a synthetic dual-route language modeling task that separates parametric memorization from in-context retrieval. The predicted signature follows the observed loss gap across 160 controlled training runs with variants independently varying token share and information content. Even models with near identical validation loss can differ hugely in in-context accuracy, including a loss-matched pair with an 188x difference in ability. A gradient-level intervention with 76M parameters reproduces the dissociation, yielding a 69.5-point in-context accuracy gap between models evaluated at nearly identical aggregate loss (ΔL = 0.0027 nats). Direct measurement of the leakage term shows the capability’s own signature (fcΔLc = 0.1601 nats, 2.5% of operating loss) is almost exactly offset by a compensating gain on the competing memorization route (Λ = −0.1574 nats), leaving aggregate loss essentially unchanged. An ablation over five intervention weights shows this trade is smooth and monotone. On WikiText-103, induction is clearly visible in validation loss because its token share is large, illustrating that the framework is conditional rather than a blanket criticism of loss. We further derive a sample-size relationship showing that role-resolved evaluation can scale more favourably than aggregate loss for rare capabilities. Our results establish a practical principle: before using validation loss to measure a capability, estimate its token share and information content. When the resulting loss signature is small, or when a competing route can absorb it, capability-specific evaluation or role-resolved loss is required.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language models, validation loss, capability evaluation, loss signatures, capability–loss dissociation, in-context learning, parametric memorization, language model evaluation, aggregate loss, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_pavEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when one validation loss can detect a capability. It splits loss by token role and defines a capability’s loss signature as token share × per-token loss reduction, plus a leakage term for changes on other roles. Evidence comes from a synthetic dual-route task (160 small runs), a 76M loss-weight intervention, loss-matched PolyPythia checkpoints, and induction on WikiText-103.

Critical strengths
  • This work is generally clearly stated, with clear motivation and experimental setting.

  • The results and findings i believe, are honestly reporting; the failed regression test (App. B) and the limitations are stated openly.

Critical weaknesses

Limited novelty. As the authors themselves note, “The identity is simple.” Measuring a capability through the loss on its own tokens is common sense and already standard practice (e.g. masking the loss to response tokens). The hard part is deciding which tokens belong to a capability, and here the paper is not convincing: its unsupervised role discovery (App. F.1) overlaps poorly with the hand-labelled role (precision 0.44–0.48).

Questions for authors

How to decide which tokens belong to a capability?

In Section 6, what happens at matched training steps and across seeds?

4 comments

Thank you. Your novelty question is the sharpest in the set, so we answer it directly rather than by assertion.

Novelty relative to response-token masking. Masking changes the training objective on a span you have already chosen. The signature measures the evaluation effect of a capability you may not have localized, and provides three things masking does not: an upper bound computable before a run (token share × marginal entropy), the compensation term on the other roles, and the sample size at which the effect becomes resolvable. The compensation is not trivial: λ_c/f_c = 0.42, 0.46, 0.58, 0.71 over four variants. This explains why a masked and an unmasked model at the same aggregate loss can differ in a capability neither was trained on.

Q1 : what tokens belong to a capability. A three-tier protocol: mechanical roles with exact f_c; ablation-defined roles via context ablation (precision 0.44-0.48, recall 0.30-0.34, ~2x lift i.e. a screening tool, not a substitute for a stated hypothesis); and capabilities with no token locus, which we now declare out of scope rather than stretch the framework to cover.

Q2 : matched steps and seeds. Now measured. At step 650 with seed, data order, corpus and budget fixed and only the role weight changed (4.0 vs 0.0): in-context accuracy 0.793 vs 0.008, memorization 0.383 vs 0.639 (the control memorizes better), and the aggregate loss differs by 0.1251 nats, the capability term predicts 0.1241 -- 99.2 %. A second matched-step comparison at 500 steps (weights 1 vs 16) costs 71.9 points of memorization for 0.056 nats. The other seed pairs are being worked on.

We should also flag a correction to Section 6, to which your Q2 refers: on a single fixed validation set the aggregate difference is −0.0235 nats and not 0.0027, and with the sign favoring the intervened model and the cancelation at an iso-loss point is an identity, not evidence. The revision replaces that comparison with the above matched-step one.

Reviewer_pavEOpenReview ↗

Response to the Authors

Thank you for the response and for reporting the correction openly.

Novelty. My concern stands. The λ_c/f_c values you cite (0.42–0.71) match Table 8, but Appendix B states this leakage was induced by construction through low learning rates, so it does not show that compensation is a real phenomenon.

If I understand correctly, the correction withdraws the paper’s main 76M claim (ΔL = 0.0027, Λ = −0.1574), and the matched-step replacement points the other way: by Eq. (8), Λ = 0.1251 − 0.1241 ≈ 0.001, so there is no compensating leakage and the capability is plainly visible in aggregate loss (≈1.9% of operating loss). This contradicts Section 6.3, which attributed Λ to the control memorizing better.

Thank you. You are right on both counts, and I would rather say so than defend either.

The λ_c/f_c numbers. Withdrawn. These are from a sweep where the failure tail was enriched with low learning rates. So the leakage in that population is induced by how the population was assembled, exactly as Appendix B says. It was wrong to point to them as evidence that compensation is a real thing; that sentence has been deleted from the revised version and from my answer.

The contradiction. Also correct, and worse than a presentational inconsistency.At matched steps ΔL=0.1251\Delta L=0.1251 and the capability term is 0.12410.1241, so Λ=0.0010\Lambda=0.0010: there is no compensation and the capability is visible at 1.9%1.9 \% of operating loss. Neither does the attribution of Λ to the control memorizing better survive the decomposition. For the matched pair: capability +0.1241 per role, memorized successors +0.0074 per role, first occurrences +0.0052 per role, background −0.0117. 6 % of the gap is due to memorization, but not the leakage. At the step-mismatched pair the leakage is ≈−0.024\approx -0.024, which is due to background drift (−0.0187-0.0187) and not to the memorization route (−0.0014-0.0014). We had inferred a memorization trade from the probe accuracies and did not check the decomposition; we remove this inference.

What this does to the paper. The capability is clearly demonstrated by the loss at f_c = 5.9 %, which is what the framework predicts for a large share. The experiment is a positive control for the conditional claim, not a proof of hiddenness. The revision now makes it out to be the latter. The real-corpus case is the case where the compensation is real. At f_c = 1.06 × 10⁻², a 46-point capability gain moved aggregate loss by +0.0139 nats against +0.0050 for an insertion control with a signature of 0.0542 and Λ = −0.0403. That compensation is diffuse, an interference across the rest of the distribution, not a trade against a named competing mechanism. We no longer make that latter claim anywhere.

Novelty. The claim is not that compensation is a mechanism anymore. It is three things: a decision rule comparing the signature of a capability to the run-to-run spread of the loss, which says before a run whether a loss comparison can resolve that capability at all; a demonstration on a real corpus that aggregate loss can rank two models the wrong way round while the role-resolved loss moves by five nats; and the sample size at which role-resolved evaluation succeeds where aggregate loss does not. If that is too thin for ICLR. I take the suggestion and I would rather ship the narrower claim than the withdrawn one.

Correction

I stated the compensating term in the real-capability experiment in my previous comment as -0.0403 nats, or 74% of the signature. This was calculated with ΔL taken as present−absent. In the paper’s convention it is −0.0681 nats, 126% of the signature: the compensation is larger than the capability, not a fraction of it. The capability’s own contribution remains 0.0542 nats, and the arm that received it is 0.0139 nats worse on the aggregate loss.

So the contrast is sharper than either of us put it. At a 5.9 % share (the 76M pair) Λ = 0.0010 and the capability is plainly visible, exactly as you said. At a 1.06 % share the compensation exceeds the capability and the sign reverses. The 0.0027/0.1574 pair stays withdrawn, as does the attribution of Λ to the memorization route as the decomposition shows that route contributing 0.0074 of a 0.1251-nat gap.

Reviewer_o8twOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalises when validation loss can detect a capability. Once scored tokens are partitioned into roles, cross-entropy is exactly a token-share-weighted average of per-role losses, so a capability confined to role cc moves aggregate loss by exactly Δ=fc(Lc0−Lc⋆)\Delta = f_c(L^0_c - L^\star_c), its token share times its per-token information gain. The author calls this the loss signature, and the point is that when this product is small, or when a competing route absorbs it, near-identical validation loss can coexist with very different capability.

Contributions. The main contribution is the loss-signature identity: once tokens are grouped into roles, a capability's effect on validation loss is exactly its token share times its per-token information gain, which tells you in advance how much loss it can possibly move. On top of this they introduce the idea of leakage, where a competing route absorbs a capability's loss so that two models can match on loss yet differ in ability. They support the framework with a synthetic two-route sandbox, a causal intervention, a practical recommendation (report role-resolved loss), and a sample-size argument that role-resolved evaluation needs far fewer tokens than aggregate loss for rare capabilities.

Validation. Most of the evidence is synthetic. Across 160 runs the predicted signature matches the observed loss gap to within 2%, and loss-matched models differ sharply in capability, up to 188x in one pair. A 76M-parameter intervention reproduces this causally, with a 69.5-point capability gap at near-identical loss offset almost exactly by a compensating gain on the other route. For natural text they show that induction is visible in loss because its token share is large, and a public-checkpoint analysis on LAMBADA shows a weaker, correlational version of the effect.

Critical strengths
  • The loss-signature identity is a clean, exact, and genuinely useful result. Decomposing loss by token role gives, before training, the most a capability can move aggregate loss (fcf_c times its information gain), turning "loss may hide capability" into something estimable in advance. The less obvious and valuable part is leakage, that a competing route can absorb a capability's loss so matched loss is stable rather than coincidental, and it is shown causally: a 69.5-point gap at ΔL=0.0027\Delta L=0.0027 nats with an almost exactly offsetting Λ=−0.1574\Lambda=-0.1574.
Critical weaknesses
  • The method's utility is bounded to capabilities that map cleanly onto tokens. Eq. 2 requires disjoint per-token roles, which holds in the sandbox by construction but not in open language, where a prediction mixes capabilities and a sub-word token need not carry one. The identity still holds; its reading as a capability signature does not, and soft roles would break the exact decomposition. The only natural language capability they actually role-decompose (induction) is itself low-level and token-separable, and even there the unsupervised role recovery for GPT-2 is weak (precision 0.476, recall 0.342); their other natural-language case (LAMBADA) is analyzed at the accuracy level, not via a token-role partition.
  • The strong dissociation is synthetic by construction, so real-world prevalence is unestablished. The sandbox is built with a small-token-share, substitutable capability and supplied competing routes, and by the paper's own identity the dissociation magnitude scales with those choices, so the 188x is what the design permits, not evidence about real models (the one natural anchor, LAMBADA, shows only ~20x and is correlational).
Questions for authors
  1. How is the per-token role assigned for a high-level natural-language capability that lacks a mechanical signature like repeated-bigram induction? The supervised rule and the unsupervised Otsu method both appear specific to token-separable, low-level capabilities.
  2. Beyond the synthetic sandbox and the LAMBADA checkpoints, can you demonstrate the dissociation on a real but controllable capability such as arithmetic or code, where ground truth and token share are still measurable but the capability is genuine?
3 comments

Thank you for raising these concerns. We agree that both points need to be addressed in the revision. The first also has a formal explanation that we had not made sufficiently clear.

Q1 is role assignment when the capability is not token-separable. Three tiers, and the revision states which tier each experiment occupies.

  1. Mechanical roles, These are roles for which a supervisor is available, such as repeated n-grams, answer spans in arithmetic or code, tool-call arguments, and cited spans. f_c exact, decomposition exact. This is what Sections V–VI do.
  2. Ablation-defined roles: tokens whose NLL rises when long-range context is destroyed. Your point here is right and we had not stated it precisely enough: Eq. 2 is exact for any weighting of the tokens, not only for hard partitions. For weights w_t ∈ [0,1], L = Σ_r π_r L_r with π_r = E[w_r] and L_r the weighted mean over role r, so soft roles preserve exactness. When a token supports multiple capabilities, those individual contributions cannot be uniquely attributed, although their combined contribution remains identifiable. The resulting signature therefore provides an upper bound on each capability's individual contribution. This is the conservative direction for our claim, since a smaller actual contribution would be harder for aggregate loss to detect.
  3. Capabilities with no token locus : Examples include multi-step derivation, refusal style, and discourse stance. These are outside the scope of the current accounting framework because it requires a defined (f_c). We will state this limitation explicitly in the revision.

Our unsupervised procedure recovers a role with approximately (2\times) lift over the base rate, with precision between 0.44 and 0.48 and recall between 0.30 and 0.34. Its signature accounts for 18.8–22.4% of the loss. We will describe this as a "high long-range dependency" role rather than claiming that the procedure has independently rediscovered induction. That description is more consistent with what the results support.

W2 : synthetic by construction. Agreed, and the honest measure is the real-checkpoint analogue: 2.66 points of LAMBADA accuracy across 33 loss-matched pairs, a 20× relative amplification rather than 188×. We also report an intervention in a 76.8M-parameter model, with training steps matched across conditions. Reweighting one role changes in-context accuracy by 78.5 percentage points and memorization by 25.6 percentage points in the opposite direction. The signature predicts 99.2% of the 0.1251-nat loss gap.

Q2 : a real but controllable capability. This is the experiment we are running now, and we agree it is the difference between a demonstration and a result. Design: GPT-2 fine-tuned on WikiText-103, with a factual-recall capability whose token share is set by construction and counted from the tokenised corpus before training. Three arms at f_c = 0, 8×10⁻⁴ and 8×10⁻³ give predicted signatures of 0, 0.004 and 0.040 nats against a measured run-to-run spread of 0.02–0.12 nats. Our prediction, registered before training, is that the middle condition will show a substantial change in capability while its aggregate loss remains largely unchanged. We also expect the transition between visible and undetectable changes to fall between the two nonzero conditions. We will share the results in this thread once the experiments are complete.

One correction we should flag, since your summary cites Section 6 which is on a single fixed validation set the aggregate difference there is −0.0235 nats, not 0.0027, and its sign favours the intervened model; the cancellation at an iso-loss point is an identity rather than evidence. Both are fixed in the revision.

Following your Q2, we ran the capability test you suggested in a form where ground truth and token share are both exact: factual recall in GPT-2 124M fine-tuned on WikiText-103, with the token share set by construction and counted from the tokenised corpus before training.

At f_c = 1.1×10⁻², exact match goes from 0 to 46.5 % and the answer-span loss falls from 7.20 to 2.06 nats. The arm that acquired the capability ends 0.0139 nats worse on aggregate validation loss as 0.4 % of the operating point against 0.0050 for a length-matched insertion control. The capability's own contribution is 0.0542 nats (1.6 % of the loss); the compensating term is −0.0681, i.e. 126 % of the signature, larger than the capability itself. At f_c = 1.2×10⁻³ the capability reaches 3 % and the aggregate stays within the seed spread of control.

The result is not the one we predicted. We expected a 0.054-nat fall and observed a 0.0139-nat rise, and we report that failure rather than the favourable half. What survives is your framing of the leakage term: on a real task the compensating degradation is larger than the capability's own contribution, so the sign of the total is set by Λ rather than by the signature. The gain also does not transfer across templates (46.5 % on the training template, 0 % on a paraphrase), so it is a learned association rather than a general faculty which we now state explicitly.

Edited: the first version of this comment quoted the compensating term as −0.045 nats, 83 % of the signature. That used the opposite sign for ΔL; the values above are the corrected ones.

Thank you for the detailed responses and for your honesty in reporting the corrections and the negative result.

Q1. This answered my question but strengthened my concern. Capabilities without a token locus, including the paper's motivating examples, are now out of scope, which reduces the significance of the framework. Even for token-level capabilities, fcf_c and the signature depend on the tokenizer, making the framework hard to control and hard to transfer across models.

Q2. The new experiment also strengthened my concern. The framework predicted a 0.0540.054-nat decrease, but loss increased by 0.01390.0139, within noise of the control (two seeds, p=0.067p = 0.067), and the gain does not transfer to a paraphrased template (46.5%46.5\% vs. 0%0\%).

My original assessment relied on the 76M leakage result, which is now withdrawn; at matched steps the capability is clearly visible in loss. The causal evidence is also limited to $\le$124M parameters. Given all the above, I will lower my score to 2.

Reviewer_bsm3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies if and when validation loss can detect a capability of a language model. They split the tokens where the loss is computed into roles and observe that the contribution of a capability to the loss is the fraction of tokens where the capability is used multiplied by how much it lowers the loss on them. They call it the "loss signature".

They claim that loss can hide capabilities that are used on just a few tokens or that can be replaced by a competing strategy. They also claim that whether this happens can be determined before training. In the experiments, they train 160 small transformer models on a synthetic task where the next token can be predicted either by memorization or by in-context learning, and show that runs with almost the same validation loss can differ a lot in in-context accuracy. They also train a 76M-parameter model where the loss on the in-context tokens is reweighted during training, and run additional analyses on GPT-2 and Pythia checkpoints.

Critical strengths

I find understanding when validation loss is a reliable measure of competence in language models a relevant question. The paper hypothesizes that the validation loss can hide a capability, and claims that this can happen for two reasons. First, because the capability is used only on a small fraction of the tokens, so not even a large improvement on those tokens can significantly move the average loss. Second, because a model that does not learn the capability of interest can compensate by getting better on other tokens, for example, by memorizing more.

They also claim that whether this happens for a given capability can be estimated before training, from how many tokens it affects and how many possible answers there are on those tokens. If the empirical evidence solidly confirmed this hypothesis in realistic settings, that would be a meaningful contribution.

Critical weaknesses

The proposed framework is not a substantial contribution.

  • Proposition 1 only states that the validation loss is the weighted average of the losses on each group of tokens. Once the change of the loss on the other tokens (the "leakage") is added, this holds for any pair of models, so the experiments cannot confirm or reject it. Propositions 2 and 3 are the standard formulas for a regression slope and a sample size.
  • The idea that the loss weights a capability by how often it is used is not new. It is a well known fact that models with the same pretraining loss can differ in downstream performance.
  • The claim that this can be determined before training holds only in the synthetic task. There, how much the capability lowers the loss on each token is known in advance, because the authors built it so that, without the capability, the model can only guess among a fixed number of equally likely answers. On real text there is no such number, and indeed the paper obtains it by measuring the loss of trained GPT-2 models. Also, when a competing strategy is present, what hides the capability is the change of the loss on the other tokens, which the paper itself says should be measured, i.e., after training.

This is important because the framework is the main contribution of the paper, but it is not new and it can be used before training only in the synthetic task.

Anyways, the experiments show the loss hiding a capability only in settings built to do so.

  • In the synthetic task, the models that learned in-context learning almost always have a lower validation loss than the ones that did not, so in most cases the loss is enough to tell them apart. The pairs of models with the same loss but very different in-context accuracy reported in the paper are the exceptions, picked exactly because they have the same loss and the largest gap in accuracy.
  • In the 76M-parameter experiment, the authors train two models, one where the errors on the in-context tokens count four times more, and one where they do not count at all, so the second model is never trained to predict those tokens. Then they compare the two models when they reach the same validation loss, which happens at different training steps, i.e., the second model has been trained longer, and only on the tokens that do not need in-context learning, for example, the memorized ones. So it is not surprising that the second model memorizes better and the first one does better in-context learning, and this does not show that the two strategies compete when all tokens count the same, as in a normal training. Also, these results have no seeds or confidence intervals.
  • On real models, the paper's own results show that the loss does track the capability, i.e., induction is clearly visible in the loss of GPT-2 on WikiText-103, and models with lower loss also have higher LAMBADA accuracy across the Pythia checkpoints. The capabilities that the paper uses as motivation, e.g., multi-step reasoning, rare factual lookup or refusals, are never tested.

This is important because, even accepting the framework, the paper does not show that the loss hides any capability outside of artificial settings.

Questions for authors

My main concerns require new experiments and a substantial rewrite, so I do not expect a rebuttal to be enough to change my recommendation. Still, the following points could affect my assessment.

  • Beyond the well known fact that the loss weights a capability by how often it is used, what does the framework add? In particular, how can one tell before training whether the loss hides a capability on real text, where it is not known in advance how much the capability lowers the loss on each token?
  • Do you observe models with the same loss but very different in-context accuracy also when all tokens count the same during training, for example, across different seeds of the 76M-parameter model? If so, could you report it with confidence intervals over seeds?
  • I am aware this is not something that can be fully addressed in a rebuttal, but to be convinced I would need at least one real capability, used on a small fraction of the tokens, for which you determine before training that the loss should hide it, and then show that it indeed does. Could you show such a case, even at a small scale?
2 comments

Your review asked for one real capability, used on a small fraction of tokens, for which we determine before training that the loss should hide it, and then show that it does. We ran it.

We fine-tuned GPT-2 124M on WikiText-103, establishing a factual recall rate (fcf_c) of 1.1×10−21.1 \times 10^{-2} based on the tokenized corpus. This intervention drove exact match scores from 0 to 46.5% and reduced answer-span loss from 7.20 to 2.06 nats. The model arm acquiring this capability incurred an aggregate validation loss penalty of 0.0139 nats, compared to 0.0050 for a length-matched insertion control. Dissecting the loss components reveals that the capability’s direct contribution is 0.0542 nats. The compensating term stands at −0.0681 nats, representing 126% of the signature. This magnitude exceeds the capability itself, contradicting expectations that the compensation would be merely a fraction of the gain.

We explicitly acknowledge the discrepancy: the model forecasted a decline of approximately 0.054 nats, yet the empirical data showed an increase. Consequently, the point prediction fails to generalize beyond the sandbox environment, even in contexts where the theoretical bound remains valid. This result appears in the revised manuscript, which details two seeds per arm and the control group. The direction of this effect is reported as suggestive rather than definitive (p = 0.067).

Edited: the first version of this comment quoted the compensating term as −0.045 nats, 83 % of the signature. That used the opposite sign for ΔL; the values above are the corrected ones.

I thank the authors for the detailed response, and for openly reporting the corrections and the negative results. However, I believe the new results do not address my main concerns, and in some cases, for example, the 76M-parameter experiment at matched training steps, they rather confirm them. Therefore, I keep my score.

Reviewer_RH9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why aggregate validation cross-entropy loss can fail to reflect substantial changes in specific language model capabilities. The authors formalize this phenomenon by partitioning evaluation tokens into disjoint roles and decomposing the aggregate loss difference between two models into a capability-specific loss signature and a leakage term capturing loss changes across all other roles. Using a synthetic dual-route language modeling sandbox that contrasts parametric memorization with in-context retrieval across 160 training runs, the paper shows that models with nearly identical validation loss can differ dramatically in in-context accuracy. The authors further scale this setup to 76M-parameter Transformers using a gradient-reweighting intervention on the capability role, and complement the synthetic study with evaluations of induction on WikiText-103, loss-matching analyses on public PolyPythia-410M checkpoints, and sample-complexity derivations comparing aggregate versus role-resolved evaluation.

Critical strengths

S1 Clear separation of the two mechanisms behind capability and loss dissociation, offering practical diagnostic value.

The paper goes beyond the standard observation that rare tokens contribute little to aggregate loss by explicitly modeling the compensating shifts across other token roles via the leakage term. This distinction is important because even when a capability accounts for a non-trivial share of tokens, aggregate validation loss can remain flat due to improvements or regressions on other roles. Decomposing loss at the role level allows practitioners to distinguish between a model that did not change and a model that traded performance across different capabilities, providing a much richer diagnostic signal for model comparison than a single scalar mean loss.

S2 Multi-faceted empirical analyses and transparent reporting of results that nuance the main narrative.

The experimental design includes several complementary settings. Four synthetic variants independently vary token share and output space size, a 76M-parameter intervention varies gradient weights during training, and natural-text experiments on WikiText-103 show that when a capability token share is large, its gains are clearly visible in aggregate loss. Moreover, the authors candidly report cases where idealized assumptions break down, specifically that the clean-partition assumption fails in the synthetic sweep and that aggregate loss and LAMBADA capability remain strongly correlated across public PolyPythia checkpoints. This makes the argument of the paper appropriately conditional rather than a blanket dismissal of validation loss.

Critical weaknesses

W1 Gap between the idealized theoretical bound and its applicability as a prior predictor for practical trained models.

While the product of token share and the log of the output space size cleanly characterizes the loss reduction from an optimal uniform predictor to an oracle predictor under idealized conditions, applying it as a prior bound for actual trained models relies on strong assumptions that rarely hold in practice. Specifically, it requires assuming that a capability-absent model is perfectly calibrated to the uniform marginal, that a capability-present model achieves near-zero loss, and that cross-role leakage is either negligible or known before training. For general neural language models, cross-entropy is sensitive to calibration error and overconfidence, meaning the log of the output space size is not a true upper bound on the empirical loss of the target role, nor is the net aggregate shift predictable without already knowing the leakage term.

W2 The 76M causal intervention demonstrates capability and loss dissociation, but the iso-loss conditioning confounds the mechanistic trade-off claim, alongside a numerical inconsistency.

In Section 6, the 76M-parameter experiment alters the backward-pass weight on the target role and selects checkpoints from different training steps that match in aggregate validation loss, specifically step 650 for the intervened model and step 950 for the control model. While this successfully proves that different optimization trajectories can yield models with similar aggregate loss but distinct capability profiles, the subsequent interpretation of the leakage cancellation is partly tautological. By conditioning on two checkpoints selected specifically to have a near-zero total loss difference, the exact accounting identity guarantees that any large positive capability term must be accompanied by an approximately equal and opposite leakage term. Therefore, observing near-exact cancellation at iso-loss checkpoints cannot serve as independent evidence of an intrinsic mechanistic capacity trade-off between memorization and retrieval. Furthermore, there is a numerical discrepancy between the text and Table 4 regarding the exact loss difference at this operating point.

Questions for authors

Q1 Why is there a discrepancy in the reported validation loss difference between Table 4 and the main text? In Table 4, the validation losses for the control and intervened models are reported as 6.4988 and 6.4911, which yields a difference of 0.0077 nats. However, the Abstract, Section 6.2, Table 3, and Figure 3 all state that the aggregate loss difference is 0.0027 nats, which matches the sum of the reported capability term 0.1601 and leakage term minus 0.1574. Could the authors clarify which value is correct and confirm whether all reported metrics in Section 6 come from the exact same pair of checkpoints?

Q2 Is the proposed framework genuinely predictive before training, or is it primarily a post-hoc diagnostic accounting tool? The paper repeatedly emphasizes that whether validation loss can detect a capability is measurable before training or computable in advance. However, in the experiments, key quantities governing the actual observable aggregate loss gap, most notably the empirical role loss shift and the leakage term, are only measured after training the models. Could the authors provide a concrete example in a realistic setting where the framework predicts the actual aggregate validation loss change prior to training and validates it experimentally, rather than decomposing the loss gap after the fact? If the leakage term cannot be predicted in advance, should the claims be tempered to frame the method primarily as a post-hoc evaluation and auditing tool?

Q3 Is the memorization and retrieval trade-off a robust phenomenon across training, or an artifact of comparing checkpoints at different training steps under iso-loss selection? In the 76M experiment, the intervened model is evaluated at step 650, whereas the control model is evaluated at step 950. Because the two models have experienced different numbers of optimization updates, the control model simply had 300 additional steps to memorize the global triggers. Could the authors clarify whether this trade-off holds across multiple random seeds, and whether increasing the intervention weight still degrades memorization accuracy when comparing models at the exact same training step or across other loss levels? In addition, within the leakage term of minus 0.1574, what is the exact breakdown of loss changes across the remaining token roles?

Comments on the paper

1 comment
Area_Chair_GxiUOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for the detailed reviews and rebuttal.

To Reviewers RH9u and o8tw: As the author–reviewer discussion deadline approaches, could you please review the authors’ responses and indicate whether your concerns have been addressed?

Kind regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.