- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when one validation loss can detect a capability. It splits loss by token role and defines a capability’s loss signature as token share × per-token loss reduction, plus a leakage term for changes on other roles. Evidence comes from a synthetic dual-route task (160 small runs), a 76M loss-weight intervention, loss-matched PolyPythia checkpoints, and induction on WikiText-103.
Critical strengths
-
This work is generally clearly stated, with clear motivation and experimental setting.
-
The results and findings i believe, are honestly reporting; the failed regression test (App. B) and the limitations are stated openly.
Critical weaknesses
Limited novelty. As the authors themselves note, “The identity is simple.” Measuring a capability through the loss on its own tokens is common sense and already standard practice (e.g. masking the loss to response tokens). The hard part is deciding which tokens belong to a capability, and here the paper is not convincing: its unsupervised role discovery (App. F.1) overlaps poorly with the hand-labelled role (precision 0.44–0.48).
Questions for authors
How to decide which tokens belong to a capability?
In Section 6, what happens at matched training steps and across seeds?
4 comments
Response to the Authors
Thank you for the response and for reporting the correction openly.
Novelty. My concern stands. The λ_c/f_c values you cite (0.42–0.71) match Table 8, but Appendix B states this leakage was induced by construction through low learning rates, so it does not show that compensation is a real phenomenon.
If I understand correctly, the correction withdraws the paper’s main 76M claim (ΔL = 0.0027, Λ = −0.1574), and the matched-step replacement points the other way: by Eq. (8), Λ = 0.1251 − 0.1241 ≈ 0.001, so there is no compensating leakage and the capability is plainly visible in aggregate loss (≈1.9% of operating loss). This contradicts Section 6.3, which attributed Λ to the control memorizing better.
Thank you. You are right on both counts, and I would rather say so than defend either.
The λ_c/f_c numbers. Withdrawn. These are from a sweep where the failure tail was enriched with low learning rates. So the leakage in that population is induced by how the population was assembled, exactly as Appendix B says. It was wrong to point to them as evidence that compensation is a real thing; that sentence has been deleted from the revised version and from my answer.
The contradiction. Also correct, and worse than a presentational inconsistency.At matched steps and the capability term is , so : there is no compensation and the capability is visible at of operating loss. Neither does the attribution of Λ to the control memorizing better survive the decomposition. For the matched pair: capability +0.1241 per role, memorized successors +0.0074 per role, first occurrences +0.0052 per role, background −0.0117. 6 % of the gap is due to memorization, but not the leakage. At the step-mismatched pair the leakage is , which is due to background drift () and not to the memorization route (). We had inferred a memorization trade from the probe accuracies and did not check the decomposition; we remove this inference.
What this does to the paper. The capability is clearly demonstrated by the loss at f_c = 5.9 %, which is what the framework predicts for a large share. The experiment is a positive control for the conditional claim, not a proof of hiddenness. The revision now makes it out to be the latter. The real-corpus case is the case where the compensation is real. At f_c = 1.06 × 10⁻², a 46-point capability gain moved aggregate loss by +0.0139 nats against +0.0050 for an insertion control with a signature of 0.0542 and Λ = −0.0403. That compensation is diffuse, an interference across the rest of the distribution, not a trade against a named competing mechanism. We no longer make that latter claim anywhere.
Novelty. The claim is not that compensation is a mechanism anymore. It is three things: a decision rule comparing the signature of a capability to the run-to-run spread of the loss, which says before a run whether a loss comparison can resolve that capability at all; a demonstration on a real corpus that aggregate loss can rank two models the wrong way round while the role-resolved loss moves by five nats; and the sample size at which role-resolved evaluation succeeds where aggregate loss does not. If that is too thin for ICLR. I take the suggestion and I would rather ship the narrower claim than the withdrawn one.
Correction
I stated the compensating term in the real-capability experiment in my previous comment as -0.0403 nats, or 74% of the signature. This was calculated with ΔL taken as present−absent. In the paper’s convention it is −0.0681 nats, 126% of the signature: the compensation is larger than the capability, not a fraction of it. The capability’s own contribution remains 0.0542 nats, and the arm that received it is 0.0139 nats worse on the aggregate loss.
So the contrast is sharper than either of us put it. At a 5.9 % share (the 76M pair) Λ = 0.0010 and the capability is plainly visible, exactly as you said. At a 1.06 % share the compensation exceeds the capability and the sign reverses. The 0.0027/0.1574 pair stays withdrawn, as does the attribution of Λ to the memorization route as the decomposition shows that route contributing 0.0074 of a 0.1251-nat gap.
Thank you. Your novelty question is the sharpest in the set, so we answer it directly rather than by assertion.
Novelty relative to response-token masking. Masking changes the training objective on a span you have already chosen. The signature measures the evaluation effect of a capability you may not have localized, and provides three things masking does not: an upper bound computable before a run (token share × marginal entropy), the compensation term on the other roles, and the sample size at which the effect becomes resolvable. The compensation is not trivial:
λ_c/f_c= 0.42, 0.46, 0.58, 0.71 over four variants. This explains why a masked and an unmasked model at the same aggregate loss can differ in a capability neither was trained on.Q1 : what tokens belong to a capability. A three-tier protocol: mechanical roles with exact
f_c; ablation-defined roles via context ablation (precision 0.44-0.48, recall 0.30-0.34, ~2x lift i.e. a screening tool, not a substitute for a stated hypothesis); and capabilities with no token locus, which we now declare out of scope rather than stretch the framework to cover.Q2 : matched steps and seeds. Now measured. At step 650 with seed, data order, corpus and budget fixed and only the role weight changed (4.0 vs 0.0): in-context accuracy 0.793 vs 0.008, memorization 0.383 vs 0.639 (the control memorizes better), and the aggregate loss differs by 0.1251 nats, the capability term predicts 0.1241 -- 99.2 %. A second matched-step comparison at 500 steps (weights 1 vs 16) costs 71.9 points of memorization for 0.056 nats. The other seed pairs are being worked on.
We should also flag a correction to Section 6, to which your Q2 refers: on a single fixed validation set the aggregate difference is −0.0235 nats and not 0.0027, and with the sign favoring the intervened model and the cancelation at an iso-loss point is an identity, not evidence. The revision replaces that comparison with the above matched-step one.