- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks how to compare language-model checkpoints when each responses score is paid for. It proposes contrast-aware feedback acquisition (or "CAFA'), which sums the report coefficients over identical scoring inputs and acquires only the scores whose aggregated coefficients is non-zero, recovering the fully-scored sample mean. Propositions 1 and 2 show that these supports are necessary for an unrestricted scorer. On controlled updates, the authors claim CAFA can remove a majority of cached inputs.
Critical strengths
The observation that the required supports depend on the requested statistic is a good contribution. I think quite naturally, a response can cancel from the mean yet remain necessary for the covariance (Figure 6; L078-079), and an absolute value restores the full support (Corollary 1).
Critical weaknesses
My concerns are attribution of the claimed computational savings, low originality, savings concentrated on ~no-ops, the use of a post-hoc precision threshold, and overall narrow scope.
-
The abstract (L025) and Figure 1b credit 'Independent Star + CAFA' with 46.1–51.3% lower cost than 'Independent AR + Cache-only', but the arms differ in the samplers as well. The sampler is the classical maximal coupling (L456-457). CAFA alone removes only ~3% of inputs and ~2% of cost for the three-gain report (L425; Table 13). A control for Independent Star + Cache-only is absent.
-
If I read Proposition 3 correctly, for base-relative zero-sum reports the saving over caching is at most one score per all-copy draw, and within-draw skipping equals batch aggregation in 72 of 72 cells (L1720). It seems to me that the rule amounts to not scoring a response every active policy produced, layered on Wen (2026), whose rule already gets one reward per matched component (Appendix F.5, L2854). That 'all 504 report cells' reproduce full scoring (L027) tests the implementation and says nothing of the method, in my view.
-
The controlled endpoints are 8 optimiser steps at learning rate 1e-7 (Tables 2 and 4); on SmolLM2 the inputs removed (Table 5) are about 95% of the 2,048 root draws. Removal falls from 83.9% at step 2 (gain -0.002) to 21.6% at step 32 (gain 0.121) in Table 14, and to 2.9-3.8% for the joint report in the GRPO runs (Table 13).
-
The 0.04 threshold is, in the authors' words, 'a post hoc readout of the completed fixed-budget grid' (Appendix D.2.3). The 46.1-51.3% claim is largely 8 versus 16 draws. MCSE also conditions on 64 fixed prompts, where the step-250 prompt-bootstrap intervals (Table 12) have half-widths of about 0.08-0.10 against an MCSE of 0.02, and coupled MCSE was checked only on synthetic simulations (Appendix F.3).
-
The evidence in this paper comes from only one recipe (LoRA GRPO), models of 1.5B-1.7B, one 7B trajectory and one fairly cheap judge. Pairwise judges are excluded (L472) and no human raters are used (L061). Because the coupling needs full-sequence probabilities of every checkpoint, it rules out API-only blackbox models. Additionally, any unrecorded overhead equal to the fairly small CNY margin would erase the n=16 advantage (Appendix G.4).
Questions for authors
- Please add Independent Star + Cache-only to Figure 1b and Table 1, and TV tree under Wen's own query rule beside TV tree + CAFA.
2.How much of the 46.1-51.3% claim remains attributable to CAFA?
-
Could cost be reported at several pre-specified thresholds (e.g., 0.02, 0.03, 0.05) from a fitted MCSE–n curve, and relative to the prompt-bootstrap interval width?
-
Across the 72 GRPO banks, how does realised error against the 64-draw reference compare with the MCSE-based intervals under coupling?
-
What savings follow where selection is hard (consecutive late checkpoints, larger learning rates, a 7B+ GRPO run)?