- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new test-time learning framework that evaluates each candidate update by its counterfactual effect on future utility. It estimates this effect through matched rollouts with and without the update, then uses a learned value model to decide whether to apply the update, where to write it, and how long to retain it.
Critical strengths
- Novel and meaningful problem formulation: The paper distinguishes immediate objective improvement from future deployment utility, addressing an important but underexplored problem in test-time learning.
Critical weaknesses
- Dependence on value-model generalization: The framework relies on a value model trained on meta-training streams to predict the future utility of candidate updates. Its reliability under distribution shifts, unseen task types, or different future horizons remains unclear.
- The theoretical guarantee relies on a strong calibration assumption. The safety bound assumes that the predicted utility intervals are simultaneously calibrated for all candidate updates and deployment steps.
Questions for authors
- Value-model generalization: How does the learned value model perform under distribution shifts, unseen task types, and future horizons different from those used during meta-training? Can the authors evaluate whether its predictions remain accurate without retraining the controller?
- Validity of the calibration guarantee: How well do the predicted utility intervals satisfy the assumed simultaneous calibration condition in practice? Can the authors report empirical coverage and harmful-update rates under distribution shifts and adversarial or corrupted streams?
We thank the reviewer for these questions. We test frozen-controller transfer across domains and horizons, and distinguish empirical coverage from Proposition 1's simultaneous-coverage assumption.
1. Value-model generalization
Meta-training uses MATH-train/OpenR1-Math, with non-IID mixtures and synthetic corruption (Appendix E). Existing tests include GPQA, BBH, and held-out competition math (Figs. 3–4). Appendix J retrains at each H, so it does not test frozen-controller horizon transfer.
A. Frozen cross-domain evaluation. We evaluate the H=8 controller, trained/calibrated only on MATH/OpenR1, on held-out domains without retraining or recalibration. Predictions are compared with matched counterfactual audits.
MATH-500 is the in-domain reference; other datasets test transfer. Sign accuracy is predicted/audited sign agreement; coverage is the fraction of audited utilities within predicted intervals. Transfer degrades on distant domains and does not guarantee arbitrary OOD reliability.
B. Frozen horizon transfer. On MATH-500, we retain the H=8 controller's unchanged predictions and compare them with matched apply/skip audits over H=4, 8, 16, and 32. Each pair uses identical future examples/seeds, with no intervening writes. We test ranking/sign transfer, not cross-horizon calibration.
Transfer degrades at H=32, indicating horizon-dependent estimates.
2. Empirical interval coverage
Proposition 1 assumes simultaneous coverage across candidates and steps. Appendix N applies per-head temperature scaling on held-out MATH/OpenR1 validation data, but this does not establish joint coverage.
C. Pointwise and finite-window coverage. We evaluate frozen, meta-validation-calibrated intervals without target-domain recalibration. The nominal 90% Gaussian intervals use temperature-scaled predictive standard deviations, with head-wise temperatures fitted on MATH/OpenR1 meta-validation data and frozen throughout evaluation. An 8-step window is covered only if all 48 intervals (6 subspaces × 8 steps) contain their audited utilities. Clean/corruption streams have 120 non-overlapping windows per 1,000-interaction stream (600 across five seeds). Each abrupt-shift stream has 500 MATH-500 then 198 GPQA-Diamond interactions; all shift metrics use post-shift data, with 24 complete windows per seed (120 total). GPQA-Diamond is used only as an unlabeled input stream in this separate distribution-shift stress test; its ground-truth labels are withheld from the learner and controller and used only for offline auditing. These runs are excluded from endpoint benchmark scores. Pointwise coverage is over candidate utilities; HUR is over accepted writes.
We use seed-stratified 50-step block-bootstrap CIs; excluding the final six post-shift interactions leaves 192 interactions (24 non-overlapping 8-step windows) per seed. Overlapping windows are not treated as independent.
Clean HUR (1.6%) is over realized writes among 420 held-out decisions (29.0% write rate; Sec. 7.1), not coverage windows. The 20%/40% corruption HURs are from Sec. 7.3. The shift uses a sequential protocol, unlike standalone GPQA. Anti-alignment corrupts 20% of learning positions, using operator (iv) exclusively at selected positions; distractors come from the bottom 10% of the gradient-similarity-ranked corruption bank, with protected-anchor categories sampled uniformly (Appendix E).
These results show a gap between marginal and finite-window simultaneous coverage.
Theoretical scope. Equation (26) is a conditional assumption, not an empirically established guarantee. Our experiments do not establish simultaneous coverage over the full deployment horizon. Temperature scaling does not prove joint coverage; split conformal provides marginal coverage under exchangeability. Proposition 1 is therefore a conditional safety result, not an unconditional guarantee under arbitrary shifts. LCB gating and reversible updates are practical safeguards, not substitutes for the assumption.