acceptodds
Under review as a conference paper at ICLR 2027

Learning Whether to Learn: Prospective Counterfactual Test-Time Learning for Language Models

Abstract

Test-time learning (TTL) promises models that improve during deployment, yet the dominant design pattern is myopic: an experience produces an update, and the update is judged by whether it fits that same experience. This creates a structural failure mode under unlabeled or weakly supervised streams—updates can reduce the current loss while degrading future decisions, amplifying pseudo-label noise, interference, and prompt-injection effects. We propose Foresight-TTL, a prospective formulation in which a candidate update is valued by its counterfactual effect on future deployment utility. During meta-training, matched rollout pairs estimate the counterfactual utility difference between applying and skipping the same update. A compact value model learns this future utility from the current experience, update geometry, uncertainty, and interference features. At deployment, a lower-confidence-bound gate decides whether to learn, a plasticity router decides where to write, and a reversible fast-weight transaction decides whether the write should survive. We further introduce multi-timescale persistence, promoting only repeatedly useful updates from episodic to persistent state. We analyze how calibrated value intervals can bound the probability of accepting harmful updates under a simultaneous calibration assumption. We evaluate the approach primarily on DeepSeek-R1-Distill-Qwen-7B across AIME 2024/2025, MATH-500, GPQA-Diamond, BIG-Bench Hard, and ARC-AGI-2, and additionally test cross-backbone deployment behavior on Qwen3-8B and Llama-3.1-8B-Instruct, with non-IID streams, corruption stress tests, and retention probes.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time training, large language models, continual learning, meta-learning, online learning, adaptive computation, counterfactual reasoning, reinforcement learning, model adaptation, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qp34OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new test-time learning framework that evaluates each candidate update by its counterfactual effect on future utility. It estimates this effect through matched rollouts with and without the update, then uses a learned value model to decide whether to apply the update, where to write it, and how long to retain it.

Critical strengths
  1. Novel and meaningful problem formulation: The paper distinguishes immediate objective improvement from future deployment utility, addressing an important but underexplored problem in test-time learning.
Critical weaknesses
  1. Dependence on value-model generalization: The framework relies on a value model trained on meta-training streams to predict the future utility of candidate updates. Its reliability under distribution shifts, unseen task types, or different future horizons remains unclear.
  2. The theoretical guarantee relies on a strong calibration assumption. The safety bound assumes that the predicted utility intervals are simultaneously calibrated for all candidate updates and deployment steps.
Questions for authors
  1. Value-model generalization: How does the learned value model perform under distribution shifts, unseen task types, and future horizons different from those used during meta-training? Can the authors evaluate whether its predictions remain accurate without retraining the controller?
  2. Validity of the calibration guarantee: How well do the predicted utility intervals satisfy the assumed simultaneous calibration condition in practice? Can the authors report empirical coverage and harmful-update rates under distribution shifts and adversarial or corrupted streams?
1 comment

We thank the reviewer for these questions. We test frozen-controller transfer across domains and horizons, and distinguish empirical coverage from Proposition 1's simultaneous-coverage assumption.

1. Value-model generalization

Meta-training uses MATH-train/OpenR1-Math, with non-IID mixtures and synthetic corruption (Appendix E). Existing tests include GPQA, BBH, and held-out competition math (Figs. 3–4). Appendix J retrains at each H, so it does not test frozen-controller horizon transfer.

A. Frozen cross-domain evaluation. We evaluate the H=8 controller, trained/calibrated only on MATH/OpenR1, on held-out domains without retraining or recalibration. Predictions are compared with matched counterfactual audits.

DomainSpearman ρ ↑Sign accuracy ↑90% coverage ↑
MATH-5000.7485.2%89%
GPQA-Diamond0.6279.8%86%
BBH0.5877.3%84%
Competition math0.5374.6%83%
ARC-AGI-20.4570.1%80%

MATH-500 is the in-domain reference; other datasets test transfer. Sign accuracy is predicted/audited sign agreement; coverage is the fraction of audited utilities within predicted intervals. Transfer degrades on distant domains and does not guarantee arbitrary OOD reliability.

B. Frozen horizon transfer. On MATH-500, we retain the H=8 controller's unchanged predictions and compare them with matched apply/skip audits over H=4, 8, 16, and 32. Each pair uses identical future examples/seeds, with no intervening writes. We test ranking/sign transfer, not cross-horizon calibration.

Audit horizonSpearman ρ ↑Sign accuracy ↑
H=40.7082.4%
H=80.7485.2%
H=160.6580.7%
H=320.5274.9%

Transfer degrades at H=32, indicating horizon-dependent estimates.

2. Empirical interval coverage

Proposition 1 assumes simultaneous coverage across candidates and steps. Appendix N applies per-head temperature scaling on held-out MATH/OpenR1 validation data, but this does not establish joint coverage.

C. Pointwise and finite-window coverage. We evaluate frozen, meta-validation-calibrated intervals without target-domain recalibration. The nominal 90% Gaussian intervals use temperature-scaled predictive standard deviations, with head-wise temperatures fitted on MATH/OpenR1 meta-validation data and frozen throughout evaluation. An 8-step window is covered only if all 48 intervals (6 subspaces × 8 steps) contain their audited utilities. Clean/corruption streams have 120 non-overlapping windows per 1,000-interaction stream (600 across five seeds). Each abrupt-shift stream has 500 MATH-500 then 198 GPQA-Diamond interactions; all shift metrics use post-shift data, with 24 complete windows per seed (120 total). GPQA-Diamond is used only as an unlabeled input stream in this separate distribution-shift stress test; its ground-truth labels are withheld from the learner and controller and used only for offline auditing. These runs are excluded from endpoint benchmark scores. Pointwise coverage is over candidate utilities; HUR is over accepted writes.

We use seed-stratified 50-step block-bootstrap CIs; excluding the final six post-shift interactions leaves 192 interactions (24 non-overlapping 8-step windows) per seed. Overlapping windows are not treated as independent.

SettingPointwise coverage (95% CI)Simultaneous coverage (95% CI)HUR (95% CI)
Clean89% [87, 91]24% [20, 28]1.6% [0.2, 5.3]
MATH-500 → GPQA-Diamond83% [79, 87]16% [11, 21]11.2% [7.6, 15.1]
20% mixed corruption86% [82, 89]18% [14, 22]8.9% [6.0, 12.2]
40% mixed corruption79% [75, 83]10% [7, 14]15.5% [11.5, 19.8]
20% targeted anti-alignment (operator iv)75% [71, 79]7% [4, 11]17.8% [13.2, 22.5]

Clean HUR (1.6%) is over realized writes among 420 held-out decisions (29.0% write rate; Sec. 7.1), not coverage windows. The 20%/40% corruption HURs are from Sec. 7.3. The shift uses a sequential protocol, unlike standalone GPQA. Anti-alignment corrupts 20% of learning positions, using operator (iv) exclusively at selected positions; distractors come from the bottom 10% of the gradient-similarity-ranked corruption bank, with protected-anchor categories sampled uniformly (Appendix E).

These results show a gap between marginal and finite-window simultaneous coverage.

Theoretical scope. Equation (26) is a conditional assumption, not an empirically established guarantee. Our experiments do not establish simultaneous coverage over the full deployment horizon. Temperature scaling does not prove joint coverage; split conformal provides marginal coverage under exchangeability. Proposition 1 is therefore a conditional safety result, not an unconditional guarantee under arbitrary shifts. LCB gating and reversible updates are practical safeguards, not substitutes for the assumption.

Reviewer_CzZ1OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FORESIGHT-TTL, a selective test-time learning framework that estimates the future utility of parameter updates through offline paired counterfactual experiments. It combines a prospective value predictor, lower-confidence-bound (LCB) gating, parameter subspace routing, reversible updates, and multi-timescale persistence. Experiments report improvements in reasoning accuracy, reduced harmful-update rates, and better knowledge retention.

Critical strengths
  1. This paper identifies an important limitation of existing TTL methods: improving current loss may not necessarily be beneficial for future tasks.
  2. Matching applications/skipping launches provides a meaningful way to evaluate updates based on future utility rather than immediate loss reduction.
  3. The evaluation covers multiple inference benchmarks, non IID flows, damage testing, and component erosion, providing useful evidence of adaptability and robustness.
Critical weaknesses
  1. The predictor is mainly trained based on mathematical task streams, and the update utility depends on the future task distribution. The same updates may be beneficial in one distribution, but harmful in another distribution. Although the cross domain results are encouraging, they have not fully established reliability under more diverse or unexpected distribution changes.
  2. Offline deployment eliminates subsequent updates (Algorithm 1), while deployment continuously accumulates parameter changes. Therefore, the isolated utility of updates may differ from their effect during the period of sustained adaptation. In addition, rollback relies on weak rewards based on self consistency rather than actual correctness, which makes it difficult to attribute when multiple updates coexist.
  3. The framework combines multiple objectives, six manually defined subspaces, LCB gating, rollback, and persistence. Although Table 6 shows benefits from individual components, it remains unclear whether simpler update-selection strategies could achieve comparable results. The particular subspace partition and feature design also lack sufficient justification.
  4. The harmful update limit depends on the synchronous calibration assumption in equation (26), but the reported calibration results cannot directly verify the required interval coverage range. The sensitivity of LCB multiplier, write threshold, rollback threshold, and persistence parameters has not been fully explored.
  5. Complete multi-task accuracy results are mainly reported for one backbone, while other backbones require separately trained controllers. Comparisons with relevant methods such as FG-TTL[A] are missing. Furthermore, the reported online efficiency excludes offline counterfactual data construction and controller meta-training costs, making the overall computational trade-off unclear.

Minor issues

  1. Algorithm 2, line 16 refers to Eq. (24) for the persistence promotion condition, whereas the actual criterion is given in Eq. (25).
  2. Eq. (12) uses (i_t^{(j)}), while Eq. (13) defines (I_t^{(j)}), without explicitly clarifying their relationship.

[A] Future-Gain Guided Test-Time Learning for Large Language Models, ICML2026.

Questions for authors

see weakness

1 comment

We thank the reviewer for identifying these gaps.

1. Generalization. Meta-training uses MATH-train/OpenR1-Math (Appendix E). We test a frozen controller without target-domain retraining or recalibration:

DomainECE ↓90% Cov.HUR ↓
MATH-5000.0489%8.9%
GPQA0.0686%10.8%
BBH0.0784%12.1%
Competition math0.0783%13.4%
ARC-AGI-20.0981%15.2%

ECE is the 10-bin, bin-size-weighted absolute gap between predicted and audited utility means; coverage is the fraction of audited utilities within nominal 90% Gaussian intervals (μ̄ ± 1.645σ̃), using the ensemble scale after head-wise temperature calibration. HUR uses 20% corruption. Coverage is marginal, not simultaneous time-uniform, and does not establish Eq. (26)'s assumption.

2. Isolated vs. sequential supervision. Algorithm 1 excludes subsequent writes within its H-step counterfactual window. We compare apply/skip branches, then let both execute the online policy on identical future examples/seeds, including new writes, commits, rollbacks, and persistence. Controllers share architecture, streams, and budgets; evaluation uses held-out seeds.

TargetSign agreement ↑Spearman ρ ↑Sequential HUR ↓
Isolated82.1%0.7211.6%
Sequential88.5%0.819.4%

Sequential HUR is the fraction of accepted writes with negative discounted apply–skip exact-correctness utility over H=8 steps, testing isolated targets against total effect under continued adaptation.

3. Rollback vs. correctness. Rollback uses vote-margin reward, not ground truth (Eqs. 19–21). For each gate-selected provisional write, we clone the pre-write state, including other active updates. Branches differ only by applying/skipping that write, then independently run the same frozen policy on identical H=8 examples/seeds, allowing subsequent adaptation.

Let cᵃₖ,cˢₖ be exact correctness in apply/skip branches. Audited utility is ΔU=100Σₖ₌₁⁸γᵏ⁻¹(cᵃₖ−cˢₖ), γ=0.95; harmful means ΔU<0. Labels are used only offline.

OutcomeEvents/1kMean ΔUHarmful
Rolled back36−2.778%
Committed218+2.418%
Timed out (uncommitted)36−0.554%

The 60-pp gap indicates alignment with future correctness harm. The audit measures decision alignment; ablation measures policy benefit. Removing rollback raises HUR 8.9%→15.1% and worsens retention −0.8→−3.1 (Table 6).

4. Design choices and robustness. We compare three selection rules on identical updates in a fixed rank-16 LoRA subspace (110 writes/1k interactions); routing, rollback, and persistence are disabled. Thresholds are set on meta-validation.

Selection ruleStream-AUC ↑HUR ↓Retention ↑
Prospective-value LCB62.414.2%−3.2
Prospective-value mean61.221.0%−4.0
Local gain58.731.6%−5.1

Stream-AUC uses the standard stream; HUR uses 20% corruption. LCB gains 3.7 AUC points and reduces HUR 17.4 pp vs. local gain; full FORESIGHT-TTL (65.7/8.9%) is reference only. Six subspaces: upper-12-block Q/V; upper-12-block attention output; MLP up/gate; MLP down; joint final-four-block LoRA; episodic adapter bank. Appendix J tests H∈1,2,4,8,16,32H\in\\{1,2,4,8,16,32\\}. One-at-a-time sweeps vary β∈1.0,1.3,1.64,2.0,2.5\beta\in\\{1.0,1.3,1.64,2.0,2.5\\}, τw∈0.00125,0.001875,0.0025,0.00375,0.005\tau _w\in\\{0.00125,0.001875,0.0025,0.00375,0.005\\}, τrollback∈0.5,0.75,1.0,1.25,1.5\tau _{\mathrm{rollback}}\in\\{0.5,0.75,1.0,1.25,1.5\\}, Kp∈4,6,8,12,16K _p\in\\{4,6,8,12,16\\}, and λp∈0.90,0.95,0.98,0.99\lambda _p\in\\{0.90,0.95,0.98,0.99\\}, holding others fixed. Across sweeps, Stream-AUC=64.8–65.9, HUR=7.4–13.1%, retention=−1.5 to −0.7.

5. Multi-backbone, FG-TTL, and cost. Qwen3-8B and Llama-3.1-8B-Instruct use separately trained/calibrated controllers; results vs. DDRL:

Backbone/methodAIME'24AIME'25MATH-500GPQA-DBBHARC-AGI-2 p@1
Qwen3 / DDRL62.253.094.052.782.66.5
Qwen3 / FORESIGHT64.355.495.054.884.17.8
Llama / DDRL18.612.470.342.076.22.9
Llama / FORESIGHT21.115.272.143.677.83.8

FG-TTL is implemented on Llama-3.1-8B with matched stream order, LoRA budget, decoding, and tuning:

MethodAIME'24MATH-500Stream-AUCHURRetentionOnline cost
FG-TTL14.269.861.917.1%−2.41.27×
FORESIGHT-TTL21.172.164.010.6%−1.01.16×

Online cost uses the same H100 hardware and matched batch sizes/token budgets. Offline meta-training (excluded from Appendix M live cost) takes 72 h on 8×H100 80GB: 68 h rollouts + 4 h controller fitting (576 GPU-hours), 160k prefixes, up to 960k targets. Online overhead is 1.16× static inference, without counterfactual branching.

Minor corrections. We clarify that it(j)i _t^{(j)} in Eq. (12) denotes the scalar It(j)I _t^{(j)} defined in Eq. (13), and correct Algorithm 2, line 16, to cite Eq. (25).

Reviewer_7bnqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of deciding whether a test-time learning update should be applied in language models, arguing that improving current objectives does not necessarily lead to better future deployment performance. The authors propose FORESIGHT-TTL, a prospective test-time learning framework that estimates the future utility of candidate updates through counterfactual rollouts and uses this estimation to gate, route, and preserve updates. The method is evaluated on several reasoning benchmarks including AIME, MATH-500, GPQA-Diamond, BIG-Bench Hard, and ARC-AGI-2, with comparisons against several test-time adaptation baselines. The paper claims that prospective update selection improves adaptation quality, robustness to corrupted streams, and retention compared with existing test-time learning approaches.

Critical strengths
  1. Interesting problem formulation: The paper identifies an important limitation of existing test-time learning methods: an update that improves immediate objectives may still harm future performance due to noise, interference, or distribution shifts. The formulation of evaluating updates based on future utility rather than local improvement is conceptually interesting.
Critical weaknesses
  1. The contribution appears over-engineered and the necessity of each component is unclear. FORESIGHT-TTL combines several mechanisms, including counterfactual value prediction, uncertainty estimation, plasticity routing, reversible transactions, and persistence mechanisms. However, the paper does not sufficiently demonstrate that the central contribution is the key factor behind the improvements, rather than the accumulation of multiple auxiliary components. Although the ablation table removes individual components, the evaluation does not isolate whether the proposed counterfactual formulation itself provides substantial benefits over simpler update-selection strategies.
  2. The experimental validation is insufficient for supporting such a broad claim. The method relies heavily on offline meta-training with counterfactual future rollouts and learned value models. However, the evaluation mainly focuses on benchmark accuracy and synthetic corrupted streams. It remains unclear whether the learned prospective value model generalizes to realistic deployment scenarios with unseen domains, longer interaction horizons, or different adaptation objectives.
Questions for authors
  1. The proposed framework introduces many components beyond prospective update evaluation. Could the authors provide stronger evidence that the counterfactual value prediction itself is responsible for the improvements, rather than the additional mechanisms such as rollback or persistence?

  2. How sensitive is the learned prospective value model to the meta-training distribution? For example, if the deployment stream differs substantially from the MATH/OpenR1-based meta-training streams, does the value estimator remain calibrated?

  3. The method requires offline counterfactual rollouts to train the update-value model. What is the computational cost of this meta-training procedure compared with existing test-time learning methods, and how does this affect practical scalability?

  4. The theoretical guarantee assumes simultaneous calibration of the predicted uncertainty intervals. How realistic is this assumption for neural value models in open-ended deployment environments?

1 comment

We thank the reviewer for the constructive feedback.

1. Does prospective value prediction drive improvements?

Appendix L, Table 6 shows that replacing prospective value with local gain lowers Stream-AUC from 65.7 to 60.8 and raises HUR from 8.9% to 27.4%. Removing uncertainty/LCB, routing, rollback, or persistence yields 63.1, 63.8, 64.2, and 64.4; unpaired targets yield 62.9. Paired rollouts reduce target SD from 5.8 to 2.6 reward points (Appendix D).

We compare three gates (prospective-value LCB, mean-only, and local gain) using identical candidate updates and a fixed rank-16 LoRA subspace, disabling routing, rollback, and persistence. All share a 110-write/1k-step budget fixed on meta-validation.

Table 1: Controlled ablation on REASONSTREAM-1K.

MethodStream-AUC ↑HUR ↓Write budget/1kRetention ↑
Full FORESIGHT-TTL65.78.9%–-0.8
Prospective-value LCB62.414.2%110-3.2
Prospective-value mean-only61.221.0%110-4.0
Local-gain gate58.731.6%110-5.1

Full FORESIGHT-TTL is a reference; the three gates form a separate controlled comparison. Under 20% corruption, LCB gains 3.7 Stream-AUC points and reduces HUR by 17.4 percentage points versus local gain.

2. Generalization and calibration under distribution shift

Meta-training uses 60% MATH-train/40% OpenR1-Math, piecewise-stationary mixtures, and 35% corrupted episodes (Appendix E). Evaluation covers MATH-500, GPQA-Diamond, 18 held-out BBH tasks, and competition math. Figure 8 retrains controllers separately, not zero-shot. We additionally evaluate a frozen MATH/OpenR1-trained and calibrated controller without target-domain retraining or recalibration.

Table 2: Cross-domain calibration with a frozen controller.

DomainCalibration error ↓90% Coverage ↑HUR ↓
MATH-5000.0489%8.9%
GPQA-Diamond0.0686%10.8%
BBH (18 held-out transfer tasks)0.0784%12.1%
Held-out competition math0.0783%13.4%

Calibration error is the bin-size-weighted mean absolute gap between mean predicted and audited utility across 10 equal-count bins. Coverage is the fraction of audited targets within nominal 90% temperature-scaled Gaussian intervals (μ ± 1.645σ̃), with head-wise temperatures fitted on MATH/OpenR1 meta-validation and frozen at evaluation. HUR uses 20% corruption. Shifted-domain coverage (83–86%) indicates degraded but nontrivial calibration.

3. Offline counterfactual meta-training cost

Appendix E reports 40k batches × four prefixes (160k) and up to 960k candidate targets. Offline compute scales with prefixes, candidate subspaces, and horizon; sequential branching caps active model states at two (Appendix M).

Table 3: Offline meta-training cost.

MetricResult
Hardware8 × H100 80GB
Paired rollout generation68 h
Controller fitting4 h
Total wall-clock72 h
Accelerator cost576 GPU-hours
Stream prefixes160,000
Candidate-value targetsUp to 960,000
Peak active model states2

This one-time cost is amortized across deployments sharing a backbone/controller. Baselines lack equivalent paired-counterfactual training, precluding direct offline-cost comparison. Online deployment needs no counterfactual branching: overhead is 1.16× static inference (DDRL: 1.31×; TTRL: 1.38×).

4. Scope of the theoretical guarantee

Proposition 1 and Eqs. (26)–(27) are conditional: under a fixed calibrated deployment distribution, if simultaneous coverage holds with probability at least 1−δ1-\delta over all candidate updates and steps, costs are nonnegative, and the gate accepts only when its cost-adjusted LCB is positive, then

\exists\,t,j: \text{candidate }j\text{ is gate-accepted at }t,\; V_t^{(j)}(\Delta_t^{(j)})<0 \right)\leq\delta.$$ This is not an unconditional guarantee for learned uncertainty. Appendix N describes temperature scaling and split conformal; the latter provides marginal coverage under exchangeability, not the simultaneous time-uniform coverage required by Proposition 1. Neither guarantees validity under arbitrary non-stationary shifts. Empirical calibration differs from the conditional guarantee; rollback is an additional safeguard, not a substitute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.