acceptodds
Under review as a conference paper at ICLR 2027

Continuous Test-Time Training for Large Reasoning Models

12 views

Abstract

Scaling test-time computation improves the reasoning performance of large language models (LLMs) without improving the models themselves. Test-time training (TTT) instead updates model parameters on unlabeled test problems. Without ground-truth answers, existing methods often use agreement or confidence as proxy rewards, achieving early gains that can plateau or reverse with further training. We investigate whether TTT can sustain improvement beyond the plateau of reinforcement learning with verifiable rewards (RLVR) on labeled data. We propose Continuous Test-Time Reinforcement Learning (C-TTRL), which uses a learned critic to reward policy updates on unlabeled test problems. At each iteration, verified responses generated by the updated policy on a separate, non-overlapping labeled reference set are used solely to recalibrate the critic. Across OLMo3 and Qwen3 models, C-TTRL continues to improve over the evaluated training horizon while representative self-rewarded baselines plateau. On AIME 2024, it raises Avg@16 from 33.0% to 51.1% for OLMo3-7B and from 42.3% to 65.8% for Qwen3-14B. Controlled comparisons support the respective roles of unlabeled test problems and critic recalibration. Code is available at https://anonymous.4open.science/r/C-TTRL-5B4F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM RL, Reasoning, Test-time Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JXuFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes C-TTRL, a semi-supervised test-time reinforcement learning framework that alternates policy updates on unlabeled target problems with critic recalibration using verified responses from a separate labeled reference set. It aims to sustain improvement beyond the RLVR plateau by maintaining reward reliability as the policy evolves. Experiments across OLMo3 and Qwen3 cover mathematical and general reasoning, with held-out evaluations and ablations examining transfer and the role of critic recalibration.

Critical strengths
  1. The paper presents a practical approach to sustaining test-time learning through critic recalibration.
  2. C-TTRL achieves substantial improvements across mathematical and general reasoning tasks, with evidence of held-out transfer.
  3. Frozen-critic and supervised-continuation ablations provide meaningful evidence for the proposed training procedure.
Critical weaknesses
  1. The comparisons are not matched by total computational budget. Appendix C reports 1.3–1.6× higher per-step cost than TTRL, while Table 6 uses different training durations across methods, leaving their relative performance at equal GPU-hour or rollout-token budgets unclear.

  2. The extended-horizon evidence is uneven. Figure 8 follows C-TTRL to approximately 350 steps, whereas Figure 9 shows Self-Harmony only through 128–136 steps, limiting the evidence for C-TTRL’s longer-term advantage over this competitive baseline.

  3. Discussion of recent semi-supervised methods is incomplete. GeoMin[1] also uses labeled rollouts to update reliability estimates for unlabeled data as the policy evolves, but is not discussed; given its recent release, a discussion of its geometric sample selection versus C-TTRL’s critic-based reward estimation would be sufficient without requiring an experimental comparison.

  4. Sampling-coverage results need clearer reporting. For Qwen3-14B on AIME 2024, C-TTRL’s Pass@8 is 73.3% in Table 2 but 71.3% in Table 10; the paper should clarify whether these values use different checkpoints or evaluation samples.

References

[1] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling.

Questions for authors

See the Critical Weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer JXuF

Thank you for your valuable comments and for recognizing C-TTRL as practical and effective, with substantial empirical gains and thorough ablations.


W1: Comparisons are not matched by total computational budget

Thank you for this suggestion. We compare TTRL with C-TTRL at the corresponding budget. The budget is set by TTRL’s best checkpoint.

ModelGPU hoursMethodAIME 2024AIME 2025
Qwen3-14B~505TTRL55.440.6
Qwen3-14B~505C-TTRL56.242.1

At matched GPU hours, C-TTRL slightly outperforms TTRL on both benchmarks; with additional training, C-TTRL continues to improve while TTRL plateaus or degrades (more detail in Figure 8).


W2: The extended-horizon evidence is uneven

"The extended-horizon evidence is uneven. Figure 8 follows C-TTRL to approximately 350 steps, whereas Figure 9 shows Self-Harmony only through 128–136 steps, limiting the evidence for C-TTRL's longer-term advantage over this competitive baseline."

Thank you for pointing this out. We acknowledge the difference in training horizons. We stopped the baseline runs after observing that their performance plateaued or degraded with further training. In addition, the reported comparison uses a fixed checkpoint at step 224 for C-TTRL, while selecting the best checkpoint for each baseline based on overall performance across the adaptation benchmarks (Appendix B.3).

We illustrate the evidence for stopping with Self-Harmony as an example.

1. Self-Harmony plateaus within the observed training horizon.

To show this trend, we report Self-Harmony's AIME 2024 Mean@8 from online evaluations during training.

ModelFinal stepSelected checkpoint (Mean@8)Training trend (Mean@8)
OLMo3-7B128Step 96: 40.0Peaks at 40.4 at step 64, then fluctuates between 38.8 and 40.4, with no upward trend.
Qwen3-8B224Step 32: 32.5Reaches 32.9 at step 24, then fluctuates between 28.7 and 33.8, ending at 32.5.
Qwen3-14B128Step 80: 61.3Fluctuates between 61.3 and 62.9 from step 60 onward.

Overall, the AIME 2024 training curves show no sustained upward trend over the observed intervals. Appendix E.3 of the revision provides additional evidence from training logs.

2. Later checkpoints within the plateau can perform worse

To examine whether a later checkpoint performs better, we re-evaluated Qwen3-8B at step 176, where its online AIME 2024 score peaks.

Qwen3-8B checkpointAIME 2024AIME 2025BeyondAIMEAdaptation avgAIME 2026OlymMATHHeld-out avg
Step 32 (selected)32.930.414.926.121.914.918.4
Step 176 (144 steps later)32.529.812.925.117.112.114.6

The later checkpoint performs similarly on the adaptation sets but worse on the held-out benchmarks. Compared with the selected checkpoint, step 176 is 1.0 point lower on the adaptation-set average and 3.8 points lower on the held-out average. Therefore, when multiple checkpoints within the plateau perform comparably well, we favor the earlier one for evaluation.


W3: Discussion of recent semi-supervised methods is incomplete

Thank you for pointing us to the missing baseline GeoMin [1], we have added a detailed discussion in the revised manuscript (highlighted in Lines 131–133).

GeoMin and C-TTRL both use a labeled reference set, but in different ways:

  1. Role of labeled data. GeoMin uses labeled data to model geometric distributions and select reliable unlabeled problems through hard filtering. C-TTRL uses labeled data to calibrate a critic model.

  2. Reward source. GeoMin uses majority-vote pseudo-labels as reward source, whereas C-TTRL uses critic scores.

  3. Advantage estimation. GeoMin uses GRPO to compute response-level advantages from group-normalized rewards. C-TTRL uses a PPO-like approach, computing token-level advantages.

[1] GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling.


W4: Sampling-coverage results need clearer reporting

We thank the reviewer for catching this inconsistency. The two tables report values from different checkpoints:

  • Table 2 uses the checkpoints specified in Table 6 (C-TTRL at step 224, TTRL at step 164).
  • Table 10 in the original manuscript reports the last online evaluation of each training curve (C-TTRL at step 352, TTRL at step 176).

We have updated the original Table 10 to use the same checkpoints as Table 2:

MethodPass@2Pass@4Pass@8Pass@16
TTRL (step164)56.456.656.756.7
C-TTRL (step224)68.571.172.373.0
Δ+12.1+14.5+15.6+16.3

The conclusion is unchanged: C-TTRL outperforms TTRL at every sampling budget. We have updated the results in the revised manuscript (Table 12, previously Table 10).

Thanks for your response. My concerns have been addressed, and I continue to support accepting this paper.

Thank you for your feedback and continued support! We are glad that our additional experiments and clarifications have addressed your concerns. We appreciate your thoughtful comments and constructive suggestions, which helped us strengthen the evaluation and improve the clarity of the paper.

Reviewer_ckhuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of performance plateauing or degradation in existing test-time training (TTT) methods due to unreliable reward signals. The authors propose C-TTRL, which optimizes the policy on unlabeled test problems using critic-provided rewards, while continuously recalibrating the critic with verified responses generated by the updated policy on a labeled reference dataset. Experiments across OLMo3 and Qwen3 models of different scales, covering both mathematical and general reasoning tasks, demonstrate that C-TTRL can further improve model performance after RLVR reaches a plateau, outperforming several existing TTT methods.

Critical strengths
  1. A simple yet effective approach to a fundamental TTT challenge. The paper identifies reward reliability under policy evolution as a key bottleneck in existing TTT methods. C-TTRL addresses this challenge through a simple yet effective iterative critic recalibration mechanism, enabling sustained improvements beyond the performance plateau of self-rewarding baselines. This offers valuable insights into scaling test-time training and improving model capabilities when ground-truth answers to target problems are unavailable.

  2. Solid and comprehensive experimental validation. The paper evaluates C-TTRL across multiple model families and scales, covering both mathematical and general reasoning tasks. The experiments include comparisons with competitive TTT baselines, continued RLVR training, frozen-critic ablations, and computational cost analysis. These results provide convincing empirical evidence for the effectiveness of the proposed approach and its key design choices.

Critical weaknesses
  1. Limited generalization on held-out benchmarks. While C-TTRL achieves substantial gains on benchmarks used for TTT, improvements on unseen benchmarks are much smaller. For example, Qwen3-14B gains 23.5 points on AIME 2024, but only 0.7 and 1.6 points on AIME 2026 and OlymMath. This raises concerns about whether the improvements reflect transferable reasoning abilities or primarily adaptation to specific test problems.
  2. Limited analysis of continuous improvement boundaries. Although C-TTRL shows sustained improvements within the evaluated training horizon, the paper does not sufficiently explore when or why these gains might plateau or degrade. Further analysis of critic reliability, distribution shifts, and longer training horizons would strengthen the paper's claims about continuous scaling.
Questions for authors
  1. Why are the improvements on held-out benchmarks much smaller than those on TTT adaptation benchmarks? Could the authors further investigate whether C-TTRL improves transferable reasoning capabilities beyond adaptation to specific test problems?

  2. Under what conditions might C-TTRL's improvements plateau or degrade? Could the authors further analyze how critic reliability and distribution shifts affect long-term training stability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ckhu(1/2)

Thank you for the helpful comments and for recognizing C-TTRL as a simple yet effective approach with solid and comprehensive experimental validation. Your comments are very helpful for improving the quality of our manuscript.

W1 & Q1: Limited generalization on held-out benchmarks

"Why are the improvements on held-out benchmarks much smaller than those on TTT adaptation benchmarks?"

We acknowledge that C-TTRL's gains on held-out benchmarks are smaller than those on the adaptation sets. As defined in Eq. (1), the objective is to improve accuracy on the target distribution μu\mu_u without supervision, and the policy is updated only on DuD_u. However, improvements on held-out benchmarks depend on how well the adapted policy generalizes to held-out problems. This is a common challenge for TTT methods, we have made this limitation clearer in the revision.

Nevertheless, C-TTRL demonstrates stronger generalization than existing baselines. While most baselines degrade on held-out benchmarks, C-TTRL improves across all held-out benchmarks and models. For example, on AIME 2026 with Qwen3-8B, TTRL drops by 8.1 points, while C-TTRL gains 2.3 points.

"Could the authors further investigate whether C-TTRL improves transferable reasoning capabilities beyond adaptation to specific test problems?"

C-TTRL improves performance on held-out benchmarks in both math and general reasoning. In the math experiment, OLMo3-7B gains 3.4 Avg@16 points on AIME 2026 and 4.6 points on OlymMath. In the general reasoning experiment, GPQA-Diamond Avg@8 improves by 10.5 points on OLMo3-7B and 5.0 points on Qwen3-8B (Table 3).

C-TTRL also shows cross-domain transfer from math to code. A checkpoint adapted only on math improves the average score across three code benchmarks from 43.9 to 48.1 on OLMo3-7B, without using code data during adaptation. Results from Appendix D are shown below:

Method (OLMo3-7B)MBPP+HumanEval+LiveCodeBench v6Average
Zero-RL53.759.818.343.9
C-TTRL (math-only adaptation)60.862.820.648.1

These results provide evidence of transfer beyond the adaptation problems, although the magnitude of the gains varies across models and benchmarks.

"Qwen3-14B gains ... only 0.7 and 1.6 points on AIME 2026 and OlymMath."

Thank you for pointing this out. Qwen3-14B achieves smaller gains on the held-out benchmarks than OLMo3-7B and Qwen3-8B. We found that its responses were often truncated: C-TTRL's truncation rate was 26.46%. To investigate whether this affected performance, We conducted an additional evaluation on AIME 2026:

MethodMax generation lengthAvg@16 at AIME2026Truncation rate
Zero-RL16K36.8811.46%
Zero-RL32K36.671.04%
C-TTRL16K38.5426.46%
C-TTRL32K40.421.88%

With the larger budget, C-TTRL's Avg@16 improves from 38.54 to 40.42, while truncation falls substantially. This suggests that evaluation with a 16K budget may underestimate C-TTRL's gains on Qwen3-14B. We have added this analysis to the revision.

AuthorsOpenReview ↗

Response to Reviewer ckhu(2/2)

W2 & Q2: Limited analysis of continuous improvement boundaries

Q2.1 "Under what conditions might C-TTRL's improvements plateau or degrade?"

As shown in Proposition 1, C-TTRL’s performance may plateau or degrade due to problem saturation and weakened critic discrimination.

  1. Problem saturation. The improvement for a problem depends on pk(x)(1−pk(x))p_k(x)(1-p_k(x)), which vanishes when the problem is always solved or never solved.
  2. Weakened critic discrimination. The critic must assign correct responses an average weight no lower than that assigned to incorrect ones. If this condition fails, performance may degrade.

Within the observed training horizon, we do not see a clear plateau, but accuracy gains gradually slow down. Meanwhile, critic AUROC shows a slight decline in later stages (from approximately 0.90 to 0.88 for OLMo3-7B and from 0.93 to 0.90 for Qwen3-14B). This weakening in critic discrimination may contribute to the slower gains and suggests that performance could approach a plateau with further training. We have added the full critic trajectories for all three models to Appendix F.2 of the revision.

Q2.2 "Could the authors further analyze how critic reliability and distribution shifts affect long-term training stability?"

We study how critic reliability and distribution shifts affect training stability through three experiments: reducing the reference set to weaken the critic, freezing the critic while the policy evolves, and cross-domain shift for the critic.

  1. A rather small reference set to weaken the critic. With only 128 reference examples, critic AUROC stays between 0.90 and 0.95 during the early stages of training, while AIME 2024 Mean@8 rises from 29.6% to 38.3% at step 44. After approximately 44 steps, both metrics decline: by step 104, AUROC falls to 0.75 and Mean@8 to 31.2%. In contrast, with the full reference set, the critic remains stable and accuracy continues to improve. We have added this result in the highlighted text in Appendix G of the revision.

  2. A frozen critic while the policy evolves. Performance initially matches that of C-TTRL but plateaus after approximately 100 steps, while C-TTRL continues to improve (Figure 3). This ablation supports our design choice of continuously recalibrating the critic.

  3. Cross-domain shift for the critic. Following your suggestion, we added an experiment: the critic is calibrated on labeled math problems and applied to unlabeled code problems. Despite the domain shift, C-TTRL improves performance on both the adaptation set and held-out code benchmarks:

Method (OLMo3-7B)MBPP+ (adaptation)HumanEval+ (held out)LiveCodeBench v6 (held out)
Zero-RL53.759.818.3
C-TTRL, math reference → code adaptation67.265.924.0

These results suggest that the math-calibrated critic provides a useful reward signal for code. We have added this experiment to Appendix D.2.

Thank you for the clarification and detailed response. My concerns have been addressed, and I am happy to maintain my recommendation for acceptance.

We’re glad that the discussion of generalization and training stability addressed your concerns. Thank you for your constructive feedback and continued support for acceptance!

Reviewer_aCstOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new semi-supervised approach for LRMs that exploit unlabelled problems to improve reasoning capabilities. The idea is to start from a base model, train it with RLVR with PPO to obtain a better policy and a critic, and then continue training the model on the unlabelled set with a recalibrated critic used as the pseudo-reward. This approach outperforms methods that use majority voting as pseudo-reward for example, and show continual improvement rather than plateauing or collapse of the compared baselines.

Critical strengths

The core strength of the paper is the simplicity of its idea and its solid empirical results on a nontrivial problem. The approach is simple, easy to understand and motivate, and integrates naturally into existing RLVR pipelines that use PPO or related RL methods that learn a critic. The problem it addresses is challenging, and the proposed method presents nice empirical results although the evaluation is conducted in moderately difficult scenarios.

Critical weaknesses

The main weaknesses that the approach can suffer from are the following:

  • This method relies heavily on RL approaches that learn a critic, and was only tested alongside a full RLVR + TTT training. The other TTT methods can already start from a post-trained checkpoint and do not need a critic to improve the base model. How can we use the method if we do not have access to a pre-trained critic?

  • This method trains a critic and makes it naturally domain dependent. This means that if DLD_L has math problems, and DuD_u has coding problems, I would assume that calibrating the learned critic on math problems will not work so well. TTRL uses majority voting, which is a pseudo label that is domain agnostic. I would at least want to see an experiment testing this scenario, where DLD_L and DuD_u have different domains.

  • With the current experiments, it is hard to decouple the gain from using the critic compared to the RL algorithm itself. If we compare to TTRL, not only the pseudo label is changing, but also the base RL algorithm used to learn from these pseudo labels. For instance, [1] shows that if we change the RL algorithm from GRPO and keep majority voting pseudo labels like TTRL, we still can achieve a huge gain. I would like an ablation testing changing the RL algorithm for TTRL to decouple the effect of pseudo labels from the RL algorithm.

[1] RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning. Arzhantsev et. al.

Questions for authors

I have a few question for the authors, aligned basically with the weaknesses identified:

  • Could you please discuss how the method would behave when starting from a post-trained model directly?

  • To isolate the effect of the RL algorithm, could you add RoiRL to the experiments and include it in the results table? Could you also evaluate TTRL with PPO instead of GRPO?

  • How does the method perform under a domain mismatch between DLD_L and DuD_u?

Minor points/questions:

  • Figure 1 disrupts the flow of the paper. I would recommend a figure that describes the general method as it can be hard to understand at the beginning.
4 comments
AuthorsOpenReview ↗

Response to Reviewer aCst (1/2)

Thank you for the constructive feedback and for recognizing C-TTRL’s simplicity and strong empirical performance. In response to your questions, we conducted three new experiments covering TTRL with PPO, RoiRL, and a cross-domain generalization setting.

W1&Q1 "How can we use the method if we do not have access to a pre-trained critic?"

C-TTRL requires a reward signal that can be calibrated against verified outcomes on labeled reference data. Without a pre-trained critic, C-TTRL could use an open-weight reward model or a self-verification score:

  1. An open-weight reward model. C-TTRL could use a model such as Qwen2.5-Math-PRM-7B to score the actor’s responses. As the actor evolves, the reward model could be periodically updated and recalibrated using verified outcomes of fresh responses on DLD_L . This may help reduce overfitting to a fixed reward model.

  2. Self-verification. Recent work such as LaSeR [1] and RLCR [2] shows that models can be trained to produce self-reward scores. These scores could be used as V(x,y)V(x,y) and recalibrated on DLD_L using verified outcomes.

We view both as promising directions for applying C-TTRL without a pre-trained critic and plan to explore them in future work.

[1] LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR2026).

[2] Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR2026).

W2&Q3 "How does the method perform under a domain mismatch between DLD_L and DuD_u?"

Thank you for this suggestion. We conducted a cross-domain experiment using mathematics as the labeled reference domain and code as the unlabeled adaptation domain.

MBPP+ pass@1 during adaptation:

step01632486480
pass@153.759.061.462.764.067.2

Although the critic is calibrated only on mathematics, it provides a useful reward signal for code: MBPP+ pass@1 increases from 53.7 to 67.2 at step 80, a gain of 13.5 points.

We also evaluate the best MBPP+ checkpoint on two held-out code benchmarks:

Method (OLMo3-7B)MBPP+ (unlabeled target)HumanEval+ (held out)LiveCodeBench v6 (held out)
Zero-RL53.759.818.3
C-TTRL, mathematics-only TTT60.862.820.6
C-TTRL, math reference → code target67.265.924.0

Compared with Zero-RL, the cross-domain setting improves the three benchmarks by 13.5, 6.1, and 5.7 points, respectively. The gains on held-out benchmarks suggest that the improvements generalize beyond the MBPP+ adaptation set. It also outperforms the mathematics-only TTT checkpoint on all three benchmarks, suggesting that adaptation on unlabeled code problems provides additional benefits. We have added these experiments and the corresponding discussion to Appendix D.2.

W2 "TTRL uses majority voting, which is a pseudo label that is domain agnostic."

We note that TTRL implements majority voting through exact string matching. This works for mathematical problems with standardized final answers, but in coding and general reasoning, equivalent responses may take different forms, requiring an additional judge to assess their equivalence. C-TTRL does not rely on answer matching, making it applicable to a broader range of generation tasks.

AuthorsOpenReview ↗

Response to Reviewer aCst(2/2)

W3&Q2:Isolating the effect of the RL algorithm (TTRL with PPO, RoiRL)

"Could you also evaluate TTRL with PPO instead of GRPO?"

Thank you for this suggestion. We conducted an additional experiment combining TTRL’s majority-vote pseudo-labels with a PPO-like actor–critic framework. We use the same experimental setup as C-TTRL, changing the reward source: TTRL with PPO uses binary majority-vote pseudo-rewards, computes advantages with GAE, and the critic fits the majority-vote returns.

MethodAIME 2024 Avg@16AIME 2025 Avg@16BeyondAIME
Zero-RL33.026.317.6
TTRL (PPO)39.228.519.3
TTRL (GRPO)40.827.121.8
C-TTRL51.137.024.5

TTRL with PPO and GRPO achieve comparable performance across the three benchmarks, while C-TTRL consistently outperforms both. We thank the reviewer for suggesting this baseline, which helps clarify the contribution of C-TTRL beyond the choice of RL algorithm.

"To isolate the effect of the RL algorithm, could you add RoiRL to the experiments and include it in the results table?"

Thank you for mentioning this missing baseline. RoiRL uses majority-vote pseudo-labels to select responses for supervised fine-tuning. We evaluated it on OLMo3-7B, Qwen3-8B, and Qwen3-14B.

ModelMethodBeyondAIMEAIME 2024AIME 2025AIME 2026 (held out)OlymMath (held out)
OLMo3-7BZero-RL17.633.026.326.718.9
OLMo3-7BRoiRL17.836.726.527.921.1
OLMo3-7BC-TTRL24.551.137.030.123.5
Qwen3-8BZero-RL15.626.325.421.915.0
Qwen3-8BRoiRL15.429.425.622.313.9
Qwen3-8BC-TTRL20.042.740.824.218.7
Qwen3-14BZero-RL24.942.337.138.124.2
Qwen3-14BRoiRL24.950.038.137.324.1
Qwen3-14BC-TTRL29.365.844.638.825.8

In our experiments, RoiRL demonstrates a clear advantage in training efficiency and substantially improves performance on the adaptation sets. For example, it improves Avg@16 from 42.3 to 50.0 on AIME 2024 with Qwen3-14B, while C-TTRL still achieves the best result of 65.8. We have added RoiRL to Table 2 and the related-work discussion in the revised manuscript, and provide a detailed description of our reproduction setup in Appendix B.2.

I thank the authors for their answer, providing enough evidence for me to maintain my positive assessment of the work. While I do not think that this is a groundbreaking contribution, I still consider that it is a simple and interesting idea that deserve to be shared with the community at ICLR.

Thank you for your positive assessment and constructive feedback. Before the discussion window closes, we would love to briefly share the broader vision behind our work.

C-TTRL is also motivated by the challenge of scalable oversight and weak-to-strong supervision: as frontier models such as Fable 5.2 and GPT Astra surpass human capabilities, neither a stronger teacher nor reliable human verification will be available, especially in complex domains like Terminal-Bench-Science. While model-based judges offer an alternative, fixed judges are easily exploited by increasingly capable actors. C-TTRL addresses this by continually recalibrating the critic on labeled data with fresh actor rollouts to maintain trustworthy supervision on harder, unlabeled tasks. Extending and scaling this paradigm is an exciting next step for us.

Thank you again for your suggestions that significantly strengthened our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear Reviewers, AC, SAC, and PC,

We sincerely thank all reviewers for their thoughtful feedback and constructive engagement, as well as for recognizing C-TTRL’s simplicity, practical value, and strong empirical performance. During discussion, all reviewers actively engaged and reaffirmed their positive ratings (3, 3, 3 →\rightarrow 3, 3, 3), reaching a unanimous consensus in favor of acceptance.

During the discussion phase, we resolved the remaining questions through targeted experiments and clarifications:

  1. Computational cost. Added a matched-GPU-hour comparison between C-TTRL and TTRL, confirming that C-TTRL consistently retains its performance advantage under an identical compute budget (Table 13).
  2. Ablation. Added comparisons against TTRL with PPO and RoiRL, verifying that our gains stem from continual critic recalibration rather than the choice of RL algorithm alone (Tables 2 and 14; details in Appendix B.2).
  3. Cross-domain generalization. Strengthened our transfer analysis—demonstrating effective adaptation to coding tasks using a math-calibrated critic—and analyzed how generation-length limits affect held-out performance (Section 4.2 and Appendix D, particularly D.2).
  4. Training and evaluation details. Provided a detailed analysis of how critic reliability and reference-set properties govern training stability (Appendices F.2 and G). Clarified baseline training horizons and checkpoint selection criteria, and unified the evaluated checkpoints across Tables 2 and 12 (Appendices B.3 and E.3).
  5. Related work. Expanded our discussion of concurrent and related test-time RL methods, including GeoMin and RoiRL (Sections 2 and 5).

All major revisions are highlighted in blue in the updated manuscript. In the remaining short time of the discussion period, we would be delighted to engage in any open-ended scientific exchange and share insights on broader topics around foundation models and RL post-training. We deeply appreciate the reviewers’ time and constructive feedback, which have substantially strengthened our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.