acceptodds
Under review as a conference paper at ICLR 2027

Continual Test-Time Learning with Evolution Strategies

Abstract

Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. However, existing continual learning approaches often rely on external supervision, such as labeled data, demonstrations, or reward signals, which can be costly and unavailable in practice. In contrast, repeated test-time adaptation using self-generated supervision can lead to policy collapse, degraded reasoning, and catastrophic forgetting. In this paper, we propose **Evolution Strategy Reinforcement Learning (ESRL)**, a continual test-time learning framework designed to mitigate catastrophic forgetting during label-free continual adaptation. Specifically, ESRL constructs a population of locally perturbed policies around the current model to explore multiple neighboring parameter regions throughout adaptation, and then samples trajectories from those regions. Unlike trajectory-space exploration from a single evolving policy, this parameter-space population maintains diverse sources of self-generated supervision even when the central policy begins to drift. The resulting trajectories are aggregated into label-free group-relative learning signals to update the central student. Through extensive experiments on single-task test-time adaptation and multi-task continual learning across mathematical reasoning, general knowledge, and medical domains, ESRL achieves relative gains of up to 31.7% over the base model and 16.6% over the second-best method. Notably, ESRL substantially reduces catastrophic forgetting and maintains the strongest final performance at the end of continual learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Test-Time Learning, Evolution Strategies, Catastrophic Forgetting, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XyytOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ESRL for label-free test-time adaptation and continual learning. A population of parameter-perturbed policies generates responses around a periodically refreshed anchor. Majority agreement supplies group-relative advantages, which train a central student through an advantage-weighted likelihood objective with reference-policy regularization. Experiments cover four backbones and nine benchmarks across mathematics, general knowledge, and medicine. A three-task sequential experiment studies forgetting, while additional analyses examine population diversity, perturbation strength, optimization choices, and computational cost.

Critical strengths
  • ESRL achieves the highest reported average accuracy across all four backbones in Table 1, with gains extending to related evaluation benchmarks. This breadth makes the empirical contribution more convincing than an improvement on a single adaptation task.
  • The parameter-space exploration mechanism has useful supporting evidence: removing perturbations reduces the general-knowledge average from 50.3 to 46.2 (Table 6), while the reported 0.6B cost comparison shows approximately 14% runtime overhead with nearly unchanged peak memory.
Critical weaknesses
  • The central continual-learning claim is not sufficiently validated. Figure 3 covers one small backbone and one task order, without a matched zero-perturbation control or repeated-seed results for that sequence. The broader single-domain results do not isolate whether population exploration reliably improves retention across tasks.
  • Evaluation details needed to interpret the reported gains are missing: MATH-500/MMLU split construction and sizes, evaluated subset counts, and checkpoint-selection rules are unspecified. Given the substantial training-time fluctuations, these omissions prevent a clear assessment of selection effects and reproducibility and weigh against acceptance in the current form.
  • Figure 3 shows substantial deterioration during the first task, before any task switch. Without a same-task continuation control and forgetting measured from each pre-switch checkpoint, the contribution of reduced cross-task interference is difficult to separate from differences in prolonged self-training stability.
Questions for authors
  1. Please specify the adaptation/evaluation splits and evaluated sample counts for MATH-500, MMLU, and MedQA. How were checkpoints, stopping points, and hyperparameters selected, and did any of these decisions use held-out answer accuracy, particularly given the performance fluctuations in Figure 3?

  2. Does the continual-retention advantage persist against the otherwise identical zero-perturbation variant across seeds and task orders under matched training budgets? The existing perturbation ablation and three-seed study concern single-domain adaptation, so they do not directly establish the robustness of the continual-learning result.

  3. In Figure 3, how much of the earlier-task degradation occurs before versus after each task switch? Reporting forgetting relative to the pre-switch checkpoint, alongside continued adaptation on the same task, would help separate cross-task interference from deterioration caused by prolonged self-training.

  4. In Equations (5)-(6), trajectories are sampled from perturbed anchor policies rather than the current student. Under this sampling distribution, what KL quantity does the unweighted k3 term estimate, or is it intended as a heuristic regularizer? How is behavior-policy mismatch controlled as the student changes within a block?

  5. Does a "clear majority" mean more than half of the valid responses or a unique plurality, and how are ties and unparsable responses handled? Can retrospective measurements of accepted-group frequency and pseudo-label accuracy clarify whether consensus remains reliable during continual adaptation?

8 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (1)

W1 and Q2

zero-perturbation control / mutli seed resutls on continual leanring

We thank the reviewer for raising this point. The following results show the continual learning performance across three seeds for ESRL, TTRL, and ESRL (Zero-Perturbation) using the Qwen3-0.6B-Base model. For ESRL (Zero-Perturbation), no parameter perturbation is applied, so the trajectories are sampled from the same anchor policy and therefore lack parameter-space exploration, resulting in narrower exploration coverage. Overall, ESRL shows the same trend as reported in the main draft.

MATH-500

Training StepESRLTTRLESRL (Zero-Perturbation)
50048.67 ± 2.0839.00 ± 1.7336.33 ± 2.52
100026.00 ± 2.6512.00 ± 2.6512.67 ± 2.89
150020.33 ± 3.5114.00 ± 1.0011.67 ± 2.08
200019.33 ± 1.5314.33 ± 2.5212.33 ± 3.06
250016.33 ± 3.2111.33 ± 2.528.67 ± 2.52
300017.33 ± 2.529.33 ± 1.537.33 ± 2.08

MMLU

Training StepESRLTTRLESRL (Zero-Perturbation)
50046.83 ± 0.5845.83 ± 1.2644.17 ± 1.61
100049.50 ± 0.5046.67 ± 1.8944.83 ± 2.25
150049.00 ± 0.8748.33 ± 1.5346.50 ± 2.00
200052.83 ± 1.5345.50 ± 1.5043.67 ± 2.25
250047.67 ± 1.2647.67 ± 1.8945.50 ± 2.50
300048.17 ± 2.0244.00 ± 3.4644.83 ± 3.75

MedQA

Training StepESRLTTRLESRL (Zero-Perturbation)
50026.00 ± 1.3224.33 ± 1.5324.83 ± 1.76
100029.83 ± 1.1523.33 ± 2.5721.50 ± 2.78
150032.67 ± 3.7928.33 ± 1.5326.33 ± 2.52
200031.83 ± 1.6130.83 ± 1.8928.67 ± 2.57
250035.83 ± 1.7631.83 ± 2.7529.83 ± 3.18
300033.83 ± 2.9329.33 ± 2.0827.17 ± 2.75

W2 and Q1

Please specify the adaptation/evaluation splits and evaluated sample counts for MATH-500, MMLU, and MedQA. How were checkpoints, stopping points, and hyperparameters selected, and did any of these decisions use held-out answer accuracy,

Thank you for raising this point. We clarify the adaptation/evaluation protocol and model-selection procedure below.

Data splits and sample counts. We use a fixed split (data seed 0) shared across all methods and training seeds. For MATH-500, 400 examples are used for adaptation and 100 disjoint examples for evaluation. For MMLU, 500 examples are used for adaptation and 200 disjoint examples for evaluation. For MedQA, we use 500 examples from the training split for adaptation and 200 examples from the test split for evaluation. Thus, there is no example-level overlap between adaptation and evaluation. MATH-500 and MMLU therefore correspond to our held-out subsets rather than the untouched official test sets.

Checkpointing and stopping. Training uses a fixed budget with no validation-based early stopping: 1,000 training step udates per domain and 3,000 traning updates in total for MATH-500 → MMLU → MedQA. Checkpoints are stored every 100 visits, while the dense continual-learning analysis is reported at predefined intervals. The stage-boundary checkpoints at 1,000, 2,000, and 3,000 accumulated training updates are used rather than selecting the locally best-performing checkpoint.

Importantly, held-out accuracy is never used for training, early stopping, checkpoint selection, or hyperparameter updates. Evaluation is performed independently of the adaptation process. Therefore, the fluctuations shown in Figure 3 are diagnostic measurements only and do not affect the optimization trajectory or the reported terminal checkpoints.

W3 and Q3

In Figure 3, how much of the earlier-task degradation occurs before versus after each task switch? Reporting forgetting relative to the pre-switch checkpoint, alongside continued adaptation on the same task, would help separate cross-task interference from deterioration caused by prolonged self-training.

The following table shows the comparison between continual learning and self-training on MATH-500 under Qwen3-0.6B-Base. In the follwoing table, we show two task swith from MATH-500 to MMLU and from MMLU to MedQA. The accuracy gap between the two settings shows the deterioration caused by the task switch.

Training StepDatasetContinual Learning Accuracy ↑Self-Training Accuracy ↑Gap (pp)
1,000MATH-50027.0%27.0%0.0
1,100MMLU27.0%32.0%5.0
1,200MMLU22.0%35.0%13.0
1,300MMLU28.0%36.0%8.0
1,400MMLU17.0%34.0%17.0
1,500MMLU20.0%35.0%15.0
1,600MMLU22.0%35.0%13.0
1,700MMLU19.0%33.0%14.0
1,800MMLU23.0%31.0%8.0
1,900MMLU20.0%30.0%10.0
2,000MMLU20.0%32.0%12.0
2,100MedQA17.0%31.0%14.0
2,200MedQA18.0%33.0%15.0
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (2)

Q4

what KL quantity does the unweighted k3 term estimate, or is it intended as a heuristic regularizer? How is behavior-policy mismatch controlled as the student changes within a block?

The KL term is designed as a heuristic regularizer to prevent the student model from deviating too far from the initial policy. The block-wise design is introduced to control the policy mismatch that would arise if the original model were used to continuously update the evolving student. As shown in Table 2, the mismatch introduced within a single block does not cause substantial differences in accuracy when the block size is within 20. Overall, a block size within this range provides a better balance between computational efficiency, exploration stability, and alignment with the evolving student.

Q5

Does a "clear majority" mean more than half of the valid responses or a unique plurality, and how are ties and unparsable responses handled? Can retrospective measurements of accepted-group frequency and pseudo-label accuracy clarify whether consensus remains reliable during continual adaptation?

The “clear majority” refers to a unique plurality. Tied groups are not counted as accepted groups; in such cases, no pseudo-label is assigned and the group is excluded from the subsequent policy update. The accepted-group frequency is shown below. In addition, we report the pseudo-label accuracy, which is estimated at each checkpoint as the number of correct pseudo-labels divided by the number of accepted groups. Overall, the pseudo-label accuracy remains relatively stable throughout continual training, indicating that the consensus signal remains reliable during adaptation.

TaskBenchmarkPercentage
1MATH-50080.2%
2MMLU81.5%
3MedQA84.9%
Training StepMATH-500 Pseudo-label AccuracyTraning StepMMLU Pseudo-label AccuracyTraining StepMedQA Pseudo-label Accuracy
10048.2%110052.70%210034.83%
20053.7%120049.33%220037.65%
30043.9%130046.15%230039.47%
40045.1%140054.55%240040.51%
50050.6%150055.26%250034.12%
60044.6%160045.76%260037.04%
70044.3%170047.06%270037.36%
80046.8%180051.43%280037.50%
90044.3%190050.67%290036.90%
100039.2%200052.05%300035.87%

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of ESRL

Reviewer_LsYPOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new way to do test-time learning, based on population-based search. Population diversity and achoring reduces forgetting, and the system can significantly improve performance in several benchmarks, especially in the pass@n settings.

Critical strengths

Both the test-time learning topic and the evolution strategy approach are topical, and their combination is intriguing. It seems ES has inherent robustness properties in high-dimensional spaces that are just now beginning to be understood, and this paper builds on them and illustrates them further. These kinds of novel directions are exactly what should be discussed at the conference.

Critical weaknesses

I don't see any critical weaknesses, but I have several questions that should be further clarified, both to increase the impact and to confirm that the results are sound. See details below.

Questions for authors

One advantage of ES (as opposed to e.g. RL) is that it can naturally be multiobjective. In TTL the natural objectives could be to maintain base performance and to improve upon multiple tasks with separate objectives. Would such an approach make sense as the next step here?

A recent paper (arXiv:2605.30148) suggests that adaptive weight decay AWD can serve as a powerful anchor to mitigate forgetting. How is your anchoring mechanism different from AWD? Could it be used here as well? Or combined with your method?

One dimension of performance that is often lost when doing fine-tuning and test-time adaptation is safety. It would be good to measure not just whether the base performance is retained, but whether safety is retained as well. Standard safety benchmarks can be used for evaluation (such as AILuminate and SORRY-Bench).

The curve in Figure 5 is actually commonly observed in fine-tuning. There's a recent study that explains it in terms of curvature and accumulation of noise (arXiv:2602.00170). Is that explanation consistent with yours? Does this study add depth to that explanation, or the other way around?

4 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback, as well as your positive assessment of our work.

Q1

Extension to multiobjective settings

We thank the reviewer for this insightful suggestion. We agree that multi-objective optimization is a promising extension of our method. In particular, ES could jointly optimize task-specific objectives while preserving the model's original capabilities, potentially mitigating catastrophic forgetting during continual test-time learning. However, one potential challenge is that combining trajectories from different objectives may introduce conflicting optimization signals, as observed in multi-teacher distillation, potentially leading to signal interference or collapse. A possible solution would be to introduce an adaptive weighting strategy that dynamically assigns different weights to the corresponding trajectories based on their objectives, thereby balancing their contributions during optimization.

Q2

connection to decay AWD

We thank the reviewer for highlighting this relevant work. AWD mitigates forgetting by constraining parameter-space drift, whereas our re-anchoring mechanism periodically updates the anchor policy after each block to reduce the mismatch between the policy generating trajectories and the evolving student model. As demonstrated in Table 2 of our ablation study, different block sizes affect the balance between computational efficiency, exploration stability, and alignment with the evolving student. Separately, our KL regularization constrains the student's deviation from the initial model, serving a role closer to AWD. We agree that AWD could be incorporated into ESRL or compared directly against KL regularization. Exploring their complementary effects on mitigating catastrophic forgetting is a promising direction for future work.

Q3

Standard safety benchmarks can be used for evaluation (such as AILuminate and SORRY-Bench)

The following table shows the continual learning results using Qwen3-0.6B-Base on the sequence MATH-500 →\rightarrow MMLU →\rightarrow MedQA.

We additionally evaluate the intermediate checkpoints on SORRY-Bench using its official safety evaluator and report the compliance rate.The results are shown below. ESRL demonstrates better safety retention overall, with the compliance rate decreasing during MMLU training, while TTRL shows a clearer degradation in safety retention as the continual learning process progresses.

Compliance Rate (%) ↓

Training StepESRLTTRL
087.5%87.5%
50088.0%89.5%
100088.2%90.0%
150087.0%90.5%
200086.1%90.9%
250085.9%90.0%
300085.5%90.9%

Q4

There's a recent study that explains it in terms of curvature and accumulation of noise (arXiv:2602.00170). Is that explanation consistent with yours? Does this study add depth to that explanation, or the other way around?

We sincerely thank the reviewer for highlighting this relevant study. Both works aim to explain the same phenomenon: informative optimization signals drive early improvements, while accumulated noise gradually dominates the training process, leading to performance degradation. The referenced study provides a complementary theoretical explanation for our observations.

Our work extends these insights in two aspects. First, we demonstrate that the same phenomenon also occurs in test-time learning without external supervision, where stochastic noise itself plays a crucial role in exploring useful update directions and generating self-supervision, rather than being purely detrimental. Second, as discussed in Section 5.2, we investigate the trade-off between signal quality and noise accumulation through batch size. Increasing the batch size produces smoother updates and reduces stochastic noise, but may also dilute informative learning signals, making it harder to identify and reinforce useful optimization directions.

Therefore, the two studies offer complementary perspectives, with the referenced work providing a theoretical explanation and ours extending the empirical observations to self-supervised test-time adaptation while investigating this practical trade-off. A promising next step would be to develop an adaptive early-stopping mechanism that identifies when accumulated noise begins to outweigh useful learning signals, thereby preventing performance degradation while preserving the benefits of stochastic exploration.

We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.

Kind regards

The authors of ESRL

Reviewer_LsYPOpenReview ↗

Fine by me.

Yes that's fine, just make sure to include these points in the revision.

Many thanks for your guidance. We will make sure to include these points clearly in the revised version.

Reviewer_2WBYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the new method ESRL for continual test-time learning for LLMs. It's based on the previous work TTRL for voting, where the current student samples few different parameter preturbations which produced one rollout per prompt. Majority voting over the rollaouts gives the reward which is then used for update of the student in an advantage-weighted mean log-likelihood loss + KL penalty towards the initial model. In the experimental sections the authors proposed experiments with single task adaptation (Llama and Quwen), three tasks sequences with Qwen, and ablations (block size, group size, loss, cost and seeds).

Critical strengths
  1. Paying attention to the memory footprint of the proposed ESRL method (borrowed from different work, based on seed generated variants).
  2. Relevant problem of self-trainin on an unlabeled test stream, where many methods collapse and forget.
  3. Analysis of the failure modes in Fig. 4 - good visualization and intuition for what is happening.
Critical weaknesses
  1. First, maybe not the most important thing, but somehow I find the relation to Evolution Strategies (ES) very artificial in this work. Why? As there is no fitness-based selection/aggregation and the main student is never updated from the population fitness. The population are used only for sampling distribution and then voting. The connection with ES is not clear in this work, and for me, seems to be misused.
  2. The baseline values for the vanilla model and used benchmark seem very low, e.g. Qwen3 report (arXiv:2505.09388), MATH-500: Qwen3-0.6B 55.2 non-thinking / 77.6 thinking (Tabs. 19–20) vs. 5.0 here.
  3. Additonally, Qwen3-0.6B base has Pass@32 = 45% but Pass@1 = 5% on MATH-500 -- why is this gap so high? Maybe the answer extraction/format is the problem, and what is model learning during test-time is the way how the answers should be composed, not the reasoning itself?
  4. All methods share one learning rate (1e-6), batch size 2 and group size 16, and these settings appear tuned for ESRL. Multiple methods seem to be undertuned and result in 0.0. It's not clear that the paper did a fair comparison across the methods.
  5. The continual learning experiments are very limited - one backbone, one task order, one architecture, only three tasks - no longer training sessions.
  6. No mention or relation to continual learning methods from vision test-time adaptation like: CoTTA, EATA. No metodhs to compare in CL setting, even presenting a stronger KL to init model, or EMA teacher approach would be nice to see for the comparison.
  7. Paper in many places has mistakes, lacking the precise information, e.g. where in Algorithm 1 the lambda and KL is uesed?
Questions for authors
  1. Why bringing ES without clear connection to it?
  2. Can we align/explain the base results with existing works/technical reports for selected models?
  3. Do other methods as well use KL to the initial model?
  4. Can we see the results from multiple runs and std?
  5. How were the hyper-parameters selected for other methods?
5 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (1)

W1 and Q1

The connection with ES is not clear in this work, As there is no fitness-based selection/aggregation and the main student is never updated from the population fitness. The population are used only for sampling distribution and then voting.

We thank the reviewer for raising this out. We acknowledge that our strategy shares some similarities with classical ES methods. In classical ES, parameter perturbations are used to generate a population of different parameters, and external fitness signals are then used to select or aggregate the better-performing parameters for updating the model. In our method, our perturbation part stays the same as the classical ES. Regarding the selcting and aggragating the better-performing parameter part, our method utilitze these perturbed parameters to generate on-policy trajectories, which then will be used to calculate the advantages and update the parameters. In the selecting and aggragating part, the on-policy strategy is the key connection between our method and classical ES, because this is how the perturbed parameters directly contribute to the parameter update in both methods.

W2 and Q2

Can we align/explain the base results with existing works/technical reports for selected models?

We thank the reviewer for pointing this out. The model selected in the draft was Qwen3-0.6B-Base (https://huggingface.co/Qwen/Qwen3-0.6B-Base), which is a pretrained model and is quite different from Qwen3-0.6B (https://huggingface.co/Qwen/Qwen3-0.6B), a post-trained model. The same applies to the other selected models. We have revised this typo in the latest draft. The use of pretrained models in the test-time learning framework directly follows TTRL [1], a foundational method in TTL.

The rationale behind choosing the pre-trained model is that this setting reflects an important deployment scenario of test-time learning: when a model encounters a new environment or distribution after deployment, it may not have access to additional environment-specific post-training before adaptation. Instead, it must adapt directly from the newly encountered test-time data. Starting from a checkpoint that has not been specifically adapted to the target environment therefore provides a useful setting for studying this cold-start adaptation process.

Additionally, to provide a broader evaluation, we also conduct experiments using Qwen3-0.6B, and the results are shown below. As shown in the table, ESRL achieves the best performance on 8 out of 9 datasets and the best average performance, while maintaining the same overall trend as reported in the paper.

ModelMethodMath MATH-500AMC23AIME25General Knowledge MMLUARC-CGPQA-DMedical MedQAMedMCQAPubMedQAAvg.
Qwen3-0.6BBase model55.027.73.354.072.024.236.047.057.541.9
TTRL61.050.66.753.577.027.838.548.558.546.9
ECHO64.053.010.056.575.528.337.046.557.547.6
EVOL-RL58.054.23.355.573.530.340.549.059.047.0
ESRL68.056.610.060.080.029.342.049.059.550.5

[1] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-Time Reinforcement Learning. Advances in Neural Information Processing Systems, 38, 2025.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (2)

W3

what is model learning during test-time is the way how the answers should be composed, not the reasoning itself?

We thank the reviewer for this insightful question. We interpret the large gap between Pass@1 and Pass@32 as a difference between reasoning coverage and the probability assigned to successful reasoning trajectories rather than an answer extraction or formatting issue. A similar phenomenon is also observed in TTRL [1] and ECHO [2]. A Pass@32 of 45% indicates that correct solutions remain reachable for a substantially larger set of MATH-500 questions under repeated sampling, whereas the much lower Pass@1 suggests that these successful trajectories are assigned relatively low probability by the base policy.

This is consistent with our analysis in Section 5.1. Test-time learning does not necessarily need to create entirely new reasoning modes from scratch; instead, self-generated signals can identify useful reasoning modes that are already reachable but underrepresented by the current policy, and increase their probability.

However, different TTL methods behave differently during continual adaptation. TTRL and ECHO tend to preserve broader reasoning coverage but maintain a more diffuse output distribution and lower Pass@1, while EVOL-RL can become overly concentrated, leading to reduced Pass@32 and signs of policy collapse. ESRL instead concentrates probability mass on useful outputs under the center policy while preserving broader reasoning coverage through population-level exploration. From the results in Tables 1 and 5, the gaps between Pass@1 and Pass@32 are 21 percentage points for TTRL, 26 percentage points for ECHO, and 18 percentage points for ESRL, while EVOL-RL collapses, with both metrics dropping to 4%. The smallest gap of ESRL empirically supports that ESRL better concentrates probability mass on useful outputs under the center policy. Moreover, as shown in Figure 4, ESRL achieves the highest population-level Majority@32 among the four methods, further demonstrating that it preserves broader reasoning coverage through population-level exploration.

[1] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-Time Reinforcement Learning. Advances in Neural Information Processing Systems, 38, 2025.

[2] Chu Zhao, Enneng Yang, Yuting Liu, Jianzhe Zhao, and Guibing Guo. ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning. International Conference on Machine Learning (ICML), 2026

W4

Concerns about all model share the same learning rate

We believe there is some misunderstanding regarding the experimental settings. The learning rates are only partially shared across methods rather than being identical: TTRL, ECHO, EVOL-RL, and ESRL use the same learning rate, which is consistent with the original paper, while SFT and ES use different learning rates. Where applicable, we follow the settings used in the original methods, while keeping the batch size and group size matched to ensure a fair comparison.

The weaker performance of some baselines is also consistent with the difference in their exploration mechanisms. ESRL uses parameter-space population exploration, which maintains diverse sources of self-generated supervision and therefore provides more informative signals during test-time learning. In contrast, trajectory-space methods generate all trajectories from the same central policy, resulting in a more limited search space and making them more likely to encounter limited or unstable updates under different situations.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (3)

W5

no longer training sessions for continal learning experiment

We thank the reviewer for this comment. We would like to clarify that our continual learning experiments involve two epochs, with total 1,000 training steps for each domain, resulting in 3000 cumulative training steps across three sequential domains. This represents a relatively extensive adaptation process compared with typical continual-learning settings: approximately 700 steps for SDFT [1], 600 steps for iSDFT [2], and 170 steps for SDPO [3]., allowing us to examine longer-term adaptation and potential catastrophic forgetting.

Regarding the number of tasks, our three-domain sequential learning setup directly follows the experimental framework adopted in Self-Distillation Enables Continual Learning [1], an ICML 2026 Spotlight paper, which provides a strong and established precedent for evaluating continual learning across three sequential tasks.

[1] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-Distillation Enables Continual Learning. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306:111542–111558, 2026.

[2] Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, Rasul Tutunov, Matthieu Zimmer, Jun Wang, and Haitham Bou-Ammar. iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs. arXiv preprint arXiv:2609.24646, 2026.

[3] Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, and Fei Zhu. Denser ≠ Better: Limits of On-Policy Self-Distillation for Continual Post-Training. arXiv preprint arXiv:2607.01763, 2026.

W6

No mention or relation to continual learning methods from vision test-time adaptation like: CoTTA, EATA.

We thank the reviewer for the suggestion. Our work primarily focuses on text-based language models across mathematical reasoning, general QA, and medical domains. Methods such as CoTTA and EATA were originally developed and evaluated for vision-based test-time adaptation, which is beyond the current scope of this paper. Moreover, these methods were evaluated on vision backbones with millions of parameters, and their effectiveness and computational scalability to billion-parameter autoregressive language models remain largely unexplored. Nevertheless, extending our method to vision models is a promising research direction that we plan to explore in future work.

No methods to compare in CL setting, even presenting a stronger KL to init model, or EMA teacher approach would be nice to see for the comparison.

We introduce an EMA-regularized SFT baseline for continual learning. Under our continual test-time learning setting, for each unlabeled prompt, the model samples multiple responses and selects the majority answer as the pseudo-label. Majority-consistent responses are then used as pseudo-targets for token-level cross-entropy training.

To reduce policy drift, we maintain an EMA teacher updated as

θˉt+1=0.99,θˉt+0.01,θt+1,\bar{\theta}_{t+1} = 0.99\\,\bar{\theta}_{t} + 0.01\\,\theta_{t+1},

providing a slowly evolving historical reference. The student is regularized toward this teacher using

L=LCE+0.01,DKL(πθ,∣,πθˉ).\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + 0.01\\,D_{\mathrm{KL}}\left(\pi_{\theta}\\,\\|\\,\pi_{\bar{\theta}}\right). where LCE\mathcal{L}_{\mathrm{CE}} is token-level cross-entropy loss

The results are shown below using Qwen3-0.6B-Base and compared with ESRL (mean value across three seeds). As the results show, increasing the KL term and applying the EMA method can help alleviate catastrophic forgetting. On the other hand, these stronger constraints also prevent the student from exploring a broader parameter space, which may limit further adaptation.

MATH-500

StepEMAESRL
50014.048.67
100016.026.00
150017.020.33
200014.019.33
250015.016.33
300012.017.33

MMLU

StepEMAESRL
50046.046.83
100045.549.50
150046.549.00
200047.552.83
250046.547.67
300045.048.17

MedQA

StepEMAESRL
50024.026.00
100024.029.83
150025.532.67
200026.031.83
250027.535.83
300028.533.83

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.