- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ESRL for label-free test-time adaptation and continual learning. A population of parameter-perturbed policies generates responses around a periodically refreshed anchor. Majority agreement supplies group-relative advantages, which train a central student through an advantage-weighted likelihood objective with reference-policy regularization. Experiments cover four backbones and nine benchmarks across mathematics, general knowledge, and medicine. A three-task sequential experiment studies forgetting, while additional analyses examine population diversity, perturbation strength, optimization choices, and computational cost.
Critical strengths
- ESRL achieves the highest reported average accuracy across all four backbones in Table 1, with gains extending to related evaluation benchmarks. This breadth makes the empirical contribution more convincing than an improvement on a single adaptation task.
- The parameter-space exploration mechanism has useful supporting evidence: removing perturbations reduces the general-knowledge average from 50.3 to 46.2 (Table 6), while the reported 0.6B cost comparison shows approximately 14% runtime overhead with nearly unchanged peak memory.
Critical weaknesses
- The central continual-learning claim is not sufficiently validated. Figure 3 covers one small backbone and one task order, without a matched zero-perturbation control or repeated-seed results for that sequence. The broader single-domain results do not isolate whether population exploration reliably improves retention across tasks.
- Evaluation details needed to interpret the reported gains are missing: MATH-500/MMLU split construction and sizes, evaluated subset counts, and checkpoint-selection rules are unspecified. Given the substantial training-time fluctuations, these omissions prevent a clear assessment of selection effects and reproducibility and weigh against acceptance in the current form.
- Figure 3 shows substantial deterioration during the first task, before any task switch. Without a same-task continuation control and forgetting measured from each pre-switch checkpoint, the contribution of reduced cross-task interference is difficult to separate from differences in prolonged self-training stability.
Questions for authors
-
Please specify the adaptation/evaluation splits and evaluated sample counts for MATH-500, MMLU, and MedQA. How were checkpoints, stopping points, and hyperparameters selected, and did any of these decisions use held-out answer accuracy, particularly given the performance fluctuations in Figure 3?
-
Does the continual-retention advantage persist against the otherwise identical zero-perturbation variant across seeds and task orders under matched training budgets? The existing perturbation ablation and three-seed study concern single-domain adaptation, so they do not directly establish the robustness of the continual-learning result.
-
In Figure 3, how much of the earlier-task degradation occurs before versus after each task switch? Reporting forgetting relative to the pre-switch checkpoint, alongside continued adaptation on the same task, would help separate cross-task interference from deterioration caused by prolonged self-training.
-
In Equations (5)-(6), trajectories are sampled from perturbed anchor policies rather than the current student. Under this sampling distribution, what KL quantity does the unweighted k3 term estimate, or is it intended as a heuristic regularizer? How is behavior-policy mismatch controlled as the student changes within a block?
-
Does a "clear majority" mean more than half of the valid responses or a unique plurality, and how are ties and unparsable responses handled? Can retrospective measurements of accepted-group frequency and pseudo-label accuracy clarify whether consensus remains reliable during continual adaptation?
8 comments
We sincerely appreciate your time and thoughtful feedback (2)
Q4
what KL quantity does the unweighted k3 term estimate, or is it intended as a heuristic regularizer? How is behavior-policy mismatch controlled as the student changes within a block?
The KL term is designed as a heuristic regularizer to prevent the student model from deviating too far from the initial policy. The block-wise design is introduced to control the policy mismatch that would arise if the original model were used to continuously update the evolving student. As shown in Table 2, the mismatch introduced within a single block does not cause substantial differences in accuracy when the block size is within 20. Overall, a block size within this range provides a better balance between computational efficiency, exploration stability, and alignment with the evolving student.
Q5
Does a "clear majority" mean more than half of the valid responses or a unique plurality, and how are ties and unparsable responses handled? Can retrospective measurements of accepted-group frequency and pseudo-label accuracy clarify whether consensus remains reliable during continual adaptation?
The “clear majority” refers to a unique plurality. Tied groups are not counted as accepted groups; in such cases, no pseudo-label is assigned and the group is excluded from the subsequent policy update. The accepted-group frequency is shown below. In addition, we report the pseudo-label accuracy, which is estimated at each checkpoint as the number of correct pseudo-labels divided by the number of accepted groups. Overall, the pseudo-label accuracy remains relatively stable throughout continual training, indicating that the consensus signal remains reliable during adaptation.
| Task | Benchmark | Percentage |
|---|---|---|
| 1 | MATH-500 | 80.2% |
| 2 | MMLU | 81.5% |
| 3 | MedQA | 84.9% |
| Training Step | MATH-500 Pseudo-label Accuracy | Traning Step | MMLU Pseudo-label Accuracy | Training Step | MedQA Pseudo-label Accuracy |
|---|---|---|---|---|---|
| 100 | 48.2% | 1100 | 52.70% | 2100 | 34.83% |
| 200 | 53.7% | 1200 | 49.33% | 2200 | 37.65% |
| 300 | 43.9% | 1300 | 46.15% | 2300 | 39.47% |
| 400 | 45.1% | 1400 | 54.55% | 2400 | 40.51% |
| 500 | 50.6% | 1500 | 55.26% | 2500 | 34.12% |
| 600 | 44.6% | 1600 | 45.76% | 2600 | 37.04% |
| 700 | 44.3% | 1700 | 47.06% | 2700 | 37.36% |
| 800 | 46.8% | 1800 | 51.43% | 2800 | 37.50% |
| 900 | 44.3% | 1900 | 50.67% | 2900 | 36.90% |
| 1000 | 39.2% | 2000 | 52.05% | 3000 | 35.87% |
We would like to gently follow up on our previous response. We would greatly appreciate it if you could let us know whether our clarifications have addressed your concerns. Thank you again for your time and valuable feedback.
Kind regards
The authors of ESRL
We sincerely appreciate your time and thoughtful feedback (1)
W1 and Q2
We thank the reviewer for raising this point. The following results show the continual learning performance across three seeds for ESRL, TTRL, and ESRL (Zero-Perturbation) using the Qwen3-0.6B-Base model. For ESRL (Zero-Perturbation), no parameter perturbation is applied, so the trajectories are sampled from the same anchor policy and therefore lack parameter-space exploration, resulting in narrower exploration coverage. Overall, ESRL shows the same trend as reported in the main draft.
MATH-500
MMLU
MedQA
W2 and Q1
Thank you for raising this point. We clarify the adaptation/evaluation protocol and model-selection procedure below.
Data splits and sample counts. We use a fixed split (data seed 0) shared across all methods and training seeds. For MATH-500, 400 examples are used for adaptation and 100 disjoint examples for evaluation. For MMLU, 500 examples are used for adaptation and 200 disjoint examples for evaluation. For MedQA, we use 500 examples from the training split for adaptation and 200 examples from the test split for evaluation. Thus, there is no example-level overlap between adaptation and evaluation. MATH-500 and MMLU therefore correspond to our held-out subsets rather than the untouched official test sets.
Checkpointing and stopping. Training uses a fixed budget with no validation-based early stopping: 1,000 training step udates per domain and 3,000 traning updates in total for MATH-500 → MMLU → MedQA. Checkpoints are stored every 100 visits, while the dense continual-learning analysis is reported at predefined intervals. The stage-boundary checkpoints at 1,000, 2,000, and 3,000 accumulated training updates are used rather than selecting the locally best-performing checkpoint.
Importantly, held-out accuracy is never used for training, early stopping, checkpoint selection, or hyperparameter updates. Evaluation is performed independently of the adaptation process. Therefore, the fluctuations shown in Figure 3 are diagnostic measurements only and do not affect the optimization trajectory or the reported terminal checkpoints.
W3 and Q3
The following table shows the comparison between continual learning and self-training on MATH-500 under Qwen3-0.6B-Base. In the follwoing table, we show two task swith from MATH-500 to MMLU and from MMLU to MedQA. The accuracy gap between the two settings shows the deterioration caused by the task switch.