acceptodds
Under review as a conference paper at ICLR 2027

SUTA: Towards Online Adaptation for Self-Improving Large Language Models

Abstract

Deployed language models are typically static: experience accumulated while processing one input does not improve their behavior on later inputs. Test-time adaptation offers a practical form of online self-improvement by allowing a model to update persistent parameters from unlabeled deployment data. For autoregressive LLMs, however, treating every token in a self-generated trajectory as an equally reliable update signal can reinforce confirmation bias and cause reasoning collapse. We introduce **S**elective **U**ncertainty-aware **T**est-time **A**daptation (SUTA), a lightweight parametric self-improvement mechanism for autoregressive LLMs. SUTA uses predictive entropy to identify localized uncertainty bottlenecks in its own generations and restricts cumulative adapter updates to these high-entropy positions. An information-gain contrastive objective encourages the updated model to reduce uncertainty through the input context rather than context-free shortcuts, while a frozen-prior preservation term constrains drift during online updates. Across mathematical reasoning, domain knowledge, and coding benchmarks, SUTA consistently outperforms representative test-time optimization baselines on pre-trained and instruction-tuned models. Its gains grow with model scale and remain complementary to inference-time sampling and voting. These results show that selectively resolving uncertainty in self-generated trajectories is an effective and stable route to test-time self-improvement under distribution shift.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Adaptation, Large Language Models, Selective Uncertainty-Aware Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hVF4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that selecting high-entropy tokens for update is more effective for online update of LLMs. Besides, the proposed method also uses input-grounded contrastive regularization and frozen-prior preservation. Experiments validate the effectiveness of the proposed approach.

Critical strengths
  1. The paper provides relatively comprehensive experimental evaluations.
Critical weaknesses
  1. The paper proposes updating the model using high-entropy tokens, which appears to be the main innovation of this work. However, previous studies [1, 2] have already demonstrated that training on high-entropy tokens can improve the reasoning performance of LLMs. Therefore, I feel that the novelty of this work is limited.
  2. The title of the paper is overly broad. Prior work [3] has already explored online adaptation, and this paper also follows that approach. Therefore, I believe the title significantly overstates the actual contribution of this work.

[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS 2025

[2] SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization. arXiv:2511.17938

[3] Test-Time Learning for Large Language Models. ICML 2025

Questions for authors

see weaknesses

1 comment

We thank the reviewer for the comments on the positioning of our contribution and on the title; below we specify the differences from related work.

W1: Limited novelty relative to [1] and [2].

The difference from both works lies mainly in how the supervision signal is formed: 80/20 Rule depends on ground-truth rewards from the environment, and SPINE on a voted self-consistency reward obtained from multiple samples of the same problem; our method requires neither rewards nor voting, with the signal taken directly from trajectories the model generates on streaming data.

Table: Setting comparison with 80/20 Rule and SPINE

Dimension80/20 RuleSPINESUTA (ours)
SettingOffline RL training (real rewards)Test-time RL (multiple samples + voting)Label-free streaming online adaptation
SupervisionGround-truth rewards from the environmentVoted self-consistency rewardSelf-generated trajectories (single sample, no voting)
SelectionHigh-entropy minority tokensEntropy-band regularizationHigh-entropy positions filtered by threshold τ\tau
Extra components—Entropy-band constraintNegative-prompt contrastive term + KL to a frozen reference
GoalImprove RL training efficiencyImprove single-sample reasoningAccumulate experience over an unlabeled stream while limiting forgetting

Within this setting, our work makes two contributions: (a) it carries the observation about high-entropy positions over to label-free online adaptation and specifies the conditions under which it holds—positions the model has already resolved are not updated, avoiding the injection of bias; and (b) it introduces a negative-prompt contrastive term and a frozen-reference constraint, targeting context use and capability forgetting in label-free self-training respectively (Tables 3 and 10). On the gain of LcontrastL_{\mathrm{contrast}}: adding it on top of Ladapt+LklL_{\mathrm{adapt}} + L_{\mathrm{kl}} raises accuracy from 73.6% to 75.2% (+1.6 points); added on top of LadaptL_{\mathrm{adapt}} alone it gives 1.0 point (73.0% → 74.0%, Table 3). This magnitude is consistent with its role as a constraint that limits degradation, whose effect shows up in output behaviour and stability (Tables 10 and Figure 4) rather than in the average-accuracy increment.

W2: The title is overly broad.

We understand the reviewer's concern about the scope of the title, and would like to explain how it was chosen. The title follows from an experimental observation: under the same label-free streaming setting, SUTA is at or above Vanilla on all 8 benchmarks, whereas the compared methods degrade substantially—TLM (i.e. [3]) on Qwen3-4B-Base drops from 22.1% to 1.86% on GPQA, from 65.9% to 38.2% on MBPP+, and from 63.0% to 53.0% on MATH-500. It is precisely this contrast that led us to the possibility of self-improvement during deployment, which earlier methods do not exhibit. "Towards Online Adaptation" therefore expresses that this direction is viable and worth pursuing; it does not claim that online adaptation is introduced here, nor that the problem has been solved.

Our specific novelty claims are stated in the three contributions in Section 1: selective updating at high-entropy positions, the information-gain contrastive term, and the frozen-prior constraint. None of them concerns the online-adaptation setting itself; rather, within that setting they determine where updates are applied and how accumulated drift is limited. If our method merely followed [3], the two should behave similarly under the same setting and budget; the results in Table 1 show that this is not the case, and the difference comes from the selectivity of the updated positions.

Best regards

Reviewer_trxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SUTA (Selective Uncertainty-aware Test-time Adaptation), an online test-time adaptation framework for autoregressive LLMs. During inference over sequential batches, SUTA generates responses, identifies token positions where predictive Shannon entropy exceeds a preset threshold τ\tau, and updates a persistent LoRA adapter solely on these high-entropy tokens. The training objective integrates selective negative log-likelihood on self-generated tokens, an information-gain contrastive loss comparing the original prompt against a neutralized negative prompt, and a reference cross-entropy constraint against the frozen pre-trained model. Evaluations cover mathematical reasoning, domain knowledge, and coding benchmarks across the Qwen and Llama model families.

Critical strengths
  • Exploring token-level feedback selection rather than uniform trajectory optimization addresses an important practical question in online test-time adaptation.
  • The preliminary analysis in Figure 1 effectively illustrates that indiscriminately minimizing entropy or reinforcing low-entropy tokens can induce premature stopping and repetitive degeneration.
  • The experimental evaluation covers multiple model scales and reasoning domains, supported by extensive ablation studies and runtime measurements.
Critical weaknesses
  1. Contradictory and ambiguous adaptation protocol. A fundamental ambiguity exists regarding what data drives parameter updates. The main text and Algorithm 1 define SUTA as an online transductive test-time adaptation method where parameters are updated on incoming unlabeled test batches. In contrast, Appendix B.4 explicitly states that the adaptation stream is drawn from the MATH training split to match the evaluation set scale and prevent data leakage. If the models evaluated in Table 1 were adapted on the MATH training split prior to evaluation on out-of-domain benchmarks, the framework operates as supervised/unsupervised domain adaptation rather than test-time adaptation. The manuscript must clarify precisely which adaptation stream was used for each table.
  2. Confounded evaluation under artificial length constraints. All non-thinking evaluations enforce an arbitrary maximum generation limit of 2,048 tokens. For challenging competition mathematics such as AIME24, community feedback results typically consist of responses longer than 4,000 tokens. But Table 7 shows average response lengths of 1,824 to 1,859 tokens for instruct models, meaning a substantial fraction of solutions were prematurely truncated by the decoding cap. The observed gains on competition math may simply reflect regularizing output length within the artificial cutoff rather than improving mathematical deduction. Evaluation under standard unconstrained sequence lengths is necessary to isolate reasoning improvements from truncation artifacts.
  3. Misleading efficiency claims caused by unoptimized rollout throughput. The claim that adaptation accounts for only 3.7% of total runtime relies on an unoptimized generation baseline. In Table 4, generating responses for a batch of 8 samples on two RTX 4090 GPUs consumes 132.8 seconds, corresponding to an aggregate decoding throughput of under 100 tokens per second. Modern serving engines (such as vLLM or SGLang) deliver several times higher throughput for 4B models on identical hardware. Because the generation denominator is inflated by decoding latency, the 5.1-second adaptation step appears negligible, whereas in an optimized inference pipeline it would represent a substantial latency penalty.
  4. Limited methodological novelty and incremental formulation. The insight that high-entropy positions capture reasoning bottlenecks has already been established in LLM optimization literature (e.g., arXiv:2511.17938). SUTA's core update reduces to standard cross-entropy self-training filtered by an entropy mask, combined with a KL regularization term commonly employed in test-time adaptation. In Table 3, adding the contrastive loss LcontrastL_{\text{contrast}} yields an improvement of only 1.2% over Ladapt+LklL_{\text{adapt}} + L_{\text{kl}}, indicating that the primary driver of performance is basic masked self-distillation.
  5. Heuristic hyperparameter selection and threshold transferability. The entropy threshold τ=1.5\tau = 1.5 and contrastive weight λ=0.5\lambda = 0.5 are tuned directly on MATH-500, which is simultaneously a headline evaluation benchmark. An unnormalized absolute Shannon entropy threshold lacks theoretical invariance across differing vocabulary sizes, such as Qwen (around 152k tokens) versus Llama ( around 128k tokens), and across varying decoding temperatures. The paper provides no principled or validation-free procedure for selecting τ\tau in real-world deployment on unseen tasks.
  6. Marginal improvements on low-sample benchmarks. On small benchmark sets like AIME24 (30 problems) and AMC23 (40 problems), individual problems account for 3.33% and 2.50% of the total score, respectively. Reported gains—such as Qwen3-4B-Base improving from 10.0% to 13.3% on AIME24—correspond to solving exactly one additional problem across 3 random seeds. Without statistical significance tests or confidence intervals, these small shifts cannot be distinguished from evaluation noise.
Questions for authors
  1. Exactly what data stream is used for adaptation in Table 1 and Table 2: the evaluation test set itself, or samples drawn from the MATH training split?
  2. What proportion of generations reached the 2,048-token limit on AIME24 and AMC23 for both baseline and adapted models, and do the performance advantages persist when the maximum length is expanded to 16384 tokens?
  3. How does the relative latency overhead of the adaptation step change when rollout generation is executed using an optimized inference engine such as vLLM?
  4. How should practitioners select the absolute threshold τ\tau when encountering a novel target domain without ground-truth validation labels, particularly when migrating across model families with different vocabulary sizes?
  5. On AIME24 and AMC23, what is the exact number of unique problems corrected versus degraded across seeds, and are the improvements statistically significant under a paired bootstrap test?
3 comments

We thank the reviewer for the careful comments on the adaptation protocol, the evaluation setup and the efficiency analysis; each point is clarified below.

W1: Contradictory and ambiguous adaptation protocol.

The procedure described in the main text and Algorithm 1 is the one actually used: the adaptation stream for Tables 1 and 2 is the evaluation test set itself, processed batch by batch in streaming order. Each batch is first decoded by the pre-update model and its predictions recorded, after which the adapter is updated on that batch's unlabeled self-generated trajectories; results for batch ii are therefore unaffected by the update at batch ii, and gains come only from accumulation over previous batches. This is consistent with Algorithm 1, and there is no "adapt on the MATH training split, then evaluate" procedure; hence no data leakage is involved and the test-time adaptation positioning is unchanged.

Appendix B.4 uses a separate adaptation stream: adaptation on the MATH training split, with evaluation on data unseen during adaptation, in order to separate the in-distribution and out-of-distribution cases. This is independent of the protocol of the main tables: the main tables adapt on the test batches being evaluated, whereas B.4 holds out data and examines performance on samples that took no part in optimization. Neither stream leaks ground-truth labels.

W2: Confounded evaluation under artificial length constraints.

In non-thinking mode the responses do not need to be long, so we use a uniform 2,048-token budget for all non-thinking tasks. We also re-ran the evaluation with a 16,384-token budget; as the table below shows, the extra budget brings no appreciable gain, and SUTA remains better than Vanilla under both settings. The gains on competition mathematics are therefore not an artifact of the 2,048-token cutoff.

Table: Accuracy on AIME 2024 / 2025 / 2026

ModelAIME 2024AIME 2025AIME 2026
Qwen3-4B-Base13.3±010.0±010.0±0
Qwen3-4B-Base + SUTA15.6±1.5710.0±012.2±1.57
Qwen3-4B16.7±010.0±013.3±0
Qwen3-4B + SUTA26.7±018.9±1.5723.3±0

W3: Misleading efficiency claims caused by unoptimized rollout throughput.

The measurement in Table 4 is based on the baseline implementation of per-sample HF decoding (batch size 8, two RTX 4090s): 132.8 s for generation and 5.1 s for adaptation, i.e. 3.7%. After switching generation to vLLM, under the same batch size, hardware and generation length, generation drops to 56.4 s and adaptation accounts for 8.3%. Even with optimized generation throughput, adaptation accounts for only about one tenth of total runtime, which is an acceptable overhead.

W4: Limited methodological novelty and incremental formulation.

The difference between our method and 80/20 Rule and SPINE lies in the setting and the supervision signal: the former two rely on ground-truth rewards and self-consistency rewards respectively, whereas we adapt under label-free, reward-free streaming conditions, with supervision coming only from the model's own trajectories, plus a negative-prompt contrastive term and a frozen-reference constraint.

Table 3 shows that the main driver of performance is entropy-masked self-training, consistent with the conclusion of Table 11 that selective updating outperforms full updating; the role of LcontrastL_{\mathrm{contrast}} lies in output behaviour and stability—outputs adhere more closely to the problem conditions in Figure 4, and correct-to-wrong flips drop from 40 to 20 in Table 10. The contrastive term is a constraint that limits degradation, and its value is not reflected in the average-accuracy increment.

Table: Setting comparison with 80/20 Rule and SPINE

Dimension80/20 RuleSPINESUTA (ours)
SettingOffline RL training (real rewards)Test-time RL (multiple samples + voting)Label-free streaming online adaptation
SupervisionGround-truth rewards from the environmentVoted self-consistency rewardSelf-generated trajectories (single sample, no voting)
SelectionHigh-entropy minority tokensEntropy-band regularizationHigh-entropy positions filtered by threshold τ\tau
Extra components—Entropy-band constraintNegative-prompt contrastive term + KL to a frozen reference
GoalImprove RL training efficiencyImprove single-sample reasoningAccumulate experience over an unlabeled stream while limiting forgetting

W5: Heuristic hyperparameter selection and threshold transferability.

We use an absolute entropy threshold of τ=1.5\tau = 1.5 nats rather than a form normalized by vocabulary size, because entropy in nats has a direct meaning: τ=1.5\tau = 1.5 corresponds to a perplexity of e1.5≈4.5e^{1.5} \approx 4.5, i.e. "the model is still undecided among about 4–5 candidates". This criterion measures the concentration of the distribution and is independent of the size of the vocabulary space, so dividing by log⁡∣V∣\log |V| is unnecessary.

The effect of vocabulary size is indeed negligible: for Qwen (≈152k\approx 152\mathrm{k}) and Llama (≈128k\approx 128\mathrm{k}), log⁡∣V∣\log |V| is 11.9311.93 and 11.7611.76 respectively, a difference of 0.170.17 nats; under a normalized conversion, the corresponding absolute thresholds differ by only about 0.020.02 nats, far below the method's tolerance for τ\tau. The results are consistent with this: the same τ=1.5\tau = 1.5 yields stable gains on both vocabularies (Tables 1 and 2).

W6: Are the gains on low-sample benchmarks marginal, and has a significance test been performed?

AIME 2024/2025/2026 are three non-overlapping problem sets, 90 problems in total, so a single problem accounts for 1.11% instead of 3.33%; all three editions use the same hyperparameters and the same adaptation budget.

Comparing problems pairwise (same problem, same seed), Qwen3-4B improves from 12/90 to 20/90, 21/90 and 21/90, i.e. 8, 9 and 9 additional problems answered correctly, with none turning from correct to incorrect. Qwen3-4B-Base goes from 10/90 to 10/90, 12/90 and 12/90, i.e. 0, 2 and 2 additional correct problems, again with no degradation. Since every discordant pair is a one-directional correction, the paired test reduces to a binomial test: the two-sided pp values for Qwen3-4B are 0.008 (8 problems) and 0.004 (9 problems), with 95% confidence intervals of [3.0,14.8][3.0, 14.8] and [3.8,16.2][3.8, 16.2] percentage points estimated from the McNemar variance.

For Base the number of additional correct problems is 0 to 2, with pp between 0.25 and 1.0. The direction is consistent, but the effect is not significant at the scale of 90 problems, and we make no significance claim for Base.

A single edition contains only 30 problems; with a net gain of 3, the two-sided pp cannot fall below 0.25 even when every correction is one-directional, which is why we report the pooled evaluation. As for the number of unique corrected problems: under greedy decoding Vanilla gives identical results across the three seeds, so the union of corrected problems is at least 8 and at most 26; exact de-duplication requires per-problem judgments, and what we report here are the per-seed counts.

Best regards

Thank you for the detailed rebuttal and additional experiments. Your clarifications regarding the adaptation stream (W1), methodological comparisons (W4), entropy threshold normalization (W5), and pooled statistical testing (W6) have largely addressed my questions on those fronts.

However, the new results and claims presented in the rebuttal raise two critical follow-up concerns regarding the validity of the evaluation protocol and the reported vLLM latency measurements:

1.Baseline Discrepancy and Potential Evaluation Protocol Flaws (Follow-up on W2 & W6) Looking closely at the scores reported for Qwen3-4B (Instruct, non-thinking mode) in your rebuttal table, there appears to be a potential issue with your evaluation protocol:

  • According to the official Qwen3 Technical Report (arXiv:2505.09388, Page 18), Qwen3-4B in non-thinking mode achieves 25.0 on AIME24 and 19.1 on AIME25.
  • In contrast, your Vanilla baseline only achieves 16.7 on AIME24 and 10.0 on AIME25, while Qwen3-4B + SUTA reaches 26.7 and 18.9, respectively.

Notice that the official unadapted baseline scores (25.0 / 19.1) are higher than your reported Vanilla scores (16.7 / 10.0) and are nearly identical to your post-SUTA scores (26.7 / 18.9). This discrepancy severely undermines confidence in the empirical results and points to a much simpler alternative hypothesis: SUTA's observed gains may primarily stem from adapting the model to recover from a suboptimal or misconfigured evaluation protocol (e.g., mismatched chat templates, prompt formatting, or greedy decoding vs. the official generation configuration), rather than unlocking new latent reasoning capabilities beyond the base model's official performance.

While adapting an LLM to an unexpected or degraded evaluation setup at test time still technically falls within the scope of TTA, this is clearly different from the central claim of the paper. I strongly urge the authors to inspect the discrepancy against arXiv:2505.09388 and re-run the experiments strictly following the official Qwen3 evaluation settings.

2. Questionable vLLM Throughput Measurements (Follow-up on W3) In your response to W3, you state that switching generation to vLLM (batch size 8, two RTX 4090 GPUs, Qwen3-4B-Base) reduces generation time to 56.4 seconds per batch. I find this latency surprisingly high.

For Qwen3-4B-Base on MATH-500, Table 7 reports an average response length of ~637 tokens (and even in the worst case where a sequence hits the 2,048-token cap, a batch of 8 is at most 16,384 tokens, or ~5,100 tokens on average). A generation time of 56.4 seconds for a 4B dense model on two RTX 4090s translates to an aggregate throughput of roughly 90–290 tokens/s, which is several times slower than standard vLLM performance on a single RTX 4090 for a 4B model.

To help verify the reported 56.4s runtime, please provide the config of your vLLM setup so that I can reproduce and verify the throughput on my own RTX 4090 GPUs:

  1. The input prompt / template format fed into vLLM.
  2. The parallelism strategy across the two RTX 4090 GPUs (e.g., tensor_parallel_size=2, data_parallel_size=2, or single-GPU execution).
  3. The exact vLLM version, launch arguments, and SamplingParams (specifically, whether CUDA graphs were disabled via enforce_eager=True, gpu_memory_utilization, max_model_len, and dtype).
Reviewer_1Lh4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SUTA, a label-free method for online adaptation of large language models. For each incoming batch, the model first generates responses and then updates a lightweight LoRA adapter using its own generated trajectories. The updated adapter parameters are retained for subsequent batches.

SUTA combines three components: (1) self-training restricted to token positions whose predictive entropy exceeds a threshold; (2) a contrastive hinge loss comparing predictive entropies under the original and PAD-neutralized prompts; and (3) regularization toward the frozen base model’s predictive distribution. These components are designed to focus updates on uncertain token positions, encourage the use of the input context, and preserve existing capabilities during adaptation.

Experiments cover mathematical reasoning, scientific question answering, and code generation in multiple families and models. The reported results show improvements in the average accuracy in the model configurations evaluated. Additional analyzes examine the effects of loss components, LoRA target modules, negative prompt construction, token selection, stream order, and batch size.

Critical strengths

S1. Consistent improvements across model configurations

Table 1 shows that SUTA achieves a higher reported average accuracy than Vanilla across all four Qwen3 configurations, covering both Base and Instruct models at the 4B and 8B scales. For example, on MATH500, Qwen3-4B-Base achieves 75.4% accuracy with SUTA, compared with 63.0% for Vanilla. Table 2 shows the same pattern of higher average accuracy for Qwen3.5 and Llama3.1. These results provide evidence that the benefits extend beyond a single model configuration, supporting the practical value of the proposed label-free online adaptation method.

S2. Ablations examine key design choices and provide evidence of reduced degradation during adaptation

The paper includes several relevant controls. Table 3 evaluates the adaptation loss, the KL regularizer, the contrastive term, and their combination. Table 6 examines the choice of LoRA target modules: SUTA still achieves 72.4% accuracy on MATH500 when updates are restricted to the query and value projections, suggesting that its gains cannot be attributed solely to updating a broader set of modules (Table 3, p. 7; Table 6, p. 17).

Table 10 provides useful evidence of stability: the complete objective reduces the number of initially correct answers that become incorrect from 40 to 20. This offers direct empirical support for the role of the combined regularization in limiting degradation caused by self-training. For online adaptation, this analysis provides more direct evidence of the preservation of existing capabilities than final average accuracy alone (Table 10, p. 20).

Table 11, Table 12, and Appendix C.8 additionally examine updating all tokens, alternative negative prompt constructions, batch size, and stream order.

Critical weaknesses

W1. The evidence does not sufficiently establish the claimed mechanisms.

The core concern is that the contrastive term acts, by construction, as gated entropy minimization, while its claimed role in improving semantic use of the input context remains insufficiently established. Specifically, the negative-prompt branch in Eq. (3) is detached (Sec. 3.2). For a fixed generated trajectory and selected set (S), the contrastive loss gradient away from the hinge boundary is

∇θLcontrast=1∣S∣∑t∈S1 ⁣[Hpos,t>Hneg,t]∇θHpos,t,\nabla_\theta L_{\mathrm{contrast}} = \frac{1}{|S|}\sum_{t\in S} \mathbb{1}\!\left[H_{\mathrm{pos},t} > H_{\mathrm{neg},t}\right] \nabla_\theta H_{\mathrm{pos},t},

where Hpos,tH_{\mathrm{pos},t} and Hneg,tH_{\mathrm{neg},t} are the predictive entropies under the original and negative prompts, respectively.

The term therefore uses negative-prompt entropy to minimize gate entropy on the positive branch. A model that ignores the prompt under the same response prefix can have Hpos,t=Hneg,tH_{\mathrm{pos},t} = H_{\mathrm{neg},t} at all selected positions and incur zero contrast loss. The objective thus admits context-insensitive zero-loss solutions. Its one-sided form also provides no gradient when Hpos,t<Hneg,tH_{\mathrm{pos},t} < H_{\mathrm{neg},t}. The illustrative token-level examples in Appendix D.1 do not resolve this ambiguity or establish the stronger claim that the objective "enforces strict context adherence."

The interpretation of HnegH_{\mathrm{neg}} as a neutral baseline also requires further validation. PAD replacement may affect predictive entropy through an atypical input structure and removal of semantic information. Table 12 demonstrates sensitivity to negative-prompt construction, but does not isolate improved context use from changes in the entropy baseline and hinge activation pattern.

Similarly, the enrichment of natural-language tokens in Table 9 does not establish their causal importance to reasoning. Table 11 supports selective updating over updating all tokens, but does not isolate the importance of selecting high-entropy positions from the effects of restricting and reweighting the updated positions.

W2. Baseline competitiveness and tuning need clearer justification.

TLM and TTSV fall substantially below Vanilla in several settings in Table 1. For Qwen3-4B-Base, TLM achieves 1.86% accuracy on GPQA compared with 22.1% for Vanilla, while TTSV achieves 3.33% on AIME24 compared with 10.0% for Vanilla. These comparisons demonstrate that SUTA avoids the degradation observed with these baselines, but provide a more limited basis for establishing an advantage over competitive adaptation alternatives. SUTA’s improvements over Vanilla remain meaningful, although their magnitude varies across tasks and models. Improvements on the coding benchmarks are generally modest, as illustrated by LCB v5 (13.5% to 14.5%) and MBPP+ (65.9% to 67.3%) on Qwen3-4B-Base.

Appendix B.1 states that the published default settings were used for TLM and TTSV. Please clarify whether any model-specific tuning was performed, how the suitability of these settings was assessed, and whether comparable tuning budgets and validation procedures were used across methods. A comparison with a directly related entropy-minimization method discussed in Appendix A, where compatible with the same online setting and adaptation budget, would further strengthen the empirical evaluation.

Questions for authors

Q1. What evidence supports the context-grounding interpretation?

With the negative-prompt branch detached, Eq. (3) uses negative-prompt entropy to minimize gate entropy on the original branch. What evidence connects this gating to a more faithful use of task-relevant context? Semantic counterfactual tests, in which changing a key problem condition requires a different correct answer, would further clarify the context-grounding interpretation.

Q2. Does high-entropy selection provide benefits beyond restricting the updated positions?

Table 11 supports selective update over updating all tokens, while Table 9 demonstrates enrichment of particular token categories. Does high-entropy selection still outperform random token selection when the number of selected tokens per trajectory, loss normalization, and update budget are matched? This comparison would help establish the value of the entropy-based selection rule. What additional evidence supports the interpretation of the selected positions as “reasoning-critical" beyond their token-category composition?

Q3. How robust are the baseline comparisons to hyperparameter selection?

Appendix B.1 specifies the published settings for TLM and TTSV, while Fig. 4 examines SUTA’s hyperparameters. How were the final settings selected for each method and what data were used for that selection? Given that some baseline configurations substantially underperform Vanilla in Table 1, do these degradations persist in reasonable alternative settings? A representative comparison using comparable tuning budgets and a common validation procedure would help assess the robustness of the relative advantage of SUTA.

Q4. How robust are the gains on the small competition benchmarks?

AIME24 and AMC23 contain only 30 and 40 problems, respectively. What do the ± values in Table 1 represent and which randomness sources vary between the three seeds? For small differences such as 13.3% versus 10.0% on AIME24, how should the gains over Vanilla be interpreted in light of both run-to-run variation and uncertainty associated with the limited problem sets?

Could the authors also clarify why the AIME evaluation is limited to the 2024 edition and whether AIME25 and AIME26 were considered? Where feasible, evaluating these additional editions under the same online adaptation protocol would help assess whether improvements are consistent between competition years.

3 comments

We thank the reviewer for the detailed analysis of the mechanism and the supporting evidence; these comments helped us delimit the claims of the paper more precisely.

W1 / Q1: The evidence does not sufficiently establish the claimed mechanisms.

Before turning to the mechanism, two aspects of the setup matter.

(i) Protocol: each batch is decoded and its predictions recorded by the model before the adapter is updated on that batch's trajectories, so results for batch ii are unaffected by the update at batch ii and gains can only come from earlier batches. Since later problems are unseen, the gains are not self-reinforcement of the current trajectory—the opposite of the confirmation-bias concern in the introduction.

(ii) Decoding: all main experiments use greedy decoding, which takes the argmax at each step, so there is no "reduced sampling variance" pathway for entropy minimization to exploit; sharpening the distribution along the original path would not change the output. What changes is the relative ranking of candidate continuations at high-entropy positions, i.e. a re-ranking at the chain level rather than a global rise in confidence.

The evidence is consistent with this. In Figure 5, pass@kk is maintained or improved across sampling budgets; had adaptation merely sharpened the distribution, the explored support would shrink and pass@kk would fall with kk. Table 5 agrees: maj@kk gains persist from k=8k=8 to k=64k=64 and in most settings do not decay, i.e. adaptation improves consistency across paths rather than the confidence of one. Conversely, removing position selection and minimizing entropy everywhere (Table 11) collapses into degenerate repetition, and the full objective cuts wrong-to-wrong flips from 40 to 20 while correct-to-correct moves only from 85 to 82 (Table 10). The gains thus come from re-ranking paths at informative positions, not from lowering entropy globally.

Three objections follow. Zero-loss solutions: Hpos=HnegH_{\mathrm{pos}}=H_{\mathrm{neg}} is indeed a zero-loss fixed point, but no gradient direction drives parameters toward it—activated positions always lower positive-branch entropy, others carry no gradient—so it is attainable, not attracting. Training signal: the negative branch is forward-only and only supplies the gate HnegH_{\mathrm{neg}}; it enters neither the gradient nor the target, the sole supervision being the model's trajectories under the real prompt, so there is no pathway for learning to ignore context. No gradient when Hpos,t<Hneg,tH_{\mathrm{pos},t}<H_{\mathrm{neg},t}: this is the one-sided definition, which requires real-prompt entropy not to exceed the uninformative one rather than forcing certainty everywhere; at those positions LadaptL_{\mathrm{adapt}} and LklL_{\mathrm{kl}} update as usual.

On HnegH_{\mathrm{neg}} as a baseline: shifting it changes only how often the constraint activates, not its direction. If PAD raises HnegH_{\mathrm{neg}}, the gate triggers less and the term weakens; if it lowers HnegH_{\mathrm{neg}}, it triggers more, but the gradient still lowers real-prompt entropy. A baseline level cannot fabricate context use. Consistently, Table 12 shows sensitivity to how the negative prompt is built—had the term vanished at most positions, this would not matter—and Figure 4 shows outputs adhering more closely to the stated conditions.

"Context adherence" is operational: under the real prompt the selected positions are more concentrated than under the uninformative prompt, and this shows up as subsequent token choices following the problem conditions—an argmax change under greedy decoding. Eq. (3) fixes the direction; Figure 5 fixes that the explored support is not compressed.

W1 / Q2: High-entropy selection is not isolated from restricting and reweighting the updated positions.

On the scope of Table 11: both settings average the loss over the updated positions, so each selected position carries the same weight and the selective setting does not amplify per-position gradients by updating fewer of them; the batch budget, optimization steps and number of updates are also identical, leaving the position set as the only difference. As for the possibility that less updating is itself responsible: Figure 1 shows that the failure of all-token updating occurs specifically at low-entropy positions the model has already resolved—updates at these positions cause repetition and premature termination, whereas updates elsewhere do not produce this behaviour. The difference between the two settings thus tracks whether low-entropy positions are updated, not the total update magnitude; were the gains merely a smaller-update effect, degradation would vary smoothly with update magnitude rather than being tied to the nature of the positions.

On Table 9: the enrichment of natural-language tokens points to the same conclusion as Figure 1—the high entropy positions the model has not yet resolved coincide with the tokens that carry the reasoning. "Reasoning-critical" is an operational definition, namely positions whose predictive entropy exceeds τ\tau and where the model has not yet committed to the next step; we do not claim per-token causal importance.

W2 / Q3: Baseline competitiveness and tuning need clearer justification.

TLM and TTSV are run with the best hyperparameters reported in their own papers, with no additional model-specific tuning and without changing their default learning rate, update steps or adaptation budget; these settings reproduce the performance reported under their original evaluation conditions. SUTA's τ\tau and λ\lambda are likewise selected on the same validation set (MATH-500) over a grid of the same scale; the candidate grids and selection procedure for each method are given in Appendix B.1.

The large gap between the baselines and Vanilla is unrelated to tuning and stems from how they update. Both TLM and TTSV update all positions indiscriminately, without distinguishing the role of different positions: low-entropy positions the model has already resolved are also optimized, and the resulting drift is precisely the source of the repetition and premature termination in Figure 1; moreover, neither objective contains a reference term that constrains this drift, which therefore accumulates over the stream. Internal controls support this reading: after removing the KL term, SUTA itself degrades substantially (Tables 3 and 10), showing that the absence of a reference constraint suffices to cause collapse; all-token updating, which includes every position, is also systematically worse than selective updating (Table 11). The degradation of these baselines is thus a structural consequence of this update rule in this setting, not a matter of poor hyperparameter choice.

The absolute gains on coding benchmarks are smaller than on mathematical tasks but consistently positive; we treat them as supplementary evidence, with the main conclusions resting on MATH and AIME.

For the entropy-minimization method discussed in Appendix A, the comparison under the same online setting and the same adaptation budget is already given in the main text. "All-token updating" in Table 11 is exactly this method with position selection removed, applying entropy minimization over the entire trajectory, and it is systematically worse than updating only high-entropy positions; the ablations in Table 3 give the two further simplifications—removing the reference constraint and removing the contrastive term—both of which also fall below the complete objective. The two main simplifications of this family of methods have therefore already been measured directly in our setting, with a consistent conclusion: position selection and the reference constraint each correspond to measurable gains rather than a formal difference only.

Q4: Meaning of ±\pm, sources of randomness, and AIME editions.

The ±\pm in Table 1 denotes the standard deviation over 3 random seeds; the problem sets are fixed, and the variation across seeds comes from adapter initialization and the batch order of the adaptation stream.

Regarding the budget on AIME: since the capability of the non-thinking mode is already saturated at 2,048 tokens, the main text uses this budget uniformly. Here we raise the generation budget to 16,384; results are shown below, covering the 2024, 2025 and 2026 editions, which share a single set of hyperparameters without per-edition tuning.

Table: Accuracy on AIME 2024 / 2025 / 2026

ModelAIME 2024AIME 2025AIME 2026
Qwen3-4B-Base13.3±010.0±010.0±0
Qwen3-4B-Base + SUTA15.6±1.5710.0±012.2±1.57
Qwen3-4B16.7±010.0±013.3±0
Qwen3-4B + SUTA26.7±018.9±1.5723.3±0

After removing the length constraint, SUTA is at or above Vanilla on all three editions, and Qwen3-4B improves on all three; the direction matches the thinking-mode results in the main text, so the AIME gains cannot be attributed to the 2,048-token cutoff.

Best regards

Thank you for the clarifications and additional experiments. Could you compare high-entropy selection with random selection using the same number of tokens per trajectory, loss normalization, and update budget, and clarify whether the MATH500 examples used for tuning were separate from those used for evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.