acceptodds
Under review as a conference paper at ICLR 2027

Let it Calm: Exploratory Annealed Decoding for Verifiable RL in Mathematics

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs), yet its success hinges on effective exploration. An ideal exploration strategy must navigate two fundamental challenges: it must preserve sample quality while also ensuring training stability. While standard fixed-temperature sampling is simple, it struggles to balance these competing demands, as high temperatures degrade sample quality and low temperatures limit discovery. In this work, we propose a simpler and more effective strategy, Exploratory Annealed Decoding (EAD), grounded in the insight that exploration is most impactful on early tokens which define a sequence's semantic direction. EAD implements an intuitive *explore at the beginning, exploit at the end* strategy by annealing the sampling temperature from high to low during generation. This dynamic schedule encourages meaningful, high-level diversity at the start, then gradually lowers the temperature to preserve sample quality and keep the sampling distribution close to the target policy, which is essential for stable training. We demonstrate that EAD is a lightweight, plug-and-play method that significantly improves sample efficiency, consistently outperforming fixed-temperature sampling across various RLVR algorithms and model sizes on mathematical reasoning tasks. Our work suggests that aligning exploration with the natural dynamics of sequential generation offers a robust path to improving LLM reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, RLVR, Sampling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_diexOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Exploratory Annealed Decoding (EAD) to address the difficulty of balancing exploration, response quality, and training stability with fixed-temperature sampling in reinforcement learning with verifiable rewards for mathematical reasoning. Its core principle is “explore early, exploit late.” By gradually lowering the sampling temperature, EAD encourages the model to explore different solution paths early and complete subsequent reasoning more consistently. The method also adjusts the cooling rate as training progresses and uses TIS to control gradient fluctuations. Experiments show that EAD can improve accuracy and sample efficiency across multiple models and reinforcement learning algorithms, and can also be used with majority voting at inference time.

Critical strengths

Intuitive design and simple implementation. EAD links exploration intensity to generation position, implementing “explore early, exploit late” through gradual temperature reduction. This design is supported by entropy dynamics and forking experiments. The schedule itself introduces little computational overhead and is easy to integrate into existing training pipelines.

Experimental support for performance gains and component contributions. Single-sample accuracy improves by 1.70 and 0.93 percentage points on Qwen-1.5B and Llama-1B, respectively. Ablation studies involving reverse scheduling, fixed decay rates, and TIS also help assess the contributions of individual components.

Good compatibility and flexible use. The method has been evaluated with DAPO, GRPO, and EntropyMech, achieving good results with relatively few training rollouts. It can also be applied directly to off-the-shelf models at inference time to improve majority-voting accuracy.

Critical weaknesses

The performance claims are inconsistent with the experimental reporting. The main text states that all comparisons are statistically significant, but Table 3 reports a Pass@16 of 46.76% for EAD on Llama, below the best fixed-temperature baseline of 49.48%. The authors also acknowledge that the two are statistically comparable. Furthermore, the main text’s claim of a Pass@16 advantage for EAD with TIS on Llama is inconsistent with Figure 3. Overall, EAD does not demonstrate a clear advantage over the strongest fixed-temperature baseline on Llama in either Pass@16 or Worst@16.

Evidence of generalization across tasks and model scales is limited. The experiments primarily focus on mathematical tasks and models ranging from 1B to 8B parameters. In code generation or agentic tasks, critical decisions may occur later in generation, and continued cooling could suppress necessary exploration. Further evaluation is needed to determine whether the current position-based schedule can accommodate these scenarios.

Questions for authors

In the discussion of Pass@16, the main text states that “EAD (w/ TIS) also outperforms on the Llama model,” but Figure 3 shows that EAD-TIS has lower Pass@16 than the fixed-temperature baselines on Llama during the later stages of training. Could the authors clarify whether this discrepancy reflects an error in the model name or figure legend, and explain the respective effects of TIS on Llama’s Pass@16 and Worst@16?

Could the authors provide evaluations on code generation, agentic tasks, or larger models to examine whether EAD’s position-based temperature annealing suppresses necessary exploration when critical decisions occur late in generation, and whether its performance advantages are maintained?

2 comments

Thank you for the careful reading. We address your concerns below and have updated the paper accordingly.

W1 & Q1. Llama results: main text vs. Fig. 3 and Table 3

Sorry for the imprecise wording in Section 5.2, and thank you for catching it. On Llama, EAD with TIS outperforms every GRPO baseline on Pass@16 (46.76 vs. at most 42.06) and Worst@16 (31.65 vs. at most 30.52). Against the fixed-temperature DAPO runs, it matches the strongest one within the confidence intervals. We revised the paragraph to:

On Qwen-2.5-Math-1.5B, EAD gives the best Pass@1, Pass@16, and Worst@16. On Llama-3.2-1B-Instruct, EAD gives the best Pass@1 (40.73 vs. 39.80), outperforms all GRPO baselines, and matches the strongest fixed-temperature DAPO baseline on Pass@16 and Worst@16 within the confidence intervals.

Significance. The ±0.01 statement applies to Qwen-2.5-Math-1.5B, where every bootstrap interval in Table 3 is at most ±0.04. On Llama, the Pass@16 intervals are ±3.8 to ±4.4 and the Worst@16 intervals are ±1.9 to ±2.7, so we now report significance per model. These intervals are eval-set bootstrap on one run. Across runs (with different seeds), the standard deviation on Pass@16 is about ±2.

How TIS affects Llama. Without TIS, EAD reaches Pass@16 49.43, on par with the best DAPO run (49.48), but Worst@16 drops to 23.02 late in training. A few rollouts get extreme importance ratios, and these high-variance updates hurt the weakest samples first. TIS caps the ratios and keeps Worst@16 at 31.65, at the cost of some Pass@16. On Llama, TIS trades some best-case gain for stability. We state this in Section 5.2 and the Fig. 3 caption.

W2 & Q2. Code, agentic tasks, and larger models

As the title states, the paper targets math reasoning. Code and agentic tasks differ in a way that matters for a decoding-focused study. They rely on feedback from a compiler, terminal, or sandbox, usually run inside an agent harness these days. That feedback is appended to the context, and the harness may compact the memory. Both interrupt normal LLM autoregressive generation, and both would confound an analysis of the sampling schedule with the effects of the harness.

We still ran a simplified code setting without a harness: Qwen2.5-Coder-1.5B-Instruct trained with DAPO for 100 steps on Eurus-2 code problems, with unit-test rewards, evaluated on HumanEval+ (HE+) and LiveCodeBench (LCB). All step-100 checkpoints are decoded identically (τ = 1.0, 16 samples for each prompt):

RunHE+ Pass@16LCB Pass@16
DAPO τ = 0.387.843.8
DAPO τ = 1.088.440.2
DAPO τ = 1.286.041.9
EAD (math default)89.642.9

With the same schedule in paper (no code-specific tuning), EAD already gives the best HE+ Pass@16 and beats τ = 1.0 and 1.2 on LCB Pass@16. The new appendix reports all runs, including Worst@16 and four other schedules.

In tasks where key decisions come late, position-only cooling could suppress useful late exploration. Two design choices limit this risk:

  • The floor temperature τmin\tau_{min} > 0 keeps sampling stochastic until the end of the response.
  • The decay rate grows with training steps, so the high-temperature region grows as responses get longer.

For tasks with explicit late decision points, a schedule that re-heats at those points (e.g., before a tool call) is a natural extension. We add this to the Discussion.

On scale, Appendix D gives our largest test: Qwen-3-8B-Base with the DAPO recipe, trained on DAPO-17k and evaluated on AIME 2024. EAD learns faster than fixed-temperature sampling there. The Appendix notes that this run used reduced compute and did not train to convergence.

Thank you again for the detailed feedback.

Thank you for the additional experiments. Your response has addressed my concerns, and I will raise my score to 3.

Reviewer_g1T6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper is about explorative RLVR that adjusts the sampling temperature in rollout. The design principle ``explore early and exploit late'' is built on: 1. the sampling begins with tokens with high entropy, 2. the entropy shrinks gradually when generation continues, 3. regeneration from forking tokens in later tokens hurts more Maj@K than earlier tokens. The method is EAD, an explorative RL method that dynamically adjusts temperature in rollout. The challenges on off-policy can be mitigated by TIS.

Critical strengths

This is a good paper.

  1. The paper empirically demonstrates the effectiveness of adaptive temperature annealing in language models (i.e. 1.5B--7B models) on math reasoning tasks, with a response length of less than 3K tokens by outperforming baselines: constant-temperature decoding.
  2. (Most important): The drawback of EAD is clear: leading to off-policy discrepancy that risks training instability, which makes the introduction of TIS well-motivated, claims are supported with experiments (e.g. Figure 3).
  3. Clear ablation studies.
Critical weaknesses
  1. Lack of explorative RL baselines. The authors already show that EAD is compatible with DAPO. I am interested in EAD's comparison to other explorative RL baselines (e.g., Pass@K RL [1], Dr.GRPO [2], etc). Please consider adding compatibility testing like Sec. 5.3 on wider explorative RL to show generalization of EAD.

  2. Unknown why EAD gets better training performance under Edt background. EDT shows that dynamic temperature decoding enhances diversity. Thus, it is unknown how EAD enhances exploration in RL. Have the authors observed consistently higher Pass@K in training? Is the advantage observed at a higher ceiling? Or a more diverse final reward pattern, e.g., fewer zero advantages and a more balanced 0/1 reward distribution in rollout?

[1] Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

[2] Understanding R1-Zero-Like Training: A Critical Perspective

[3] EDT: Improving Large Language Models’ Generation by Entropy-based Dynamic Temperature Sampling

Questions for authors

Overall, I like this paper, but I have several questions. Authors are expected to discuss drawbacks or advantages of EAD frankly in responding to my questions, its ok if their methods are not optimal:

  1. Q1 questions related to observation and motivation in Sec.3: Although the evidence in Fig. 1 is inspiring, I have additional questions: 1) Does the author observe that entropy shrinks on models beyond Llama and MMLU? 2) Similarly, for Fig. 1b, does the resampling @25th > @200th still hold for wider models and datasets? And the authors may need to add settings for experiments in Sec.3 in the appendix for reproducibility of the experiment.

  2. Q2: Why is EAD good for exploration? See Weakness 2.

  3. Q3: Why not thinking mode? how difficult it is to adapt EAD on thinking mode (>=32,000 context length), for example, Qwen3-8B's RLVR? What are the major challenges here? I am curious about EAD's scalability to long CoT models.

  4. Q4: Similar to settings in Edt, if we apply EAD in test-time sampling, would it outperform Edt?

  5. Q5: Is training with EAD-TIS proven to be more stable than EAD w/o TIS beyond entropy?

3 comments

Thank you for the positive review and the specific questions.

W1. Other exploration-oriented RL methods

EAD changes only how rollouts are sampled, so it runs on top of other RL objectives. Section 5.3 shows that EAD improves both GRPO and EntropyMech [1]. Cui et al. [1] show that encouraging exploration through entropy alone does not work well in RLVR: an entropy bonus either has little effect or makes entropy explode and accuracy drop. Their method balances exploration and exploitation by restricting updates on a few high-covariance tokens. EAD still improves on it, which indicates that EAD is compatible with recent, carefully designed RL methods.

We appreciate the suggestion and ran Dr. GRPO [2]. We chose it because it also relates to generation length: it removes GRPO's per-response length normalization and the std scaling of advantages. EAD produces longer responses (Appendix G), so this tests whether EAD's gains depend on GRPO's length bias. The experiment results are demonstrated below (Pass@16 on MATH500, Qwen-2.5-Math-1.5B):

StepDr. GRPO (τ\tau = 1.0)EAD + Dr. GRPO
5066.368.9
10069.372.5
20071.272.8
35072.773.3

EAD + Dr. GRPO leads Dr. GRPO at every evaluated step from 50 to 350, by up to 3.2 points. The gain therefore survives the removal of both of these GRPO terms.

We leave the EAD combination with Pass@K training [3] to the camera-ready version.

W2 & Q2. Why EAD helps exploration

Yes. On Qwen-2.5-Math-1.5B, EAD keeps the highest Pass@16 for most of training (Fig. 3), and entropy does not collapse (Fig. 5). EAD also improves Pass@1 on both models (Table 3), so the ceiling rises too.

Our explanation: a high early temperature makes rollouts for the same prompt take different approaches, and a low later temperature keeps each one coherent. More prompts then get a mix of correct and wrong rollouts and a non-zero advantage. We do not measure the fraction of zero-advantage prompts, so we present this as a hypothesis.

On EDT [4]: EDT sets the temperature at test time from each token's entropy. EAD uses position and training step, needs no per-token entropy computation, and adds no significant compute or memory overhead to rollout. We revised our description of EDT in the related work.

Q1. Does the Section 3 evidence generalize?

Yang et al. [5] report the same decline in branching factor across several model families and tasks. Our forking test uses DeepSeek-R1-Distill-Llama-8B, a different model from Fig. 1a. We add these settings to the appendix, following Yang et al. [5]: MMLU with chain-of-thought prompting, and forking at tokens 25 and 200 (τ\tau = 1.0, top-pp = 0.9).

Q3. Long chain-of-thought models

We did not use thinking mode because RL with 32K-token responses exceeded our compute budget. In Appendix D (Qwen-3-8B-Base, 8K-token responses), EAD learns faster than fixed temperature on AIME 2024. We see three challenges in adapting our method to longer outputs:

  1. Schedule length. The step-aware decay grows the "warm region" as response length increases, but d0d_0 and α\alpha must scale with the target length (d0=500d_0 = 500 in Appendix D vs. 25 at 3K tokens).
  2. Late re-exploration. Self-reflection ("wait, let me check") creates late decision points that position-only cooling may suppress. τ_min > 0 keeps some randomness. Re-heating at reflection points is a natural extension, which we leave for future work.
  3. Off-policy gap. Importance ratios grow more extreme over long responses, so TIS matters more.

We add these points to the Discussion.

Q4. EAD vs. EDT at test time

They use different signals (position vs. entropy) and could be combined, e.g., by scaling EDT with EAD's schedule. We ran an inference-time evaluation on MATH-500: EAD outperforms at Maj@64 (1.5B: 70.4 vs. 63.0; 7B: 74.4 vs. 66.5).

Q5. Is EAD-TIS more stable beyond entropy?

Yes. On Llama (Fig. 3), Worst@16 of EAD without TIS drops late in training to 23.02, while EAD-TIS stays at 31.65. Without TIS, Fig. 10 shows clip-fraction surges, gradient-norm spikes, and a Pass@16 collapse. Truncation affects fewer than 1% of sequences, so TIS removes rare extreme updates and leaves the rest unchanged. We clarify in the Fig. 10 caption that those panels show the run without TIS.

References:

[1] G. Cui et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv 2025.

[2] Z. Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv 2025.

[3] C. Walder and D. Karkhanis. Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems. arXiv 2025.

[4] S. Zhang, Y. Bao, and S. Huang. EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling. arXiv 2024.

[5] C. Yang, S. Li, and A. Holtzman. LLM Probability Concentration: How Alignment Shrinks the Generative Horizon. TMLR 2026.

Reviewer_g1T6OpenReview ↗

Thanks for the authors' effort in Rebuttal

I thank the authors for their hard work in the rebuttal. Most of my concerns have been addressed, except for Q3: the thinking mode scalability of EDT. However, I will raise my score to 4, because it is a good paper. Although its novelty in RLVR is somewhat limited compared with the literature today, it is technically solid and worth reading to gain knowledge in RL for LLMs.

AuthorsOpenReview ↗

Thank you!

Thank you for raising the score to 4 and for the kind assessment. We are glad the other points are settled!

On Q3, we are sorry that we only have limited compute for now and could not run thinking-mode training. Still, big thanks for raising this great point (and many other great questions)! We also look forward to seeing how thinking mode, together with EAD, behaves in LLM RLVR, and we are already securing more compute to improve the next version of our paper.

Reviewer_VjhAOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Exploratory Annealed Decoding (EAD), a sampling strategy for RLVR. Instead of using a fixed temperature throughout generation, EAD gradually decreases the sampling temperature within each response, encouraging exploration during early token generation while improving the coherence of later continuations. The method also introduces a global-step-aware decay rate to accommodate changes in response length during training and uses truncated importance sampling to mitigate instability caused by the discrepancy between the sampling and target policies.

Critical strengths
  • The paper identifies an important exploration–exploitation trade-off in RLVR and exploits the observation that the role of exploration may vary across positions in an autoregressive sequence. EAD modifies the sampling procedure without changing the model architecture or introducing additional trainable components. Its compatibility with existing RLVR pipelines makes it an attractive practical technique.

  • The reverse-annealing experiment and hyperparameter studies provide some insight into the proposed design. The authors also acknowledge limitations concerning domain generalization, model-dependent temperature floors, and alternative scheduling functions.

Critical weaknesses
  • Annealed sampling differs from fixed-temperature sampling not only in where exploration occurs, but potentially also in the average temperature, entropy, and overall sampling distribution. Without carefully matched controls, it is difficult to determine whether the gains arise specifically from front-loading exploration or from differences in the total exploration budget. A mean-matched control is acknowledged as future work in the paper.

  • The explanation for the advantage of early exploration remains largely empirical. The position-wise entropy analysis and prefix-forking experiment motivate the method, but do not establish that early exploration has greater marginal value across different models, tasks, and reasoning stages. A decrease in average conditional entropy does not necessarily imply a corresponding decrease in the contribution of exploration to final correctness.

  • EAD performs well on all three reported metrics for Qwen-2.5-Math-1.5B. However, it does not achieve the best Pass@16 or Worst@16 on Llama-3.2-1B-Instruct, and the confidence intervals for these metrics are relatively wide. The claims concerning consistent multi-sample improvements should therefore be qualified.

Questions for authors

See Weaknesses.

3 comments

Thank you for the review and for the clear statement of the three concerns.

W1. Front-loading vs. total exploration budget

We appreciate the attention to controls. An exact mean-matched control, however, is hard to build in RLVR. The average temperature of an EAD rollout depends on its length, which we do not know before sampling, and the length distribution shifts during training (Appendix G, Fig. 13). Matching the mean exactly would require pilot rollouts on the fly to estimate each response's length. These estimates have high variance and add compute. The rollout stage would also have to wait for them, which breaks the parallel batched rollout that RLVR frameworks such as verl rely on.

We therefore ran a first-order control. We take the mean response length at each training step of the EAD run, compute EAD's average temperature over a response of that length at that step, and average over training. This gives τˉ\bar{\tau}= 1.15. We then train with fixed temperature τˉ\bar{\tau} and keep every other setting. Pass@16 on MATH500, Qwen-2.5-Math-1.5B:

Stepτ{\tau} = 1.0τˉ\bar{\tau}= 1.15 (mean-matched)EAD
5067.967.668.8
10069.670.172.5
20070.871.873.0
30072.572.874.0
44073.874.174.2

With the same average temperature, the mean-matched run learns much more slowly. EAD reaches 72.5 at step 100, and the control first passes that level at step 300. The two converge by the end of training, both above τ\tau = 1.0. Where the temperature is spent mainly determines how fast EAD learns.

Two existing results point the same way:

  • A larger exploration budget does not necessarilly help. The τ\tau = 1.2 baseline uses at least EAD's temperature at every position. RevEAD (Fig. 9) uses a higher average temperature than EAD. Both perform worse than EAD.
  • The fixed baselines bracket EAD's budget. EAD beats τ\tau = 0.6, 1.0, and 1.2 on Qwen, and on Pass@1 for both models (Table 3).

We add the mean-matched result to Section 5.5 (full curves in the Appendix) and the reason an exact control is impractical to the Discussion.

W2. Why early exploration helps

The paper already frames "explore early, exploit late" as an empirical principle. Section 3 introduces it as following "this empirical pattern" and states that it "is a practical heuristic grounded in the entropy dynamics of autoregressive models, not a claim that late-stage exploration is never useful."

We agree that a decrease in entropy alone does not show that exploration contributes less to correctness. This is why the paper tests correctness directly, at inference and in training:

  • Forking (Fig. 1b). We keep a prefix of a response, resample the rest from token 25 or token 200, and score the result by majority vote. Resampling at token 200 lowers Maj@K far more than resampling at token 25.
  • RevEAD (Fig. 9). Moving exploration to the end of the response hurts accuracy during RL training.

The evidence covers math reasoning with 1B–8B models, and the Discussion states this scope.

W3. Llama results

Thank you for pointing this out. We tightened the wording: Section 5.2 no longer says "consistently" for the multi-sample metrics on Llama, and the revised text reports significance per model.

We also note that the Llama results are already a positive signal. We used the same EAD hyperparameters for all 1B/1.5B models (hyperparameter table in the Appendix) and did not tune them for Llama. With these settings, EAD on Llama-3.2-1B-Instruct:

  • gives the best Pass@1 (40.73 vs. 39.80),
  • outperforms all GRPO baselines on Pass@16 and Worst@16, and
  • matches the strongest fixed-temperature DAPO baseline on Pass@16 and Worst@16 within the confidence intervals.

The Llama intervals reported in paper (Table 3, Appendix J) are evaluation dataset bootstrap on one run. Across seeds, the standard deviation on Pass@16 is about ±2\pm 2.

On Qwen-2.5-Math-1.5B, EAD is best on all three metrics, and the Pass@16 gain carries over to Qwen-2.5-Math-7B.

Thank you again for the constructive feedback.

Thank you again for the review. We have posted our response and updated the paper, with the revisions marked in cyan. In particular, we added a mean-matched control, tightened the wording of the Llama results, reported that the Pass@16 standard deviation across seeds is about ±2, and clarified the scope of the early-exploration claim.

The discussion deadline is November 18. We would be grateful if you could take a look before then. We are happy to clarify anything that is still open.

I would like to thank the authors for their rebuttal. The responses have addressed my concerns. I will keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and careful feedback. Reviewers found EAD simple, easy to add to existing RL pipelines, and supported by clear ablations (VjhA, g1T6, diex). We have updated the paper; revised text is highlighted in cyan. We summarize the main points below and answer each reviewer separately.

1. Llama results (diex, VjhA). Thank you for the careful reading of Section 5.2. We revised the wording and now report significance per model. On Qwen-2.5-Math-1.5B, EAD gives the best Pass@1, Pass@16, and Worst@16. On Llama-3.2-1B-Instruct, with hyperparameters shared across models and not tuned for Llama, EAD gives the best Pass@1, outperforms all GRPO baselines, and matches the strongest fixed-temperature DAPO baseline on Pass@16 and Worst@16 within the confidence intervals.

2. New experiments (VjhA, g1T6, diex).

  • Mean-matched control (VjhA). An exact mean-matched control is impractical in RLVR: response length is unknown before sampling and shifts during training, and estimating it with pilot rollouts adds compute and breaks parallel rollout. We ran a first-order control with a fixed temperature equal to EAD's average temperature over the logged training lengths (τˉ\bar{\tau}= 1.15). It learns much more slowly: it only reaches EAD's step-100 Pass@16 at step 300, and the two converge by the end of training.
  • EAD + Dr. GRPO (g1T6). Dr. GRPO removes GRPO's length normalization, which tests whether EAD's gains depend on response length. EAD + Dr. GRPO outperforms Dr. GRPO at every evaluated step after training starts, by up to 3.2 Pass@16 points.
  • Simplified code setting (diex). In 100-step code RL without an agent harness, with all checkpoints decoded identically, the untuned math schedule gives the best HumanEval+ Pass@16 (89.6 vs. at most 88.4), while fixed τ\tau = 1.2 is better on mean accuracy and Worst@16.

3. Scope (VjhA, g1T6, diex). As the title states, the paper targets math reasoning. Section 3 frames "explore early, exploit late" as a practical heuristic, and the paper tests its effect on correctness directly (forking in Fig. 1b, RevEAD in Fig. 9). Code and agentic tasks add environment feedback and agent harnesses that interrupt normal autoregressive generation; we discuss these settings, and long chain-of-thought training, in the Discussion, and report a simplified code result above.

4. Clearer explanations (g1T6). We add our explanation of why EAD helps exploration, the challenges for long chain-of-thought models, the TIS trade-off on Llama, and the settings of the Section 3 experiments.

5. Related work (g1T6). We revised the description of EDT, which sets temperature per token from entropy at test time. In an inference-time evaluation on MATH-500, at the starting point the EDT paper suggests (T0T_0=0.6, θ\theta=0.1), EAD is higher at Maj@64 (1.5B: 70.4 vs. 63.0; 7B: 74.4 vs. 66.5). We cite Pass@K training and Dr. GRPO, and leave the combination with Pass@K training to the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.