- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Exploratory Annealed Decoding (EAD) to address the difficulty of balancing exploration, response quality, and training stability with fixed-temperature sampling in reinforcement learning with verifiable rewards for mathematical reasoning. Its core principle is “explore early, exploit late.” By gradually lowering the sampling temperature, EAD encourages the model to explore different solution paths early and complete subsequent reasoning more consistently. The method also adjusts the cooling rate as training progresses and uses TIS to control gradient fluctuations. Experiments show that EAD can improve accuracy and sample efficiency across multiple models and reinforcement learning algorithms, and can also be used with majority voting at inference time.
Critical strengths
Intuitive design and simple implementation. EAD links exploration intensity to generation position, implementing “explore early, exploit late” through gradual temperature reduction. This design is supported by entropy dynamics and forking experiments. The schedule itself introduces little computational overhead and is easy to integrate into existing training pipelines.
Experimental support for performance gains and component contributions. Single-sample accuracy improves by 1.70 and 0.93 percentage points on Qwen-1.5B and Llama-1B, respectively. Ablation studies involving reverse scheduling, fixed decay rates, and TIS also help assess the contributions of individual components.
Good compatibility and flexible use. The method has been evaluated with DAPO, GRPO, and EntropyMech, achieving good results with relatively few training rollouts. It can also be applied directly to off-the-shelf models at inference time to improve majority-voting accuracy.
Critical weaknesses
The performance claims are inconsistent with the experimental reporting. The main text states that all comparisons are statistically significant, but Table 3 reports a Pass@16 of 46.76% for EAD on Llama, below the best fixed-temperature baseline of 49.48%. The authors also acknowledge that the two are statistically comparable. Furthermore, the main text’s claim of a Pass@16 advantage for EAD with TIS on Llama is inconsistent with Figure 3. Overall, EAD does not demonstrate a clear advantage over the strongest fixed-temperature baseline on Llama in either Pass@16 or Worst@16.
Evidence of generalization across tasks and model scales is limited. The experiments primarily focus on mathematical tasks and models ranging from 1B to 8B parameters. In code generation or agentic tasks, critical decisions may occur later in generation, and continued cooling could suppress necessary exploration. Further evaluation is needed to determine whether the current position-based schedule can accommodate these scenarios.
Questions for authors
In the discussion of Pass@16, the main text states that “EAD (w/ TIS) also outperforms on the Llama model,” but Figure 3 shows that EAD-TIS has lower Pass@16 than the fixed-temperature baselines on Llama during the later stages of training. Could the authors clarify whether this discrepancy reflects an error in the model name or figure legend, and explain the respective effects of TIS on Llama’s Pass@16 and Worst@16?
Could the authors provide evaluations on code generation, agentic tasks, or larger models to examine whether EAD’s position-based temperature annealing suppresses necessary exploration when critical decisions occur late in generation, and whether its performance advantages are maintained?
2 comments
Thank you for the additional experiments. Your response has addressed my concerns, and I will raise my score to 3.
Thank you for the careful reading. We address your concerns below and have updated the paper accordingly.
W1 & Q1. Llama results: main text vs. Fig. 3 and Table 3
Sorry for the imprecise wording in Section 5.2, and thank you for catching it. On Llama, EAD with TIS outperforms every GRPO baseline on Pass@16 (46.76 vs. at most 42.06) and Worst@16 (31.65 vs. at most 30.52). Against the fixed-temperature DAPO runs, it matches the strongest one within the confidence intervals. We revised the paragraph to:
Significance. The ±0.01 statement applies to Qwen-2.5-Math-1.5B, where every bootstrap interval in Table 3 is at most ±0.04. On Llama, the Pass@16 intervals are ±3.8 to ±4.4 and the Worst@16 intervals are ±1.9 to ±2.7, so we now report significance per model. These intervals are eval-set bootstrap on one run. Across runs (with different seeds), the standard deviation on Pass@16 is about ±2.
How TIS affects Llama. Without TIS, EAD reaches Pass@16 49.43, on par with the best DAPO run (49.48), but Worst@16 drops to 23.02 late in training. A few rollouts get extreme importance ratios, and these high-variance updates hurt the weakest samples first. TIS caps the ratios and keeps Worst@16 at 31.65, at the cost of some Pass@16. On Llama, TIS trades some best-case gain for stability. We state this in Section 5.2 and the Fig. 3 caption.
W2 & Q2. Code, agentic tasks, and larger models
As the title states, the paper targets math reasoning. Code and agentic tasks differ in a way that matters for a decoding-focused study. They rely on feedback from a compiler, terminal, or sandbox, usually run inside an agent harness these days. That feedback is appended to the context, and the harness may compact the memory. Both interrupt normal LLM autoregressive generation, and both would confound an analysis of the sampling schedule with the effects of the harness.
We still ran a simplified code setting without a harness: Qwen2.5-Coder-1.5B-Instruct trained with DAPO for 100 steps on Eurus-2 code problems, with unit-test rewards, evaluated on HumanEval+ (HE+) and LiveCodeBench (LCB). All step-100 checkpoints are decoded identically (τ = 1.0, 16 samples for each prompt):
With the same schedule in paper (no code-specific tuning), EAD already gives the best HE+ Pass@16 and beats τ = 1.0 and 1.2 on LCB Pass@16. The new appendix reports all runs, including Worst@16 and four other schedules.
In tasks where key decisions come late, position-only cooling could suppress useful late exploration. Two design choices limit this risk:
For tasks with explicit late decision points, a schedule that re-heats at those points (e.g., before a tool call) is a natural extension. We add this to the Discussion.
On scale, Appendix D gives our largest test: Qwen-3-8B-Base with the DAPO recipe, trained on DAPO-17k and evaluated on AIME 2024. EAD learns faster than fixed-temperature sampling there. The Appendix notes that this run used reduced compute and did not train to convergence.
Thank you again for the detailed feedback.