- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes EDEN, a simple input embedding-space perturbation exploration mechanism for GRPO-style RLVR. On the Qwen 2.5 series, the authors show that EDEN outperforms GRPO with respect to math benchmarks accuracy (Pass@1 -- Pass@256 on AIME24 and 25). To me, the evidence supporting EDEN helps exploration include (1. fraction of prompts with at least one correct rollout increased (Fig. 7a), 2. EDEN reduces #prompts with zero advantage, 3. Slower entropy decrease (Fig. 6), 4. diverse reasoning patterns (Fig 3), etc.
Critical strengths
- The research question of input-space perturbation for explorative RLVR is important. EDEN is a simple yet effective method for exploration. Audiences in RL and LLM reasoning may find this paper interesting.
- The experimental analysis in Section 4, with carefully designed control experiments to dive into the analysis of EDEN is inspiring.
These two reasons drive my overall score above 2. Other reasons why I lean positive on this paper:
- The ablation studies are fair.
- The method has reasonable novelty.
- The paper does not read like fully AI-generated slop.
Critical weaknesses
- The noise bank construction is not easy to understand, which makes it difficult to reproduce based on the details revealed in the paper. I recommend that the authors revise the introduction to this section and include illustrative examples in the appendix.
- The closest explorative RL baseline at the input level (Soft Tokens, PSN-GRPO, REFT, etc.) is not compared or discussed in the experiments. It's fine for EDEN not to be optimal, but providing insights by comparing it with existing methods, analysing complementary advantages through case studies, or other forms of research would likely improve the paper.
- LLMs in this paper are primarily the Qwen2.5 series with a small response length (2,560), which limited this paper from proving its scalability to thinking mode and longer CoT RL. I recommend the authors try open models from Qwen3, Olmo, etc., in thinking mode with longer CoT length if they have computational resources for the experiments. (I think 8*H100 can definitely train these models) .
Questions for authors
I am confused about several details in the paper, although some of them may be due to my own misunderstanding. I would appreciate it if the authors could answer the following questions transparently. It is completely acceptable if some design choices not fully optimal, as long as their motivations and limitations are clearly explained.
Questions related to W1: 1. How is the reference token chosen in l793, 2. whether the final performance is sensitive to this choice? 3. l799, how is the clustering done, via any algorithm? 4. l799, could the authors report the wall-clock/GPU cost of this offline screening and clarify whether this cost is included when claiming a matched rollout budget with GRPO?
Q2 Potential Bias of EDEN: Is there any potential reasoning bias for EDEN compared with GRPO? For example, longer average length, more self-reflection/verification/exploration patterns? Sometimes it's good to discuss these bias of the training algorithm for learning-related papers, which helps readers understand the algorithm from a more comprehensive perspective. For example, I noticed in the case studies that EDEN seems to prefer starting answers with “We need to xxx,” while GRPO tends to start with “To solve the problem, we need to xxx.” Is there any noticeable change in the usage of high- or low-frequency tokens in EDEN compared with GRPO?
Q3 why EDEN outperforfms GRPO even at step 0?: In Figure 7, it is surprising to see that EDEN already outperforms GRPO at step 0, and that the gap starts to shrink afterwards. This seems a bit unexpected, as I would assume that the two algorithms start from the same checkpoint and therefore should perform similarly at step 0. Am I misunderstanding something here?
3 comments
Response (2/2)
Q2. Differences in reasoning behaviour between EDEN and GRPO.
Yes, we observe several clear differences. Compared with GRPO, EDEN produces longer responses, uses self-correction and alternative-solution expressions much more often, no longer follows GRPO’s fixed “To solve …” opening, and uses slightly more low-frequency tokens. We measured these on the final Qwen3-1.7B-Base GRPO and EDEN checkpoints from W2 & W3, with eight responses per question (clean prompts, , top-p 0.95) on the same 3,293 evaluation questions. We will add this analysis to the revised paper.
| Metric | GRPO | EDEN |
|---|---|---|
| Mean response length (tokens) | 918.2 | 1,048.1 |
| Responses reaching the length limit | 5.23% | 10.04% |
| Responses containing self-correction cues | 1.05% | 8.20% |
| Responses containing alternative-solution cues | 3.31% | 13.26% |
| Responses containing verification cues | 21.36% | 21.86% |
| Mean proportion of low-frequency tokens per response | 7.17% | 8.10% |
EDEN produces longer responses and more frequently uses self-correction expressions such as “wait” and “reconsider,” and alternative-solution expressions such as “alternatively” and “another approach.” Both categories remain more frequent after normalization by response length. Verification cues have similar response-level prevalence. These measurements are based on lexical cues.
Opening styles also differ. In these Qwen3 outputs, approximately 65.26% of GRPO’s sampled responses begin with “To solve…,” compared with 0.05% of EDEN’s, which frequently begin with headings such as “Understanding the Problem” or “Problem Statement.” The opening style therefore changes, although the specific opening phrases vary across models.
We define low-frequency tokens using a common frequency ranking constructed from both models’ outputs, with the tail accounting for the last 5% of token occurrences. EDEN shows a modest increase in their proportion, and this difference persists after removing the first 32 response tokens. Token-frequency changes include Markdown and LaTeX markers, reflecting changes in formatting as well as vocabulary use.
Together with the greedy and Pass@8 gains in W2 & W3, these results show that EDEN combines improved problem-solving performance with a more varied solution style, including more frequent expressions of self-correction and alternative approaches under clean-input inference.
Q3. Step-0 gap in Fig. 7.
Both methods start from the same checkpoint, and their validation accuracy is the same at step 0 (Fig. 7c). Fig. 7(a)/(b) reports statistics of the training rollouts, which depend on the sampling conditions. GRPO generates all 16 rollouts at , whereas EDEN initially generates 12 standard rollouts at and four perturbed rollouts at . At step 0, these low-temperature rollouts tend to be more accurate than samples from the base model, so more prompts have at least one correct rollout (Fig. 7a) and fewer groups receive identical rewards (Fig. 7b). As the temperature approaches 1, the sampling conditions of the two methods become similar, and the gap closes within the first few hundred steps. The initial gap reflects differences in sampling behavior rather than an improvement already learned through RL.
The lower starting temperature serves as a warmup for perturbed sampling. Starting perturbed rollouts directly at leads to unstable early training, as illustrated by the ablation without temperature scheduling in Appendix E (Fig. 12). For Qwen2.5-Math-1.5B, we increase the perturbed-rollout temperature from 0.2 to 1 using a cosine schedule over the first 200 steps. It remains at 1 for the remaining 1,800 steps, covering 90% of training. This stabilizes the initial updates while allowing sustained exploration throughout most of training. The early curves consequently also reflect the changing sampling conditions during this warmup period. As training continues, EDEN’s advantage emerges and grows: from about step 1,000 onward, EDEN has more prompts with at least one correct rollout (Fig. 7a), fewer zero-advantage groups (0.53 vs. 0.58 at step 2,000 in Fig. 7b), and higher validation accuracy (Fig. 7c).
Thank you for your efforts in conducting additional experiments, which have further strengthened the technical soundness of your paper. I therefore maintain my positive rating. Best of luck!
Response (1/2)
We thank the reviewer for the thorough review and are glad that the reviewer found our method "simple yet effective," the analysis "inspiring" and the research question "important." We address each concern below.
W1 & Q1. Anchor-bank construction: reference token, clustering, and cost. The anchor bank is built offline in three steps: we compute directions from a fixed reference token to the other vocabulary embeddings (Eq. 4), cluster these directions into 32 candidate anchors, and screen the candidates by generation to keep eight (Eq. 5). The reference token only shapes the candidate pool, and screening is cheap and outside the matched rollout budget. We will add these details to Appendix C.
(1) Reference token. The reference token a was selected at random and held fixed. It only serves as a common origin for the directions in Eq. (4).
(2) Sensitivity. Two observations suggest that performance does not hinge on this choice. First, the reference only determines which candidates enter screening, and inclusion in the bank is decided by generation behavior (Eq. 5). Second, even a single unscreened random anchor held fixed during training improves over GRPO (Table 4), and with a different reference token on Qwen3-1.7B-Base, EDEN still improves over GRPO (W2 & W3).
(3) Clustering. We apply spherical k-means to the normalized directions ui, assigning each direction to the center with the highest cosine similarity. We use 32 centres: the first centre is chosen at random and the remaining ones by farthest-point initialization. Clustering runs for at most 20 iterations. Each cluster contributes the token closest to its centroid as a candidate anchor.
(4) Cost. The screening budget in the paper is generous. For Qwen2.5-Math-1.5B, 32 candidates × 200 prompts × 8 rollouts give 51,200 responses, about 1.25% of the rollouts in one 2,000-step training run, and take an estimated two hours on eight H100 GPUs. A much smaller budget also works: on Qwen3-1.7B-Base, screening with 64 prompts × 4 rollouts took under 40 minutes on four H100 GPUs, and the resulting bank still improved over GRPO. The bank is built once, can be reused across runs, and is not counted in the matched budget of 16 rollouts per prompt.
W2 & W3. Newer models, longer responses, and related exploration methods. To test longer CoT, we added two newer base models, Qwen3-1.7B-Base and OLMo-3-1025-7B, and extended the response limit from 2,560 tokens in the paper to 4,096 and 8,192 tokens, respectively. On Qwen3, we also compared EDEN with REFT and PSN-GRPO. EDEN achieves the highest mean greedy and Pass@8 accuracy on Qwen3 and improves mean greedy accuracy over GRPO on OLMo-3 in a preliminary run.
Qwen3-1.7B-Base. All methods were trained for 490 steps (10 epochs) on 6,354 problems from the paper’s MATH+DAPO data, with other settings following the paper. REFT branches rollouts over first tokens drawn from the policy’s 20 most likely candidates, and PSN-GRPO perturbs policy weights during rollout generation, adapted from the authors’ public implementation. PSN-GRPO degraded after about 200 steps, with eight-benchmark Avg@4 falling from 39.28% to 34.54% at step 490 (GRPO 41.32%, EDEN 43.16%), and the authors’ lower learning rate (3×10−7) did not recover it. The table reports step-490 results (%), and the six-benchmark mean excludes AIME24 and AIME25.
On the eight-benchmark mean, REFT improves over GRPO mainly in greedy accuracy, whereas EDEN improves both. This is consistent with EDEN broadening the set of solutions the policy can reach, rather than only improving its single best answer.
OLMo-3-1025-7B. GRPO and EDEN were trained on 1,024 problems from the same sources. Even with the 8,192-token limit, 35.57%/37.31% of GRPO/EDEN training rollouts and 26.81%/25.14% of greedy evaluation responses were truncated. Given the rebuttal time and compute, we report step 120 (3.75 epochs).