acceptodds
Under review as a conference paper at ICLR 2027

Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Large Language Model Post-Training, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kkM9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes EDEN, a simple input embedding-space perturbation exploration mechanism for GRPO-style RLVR. On the Qwen 2.5 series, the authors show that EDEN outperforms GRPO with respect to math benchmarks accuracy (Pass@1 -- Pass@256 on AIME24 and 25). To me, the evidence supporting EDEN helps exploration include (1. fraction of prompts with at least one correct rollout increased (Fig. 7a), 2. EDEN reduces #prompts with zero advantage, 3. Slower entropy decrease (Fig. 6), 4. diverse reasoning patterns (Fig 3), etc.

Critical strengths
  1. The research question of input-space perturbation for explorative RLVR is important. EDEN is a simple yet effective method for exploration. Audiences in RL and LLM reasoning may find this paper interesting.
  2. The experimental analysis in Section 4, with carefully designed control experiments to dive into the analysis of EDEN is inspiring.

These two reasons drive my overall score above 2. Other reasons why I lean positive on this paper:

  1. The ablation studies are fair.
  2. The method has reasonable novelty.
  3. The paper does not read like fully AI-generated slop.
Critical weaknesses
  1. The noise bank construction is not easy to understand, which makes it difficult to reproduce based on the details revealed in the paper. I recommend that the authors revise the introduction to this section and include illustrative examples in the appendix.
  2. The closest explorative RL baseline at the input level (Soft Tokens, PSN-GRPO, REFT, etc.) is not compared or discussed in the experiments. It's fine for EDEN not to be optimal, but providing insights by comparing it with existing methods, analysing complementary advantages through case studies, or other forms of research would likely improve the paper.
  3. LLMs in this paper are primarily the Qwen2.5 series with a small response length (2,560), which limited this paper from proving its scalability to thinking mode and longer CoT RL. I recommend the authors try open models from Qwen3, Olmo, etc., in thinking mode with longer CoT length if they have computational resources for the experiments. (I think 8*H100 can definitely train these models) .
Questions for authors

I am confused about several details in the paper, although some of them may be due to my own misunderstanding. I would appreciate it if the authors could answer the following questions transparently. It is completely acceptable if some design choices not fully optimal, as long as their motivations and limitations are clearly explained.


Questions related to W1: 1. How is the reference token chosen in l793, 2. whether the final performance is sensitive to this choice? 3. l799, how is the clustering done, via any algorithm? 4. l799, could the authors report the wall-clock/GPU cost of this offline screening and clarify whether this cost is included when claiming a matched rollout budget with GRPO?

Q2 Potential Bias of EDEN: Is there any potential reasoning bias for EDEN compared with GRPO? For example, longer average length, more self-reflection/verification/exploration patterns? Sometimes it's good to discuss these bias of the training algorithm for learning-related papers, which helps readers understand the algorithm from a more comprehensive perspective. For example, I noticed in the case studies that EDEN seems to prefer starting answers with “We need to xxx,” while GRPO tends to start with “To solve the problem, we need to xxx.” Is there any noticeable change in the usage of high- or low-frequency tokens in EDEN compared with GRPO?

Q3 why EDEN outperforfms GRPO even at step 0?: In Figure 7, it is surprising to see that EDEN already outperforms GRPO at step 0, and that the gap starts to shrink afterwards. This seems a bit unexpected, as I would assume that the two algorithms start from the same checkpoint and therefore should perform similarly at step 0. Am I misunderstanding something here?

3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the thorough review and are glad that the reviewer found our method "simple yet effective," the analysis "inspiring" and the research question "important." We address each concern below.

W1 & Q1. Anchor-bank construction: reference token, clustering, and cost. The anchor bank is built offline in three steps: we compute directions from a fixed reference token to the other vocabulary embeddings (Eq. 4), cluster these directions into 32 candidate anchors, and screen the candidates by generation to keep eight (Eq. 5). The reference token only shapes the candidate pool, and screening is cheap and outside the matched rollout budget. We will add these details to Appendix C.

(1) Reference token. The reference token aa was selected at random and held fixed. It only serves as a common origin for the directions in Eq. (4).

(2) Sensitivity. Two observations suggest that performance does not hinge on this choice. First, the reference only determines which candidates enter screening, and inclusion in the bank is decided by generation behavior (Eq. 5). Second, even a single unscreened random anchor held fixed during training improves over GRPO (Table 4), and with a different reference token on Qwen3-1.7B-Base, EDEN still improves over GRPO (W2 & W3).

(3) Clustering. We apply spherical k-means to the normalized directions uiu_i, assigning each direction to the center with the highest cosine similarity. We use 32 centres: the first centre is chosen at random and the remaining ones by farthest-point initialization. Clustering runs for at most 20 iterations. Each cluster contributes the token closest to its centroid as a candidate anchor.

(4) Cost. The screening budget in the paper is generous. For Qwen2.5-Math-1.5B, 32 candidates × 200 prompts × 8 rollouts give 51,200 responses, about 1.25% of the rollouts in one 2,000-step training run, and take an estimated two hours on eight H100 GPUs. A much smaller budget also works: on Qwen3-1.7B-Base, screening with 64 prompts × 4 rollouts took under 40 minutes on four H100 GPUs, and the resulting bank still improved over GRPO. The bank is built once, can be reused across runs, and is not counted in the matched budget of 16 rollouts per prompt.

W2 & W3. Newer models, longer responses, and related exploration methods. To test longer CoT, we added two newer base models, Qwen3-1.7B-Base and OLMo-3-1025-7B, and extended the response limit from 2,560 tokens in the paper to 4,096 and 8,192 tokens, respectively. On Qwen3, we also compared EDEN with REFT and PSN-GRPO. EDEN achieves the highest mean greedy and Pass@8 accuracy on Qwen3 and improves mean greedy accuracy over GRPO on OLMo-3 in a preliminary run.

Qwen3-1.7B-Base. All methods were trained for 490 steps (10 epochs) on 6,354 problems from the paper’s MATH+DAPO data, with other settings following the paper. REFT branches rollouts over first tokens drawn from the policy’s 20 most likely candidates, and PSN-GRPO perturbs policy weights during rollout generation, adapted from the authors’ public implementation. PSN-GRPO degraded after about 200 steps, with eight-benchmark Avg@4 falling from 39.28% to 34.54% at step 490 (GRPO 41.32%, EDEN 43.16%), and the authors’ lower learning rate (3×10−73\times10^{-7}) did not recover it. The table reports step-490 results (%), and the six-benchmark mean excludes AIME24 and AIME25.

BenchmarkGRPO GreedyREFT GreedyEDEN GreedyGRPO Pass@8REFT Pass@8EDEN Pass@8
MATH50068.2072.6072.0085.4087.6087.40
AMC37.3534.9440.9660.2455.4266.27
AIME246.6710.0016.6723.3323.3330.00
AIME253.333.336.6710.0016.6720.00
GSM8K83.9384.9985.0693.1094.7794.84
Gaokao58.4462.3463.3875.5876.3677.66
MinervaMath27.5728.3127.9445.5946.6945.22
OlympiadBench31.9036.9436.5054.0156.5358.31
Main six-benchmark mean51.2353.3554.3168.9969.5671.62
Eight-benchmark mean39.6741.6843.6555.9157.1759.96

On the eight-benchmark mean, REFT improves over GRPO mainly in greedy accuracy, whereas EDEN improves both. This is consistent with EDEN broadening the set of solutions the policy can reach, rather than only improving its single best answer.

OLMo-3-1025-7B. GRPO and EDEN were trained on 1,024 problems from the same sources. Even with the 8,192-token limit, 35.57%/37.31% of GRPO/EDEN training rollouts and 26.81%/25.14% of greedy evaluation responses were truncated. Given the rebuttal time and compute, we report step 120 (3.75 epochs).

BenchmarkGRPO GreedyEDEN Greedy
MATH50075.4076.60
AMC50.6045.78
AIME2410.0023.33
AIME2520.0016.67
GSM8K79.8376.72
Gaokao65.4570.91
MinervaMath26.8427.21
OlympiadBench37.2439.61
Main six-benchmark mean55.8956.14
Eight-benchmark mean45.6747.10
AuthorsOpenReview ↗

Response (2/2)

Q2. Differences in reasoning behaviour between EDEN and GRPO.

Yes, we observe several clear differences. Compared with GRPO, EDEN produces longer responses, uses self-correction and alternative-solution expressions much more often, no longer follows GRPO’s fixed “To solve …” opening, and uses slightly more low-frequency tokens. We measured these on the final Qwen3-1.7B-Base GRPO and EDEN checkpoints from W2 & W3, with eight responses per question (clean prompts, T=0.6T=0.6, top-p 0.95) on the same 3,293 evaluation questions. We will add this analysis to the revised paper.

MetricGRPOEDEN
Mean response length (tokens)918.21,048.1
Responses reaching the length limit5.23%10.04%
Responses containing self-correction cues1.05%8.20%
Responses containing alternative-solution cues3.31%13.26%
Responses containing verification cues21.36%21.86%
Mean proportion of low-frequency tokens per response7.17%8.10%

EDEN produces longer responses and more frequently uses self-correction expressions such as “wait” and “reconsider,” and alternative-solution expressions such as “alternatively” and “another approach.” Both categories remain more frequent after normalization by response length. Verification cues have similar response-level prevalence. These measurements are based on lexical cues.

Opening styles also differ. In these Qwen3 outputs, approximately 65.26% of GRPO’s sampled responses begin with “To solve…,” compared with 0.05% of EDEN’s, which frequently begin with headings such as “Understanding the Problem” or “Problem Statement.” The opening style therefore changes, although the specific opening phrases vary across models.

We define low-frequency tokens using a common frequency ranking constructed from both models’ outputs, with the tail accounting for the last 5% of token occurrences. EDEN shows a modest increase in their proportion, and this difference persists after removing the first 32 response tokens. Token-frequency changes include Markdown and LaTeX markers, reflecting changes in formatting as well as vocabulary use.

Together with the greedy and Pass@8 gains in W2 & W3, these results show that EDEN combines improved problem-solving performance with a more varied solution style, including more frequent expressions of self-correction and alternative approaches under clean-input inference.

Q3. Step-0 gap in Fig. 7.

Both methods start from the same checkpoint, and their validation accuracy is the same at step 0 (Fig. 7c). Fig. 7(a)/(b) reports statistics of the training rollouts, which depend on the sampling conditions. GRPO generates all 16 rollouts at T=1T=1, whereas EDEN initially generates 12 standard rollouts at T=1T=1 and four perturbed rollouts at T=0.2T=0.2. At step 0, these low-temperature rollouts tend to be more accurate than T=1T=1 samples from the base model, so more prompts have at least one correct rollout (Fig. 7a) and fewer groups receive identical rewards (Fig. 7b). As the temperature approaches 1, the sampling conditions of the two methods become similar, and the gap closes within the first few hundred steps. The initial gap reflects differences in sampling behavior rather than an improvement already learned through RL.

The lower starting temperature serves as a warmup for perturbed sampling. Starting perturbed rollouts directly at T=1T=1 leads to unstable early training, as illustrated by the ablation without temperature scheduling in Appendix E (Fig. 12). For Qwen2.5-Math-1.5B, we increase the perturbed-rollout temperature from 0.2 to 1 using a cosine schedule over the first 200 steps. It remains at 1 for the remaining 1,800 steps, covering 90% of training. This stabilizes the initial updates while allowing sustained exploration throughout most of training. The early curves consequently also reflect the changing sampling conditions during this warmup period. As training continues, EDEN’s advantage emerges and grows: from about step 1,000 onward, EDEN has more prompts with at least one correct rollout (Fig. 7a), fewer zero-advantage groups (0.53 vs. 0.58 at step 2,000 in Fig. 7b), and higher validation accuracy (Fig. 7c).

Thank you for your efforts in conducting additional experiments, which have further strengthened the technical soundness of your paper. I therefore maintain my positive rating. Best of luck!

Reviewer_wMf1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3(+1)
Soundness
3
Read review
Paper summary

Currently, most post-training of large models adopts GRPO. The authors notice that during GRPO sampling, for some easy problems or particularly hard problems, the samples can be all correct or all wrong, which is not good for post-training. To address this problem, the authors explore a perturbation approach that makes the sampling process artificially more diverse. This diversity does not refer to the natural diversity of sampling; rather, given a certain perturbation, the model naturally gives different outputs according to different inputs. Validation tests were conducted on some math benchmarks, and a certain improvement was found.

Critical strengths
  1. It focuses on a very important problem, and the topic is very well chosen. In current GRPO training, many very hard or very easy problems do indeed produce all-correct or all-wrong samples, and the diversity of the model (the diversity of the samples) has a very significant correlation with the effectiveness of GRPO training. Many people are currently trying to solve this problem, and there is no good way to solve it perfectly. 2. The perturbation has a certain degree of novelty. Perturbations are generally applied on the model side, whereas their approach perturbs the input side of the problem. Although I am not sure whether this is the first work to do this, the approach of adding some unrelated vectors is indeed something I did not immediately think of. However, it may have some similarity with other approaches that add perturbations in pure-text form.
Critical weaknesses
  1. The model used is too old, and this model may also have test-set leakage. That is, for the base model they use, there have been related reports that it has some problems in its own training process. 2. They did not test on tasks with more generalization. Because this model may have already seen similar problems during its pretraining, testing on other kinds of tasks should be considered.
Questions for authors
  1. Why was a relatively old model that carries certain risks used for the experiments? 2. I would like to see comparisons with other methods (not only perturbation methods, but also other diversity methods). If the scope of the comparison is broad, I will consider raising my score.
4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the thoughtful review and for recognizing that the paper "focuses on a very important problem," that the topic is "very well chosen," and the novelty of our perturbation strategy. Following the reviewer's suggestions, we have added experiments on two newer base models, evaluations beyond mathematics and comparisons with two recent exploration methods. We address each point below.

W1 & Q1. Model choice and results on newer models.

We use Qwen2.5-Math as our main setting because it is widely adopted in RLVR research on mathematical reasoning, including analyses of what RLVR learns [A], exploration methods such as PSN-GRPO [B], and rollout-efficiency methods [C, D]. Regarding possible pretraining contamination, EDEN and GRPO start from the same checkpoint and share the same training data, rollout budget, and evaluation protocol, which ensures a fair comparison between them. To check that the improvement is not specific to this model family, we added experiments on Qwen3-1.7B-Base and OLMo-3-1025-7B. On Qwen3-1.7B-Base, EDEN outperforms GRPO on all eight benchmarks in greedy accuracy. On OLMo-3-1025-7B, EDEN also improves mean accuracy in a preliminary run.

Given the rebuttal time and compute budget, we trained each method on Qwen3-1.7B-Base using 4 H100 GPUs and a fixed subset containing approximately one quarter of the paper's MATH+DAPO training mixture, preserving its source proportions. This subset contains 6,354 problems: 1,875 from MATH and 4,479 from DAPO. We trained for 10 epochs, totaling 490 training steps, with a training batch size of 128, a PPO mini-batch size of 64, 16 rollouts per problem, and a learning rate of 10−610^{-6}. The response limit was 4,096 tokens. We evaluated both methods on the same eight mathematical benchmarks used in the paper. All scores below are percentages. Following the main tables in the paper, the six-benchmark mean excludes AIME24 and AIME25.

BenchmarkGRPO GreedyEDEN GreedyGRPO Pass@8EDEN Pass@8
MATH50068.2072.0085.4087.40
AMC37.3540.9660.2466.27
AIME246.6716.6723.3330.00
AIME253.336.6710.0020.00
GSM8K83.9385.0693.1094.84
Gaokao58.4463.3875.5877.66
MinervaMath27.5727.9445.5945.22
OlympiadBench31.9036.5054.0158.31
Main six-benchmark mean51.2354.3168.9971.62
Eight-benchmark mean39.6743.6555.9159.96

We also conducted an experiment on OLMo-3-1025-7B, using 4 H100 GPUs per method. We used a fixed subset of 1,024 problems from the same training data sources. Both the training batch size and PPO mini-batch size were 32, with 16 rollouts per problem and a learning rate of 10−610^{-6}. The response limit was increased to 8,192 tokens. Because of the rebuttal time limit, we report the latest checkpoint with completed full evaluation, at training step 120 (3.75 epochs).

BenchmarkGRPO GreedyEDEN Greedy
MATH50075.4076.60
AMC50.6045.78
AIME2410.0023.33
AIME2520.0016.67
GSM8K79.8376.72
Gaokao65.4570.91
MinervaMath26.8427.21
OlympiadBench37.2439.61
Main six-benchmark mean55.8956.14
Eight-benchmark mean45.6747.10

The response-length budget remains a constraint in the OLMo-3 experiment: over the first 120 training steps, 35.57% of GRPO rollouts and 37.31% of EDEN rollouts reached the 8,192-token limit, and in the full greedy evaluation the rates were 26.81% and 25.14%. We therefore regard the OLMo-3 results as preliminary.

References

[A] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?" Advances in Neural Information Processing Systems (2025).

[B] Bai, Bizhe, et al. "Learning to explore with parameter-space noise: A deep dive into parameter-space noise for reinforcement learning with verifiable rewards." arXiv preprint arXiv:2602.02555 (2026).

[C] Zheng, Haizhong, et al. "Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts." Advances in Neural Information Processing Systems (2025).

[D] Sun, Yifan, et al. "Improving data efficiency for llm reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay." Advances in Neural Information Processing Systems (2025).

[E] Kim, Soeun, and Albert No. "Diversifying RLVR Rollouts via First-Token Exploration." arXiv preprint arXiv:2605.28295 (2026).

AuthorsOpenReview ↗

Response (2/2)

W2. Generalization beyond mathematics.

Our experiments mainly focus on mathematical tasks because their answers can be automatically verified and problems can be solved through multiple reasoning paths, making mathematics a natural setting for studying rollout exploration. To assess whether EDEN’s gains extend beyond mathematics, we took the GRPO and EDEN checkpoints from W1 & Q1, trained from Qwen3-1.7B-Base on MATH+DAPO only, and evaluated them without further training on general reasoning (BBH with 23 task families, 27 configurations, and 6,511 questions) and code generation (HumanEval+ and MBPP+), using the same prompts and scoring procedure for all models. EDEN outperforms GRPO by 3.52 points in greedy accuracy and 3.86 in Pass@8 on BBH, and the base model by 4.44 and 8.56, indicating that the gains from math training transfer to general reasoning. On HumanEval+ and MBPP+, EDEN stays at or above the base model and is comparable to GRPO overall.

BenchmarkMetricBaseGRPOEDEN
BBHGreedy43.6844.6048.12
BBHPass@872.2176.9180.77
HumanEval+Greedy41.4641.4645.73
HumanEval+Pass@863.4168.9064.63
MBPP+Greedy59.2660.0559.79
MBPP+Pass@877.2576.7277.25

Q2. Comparison with other exploration and diversity methods.

Together with the methods in the paper, EDEN is now compared with methods that introduce diversity at five different parts of the RL pipeline, including two recent exploration methods added for this response. On Qwen3-1.7B-Base, EDEN achieves higher mean greedy and Pass@8 accuracy than REFT, while PSN-GRPO degrades under the shared training setup.

Where diversity is introducedMethodWhere reported
Sampling temperatureHigher temperatureTab. 1, Tab. 10
Training objectiveClip-Higher, entropy bonusTab. 1, Fig. 6
Number of rolloutsGRPO with 8 / 16 / 32 rollouts per promptTab. 5
First response tokensREFT [E]New, below
Policy parametersPSN-GRPO [B]New, below
Input embeddingsEDEN—

In the paper, raising the temperature or using Clip-Higher changes GRPO’s six-benchmark Pass@1/Pass@8 on Qwen2.5-Math-1.5B by +0.07/−0.21 and +0.39/+0.97, respectively, compared with +2.44/+3.06 for EDEN (Tab. 1). A larger entropy bonus does not improve validation accuracy (Fig. 6). Doubling GRPO’s rollouts from 16 to 32 on Qwen2.5-0.5B changes Pass@1/Pass@8 by +0.11/−0.15, whereas EDEN with 16 rollouts gains +2.28/+2.65 (Tab. 5).

REFT diversifies rollouts by branching over different first tokens, and PSN-GRPO perturbs policy parameters during rollout generation. We reimplemented REFT following its paper and adapted the authors’ public fixed-noise implementation of PSN-GRPO, using MLP-only Gaussian perturbations with σ=0.001\sigma=0.001 and token-level truncated importance sampling. All methods use the training setup in W1 & Q1, and other settings follow the paper.

Under this shared setup, PSN-GRPO degraded after about 200 steps, with increasingly repetitive outputs and more responses reaching the length limit. Its eight-benchmark Avg@4 fell from 39.28% at step 200 to 34.54% at step 490, compared with final scores of 41.32% for GRPO and 43.16% for EDEN. We also resumed the step-200 checkpoint with the learning rate lowered to 3×10−73\times10^{-7}, the default in the authors’ public launch script, keeping all other settings unchanged. After 100 more updates, this branch reached 15.04% at step 300, compared with 26.36% for the original branch, so the lower learning rate did not recover performance. We therefore report results for GRPO, REFT, and EDEN below at their final 490-step checkpoints (10 epochs).

BenchmarkGRPO GreedyREFT GreedyEDEN GreedyGRPO Pass@8REFT Pass@8EDEN Pass@8
MATH50068.2072.6072.0085.4087.6087.40
AMC37.3534.9440.9660.2455.4266.27
AIME246.6710.0016.6723.3323.3330.00
AIME253.333.336.6710.0016.6720.00
GSM8K83.9384.9985.0693.1094.7794.84
Gaokao58.4462.3463.3875.5876.3677.66
MinervaMath27.5728.3127.9445.5946.6945.22
OlympiadBench31.9036.9436.5054.0156.5358.31
Main six-benchmark mean51.2353.3554.3168.9969.5671.62
Eight-benchmark mean39.6741.6843.6555.9157.1759.96

REFT improves both aggregate metrics over GRPO. EDEN achieves further gains over REFT, increasing the eight-benchmark mean by 1.97 percentage points in greedy accuracy and 2.79 points in Pass@8.

Thanks to the authors for the response, which has partially addressed my concerns. However, I would still like to see more comparisons with other, more common methods for increasing diversity. I will raise my score accordingly.

Thank you for your follow-up and for the opportunity to clarify the scope of our comparison. The paper and our response compare EDEN with a broad set of common methods that add rollout diversity to make RL training more effective, as summarized below.

The problem we address. With a limited rollout budget, rollouts from the same policy are often highly similar, so many groups receive identical rewards and provide no learning signal. EDEN is designed to make RL training more effective in this setting. Rollout diversity during training is the means to this goal rather than the goal itself. Our method introduces variation within the same rollout budget through anchored perturbations of the input embeddings, so that more groups receive mixed outcomes and provide a learning signal.

Common diversity methods compared. We therefore examine whether common ways of adding rollout diversity make RL training more effective, measured by their accuracy gains over GRPO. Methods whose main goal is to increase the diversity of the trained model's outputs address a different objective. We compare EDEN with representative methods from four approaches: sampling, the training objective, the rollout budget, and rollout generation. Together, these approaches give a broad comparison that covers the stages of RL training at which rollout diversity is commonly introduced, from how rollouts are generated and sampled to how the policy is updated.

ApproachMethodHow it adds diversityMethod vs. GRPOEDEN vs. GRPOSource
SamplingHigher temperature (T=1.2)Flattens the token distribution+0.07 / −0.21+2.44 / +3.06Tab. 1
Training objectiveClip-HigherRelaxes the upper clipping bound+0.39 / +0.97+2.44 / +3.06Tab. 1
Training objectiveLarger entropy bonusRewards higher policy entropyDeclines at larger bonuses—Fig. 6
Rollout budget32 instead of 16 rolloutsSamples more rollouts per prompt+0.11 / −0.15+2.28 / +2.65Tab. 5
Rollout generationREFTBranches over different first tokens+2.01 / +1.26+3.98 / +4.05Response
Rollout generationPSN-GRPOPerturbs policy parameters−6.78 (Avg@4)+1.84 (Avg@4)Response

Rows 1–2: six-benchmark Pass@1 / Pass@8 on Qwen2.5-Math-1.5B. Row 4: Pass@1 / Pass@8 on Qwen2.5-0.5B. Rows 5–6: eight-benchmark greedy / Pass@8 and Avg@4 at step 490 on Qwen3-1.7B-Base. Table and figure numbers refer to the revised PDF. EDEN gives a larger improvement over GRPO than each compared method, so the variation it adds translates into accuracy gains.

After training, our method also produces more diverse correct solutions, with more of its correct AIME rollouts coming from problems solved with at least four distinct approaches (Fig. 3). It obtains this diversity without directly optimizing for it. Our method increases rollout diversity during training as a means to more effective learning, whereas methods that target the diversity of the outputs of the final trained model treat this diversity as the objective itself. These methods optimize a different objective from EDEN. Our comparison therefore covers a broad set of common methods that share EDEN's goal of more effective RL training.

Reviewer_KowJOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EDEN, an exploration method for RLVR. The key idea is to diversify a subset of rollouts by appending a perturbed embedding to the prompt, where the perturbation combines a shared anchor embedding with independent Gaussian noise. The resulting trajectories are incorporated into GRPO updates through batch-level advantage filtering and downweighted advantages, while the model architecture and test-time inference remain unchanged.

Critical strengths
  • The paper addresses rollout homogeneity and weak group-relative learning signals in RLVR, which are important practical challenges when rollout budgets are limited. Exploring alternative input conditioning is a reasonable and interesting direction.

  • EDEN introduces perturbations at a single appended embedding position without changing the model architecture or test-time inference. Its compatibility with the GRPO pipeline makes the method relatively straightforward to implement and potentially combine with existing RLVR techniques.

  • The method is evaluated across three model scales and multiple mathematical reasoning benchmarks. The ablations on noise, anchor sharing, anchor-bank size, and noisy-rollout fraction provide useful evidence about the roles of individual design choices.

Critical weaknesses
  • The generation-update distribution mismatch requires further justification. Perturbed rollouts are generated under the perturbed input condition but are evaluated and optimized under the clean prompt condition. The paper explicitly does not apply an importance-sampling correction, instead relying on advantage filtering and downweighting. While this may be an effective practical heuristic, it is unclear under what conditions the resulting update provides a reliable learning signal for the clean-input policy.

  • The mechanism and robustness of anchor selection remain somewhat unclear. The anchor bank is constructed from vocabulary embeddings using a screening procedure based on a limited set of prompts. Although the ablations support the usefulness of anchoring, they do not fully isolate whether the gains arise from directional structure, particular token-embedding properties, or more general changes to the prefix conditioning.

Questions for authors

See Weaknesses

2 comments

We thank the reviewer for the careful review. We are glad that the reviewer found that the paper addresses "important practical challenges when rollout budgets are limited," that the method is "straightforward to implement," and that the ablations provide "useful evidence" for the individual design choices. We address both concerns below.

W1. Conditions under which perturbed rollouts give a useful signal for the clean policy. EDEN uses a biased surrogate update, and our measurements and ablations identify two conditions under which this update is effective and stable. (i) The perturbation mainly changes the early response tokens, and later tokens have nearly the same probabilities under the clean and perturbed prompts. Importance-sampling correction is therefore unnecessary for the later tokens and would remove most of the perturbed signal in the early tokens. (ii) Perturbed rollouts form a small, filtered, and down-weighted part of each update, which keeps training stable.

For condition (i), we analyzed the final EDEN Qwen2.5-Math-1.5B checkpoint with four perturbed rollouts at temperature 1 for each of 64 training questions. At every response position, we computed the KL divergence from the perturbed-prompt to the clean-prompt next-token distribution, given the same sampled prefix. These KL values sum to 63.4 nats per rollout on average, and the first 32 tokens account for 95.9% of the total (bootstrap 95% CI: [94.0%, 97.5%]). After these tokens, the KL averages 0.003 nats/token, and about 98% of all sampled tokens have a clean-to-perturbed probability ratio within [0.86, 1.11], so later tokens receive a nearly on-policy update given the early tokens. At the first token, however, the median ratio is about 1.3×10⁻²², and conditional on the sampled perturbation, the sequence log-weight has a median of −59.1 and a maximum of −18.6. Token-level or sequence-level importance weighting would therefore remove the early-token changes or entire perturbed trajectories, which are the signal EDEN aims to bring into clean-policy learning.

For condition (ii), EDEN controls how perturbed rollouts enter the update. All rollouts are scored against the original question using the same verifier, so perturbation does not change the task reward definition. Advantages are computed within the mixed group. Of 16 generated rollouts per question, 12 are standard and 4 are perturbed. Batch-level filtering retains half of the perturbed rollouts by absolute advantage, and their advantages are scaled by 0.5. Removing the filter raises the standard deviation of gradient norms from 0.138 to 0.319 and lowers Avg@4 from 47.03 to 45.62 (Fig. 8). Increasing the perturbed fraction to 50% yields performance comparable to GRPO in the Qwen2.5-0.5B ablation (Fig. 10). EDEN’s design keeps the perturbed share moderate and filtered, which is what makes the update stable and effective.

W2. Source of the anchor gains and robustness of anchor selection. Our ablations indicate that the gains depend on the directional structure of the perturbation, not on a general change to the prefix or on a particular token. The key part of this structure is that all perturbed rollouts of a prompt share one anchor, while each rollout receives its own noise. A fixed, unscreened random anchor already improves over GRPO, and screening raises the gains further. In additional experiments on Qwen3-1.7B-Base, screening on only 64 prompts, compared with 200 in the paper, also produced a bank that improved over GRPO.

General prefix changes do not explain the gains. In Table 4 (Qwen2.5-0.5B, averaged over MATH500 and GSM8K), appending the screened anchors without noise gives Pass@1/Pass@8 of 45.42%/66.51%, and adding norm-matched random noise without an anchor gives 43.39%/65.45%. Neither improves Pass@1 over GRPO (45.67%/66.36%), whereas full EDEN reaches 47.95%/69.01%.

Sharing one anchor within a prompt group is necessary. A random vocabulary anchor held fixed improves over GRPO (46.37%/67.78%, three anchor draws), whereas drawing a new random anchor for every rollout falls below GRPO (43.86%/65.81%). These two settings differ in whether the perturbed rollouts of a prompt share one conditioning direction. Using different directions across prompt groups adds further gains: on Qwen2.5-Math-1.5B, increasing K from 1 to 8 raises greedy Pass@1 from 48.45% to 49.33% and Pass@8 from 61.96% to 62.13% (Table 6).

Screening raises the gains beyond a random anchor. Screening for generation viability, solution coverage, and trajectory complementarity raises Pass@1/Pass@8 from 46.37%/67.78% with the fixed random anchor to 47.95%/69.01% (Table 4). On Qwen2.5-Math-1.5B, a single screened anchor raises Pass@8 from GRPO’s 59.43% to 61.96% (Table 6).

I would like to thank the authors for their rebuttal. The responses have addressed my concerns. I will keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised version of the paper, with all changes marked in blue.

  • Sec. 3.2.1 now describes the anchor-bank construction step by step, and Appendix C adds details on the reference token, the clustering procedure, and the bank for Qwen3-1.7B-Base.
  • Sec. 3.2.3 and Sec. 4.2 explain why EDEN learns stably from perturbed rollouts without an additional importance-sampling correction, with the token-level analysis in Appendix F.
  • Sec. 4.2 states the conclusion of the anchor ablation explicitly.
  • Appendix G reports results on Qwen3-1.7B-Base, evaluation on BBH, HumanEval+, and MBPP+, and an analysis of reasoning behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.