acceptodds
Under review as a conference paper at ICLR 2027

Higher-Order Exploration in RLOO with Pairwise-Independent Rollouts

Abstract

Sparse rewards can leave an entire rollout group without a learning signal. We introduce HO-RLOO, an autoregressive sampler that induces higher-order dependence while preserving the independent law of every trajectory pair and therefore the expected raw RLOO gradient under ideal continuous randomness. A calibrated mixture of distinct and shared probability strata changes interactions among three and four trajectories without changing pairwise laws. In an aligned sparse-reward model, HO-RLOO reduces gradient variance and achieves maximal group coverage among pairwise-independent samplers. On mathematics and code, HO-RLOO improves domain-balanced pass@1 over iid-RLOO by 2.51 and 2.78 points at two Qwen3 model scales under a common training-token cap. Frozen-policy diagnostics show increased mixed-reward groups and rollout diversity on the evaluated prompt panel. These results demonstrate that preserving pairwise independence provides a principled way to shape higher-order exploration while retaining the original RLOO objective.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Reinforcement Fine-Tuning, Dependent Sampling, Pairwise Independence, Policy Gradient Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_sPaHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a way to improve RLOO’s performance by changing only the sampling strategy, rather than modifying the algorithm itself. The experimental results demonstrate the effectiveness of this approach.

Critical strengths
  1. The paper is easy to follow, and the proposed method is practical, especially in settings where modifying a well-established algorithm is difficult.
  2. The paper takes a novel perspective by shifting the focus from changing the optimization objective to improving the sampling procedure.
Critical weaknesses
  1. The paper could be strengthened by evaluating the proposed sampling strategy on a wider range of baseline models, as its effectiveness may depend on the specific model used.
  2. Figure 1 could be clearer, particularly panel (a), as I found it difficult to grasp the key idea of the proposed method on first reading.
Questions for authors
  1. I wonder whether this method would also be effective for other RL algorithms, such as GRPO, in sparse-reward settings.
3 comments

Thank you for the suggestions on model coverage, Figure 1, and extending the sampler to GRPO. We expanded the model comparison, revised the illustration to make the rollout-level construction easier to follow, and tested the sampler in GRPO-style training.

Highlights

  • We tested the sampler on two additional model families: twelve runs on MiniCPM5-2B and LFM2.5-2.6B yield positive mean gains on AIME25 and LiveCodeBench.
  • Figure 1 now connects numbered rollout identities and token sequences to the stratum assignments in panel (a).
  • We tested transfer to a group-normalized update: six matched GRPO-style runs improve the equal-weight two-benchmark mean by 1.048 pp.

Additional model families. To test the sampler on other model families, we added 12 raw-RLOO runs on MiniCPM5-2B and LFM2.5-2.6B: HO and iid with three matched seeds per model, following the main training protocol. Table 2 in Section 6.1 presents the new results, and Appendix F gives every run. With iid test decoding for both methods, mean pass@1 improves in all four comparisons:

ModelBenchmarkMean HO−iid gain (pp)
MiniCPM5-2BAIME25+0.66
MiniCPM5-2BLiveCodeBench-v5+0.88
LFM2.5-2.6BAIME25+1.35
LFM2.5-2.6BLiveCodeBench-v5+1.18

These results extend the training gains to two further model families. The larger mean gain is on code for MiniCPM5-2B and on AIME25 for LFM2.5-2.6B; Appendix F discusses this model–task variation in relation to reward geometry.

Figure 1. We revised the illustration to show how stratum assignments operate within autoregressive rollout generation. A new top strip identifies four rollouts with numbered, color-coded circles and shows their token sequences in a 2×2 layout. The instruction “At each token, assign strata using (a)” directly connects rollout generation to the distinct/shared mixture. The caption explains how the calibrated mixture yields uniform pairwise cell probabilities, and panel (b) connects fresh tokenwise sampling to complete-trajectory pair independence.

GRPO-style training. To test whether the sampling benefit carries over to a group-normalized learning rule, we added six Qwen3-4B runs using group-standardized advantages and one on-policy update per rollout batch, with HO and iid sharing the training stream, optimizer, and token cap. Both methods use iid endpoint evaluation. The results are:

BenchmarkMean HO−iid gain (pp)
MATH-5000.567
LiveCodeBench1.529
Equal-weight two-benchmark mean1.048

All three run differences are positive on each benchmark. Section 6.3, Table 4, and Appendix E report this group-normalized experiment. Equation (11) shows that normalization rescales mixed-group reward contrasts, while groups with identical rewards retain zero advantages; higher-order coordination therefore remains relevant to the normalized update. At the frozen iid-RLOO halfway checkpoint, the same-group diagnostic shows 7.1% lower normalized-gradient variance, complementing the positive training gains. The added model-family experiments and this tested GRPO-style configuration jointly support the sampler's practical use across the evaluated models and training settings.

The authors' response has addressed my concerns, and I will maintain my original score.

Thank you for your time and constructive feedback. We are glad that our response has addressed your concerns.

Reviewer_Gph4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an interesting sampling scheme (for rollout) -- HO-RLOO, which is a modification of Reinforce with Leave-One-Out (RLOO) that improves exploration through coordinated sampling.

Critical strengths

Theoretically interesting paper with a novel sampling construction, a clean mathematical motivation, and encouraging empirical results. Experimentally, HO-RLOO improves domain-balanced pass@1 over standard RLOO by 2.51 percentage points for Qwen3-1.7B and 2.78 points for Qwen3-4B, with little measured sampling overhead.

Critical weaknesses

One of my main concern is that the "theoretical" variance reduction is not general -- the variance improvement holds only when p(1-p) < (G-1)/(5G-6). In LLM training, rewards are complicated functions of complete trajectories, while policy scores are high-dimensional vectors. So the Bernoulli assumption may therefore be quite restrictive.

For experiments, the authors only use two Qwen3 model sizes, mathematics and coding tasks, and three training seeds. While the reported improvements are promising, the generality remains uncertain.

Questions for authors

See weakness.

3 comments

Thank you for raising the connection between the scalar analysis and LLM training, as well as the model coverage. We strengthened this connection with direct measurements over all trainable LoRA parameters and broadened the training evaluation to two further model families.

Highlights

  • We clarify the role of the scalar model and add direct LLM-gradient evidence for the variance benefit.
  • New measurements over all trainable LoRA parameters show 14.0% and 24.5% lower raw gradient variance at two frozen Qwen3-4B policy stages.
  • Twelve added runs on MiniCPM5-2B and LFM2.5-2.6B show positive mean gains on both AIME25 and LiveCodeBench.

Scope of the theory. The theory separates a general raw-RLOO mean guarantee from an exact scalar analysis of the variance mechanism. The condition p(1−p)<(G−1)/(5G−6)p(1-p)<(G-1)/(5G-6) characterizes the benefit regime in the aligned scalar Bernoulli model. That model assumes success occupies whole probability strata and the score is s=R−ps=R-p, allowing an exact variance calculation. Theorem 4 now states these assumptions explicitly. Theorem 1's mean-preservation result applies to general trajectory rewards and vector-valued policy scores under its pair-law and regularity assumptions. For variance, the general kernel decomposition identifies the three- and four-trajectory covariance terms that HO can change; the scalar result provides an analytically tractable benefit regime. We test the high-dimensional variance effect directly below.

Gradients over all trainable LoRA parameters. To test the variance effect in the actual training parameter space, we added autograd measurements at initialization and an iid-RLOO halfway checkpoint of Qwen3-4B, using 32 fixed validation prompts, equally split between mathematics and code, and 256 groups per prompt and sampler. These are complete trainable-LoRA loss gradients before optimizer preconditioning. Variance is measured within prompts and averaged equally across domains.

Policy stageHO/iid raw gradient variance ratioVariance reduction
Initialization0.86014.0%
Halfway0.75524.5%

Both domain-level ratios favor HO at both stages, with approximately unchanged generated-token cost. Figure 4 and Appendix E report these measurements, extending the earlier projection diagnostic to all trainable LoRA parameters.

Model coverage. To check whether the training gains extend across model families, we added 12 matched raw-RLOO training runs on MiniCPM5-2B and LFM2.5-2.6B, with three seeds per method and model. The respective mean HO−iid gains are 0.66/0.88 pp and 1.35/1.18 pp on AIME25/LiveCodeBench. Both families improve on both benchmarks; LFM2.5-2.6B has lower iid endpoint scores and larger mean gains. Table 2 and Appendix F report the results and discuss how the model–task reward structure can affect the sampling benefit.

I thank the authors for the response. I'm still a bit concerned with the experimental results, so I choose to keep my score as it is.

Thank you for your follow-up and for considering our response.

We have provided additional experimental evidence addressing the concerns raised in your original review. We would appreciate it if you could clarify which specific experimental concerns remain unresolved in light of these new results.

Your clarification would help us better understand the remaining limitations and address them more effectively.

Thank you again for your time and feedback.

Reviewer_E88hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HO-RLOO, a calibrated stratified sampling method that preserves independence between every pair of complete rollouts while introducing higher-order dependence among three or more trajectories. Under the stated assumptions, this preserves the expected raw RLOO gradient while allowing higher-order covariance terms to change its variance. The authors analyze coverage and variance in an aligned sparse-reward model and evaluate the method through mathematics and code training experiments with Qwen3-1.7B and Qwen3-4B. Under a common eight-million-completion-token cap, HO-RLOO improves domain-balanced pass@1 over iid-RLOO by 2.51 and 2.78 percentage points, respectively.

Critical strengths

1.The sampling perspective clearly connects theory and algorithm. The paper distinguishes marginal preservation from pairwise independence and uses this distinction to modify higher-order dependence while preserving the expected raw RLOO gradient. The calibrated stratum mixture provides a concrete autoregressive construction, and the analysis explicitly identifies conditions under which variance improves or worsens.

2.The experiments provide credible evidence of end-to-end gains. The 18 training runs use matched seeds, initialization, prompt streams, optimizers, and token caps. Unified iid evaluation helps attribute improvements to the learned policy. Positive paired-seed confidence intervals, resource measurements, and unfavorable diagnostic outcomes further strengthen the reporting.

Critical weaknesses

The theoretical benefit depends on reward geometry that is not sufficiently characterized in realistic tasks. Theorem 2 assumes alignment between success regions and probability strata, but fixed token-ID intervals have no inherent correspondence with sequence-level success. Although this limitation is acknowledged, sensitivity to token ordering remains unclear. Testing alternative fixed vocabulary permutations would help establish robustness.

Questions for authors

How sensitive are the coverage and variance improvements to the fixed token-ID ordering? Could the authors evaluate a few alternative vocabulary permutations using the existing frozen-policy protocol?

3 comments

Thank you for highlighting vocabulary-order sensitivity. We added a controlled comparison of fixed vocabulary permutations to examine how coverage and variance effects change with the inverse-CDF ordering while holding the policy fixed.

Highlights

  • We directly tested vocabulary-order sensitivity using the original token-ID order and seven fixed random permutations at two frozen policy stages.
  • Positive coverage-gain estimates persist across all tested orders at both policy stages; projected variance and cost–variance point estimates also favor HO in all 16 settings.
  • Mean coverage gains across the seven random orders are 3.990 pp initially and 1.869 pp halfway.

Vocabulary-order sensitivity. To separate ordering sensitivity from policy learning, we run the comparison at initialization and the halfway iid-training checkpoint of Qwen3-1.7B. At each stage, we compare identity token-ID order with seven fixed random permutations (R1–R7), using the same eight prompts, G=16G=16, and 64 complete groups per prompt and sampler. Coverage is the fraction of groups containing at least one success. The summary below reports HO−iid coverage gains in percentage points; the random-order column gives the mean across R1–R7.

Policy stageIdentity: coverage gain (pp)R1–R7: mean coverage gain (pp)
Initialization4.1023.990
Halfway2.1481.869

Positive coverage-gain estimates persist across all tested orders at both stages on this evaluated panel. Across the seven random orders, they range from 2.734 to 5.078 pp initially and from 1.367 to 2.344 pp halfway. Projected variance and token-cost–variance point estimates also favor HO in all 16 settings.

The theory explains why each fixed permutation preserves the ideal marginal and pairwise trajectory laws under the construction's assumptions; the added experiment measures how coverage and variance respond to the ordering change. Section 6.2 and Figure 3 present this diagnostic; Appendix D.1 contains the results for all eight orders and the direct identity contrasts.

Thank you for adding the vocabulary-permutation experiments. These experiments directly address my question, and the positive coverage gains across all tested orders at both policy stages substantially alleviate my concern that the results depend on a particular token-ID ordering.

To better assess the strength of this evidence, could you please use the existing experimental samples to provide the following statistics directly in your response?

  1. Per-order estimates with 95% confidence intervals. For the identity order and R1–R7 at each policy stage, please report the HO−iid coverage difference, the HO/iid projected-gradient variance ratio, and the HO/iid token-cost–variance ratio, together with their confidence intervals. This would clarify which improvements have clear statistical support and which remain favorable point estimates.

  2. Direct identity-versus-random comparisons. At each policy stage, please report the difference between the gain under the identity order and the mean gain across R1–R7, with a 95% confidence interval. This would quantify the magnitude of ordering sensitivity beyond the consistency of the observed directions.

  3. A brief description of the uncertainty estimation. Please preserve complete rollout groups as the resampling unit, rather than treating responses within a group as independent, and clarify how variability across the eight prompts is accounted for.

These clarifications would help distinguish consistent positive trends across the tested orders from stronger statistical evidence that the magnitude of the benefits is robust to vocabulary ordering.

Thank you for the constructive follow-up. We are glad the permutation experiments addressed your concern. The main ordering results were included in Appendix D.1, Tables A20–A21, of our previously uploaded revision. Below, we provide all requested estimates and confidence intervals directly, using the same experimental samples, and clarify how the intervals account for variation across groups and prompts.

Highlights

  • Coverage gains are positive in all 16 settings; the 95% CIs exclude zero for all eight orders at initialization and six of eight orders halfway through training.
  • The intervals incorporate complete-group resampling and prompt resampling stratified by mathematics and code.

1. Per-order estimates with 95% confidence intervals.

Coverage is the fraction of groups containing at least one success. All brackets below are 95% CIs. Positive coverage differences and ratios below one favor HO; VV denotes projected-gradient variance and cVcV denotes token-cost–variance.

Policy stageOrderCoverage difference, HO−iid (pp)VV ratio, HO/iidcVcV ratio, HO/iid
InitializationIdentity4.102 [0.781, 8.594]0.911 [0.784, 1.044]0.916 [0.781, 1.054]
InitializationR13.906 [0.586, 8.398]0.913 [0.786, 1.046]0.919 [0.784, 1.058]
InitializationR24.883 [0.977, 9.766]0.913 [0.785, 1.046]0.917 [0.783, 1.057]
InitializationR32.734 [0.391, 6.250]0.917 [0.790, 1.048]0.922 [0.786, 1.059]
InitializationR44.492 [0.977, 9.180]0.911 [0.783, 1.045]0.916 [0.781, 1.055]
InitializationR53.906 [0.586, 8.398]0.918 [0.790, 1.051]0.925 [0.789, 1.063]
InitializationR65.078 [0.977, 10.352]0.909 [0.780, 1.043]0.914 [0.778, 1.054]
InitializationR72.930 [0.391, 6.641]0.918 [0.791, 1.048]0.923 [0.787, 1.059]
HalfwayIdentity2.148 [0.195, 5.078]0.887 [0.723, 1.045]0.878 [0.700, 1.052]
HalfwayR11.953 [0.195, 4.492]0.890 [0.725, 1.048]0.881 [0.702, 1.054]
HalfwayR22.148 [0.195, 5.078]0.887 [0.723, 1.045]0.878 [0.700, 1.052]
HalfwayR31.367 [0.000, 3.516]0.902 [0.738, 1.056]0.890 [0.712, 1.061]
HalfwayR42.148 [0.195, 5.078]0.887 [0.723, 1.045]0.878 [0.700, 1.052]
HalfwayR51.758 [0.195, 4.102]0.890 [0.726, 1.048]0.881 [0.702, 1.054]
HalfwayR62.344 [0.391, 5.273]0.886 [0.722, 1.045]0.876 [0.699, 1.050]
HalfwayR71.367 [0.000, 3.516]0.901 [0.737, 1.056]0.888 [0.710, 1.060]

The halfway coverage intervals for R3 and R7 reach zero. Both variance-related ratios favor HO in all 16 point estimates, while their intervals include one. This distinguishes the statistically supported coverage improvements from the favorable variance and cost–variance estimates.

2. Direct identity-versus-random comparisons.

For each stage, we report the identity coverage gain minus the mean coverage gain across R1–R7, with its directly estimated confidence interval, as reported in Table A21.

Policy stageIdentity gain − mean R1–R7 gain (pp)95% CI
Initialization0.112[−0.502, 0.809]
Halfway0.279[−0.056, 0.949]

Both estimated differences are small, and neither interval excludes zero.

3. Uncertainty estimation.

We use 10,000 nested bootstrap replicates. At the prompt level, each replicate samples four mathematics prompts with replacement from the four mathematics prompts and four code prompts from the four code prompts. Aggregation gives equal weight to prompts within each domain and equal weight to the two domains. All orders and both samplers share the same prompt draws, so ordering comparisons use the same prompt composition.

Within each selected prompt, we resample 64 complete rollout groups with replacement from its 64 groups. Each group's 16 responses, rewards, group gradient, and token cost remain together, preserving the dependence structure used by coverage and the RLOO estimator. The requested statistics are recomputed in each replicate. Thus, the intervals account for both rollout-group sampling variation and variation in the contributions of the eight prompts.

Reviewer_wggkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on the problem in LLM policy optimization where the same reward for all rollouts in a group causes there not to be any gradient signal. Authors propose a method to sample from the policy so that this is less likely to happen, called HO-RLOO.

Critical strengths
  1. This is an important problem. Spending lots of compute on rollouts only not to obtain gradient signal can be costly.
  2. The results are promising. Experiments feature two different model sizes and many benchmarks tested.
  3. The algorithm appears to be lightweight in terms of implementation and overhead.
Critical weaknesses
  1. I find the paper very difficult to follow, starting from section 3. The notation and some definitions are not clear or explained, such as the symbol ⊗\otimes in "admissible group laws." The ordering of results also makes the paper harder to read: Corollary 1 appears before any theorem is stated in the main text, and several theorems are referenced in the main text but stated and proved only in the appendix. I'd encourage the authors to (i) add a short plain-language summary of each main result, (ii) state all assumptions explicitly in the theorem statements, and (iii) reorder the results so each corollary follows the theorem it depends on.
  2. For training, empirical details are missing. Is there KL regularization? Why is the objective unclipped? It would be good to justify such choices, and to state all choices that with respect to implementation in the main paper rather than only in the appendix.
  3. The benchmarks are evaluated at different temperatures: 0.6 for MATH-500, AIME 2024 and AIME 2025, and 0.2 for LiveCodeBench. What justifies this choice?
Questions for authors

See above.

5 comments

Thank you for identifying the definitions, result order, and training choices that needed clearer explanations. We improved the presentation of the formal argument, made the key training choices explicit, and added a direct temperature-sensitivity check.

Highlights

  • We improved the presentation by reorganizing the formal development around its logical dependencies, with explicit assumptions and plain-language explanations.
  • We made the training objective and its connection to the theory explicit in Section 5.2, including KL, clipping, and the behavior-to-target correction.
  • We explain the task-specific temperature choices and show positive HO−iid domain-average gains at all three tested temperatures and both model sizes.

Definitions and result order. We reorganized Sections 3–4 to make the formal path easier to follow: from the raw RLOO estimator, through the pairwise constraint and uniform construction, to the complete-trajectory guarantee. Section 3 defines Q2(pθ)\mathcal Q_2(p_\theta) as the set of rollout-group laws for which every distinct complete-trajectory pair has law pθ(⋅∣x)⊗pθ(⋅∣x)p_\theta(\cdot\mid x)\otimes p_\theta(\cdot\mid x). Theorem 1 states the pair-law and regularity assumptions explicitly. Its plain-language explanation is that raw RLOO averages two-trajectory kernels: preserving each pair's law preserves the expected gradient, while higher-order dependence can change variance. Section 4 presents the uniform construction theorem (Theorem 2) before its trajectory corollary (Corollary 3), and explains how fresh tokenwise randomness carries pairwise independence to complete rollouts. The statements include the assumptions on within-stratum randomness, logical lanes, and prefix-local decoding.

KL and clipping. Our primary comparison isolates the sampling intervention under the raw-RLOO objective analyzed in the theory. Section 5.2 now explicitly specifies raw, unclipped RLOO without KL regularization (β=0\beta=0), group normalization, or length normalization. We also explain the distinction between the cached behavior law Q\mathsf Q and scoring target P\mathsf P: the detached, unnormalized proposal weight P/Q\mathsf P/\mathsf Q corrects their difference and equals one when they coincide.

Evaluation temperatures. We clarified why evaluation uses task-specific temperatures and added a sensitivity check. We selected T=0.6T=0.6 for mathematics to retain variation in reasoning paths and T=0.2T=0.2 for code to favor concentrated generation for programs with strict syntax and execution requirements. These task-level choices follow customary benchmark practice and our evaluation experience. All methods train at T=1.0T=1.0 and share the same iid evaluation protocol within each benchmark, including temperature, top-p, sample count, and token cap. We additionally tested evaluation temperatures 0.2,0.6,1.00.2,0.6,1.0. HO−iid domain-average pass@1 gains are positive throughout this grid at both model sizes. Appendix D.3–D.4 reports the complete sensitivity results.

Thank you for the clarifications and additional experiments. These address some of my concerns, although aspects of the notation could still be explained more explicitly, such as the meaning of ''⊗\otimes''.

I remain concerned that the empirical gains are modest and their strength depends on evaluation temperature. For each of the eight model–benchmark combinations (1.7B and 4B model, and the four benchmarks), the originally selected evaluation temperature yields the largest observed HO−iid accuracy gap among the three temperatures tested. Further, the appeal to “customary practice” does not identify a specific benchmark protocol supporting these choices. Although aggregate gains remain positive at other temperatures, clarifying how the original temperatures were selected would help address this particular concern.

I also find the connection between the theoretical improvement guarantees and the LLM experiments limited. While unbiasedness is established, the variance reduction guarantee relies on a simplified Bernoulli model.

Thank you for the follow-up and for identifying the remaining questions. We have made the requested notation explicit and added a continuous-reward variance result. We also clarify how the original evaluation temperatures were selected and provide specific public evaluation references.

Highlights

  • Section 3 now explicitly defines ⊗\otimes and explains the distinction between pairwise independence and dependence within the full rollout group.
  • A new Gaussian result extends the exact variance analysis beyond Bernoulli rewards; direct measurements over all trainable Qwen3-4B LoRA parameters show 14.0% and 24.5% lower raw-RLOO gradient variance.
  • The original temperatures were fixed before the revision-stage sensitivity study. Public evaluation protocols support both choices, and aggregate gains persist across the tested temperatures.

Notation. In Section 3, ⊗\otimes denotes the product distribution: Qij=pθ⊗pθQ_{ij}=p_\theta\otimes p_\theta means that responses ii and jj are independent draws from pθp_\theta. Each pair therefore has the iid policy law, while the full group may retain higher-order dependence. We now state this directly after the definition of Q2(pθ)\mathcal Q_2(p_\theta).

Variance analysis and its connection to LLM training. To extend the explicit improvement result beyond Bernoulli rewards, we added Proposition 9 in Appendix A.13, with a pointer in Section 5.1. For a Gaussian location policy with reward R(X)=XR(X)=X, raw RLOO becomes a standardized sample-variance estimator. Applying the same calibrated HO sampler, including independent within-stratum randomness, gives an exact variance formula. At G=16G=16, its variance is 0.08962 versus 0.13333 under iid sampling, a 32.78% reduction with the same expected gradient. This complements Section 5.1's general covariance decomposition, which applies to general rewards and vector-valued policy scores.

The connection to LLM training is also measured directly in Section 6.2, Figure 4, and Appendix E. At initialization and the halfway iid-RLOO checkpoint of Qwen3-4B, HO reduces raw-RLOO gradient variance over all trainable LoRA parameters by 14.0% and 24.5%, respectively, with essentially unchanged average generated-token cost. These measurements use 32 validation prompts and 256 groups per prompt and sampler, and evaluate gradients before optimizer preconditioning. They directly test the variance mechanism in the LLM training parameter space.

Temperature selection and sensitivity. The original temperatures, T=0.6T=0.6 for mathematics and T=0.2T=0.2 for code, were fixed based on prior evaluation experience and external evaluation code. The temperature sweep was added during revision to assess sensitivity. The fact that the original settings give the largest observed gaps in this grid was observed afterward; it was not the criterion used to select them.

For concrete public precedents:

The sensitivity study tests whether the overall advantage persists beyond those settings. Applying a single temperature to all benchmarks at T∈{0.2,0.6,1.0}T\in\{0.2,0.6,1.0\} gives positive mean macro gains throughout: 1.69–2.12 pp for 1.7B and 1.99–2.65 pp for 4B. Domain-average gains also remain positive at every tested temperature for both model sizes. Appendix D.3–D.4 reports the complete results, showing that the aggregate advantage persists across alternative evaluation temperatures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.