acceptodds
Under review as a conference paper at ICLR 2027

How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors

Abstract

Reasoning diversity is crucial for large language models: exploring multiple high-quality reasoning paths broadens solution coverage at inference time and increases the chance of discovering high-reward trajectories during reinforcement learning with verifiable rewards (RLVR). However, a model's reasoning distribution often concentrates on a few solution modes, and RL optimization can further exacerbate this tendency, making exploration increasingly ineffective. The challenge is thus to promote structured reasoning diversity without sacrificing solution quality. To this end, we propose Information-Maximizing Augmented eXploration (IMAX), a framework that learns a lightweight pool of soft prefixes to elicit diverse reasoning paths from a frozen language model. IMAX combines a verifiable correctness reward to steer each prefix toward successful reasoning with a response-level Information Maximization (InfoMax) reward that prevents different prefixes from inducing redundant behaviors. Once learned, these prefixes can broaden solution coverage at inference time and diversify rollouts for subsequent full-policy RLVR. Experiments across three backbone scales and multiple mathematical reasoning benchmarks demonstrate that IMAX improves over matched prefix-tuned RLVR baselines, yielding gains of up to 11.60% in Pass@4 and 8.08% in Avg@4. Beyond these inference-time gains, the learned prefixes also enhance subsequent full-policy RLVR by promoting more effective exploration during training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UMj5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IMAX to improve reasoning diversity through a pool of learned soft prefixes. The backbone remains frozen. Prefixes are optimized using correctness rewards and a response-level mutual information objective. A classifier predicts the prefix identity from each generated response. The paper evaluates inference-time performance and uses the learned prefixes to guide subsequent full-policy RLVR.

Critical strengths
  1. Learning reusable prefixes on a frozen backbone is a practical approach to controlling reasoning diversity. Their use in subsequent RLVR training extends the method beyond inference-time prompting.

  2. Some improvements are substantial. On Qwen3-4B-Base, IMAX improves MATH-500 Pass@4 by 11.60 points and Avg@4 by 7.00 points over GRPO-prefix.

Critical weaknesses
  1. The novelty over closely related work is not sufficiently established. LTPO [1] already uses policy gradients to optimize continuous input vectors while keeping the LLM frozen. Model Whisper [2] learns continuous input prefixes on a frozen backbone through entropy minimization. Their objectives differ from IMAX, but they are directly relevant to its adaptation mechanism. SoftCoT++ [3] explicitly diversifies continuous thought representations. Random Soft Prompts [4] studies embedding-based diversification during inference and RL rollouts. These works are not discussed. UpSkill is discussed but lacks an empirical comparison. The paper needs clearer positioning and matched comparisons to establish the benefit of its learned prefix pool and InfoMax objective.

  2. Unresolved reporting issues weaken the experimental evidence. Tables 1 and 3(a) report very different results for what is described as Qwen3-8B prefix-only IMAX. MATH-500 Avg@4 is 41.00 in Table 1 and 66.80 in Table 3(a). The configuration difference is unclear. Table 6 also reports IFEval Pass@4 as 47.87±0.00, which conflicts with the per-example standard error definition in Eq. (9). Evaluation templates and grading details are incomplete. Aditionally, the appendix shows mixed AIME results that receive little discussion. For example, IMAX reduces 8B AIME24 Pass@4 from 16.67 to 10.00 relative to GRPO-prefix. These issues limit confidence in the claimed robustness. The base-model scores also need clarification. MATH-500 Avg@4 is only 30.00% for Qwen3-4B-Base and 33.65% for Qwen3-8B-Base. The official Qwen3 report gives MATH scores of 54.10% and 60.80%, respectively. These results are not directly comparable because the official evaluation uses full MATH with 4-shot CoT. Still, the large gap warrants an explanation of the evaluation setup.

[1] Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization

[2] Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time

[3] SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning

[4] From Noise to Diversity: Random Embedding Injection in LLM Reasoning

Questions for authors
  1. What checkpoints and training configurations produced Tables 1 and 3(a)? Please explain the large discrepancy in their prefix-only results.

  2. Why are several nonzero, nonperfect Pass@4 scores reported with zero standard error? Please provide corrected uncertainties and clarify whether independent training runs were conducted.

  3. Please specify the evaluation prompts, few-shot settings, generation limits, stopping criteria, and answer extraction rules. What are the truncation and parsing failure rates?

  4. What evidence establishes an advantage over independently resampled random soft prompts and UpSkill under comparable budgets?

  5. Why are AIME results omitted from Table 1? How do the mixed results in Tables 4–6 affect the claimed improvements in reasoning quality and coverage?

  6. Can you conduct experiments using more frontier LLMs, such as Qwen3.5, MiniCPM5?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UMj5

W2 / Q1: Different Checkpoints in Tables 1 and 3(a)

W2: Tables 1 and 3(a) report very different results for what is described as Qwen3-8B prefix-only IMAX. MATH-500 Avg@4 is 41.00 in Table 1 and 66.80 in Table 3(a). The configuration difference is unclear.

Q1: What checkpoints and training configurations produced Tables 1 and 3(a)? Please explain the large discrepancy in their prefix-only results.

The two tables evaluate different models. Table 1 uses Qwen3-8B-Base, as specified in Section 5.1, whereas Table 3(a) uses the post-trained Qwen3-8B, as stated in its caption. Due to space constraints, Table 1 abbreviates the model names and omits the “-Instruct”/“-Base” suffixes, which made this distinction unclear. Thus, 67.20/41.00 and 76.40/66.80 are not results from the same checkpoint. We will restore the full model names in Table 1 and clarify the model used in Table 3(a).

W2 / Q2: Corrected IFEval Standard Errors

W2: Table 6 also reports IFEval Pass@4 as 47.87±0.00, which conflicts with the per-example standard error definition in Eq. (9).

Q2: Why are several nonzero, nonperfect Pass@4 scores reported with zero standard error? Please provide corrected uncertainties and clarify whether independent training runs were conducted.

We thank the reviewer for pointing out this error. The ±0.00 entries are reporting errors, not measured zero variance. These late IFEval runs were not merged into the main uncertainty computation, so missing values defaulted to 0.00. Using Eq. (9) over 541 examples, the corrected Pass@4 standard errors are: Table 4—GRPO-full 2.15, DAPO-prefix 2.15, DAPO+IMAX 2.15; Table 5—GRPO-full 2.13, MERCI-prefix 2.15, DAPO+IMAX 2.14; Table 6—GRPO-full 2.15. We will update these entries in the next revision.

W2 / Q3: Evaluation Protocol and Comparison with Official Qwen Results

W2: Evaluation templates and grading details are incomplete. The base-model scores also need clarification. MATH-500 Avg@4 is only 30.00% for Qwen3-4B-Base and 33.65% for Qwen3-8B-Base. The official Qwen3 report gives MATH scores of 54.10% and 60.80%, respectively. These results are not directly comparable because the official evaluation uses full MATH with 4-shot CoT. Still, the large gap warrants an explanation of the evaluation setup.

Q3: Please specify the evaluation prompts, few-shot settings, generation limits, stopping criteria, and answer extraction rules. What are the truncation and parsing failure rates?

Our results and the official Qwen numbers use different evaluation protocols. We use MATH-500 with four sampled responses, a 2,048-token limit, temperature 0.7, top-k 20, and top-p 0.8 (Table 10), whereas the cited Qwen results use full MATH with 4-shot CoT. All methods in our paper use the same decoding budget.

W2 / Q5: Mixed AIME Results

W2: Aditionally, the appendix shows mixed AIME results that receive little discussion. For example, IMAX reduces 8B AIME24 Pass@4 from 16.67 to 10.00 relative to GRPO-prefix.

Q5: Why are AIME results omitted from Table 1? How do the mixed results in Tables 4–6 affect the claimed improvements in reasoning quality and coverage?

The prefix-only AIME estimates are high variance and are not used as central evidence. AIME has few problems, and our compute-limited prefix-only evaluation uses only four rollouts with a 2,048-token limit. We therefore report the complete results in Tables 4–6 but omit them from Table 1. We agree that the prefix-only AIME results are mixed, so our claim is “most,” not all, benchmark pairs. Separately, the downstream policy improves both AIME24 and AIME25 in Table 2 and AIME24 across all reported K in Table 7.

Reviewer_UMj5OpenReview ↗

Response to the authors

Thanks for your rebuttal. Most of my concern has not been addressed (e.g., Weakness 1, Question 4 & 6). I would like to maintain my rating.

Reviewer_V7R3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the loss of reasoning diversity in reinforcement learning with verifiable rewards (RLVR), and proposes IMAX, which freezes the backbone LLM and learns a pool of soft prefixes that act as distinct reasoning priors. The prefixes are optimized with both a verifiable correctness reward and a response-level variational InfoMax reward that encourages different prefixes to induce distinguishable reasoning trajectories. Experiments across multiple Qwen backbones and mathematical reasoning benchmarks show improved inference-time solution coverage and rollout quality.

Critical strengths
  • The paper is overall well written.

  • The proposed IMAX empirically outperforms vanilla RLVR baselines.

Critical weaknesses
  • There is a series mismatch between the title on Openreview page and the pdf-version title.

  • My major concern is on the novelty of this paper. It seems that the authors simply apply the prefix tuning technique to enhance reasoning diversity of RLVR methods.

  • Experimental comparisons with existing reasoning diversity baselines are lacking, such as UpSkill, SIREN (Rethinking Entropy Regularization in Large Reasoning Models), SCOPE (A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in LLM Reasoning), etc.

Questions for authors
  • How does IMAX compared empirically with SOTA reasoning diversity baselines?

  • How does IMAX perform on all datasets under the Pass@1 measure?

  • How does IMAX perform on other model families beyond Qwen?

2 comments
AuthorsOpenReview ↗

Response to Reviewer V7R3

W2: Beyond Standard Prefix Tuning

W2: My major concern is on the novelty of this paper. It seems that the authors simply apply the prefix tuning technique to enhance reasoning diversity of RLVR methods.

Our contribution is not prefix tuning itself. IMAX learns a pool of controllable reasoning priors and jointly optimizes correctness with a response-level InfoMax reward derived from a variational lower bound on I(Z;Y∣X)I(Z;Y\mid X), explicitly discouraging different prefixes from collapsing to redundant trajectories (Sections 4.1–4.3). The learned prefixes are reusable both for inference-time coverage and for diversifying subsequent full-policy RLVR rollouts, whose resulting policy is evaluated without prefixes. We will revise the Introduction and Related Work to state this distinction more explicitly.

Thank you for the rebuttal. However, the clarification does not sufficiently convince me of the methodological novelty.

Moreover, my remaining concerns have not been addressed, with no additional experiments provided.

I maintain my original rating of 1.

Reviewer_ZTwsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IMAX trains a pool of soft prefixes while freezing the language model. Correctness rewards are supplemented by a classifier-based mutual-information objective that makes responses distinguishable by prefix. The prefixes improve sampled solution coverage and are also used to generate rollouts for subsequent full-policy RL, with prefix-free evaluation.

Critical strengths
  1. Prefix tuning is a practical way to steer exploration without updating the backbone. Random, text-initialized, and reward-trained prefixes provide useful comparisons, and runtime costs are measured.
  2. The second-stage experiment evaluates without prefixes. This supports a training benefit beyond merely changing the test-time prompt.
Critical weaknesses
  1. Prefix identification can be improved through stylistic signatures without creating different solution strategies. The embedding clusters and reasoning-component counts do not separate these possibilities; the counting procedure is also insufficiently specified. The semantic-diversity interpretation remains tentative.
  2. Qwen3-8B prefix-only IMAX reaches 67.2 MATH Pass@4 in Table 1 but 76.4 in Table 3(a). The setup does not explain this gap, making the main results and initialization ablation difficult to reconcile. The latter table also lacks a GRPO-prefix control under its own setting.
Questions for authors
  1. Which training or evaluation settings differ between Tables 1 and 3(a), and what is the matched GRPO-prefix result?
  2. Do responses still exhibit distinct mathematical strategies after removing stylistic markers, as judged independently of the prefix classifier?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ZTws

W2 / Q1: Different models in Tables 1 and 3(a)

W2: Qwen3-8B prefix-only IMAX reaches 67.2 MATH Pass@4 in Table 1 but 76.4 in Table 3(a). The setup does not explain this gap, making the main results and initialization ablation difficult to reconcile. The latter table also lacks a GRPO-prefix control under its own setting.

Q1: Which training or evaluation settings differ between Tables 1 and 3(a), and what is the matched GRPO-prefix result?

The discrepancy comes from different models, rather than inconsistent results. Table 1 uses Qwen3-8B-Base, as specified in Section 5.1, whereas Table 3(a) uses the post-trained Qwen3-8B, as stated in its caption. Due to space constraints, Table 1 abbreviates the model names and omits the “-Instruct”/“-Base” suffixes, which made this distinction unclear. We will restore the full model names in Table 1 and clarify the Table 3(a) caption. “The same prefix-only training protocol” only compares random and text initialization within Table 3(a), which does not include an additional matched GRPO-prefix run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.