acceptodds
Under review as a conference paper at ICLR 2027

MIGE: Multi-Level Influence-Guided Exploration in Reinforcement Learning for LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves reasoning in large language models, but policies can collapse to a small set of familiar solution paths. Existing exploration methods mainly preserve token entropy or reward trajectory diversity. Yet high token uncertainty need not mark a consequential reasoning choice, and textually diverse trajectories may follow the same underlying strategy. To address this limitation, we propose Multi-Level Influence-Guided Exploration (MIGE), which guides exploration according to how decisions affect subsequent reasoning. At the strategy level, MIGE computes the log-likelihood of complete executions under alternative strategy contexts to reward strategies with distinct downstream effects. At the step level, it resamples continuations at candidate token positions and rewards forks that change the next intermediate conclusion. Across six mathematical and scientific reasoning benchmarks and two model scales, MIGE achieves the best average performance among standard RLVR and representative exploration baselines while largely preserving successful-solution coverage at Pass@. Controlled analyses show that influence scores distinguish strategy changes from surface rewrites and locate reasoning forks more reliably than output-distance and uncertainty baselines, supporting downstream influence as an effective signal for meaningful exploration.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6jM7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MIGE rewards exploration based on how a decision affects later reasoning. At the strategy level, it swaps strategy contexts and compares the probabilities of the full reasoning sequences. At the local level, it generates branches at high-entropy positions and measures their effect on the next intermediate conclusion. These rewards are assigned to strategy tokens and branching tokens, respectively. Experiments on Qwen3 1.7B and 4B report gains on math and science question answering, with metric validation and ablation studies.

Critical strengths
  1. The paper addresses a useful question: token uncertainty or differences in wording do not necessarily mean different solution strategies. Measuring changes in later reasoning offers a more concrete way to study exploration.

  2. Beyond final accuracy, the paper examines strategy rewrites, local branches, and the two reward components. These analyses help explain what the proposed influence measures capture.

Critical weaknesses
  1. The method depends heavily on hand-designed rules for identifying strategies and reasoning steps. The reported parsing coverage is useful, but does not establish whether the extracted content is semantically correct. It is also unclear whether these rules generalize to other models and output formats.

  2. The main experiments use different prompts for MIGE and the baselines. The same-prompt comparison in Table 7 examines solution routes on only one problem, so it does not resolve this concern for the benchmark results. The extra computation also makes gains at the same compute budget unclear.

Questions for authors
  1. Beyond the single-problem comparison in Table 7, do the benchmark accuracy gains remain when the baselines use the same strategy-first prompt?

  2. How do the authors check whether the extracted strategies and intermediate conclusions are semantically correct, rather than merely matching the expected format? After accounting for branch generation and cross-scoring, how much does MIGE improve over additional ordinary sampling or existing branching methods?

Reviewer_uzwgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIGE, a method designed to encourage meaningful exploration in LLM reasoning by quantifying the influence of high-level strategies and low-level reasoning forks. MIGE integrates both types of influence-based rewards into GRPO through a multi-level credit assignment mechanism, assigning strategy-level rewards to strategy tokens and fork-level rewards to selected reasoning tokens. This design aims to provide more fine-grained learning signals that encourage effective exploration at both the strategic and local reasoning levels.

Critical strengths
  1. Importance of the Research Problem. The paper addresses an important problem in RLVR by distinguishing meaningful reasoning exploration from superficial token-level uncertainty and textual diversity. This distinction is valuable for developing more effective exploration mechanisms in LLM reasoning.

  2. Comprehensive Analysis and Ablation Studies. The paper provides additional analyses of strategy influence and fork localization, along with ablation studies examining the individual contributions of the two exploration components. These analyses help characterize the proposed method and investigate the roles of its key design choices.

Critical weaknesses

Weakness 1: Limited Methodological Novelty

The novelty of the proposed method appears relatively incremental. In reasoning tasks, prior work has explored using token-level signals, such as entropy, to improve exploration and diversity. In open-ended generation, methods such as DPWriter have investigated the influence of high-level strategies on output diversity. MIGE appears to combine these two research directions, while its methodological contribution beyond this combination remains insufficiently established.

Weakness 2: Unclear Alignment Between the Explicitly Generated Strategy and the Actual Reasoning Process

MIGE requires the model to explicitly generate an overall strategy before performing step-by-step reasoning. However, it is unclear whether this initial strategy faithfully represents the reasoning process the model ultimately follows.

For example, what happens when the model discovers a more effective approach during execution that deviates from its initially generated strategy? Does MIGE recognize and accommodate such deviations, or does the mismatch between the initial strategy and subsequent reasoning undermine the reliability of strategy-level influence estimation and credit assignment? Further clarification and empirical analysis are needed to establish the robustness of the method to dynamic changes in reasoning strategy.

Weakness 3: Insufficient Comparisons with Closely Related Methods

The paper primarily compares MIGE against vanilla GRPO, DIVER, and GRPO with entropy-based advantages. However, recent methods such as ROSE, DSDR, and ExTra have also investigated related aspects of semantic diversity, branching-based exploration, or multi-level exploration.

The absence of comparisons with these closely related approaches makes it difficult to assess MIGE's effectiveness and novelty relative to the existing state of the art. Additional comparisons, or a more systematic discussion of the methodological differences and complementary contributions of these methods, would strengthen the empirical evaluation and clarify MIGE's position within the broader literature on exploration in LLM reasoning.

Questions for authors

Please refer to the weaknesses.

Reviewer_juHrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MIGE, which adds two exploration rewards to GRPO that measure how a decision affects later reasoning rather than token entropy or output diversity. At the strategy level, it compares the length-normalized likelihood of each execution under its own strategy and under other rollouts' strategies, and rewards strategy tokens by the average distance to the nearest neighbors. At the step level, it resamples branches at high-entropy positions and rewards positions where each branch's next intermediate conclusion is supported mainly by its own context (Eq. 5). On Qwen3-1.7B and Qwen3-4B, MIGE gives the best average over six math and science benchmarks (31.3 and 45.4, vs. 30.1 and 42.9 for Vanilla GRPO) while largely preserving Pass@64 coverage.

Critical strengths

S1. A well-motivated exploration signal. Defining meaningful exploration by downstream influence addresses a real weakness of entropy- and diversity-based methods, which can reward lexical variation that follows the same reasoning. The fork-level reward (Eq. 5, L234–244) is a natural, information-theoretic way to separate genuine reasoning forks from wording changes, and Figure 4 (L450) illustrates the difference clearly: entropy peaks after the decision is made, while influence peaks at the operation token. S2. Both levels are analyzed and contribute. Each level is examined separately (Section 4.3), and the ablation shows complementary gains: removing high-level or low-level exploration lowers the 4B average from 45.4 to 44.2 or 43.4 (Table 4, L456–460). At 4B, MIGE ranks first on all six benchmarks (Table 1, L299–315).

Critical weaknesses

W1. MIGE and the baselines use different prompts, which confounds the main comparison. MIGE appends "Output your overall strategy first..." (Appendix C.1, L721–727), whereas all baselines use a plain step-by-step prompt (Appendix C.2, L730–735). Section 4.1 (L290–294) does not mention this difference. A strategy-first format can change both accuracy and diversity independently of the influence rewards. The only matched-format comparison (Table 7, L1059–1068) measures route frequencies on a single problem with 128 samples (18 vs. 25 cubic-route responses). The GRPO checkpoint in that table is evaluated with, but as far as I can tell not trained with, the strategy-first prompt. Table 1 (L299–315) therefore needs a GRPO baseline that is trained and evaluated with the same strategy-first prompt. Without it, the average gains over Vanilla GRPO (30.1→31.3 and 42.9→45.4) cannot be attributed to the influence rewards. W2. The validation experiments in Section 4.3 are small, and the fork-localization test is close to circular. Table 2 (L370–377) does not state the number of examples. The reported accuracies (88.89%, 70.37%, 62.96%) correspond to 24/27, 19/27 and 17/27, so the comparison appears to rest on about 27 pairs, with no confidence intervals. In Table 3 (L404–409), a position is labeled a fork when resampling from it produces substantively different next conclusions (L393–395). Fork influence (Eq. 5, L234–244) is computed by resampling from the same position and measuring how differently the branches support their next conclusions. The label and the score therefore measure nearly the same quantity, so a high hit rate mainly shows internal consistency, not that fork influence finds decisions that matter for correctness. An external criterion would make this evidence much stronger, for example whether intervening at the top-ranked position fixes the error (as in the Figure 4 example), or human-annotated error locations.

Questions for authors

Q1. Baselines. The related-work section discusses FR3E (L111–112), ROSE (L113–115), DSDR (L127–128) and ExTra (L129–131) as the closest entropy- and diversity-based alternatives, and ROSE's semantic entropy is even implemented for Table 3 (L398). However, the RL comparison (L290–292) includes only DIVER and GRPO with entropy advantage. Could you add at least ROSE and one of DSDR or ExTra, under the same prompt and compute budget?

Comments on the paper

1 comment

Thanks for the authors' response and the extended experiments, I maintain my original positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.