acceptodds
Under review as a conference paper at ICLR 2027

STRIDE: Step-Level Trajectory Reasoning Induced Diversity in Exploration

Abstract

Reinforcement learning with verifiable rewards (RLVR) rewards whether a final answer is correct but does not distinguish between different correct reasoning paths. As a result, training can concentrate probability on a small set of solutions, limiting the benefit of repeated sampling. Existing diversity objectives typically score complete responses and assign the same auxiliary credit to every token, even when only a few reasoning steps make a response distinct. We introduce STRIDE, a post-training method that assigns diversity credit at the level of individual reasoning steps. STRIDE represents each parsed step using a frozen reference model and measures its novelty through regularized reconstruction against the steps of the other eligible rollouts and the preceding steps of its own trajectory. Both correct and incorrect rollouts characterize previously explored directions, but auxiliary credit is assigned only to correct responses. We show that the log-transformed step scores exactly decompose a response's marginal contribution to a regularized log-determinant measure of group diversity. STRIDE converts these scores into bounded bonuses and applies them to the corresponding step tokens during GRPO, requiring neither step-level correctness labels nor additional rollouts. Across five mathematical reasoning benchmarks, STRIDE improves macro Pass@32 from 58.6% to 62.4% on Qwen3-1.7B and from 77.8% to 79.9% on Qwen3-4B-Instruct-2507, while maintaining comparable Pass@1. It also increases macro DistinctStrategies@32 from 0.82 to 0.97 and from 0.92 to 0.97, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Reinforcement learning, Post-training, Diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o2g6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on diversity collapse in LLM post-training, particularly the loss of diversity among correct reasoning trajectories. It proposes STRIDE, which measures the novelty of individual reasoning steps through regularized reconstruction and adds step-specific advantages to GRPO. Experiments on five mathematical reasoning benchmarks report improvements in Pass@32 and reasoning diversity, while maintaining comparable Pass@1.

Critical strengths

The paper examines an important problem concerning the diversity of reasoning paths. STRIDE also achieves competitive results against the evaluated baselines, particularly on Pass@32 and the reported diversity measures.

Critical weaknesses

1.The paper discusses diversity collapse in existing post-training methods and reviews approaches to promoting diversity, but does not sufficiently explain why assigning diversity credit to individual reasoning steps would help address this problem. The experiments also do not clearly establish how much of the improvement comes from the step-level design. In Table 3, both Token-uniform and Single-step use only correct rollouts as the novelty reference set, whereas STRIDE includes both correct and incorrect rollouts. Since the reference set also affects the novelty scores, these comparisons do not isolate the contribution of the step-level design.

2.The paper discusses representation-based exploration and existing step-level credit assignment methods, but the experimental baselines are limited to GRPO and its likelihood- and entropy-based variants. Without direct comparisons with these closely related methods, it remains unclear what additional benefits STRIDE provides over existing approaches.

Questions for authors

Equation (5) measures step novelty using a reconstruction cost that combines reconstruction error with an L2 penalty on the coefficients. What motivates including the coefficient penalty in the novelty measure, and how does this reconstruction-based novelty relate to meaningful differences in reasoning strategies?

1 comment
AuthorsOpenReview ↗

Response to the Reviewer.

Thank you for your time and constructive feedback.

  1. Why step-level credit helps. Correct solutions to the same problem usually share most of their steps and differ at only a few points. A response-level bonus gives every token the same credit, so it also rewards the shared steps and weakens the signal at the point where a solution actually diverges. STRIDE splits a response’s novelty across its steps, so the credit goes to the steps that make the solution different. Proposition 3 shows that local and token-uniform allocation differ only through this within-response variation in the bonus.

Matched reference set. We reran both variants with the same all-eligible reference set and settings as STRIDE:

MethodStepLocalP@1P@32S@32
1.7B model
Token-uniform✓×31.1157.310.79
Single-step××31.3258.450.87
STRIDE✓✓31.8062.440.97
4B model
Token-uniform✓×58.8277.220.91
Single-step××59.0977.390.94
STRIDE✓✓59.5279.860.97

STRIDE outperforms both variants on every metric. Token-uniform, which has step scores but no local allocation, does no better than Single-step. The gain therefore comes from local allocation. We have updated Table 3 with these results in the revised version.

  1. STRIDE addresses a specific question: how to assign diversity credit to the reasoning steps that distinguish successful trajectories. Its novelty is conditioned jointly on other eligible rollouts and the response's own prefix, then reinforced locally using final-answer supervision, without step-correctness labels, a learned process verifier, or additional sampled rollouts. This differs from methods such as SSVPO, which estimate steps' contributions to correctness. Our comparisons are organized around this contribution: GRPO provides the outcome-based reference, likelihood and entropy variants represent general exploration incentives, and Table 3 examines alternative granularity and allocation configurations. In particular, the Single-step variant is a controlled response-level counterpart of representation-based exploration, with the same representation, kernel, reference set, and training budget. STRIDE improves over it by 4.0 (1.7B) and 2.5 (4B) points in P@32 and raises S@32 from 0.87/0.94 to 0.97, which isolates the benefit of step-level credit. We have clarified this positioning in the revision (Section 4.3, L436).

Questions For Authors

Thanks for the question. Without the penalty, any target in the span of the comparison steps is reconstructed with zero error, even if that takes large coefficients that cancel each other out. A step could then look "not novel" just because some odd combination of other steps happens to reproduce it. The L2 penalty rules out these fragile fits, so the score reflects how well the comparison steps actually support the target. It also keeps the problem well-posed: for >0, the regularized normal matrix is positive definite and the solution is unique. From Eq. (6),

c(ϕ∣Z)=λϕ⊤(Z⊤Z+λI)−1ϕ,c(\phi \mid Z) = \lambda \phi^\top \left( Z^\top Z + \lambda I \right)^{-1} \phi,

so directions the comparison steps cover well get low scores, and directions they barely cover get high scores. This is closely related to the regularized elliptical bonuses used in exploration.

On the second part, the score measures novelty in the frozen model's representation space, and we do not claim it guarantees a different mathematical strategy. What we want it to pick up is a step that brings in a new reduction or argument compared to the other rollouts and the earlier steps of the same response. A step that only restates something already there is easy to reconstruct and scores low. Our evidence for this is indirect but consistent. With the reference set matched, local allocation gives higher S@32 (point 1 above), and the S@32 judge agrees with blinded human labels on 78.9% of pools. We have made this distinction clear in the revision (line 202, 214).

We hope this answers your questions and would be grateful if you would kindly reconsider your rating.

Reviewer_UGPEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STRIDE adds step-level diversity credit to outcome-based GRPO. It pools contextual hidden states from a frozen reference over parsed reasoning steps, scores each step by its regularized reconstruction residual against other eligible responses and its own earlier steps, and gives the bounded bonus only to correct responses. Incorrect responses still contribute to the comparison set. A Cholesky identity exactly decomposes a response's log-determinant marginal into step terms; the implemented bonus is a transformed version of these terms. Across two Qwen models and five mathematical benchmarks, the paper reports higher macro Pass@32 and broadly unchanged Pass@1, accompanied by higher aggregate semantic and judged strategy diversity.

Critical strengths
  1. A precisely specified novelty heuristic without process labels. STRIDE uses correct and incorrect rollouts as the comparison set while applying the auxiliary bonus only to eligible, verifier-correct responses. The reconstruction score has an explicit definition and bounds, and the Cholesky construction makes its calculation auditable. This is a concrete, implementable approach to auxiliary diversity rewards. I credit the specification and computation, while treating the standard matrix identities as a characterization of the selected geometry rather than independent evidence of effective credit assignment.
  2. Consistent repeated-sampling gains under a shared training format. Relative to GRPO, macro Pass@32 rises from 58.62 to 62.44 for 1.7B and from 77.82 to 79.86 for 4B; the direction is positive on all five benchmarks at both sizes. These ten comparisons share models and evaluation data, so the sign pattern is descriptive rather than an independent significance test. Macro Pass@1 changes only from 31.44 to 31.80 and 59.56 to 59.52. Likelihood and entropy baselines, two diversity measures, a common numbered-step training prompt, and explicit disclosure of single-seed/LoRA/single-domain limitations make the empirical study more informative than a single leaderboard comparison.
Critical weaknesses
  1. Exact geometric decomposition does not establish effective step credit, and the key ablation leaves this gap unresolved. Proposition 2 decomposes a response's log-determinant marginal for fixed representations. The identity holds independently of whether those representations capture meaningful reasoning strategies. Moreover, Eqs. (11)–(12) threshold the reconstruction scores and broadcast the bonus to every token in a step; the resulting auxiliary mass is length-weighted (Eq. (26)), rather than the log-increment sum in Eq. (10). Proposition 3 characterizes the change in the surrogate gradient, but supplies no improvement guarantee. These are valid constructions, and the paper appropriately distinguishes diversity credit from correctness credit. Their empirical justification must nevertheless show why geometric novelty is a useful signal to reinforce locally, rather than merely a response-level diversity heuristic.

  2. Table 3 changes both allocation and the novelty reference set: the variants use correct responses only, whereas full STRIDE uses all eligible responses. The authors acknowledge this. Consequently, the token-uniform gaps (61.72 versus 62.44 Pass@32 at 1.7B; 77.30 versus 79.86 at 4B) support the complete configuration but do not isolate the claimed benefit of local credit. This is central to my recommendation: neither the identity nor the available experiment establishes the paper's main distinction over response-level reinforcement.

  3. The strategy evidence is insufficient to validate the interpretation of the novelty bonus. DistinctStrategies@32 uses GPT-OSS-120B judgments on responses passing correctness and validity checks, without reported human validation or method-wise post-filter counts. This limits our ability to separate substantive strategy variation from judgment or filtering effects. For example, AIME25 1.7B GRPO-entropy has S@32 = 0.13 versus Pass@32 = 30.0%; validity filtering could explain the difference, but its impact should be visible. The paper also gives no training-time distribution of positive step bonuses or their activation frequency, which would help establish how often the proposed signal is material. These limitations concern the diversity mechanism; the independently graded Pass@32 gains remain favorable evidence for the complete method.

Questions for authors
  1. For one representative model, can you compare full STRIDE with a variant that keeps the same all-eligible reference set, step scores, mapped tokens, and per-response auxiliary mass, but distributes that mass uniformly within each response? This single control would isolate whether local allocation contributes beyond the geometric score and total auxiliary reinforcement. It would provide the empirical link that Propositions 2–3 alone cannot establish, and could change my recommendation; no general credit-assignment or convergence theorem is required.

  2. Please summarize bonus activation and nonzero magnitudes from an existing run, give method-wise post-validity-filter and solved-question counts, and confirm shared evaluation prompts, decoding settings, and validity rules. These clarify the AIME25 example and the materiality of the bonus. A small blinded check of existing solutions and judge outputs would further help establish whether high-bonus steps introduce substantive strategy differences rather than restatements. This validation addresses the diversity claim directly, without requiring a broad new experimental suite.

  3. Which verifier and answer-normalization rules were used for non-integer benchmark answers, and were they identical across methods? Can you provide the implementation, executable numerical checks, recorded results, and relevant evaluation code described in the reproducibility statement? These materials would allow the reported construction and evaluation to be checked.

4 comments

Response to the Questions For Authors.

Q1. thank you for your time and constructive feedback. We ran all variants under the same settings on the same all-eligible reference set. We report the result below :-

MethodStepLocalP@1P@32S@32
1.7B model
Token-uniform✓×31.1157.310.79
Single-step××31.3258.450.87
STRIDE✓✓31.8062.440.97
4B model
Token-uniform✓×58.8277.220.91
Single-step××59.0977.390.94
STRIDE✓✓59.5279.860.97

These results support the contribution of local allocation itself, providing the empirical comparison that complements Propositions 2–3. We have revised table 3 in the revised version of the paper.

Q2. bonus activation and nonzero magnitudes from an existing run

ModelActive groupsPositive / eligible correct steps within active groupsMedian nonzero uP90
1.7B16/64 (25.00%)320/337 (94.96%)0.023890.08885
4B44/64 (68.75%)534/544 (98.16%)0.012170.03719

Positivity is common within active groups, while bonus magnitudes vary; we therefore report both activation and magnitude statistics rather than treating threshold passage alone as evidence of substantive novelty.

method-wise post-validity-filter and solved-question counts

For qwen 1.7B, each cell shows post-filter correct responses / solved questions

MethodAIME24AIME25MATH500MinervaOlympiadBench
GRPO104 / 1282 / 1011,312 / 4692,524 / 1476,111 / 360
GRPO-likelihood109 / 1385 / 1011,280 / 4682,507 / 1516,016 / 359
GRPO-entropy58 / 1037 / 910,272 / 4612,263 / 1524,992 / 334
STRIDE116 / 1594 / 1111,184 / 4732,585 / 1596,000 / 362

confirm shared evaluation prompts, decoding settings, and validity rules

The recovered configurations and shared evaluator specify a common prompt requesting reasoning and a boxed answer, with K=32, temperature=1, top-p=1, unrestricted top-k, non-thinking mode, and an 8,192-token total context. Strategy counting includes only final-answer-correct, non-truncated responses, with no additional step-format or proof-validity filter. Questions without qualifying responses remain in the denominator.

A small blinded check of existing solutions and judge outputs

We conducted a blinded human review of the existing solution pools for 40 questions across both methods (80 pools). The pools were unchanged, and method names and the judge’s counts were hidden to avoid influencing the manual assessments. Of the 80 pools, 76 were fully classified and four were flagged for further review. Human and judge counts matched on 60/76 pools (78.9%), with Spearman =0.611. This agreement provides human support for the strategy counts used in S@32 on the audited sample.

Q3. We used Math-Verify 0.9.0 with SymPy 1.14.0 for non-integer answer equivalence. The grader extracts the last boxed answer, using LaTeX/expression extraction when no box is present. Normalization applies LaTeX and notation cleanup and removes only allowlisted trailing unit annotations, without unit conversion. Ordered components retain their order, and multi-answer questions require all components to match. Numeric comparison uses float_rounding=6 and numeric_precision=15; source-specified tolerances additionally allow absolute error up to 1.01. Also, we have updated the github as mentioned in the reproducibility statement. The grader code and configuration were identical across methods.

We hope this answers your questions and would be grateful if you would kindly reconsider your rating.

Thank you for the detailed responses and additional evidence. Your reply addresses most of my concerns, and I am considering increasing my score.

If resources permit, it would be helpful to provide comparative training-loss curves and investigate whether the performance gaps persist when all methods are trained for more steps, or whether they eventually reach similar performance. This would help distinguish faster learning from a sustained advantage. Also,experiments on a larger size base model, such as Qwen3.5-27B, would be valuable for assessing whether the conclusions hold at a larger scale. I view these as useful additional analyses, subject to feasibility, rather than requirements for a broad new experimental suite.

AuthorsOpenReview ↗

Thank You for acknowledging our rebuttal and considering a score increase

We thank the reviewer for their time and careful assessment of our efforts, and for considering increasing the score. We will include the training-loss curves and their analysis in the camera-ready version. If resources permit, we will also extend the experiments on Qwen3.5-27B. Additionally, we have already conducted experiments with a larger base model and on different reasoning tasks (please refer to our response to Reviewer UQ6H). In particular, we evaluated Qwen3-8B on procedural Knights-and-Knaves and house-grid deduction tasks. These experiments provide further evidence of the generalizability of our approach across model sizes and reasoning domains.

To clarify, my suggestion to evaluate a larger model is motivated by the Q1 results: across the two tested model sizes, STRIDE’s gains over the control variants in Pass@32 and strategy diversity decrease as model size increases. It would therefore be useful to assess whether these benefits persist at larger scales.

Reviewer_UQ6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STRIDE, a step-level diversity credit assignment method for RLVR/GRPO post-training that aims to improve reasoning-trajectory diversity under repeated sampling without sacrificing single-response accuracy. STRIDE encodes each parsed reasoning step with a frozen reference model and measures its novelty relative to steps from other eligible rollouts (including incorrect ones) and preceding steps in the same trajectory using a regularized reconstruction score; a Cholesky-based log-determinant decomposition exactly converts a correct rollout’s marginal diversity contribution into per-step bonuses, which are added only to the corresponding step-token advantages of verified-correct rollouts while incorrect rollouts serve as reference directions without receiving credit. Experiments on Qwen3-1.7B and Qwen3-4B across five math reasoning benchmarks show that STRIDE improves Pass@32 and semantic/strategy diversity while keeping Pass@1 roughly comparable to GRPO, though the evaluation is limited to two Qwen models, one seed, math tasks, LoRA training, and an LLM-judge-based strategy metric.

Critical strengths

I think the strengths can be summarized as follows:

Decomposing diversity at the step level is a very reasonable direction.

There are some neat design choices—for example, incorrect trajectories “participate in the comparison but receive no credit,” and the log-determinant marginal can be exactly decomposed into per-step increments. Both are good design ideas.

It appears to yield some improvements in empirical results.

Critical weaknesses

I think the core weaknesses are as follows:

Step-level credit assignment is not particularly novel per se, and step-level granularity is a natural fit for agentic settings—yet this work only evaluates on math tasks.

The experimental scale is too small: training is only conducted on 1.7B and 4B models, and the empirical gains do not look especially pronounced.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to the Reviewer.

thank you for your constructive suggestions. STRIDE’s contribution is its conditional representation-based novelty signal and its local allocation within answer-correct responses.

To examine transfer beyond mathematical benchmarks and the original model sizes, we extend evaluation to Qwen3-8B on procedural Knights-and-Knaves and house-grid deduction, broadening both model scale and task coverage. The 512-puzzle test includes 192 questions above the training difficulty range of levels 1–5.

Evaluation subsetQuestionsGRPO Pass@1STRIDE Pass@1Improvement, pp [95% CI]
All puzzles51294.1095.45+1.35 [0.79, 1.92]
Levels above the training range19289.0692.15+3.09 [1.87, 4.38]
Knights-and-Knaves, levels 6–912893.7795.39+1.61 [0.39, 2.83]
House grids, levels 6–76479.6485.69+6.05 [3.52, 8.84]

Scores are percentages, with 32 sampled responses per puzzle and one trained checkpoint per method. Confidence intervals use 5,000 paired bootstrap resamples over puzzles. The final two rows partition the 192-question harder subset.

Pass@32 is near the ceiling: 99.80% for STRIDE and 100% for GRPO. STRIDE’s advantage is in per-sample accuracy, with a larger 3.09-point gain above the training difficulty range.

For strategy diversity, we use a separate, matched 484-question evaluation and distinguish standard S@32 from a comparison that controls the number of correct responses:

MethodStandard S@32Correct-count-matched diversity
GRPO1.09091.0950
STRIDE1.13841.1393

The strategy set retains 484 of a separate 500 questions after excluding the same 16 grid questions from both methods for judge-context limits. Standard S@32 uses all completed responses with correct final answers.

We would be grateful if you would kindly reconsider your rating in light of them.

Comments on the paper

3 comments
Area_Chair_3P6dOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

AuthorsOpenReview ↗

Please read the response.

Dear Reviewer,

With 1 day remaining of the author–reviewer discussion period, we request you to take a moment to read our response. We have addressed your concerns through additional analyses, experiments, and clarifications.

Thank you

Best regards,

The Authors

AuthorsOpenReview ↗

Gentle Reminder: Request to Consider Our Rebuttal

Dear Reviewer,

As the author–reviewer discussion period is coming to a close, we kindly request you to take a moment to read our response. We have tried to address your concerns through additional analyses, experiments, and clarifications.

Thank you

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.