acceptodds
Under review as a conference paper at ICLR 2027

STRATA: Preserving Rare Solution Strategies in Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) is usually audited with pass@k, which overlooks the diversity of correct solution strategies. On a controlled benchmark with exhaustively enumerated, generator-verified strategy sets, we measure rare-strategy survival and find a dose-response decoupling: as GRPO training grows from 0 to 14.9M supervised tokens, AIME24 pass@64 rises (.400 → .467) while survival falls overall (1.000 → .122). DPH-RL preserves pass@k but retains only .146 of rare strategies; WiSE-FT weight interpolation reaches .390. We introduce STRATA (STrategy-Targeted Rehearsal Anchoring): verified teacher traces are re-scored under the current policy every R steps, and cross-entropy rehearsal targets low-likelihood traces at risk of being forgotten. At 3M supervised tokens—one fifth of overtrained GRPO’s budget—STRATA matches its best AIME24 pass@64 (.467), ties best AIME25 (.433) and AMC23 (.925) in our twelve-condition matrix, and retains 2.6× standard GRPO’s rare strategies (.626 vs .244) and 5.1× overtrained GRPO’s (.626 vs .122). Five-seed results, matched-compute controls, selector validation, and a manually annotated natural-solution audit support the benefit across model families and evaluation regimes. We release all code, the benchmark, and the matched audit protocol for reuse.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, solution strategy diversity, rare-strategy preservation, targeted rehearsal, catastrophic forgetting, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JoytOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper points at a problem that mentions that for RLVR, pass@K metrics measure whether a model is able to preserve correctness, but not whether it is capable of preserving diverse solution strategies. The authors experimentally demonstrate that even though GRPO is able to preserve pass@K in their settings, the diversity of solutions reduces significantly.

For this, they provide a method STRATA which preserves such rare solution strategies via a weighted SFT-like objective on a teacher's rollouts during the online RL phase. The authors demonstrate improvements using this technique over other training objectives such as vanilla GRPO and variants of On policy distillation (OPD).

Critical strengths

The papers strength is that the idea is simple enough to use in practice, and can potentially be useful in preserving coverage measured by in solution strategy diversity. The experimental design is also clear and evaluations are done on multiple benchmarks.

Critical weaknesses

There are a couple of weaknesses in the paper:

  1. It is very difficult to parse the paper owing to many undefined terms such as 'survival', 'dose-response decoupling', 'cadence', 'audit', and 'anchors'.
  2. The experiments use Qwen3-1.7B which I'm assuming is the thinking version of the model and not the base model. However, this points to a clear flaw in the experimental design which is the choice of number of generated tokens during inference which is 1024. Thinking model generates far longer CoT of the order of 32k tokens, and this is required to match the official numbers of the Qwen3 technical report. Such a short context is only going to train the model to compress its CoT and not learn anything meaningful. If the authors wish to show genuine improvement, it would be a better choice to train the base model which has much shorter CoT and can learn meaningfully.
  3. The experiments use a teacher model, which is Qwen3-8B in their experimental setup. I wonder if this choice changes the interpretation of the results. For instance, can the teacher model be an earlier checkpoint of the student model itself which is more diverse in terms of solution strategies. It would be impressive if the authors can demonstrate gains in settings where there is no need for extra models to generate diverse traces.

My concerns are written in the same order of their priority, 1 > 2 > 3. 3 is not a critical weakness but I would appreciate if the authors could shed more light on this question. Overall I like the idea but the execution and especially the writing need more consideration.

Questions for authors

My primary question would be point 3 in the weaknesses. How would the results look like if there is no separate teacher model?

Reviewer_h3qnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how RLVR can reduce the diversity of correct solution strategies despite improving answer-level performance. The authors introduce Rewrite-Math to measure strategy retention and propose STRATA, which supplements GRPO with selective rehearsal of low-likelihood, verified teacher solutions.

Critical strengths

The problem formulation is well motivated. The paper highlights an limitation of pass@k, where improved answer accuracy does not necessarily imply preservation of diverse solution strategies. And I agree with that point.

Critical weaknesses
  1. Performance is sensitive to hyperparameters, and the ablation metrics are selective.

The strongest results are concentrated at the default λ=0.3\lambda=0.3. In Table 4, changing λ\lambda to 0.15 or 0.6 reduces MATH-500 pass@1 from 0.713 to 0.691 and 0.655, both below the token-matched GRPO baseline of 0.700. Table 13 shows similar sensitivity to selector configurations.

Furthermore, these ablations (Table 4) report only AIME24 pass@64 and MATH-500 pass@1, rather than consistent pass@1/pass@64 results across benchmarks. This makes it difficult to assess whether the accuracy-diversity trade-off remains favorable under alternative settings. The paper also does not clearly explain how the default hyperparameters were selected or whether baselines received comparable tuning.

  1. The efficiency claims rely on limited computational comparisons.

Table 8 provides a useful wall-clock comparison and accounts for online rehearsal overhead. However, it is unclear whether the reported GPU-hours include offline teacher-trace generation and verification.

The reported 4.6× GPU-time reduction compares STRATA’s 3.7 A100-hours with overtrained GRPO’s 17.2 A100-hours (Table 9), but it does not establish the same speedup over the standard 5.3M-token GRPO run. Could the authors report preparation-inclusive GPU costs, including the standard 5.3M-token GRPO run, alongside their accuracy and strategy retention?

  1. Training specifications and length-related effects remain insufficiently documented.

The paper does not report the total optimizer steps for each run. Also, the paper uses a non-thinking model with a 1024-token rollout budget. Depending on the task, this limit could prevent valid strategies from completing, confounding strategy availability with completion within the length budget. Effectiveness in longer, thinking-mode training remains untested.

Questions for authors
  1. How were the default hyperparameters selected? Could the authors report pass@1 and pass@64 results for more benchmarks (e.g., MATH-500 pass@1/pass@64 and AMC23 pass@1/pass@64) in Tables 4?

  2. Could the authors provide total GPU-hours, optimizer steps, generated rollout tokens, and rehearsal tokens, including offline teacher preparation costs? How do GRPO and STRATA compare in GPU-hours at the lower target of AIME24 pass@64 = 0.433?

  3. Is STRATA effective in thinking mode with longer reasoning traces (more than 1,024-token budget)? How do the strategy retention benefits and computational overhead change?

  4. (Not intended to point out weaknesses) Beyond the existing human-validated audit, do the gains persist under an LLM-as-a-judge evaluation of underlying solution strategies? I don’t consider that the absence of this to be a weakness.

Reviewer_mV7ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper describes a modification of the GRPO loss function, called STRATA, which encourages the diversity of correct solution strategies. As a metric for diversity of solution strategies, the authors introduce rare-strategy survival metric defined as the fraction of retained low-likelihood successful strategies sampled from baseline policy. In particular, the authors introduce an additional rehearsal-loss term Lreh\mathcal{L}_{\text{reh}} (weighted by a parameter λ=0.3\lambda = 0.3) maximizing the policy likelihood of selection of strategies with smallest log probabilities from verified teacher traces. The authors report that STRATA retains overtrained GRPO's AIME24 pass@64, while maintaining larger rare-strategy survival rate (defined relative to the base GRPO policy).

Critical strengths
  • Loss regularization using the rehearsal term Lreh\mathcal{L}_{\text{reh}} appears to shape training dynamics in a measurable way without adversely affecting the performance.
Critical weaknesses
  • The definition of rare-strategy survival metric may lead to contradictory conclusions/rankings for different values of nn. In particular, in Table 10, for sample size n=64, On-policy FKL and STRATA are reported to have rare-strategy survival of 0.610 and 0.626, respectively, while for n=128n=128, the reported rare-strategy survival rates of 0.637 and 0.512 lead to opposite conclusion. Furthermore, the definition is heavily influenced by the base policy, and it is unclear whether is might be possible to obtain substantially different ordering of any collection of policies by simply changing the base policy relative to which the rare-strategy survival rate is computed.
  • The training curves for STRATA in Figure 2 are limited to a single checkpoint at around 3M supervised tokens, whereas the training curves for GRPO span the full domain. The paper might benefit from a more comprehensive analysis of the training curves of STRATA.
Questions for authors
  1. How does STRATA compare to standard entropy regularization methods for GRPO?
  2. The paper might benefit from having the training curve spanning the same range as the baseline GRPO in Fig. 2. This would help understand the asymptotic behavior of the rare-strategy survival metric for STRATA.
  3. How much does the length ∣t∣|t| of the traces vary within a typical randomly selected pool P⊆T\mathcal{P}\subseteq \mathcal{T}, and how does this influence the performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.