acceptodds
Under review as a conference paper at ICLR 2027

The Path Matters: Trajectory-Guided Semi-Supervised Reinforcement Learning for LLM Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs), but its reliance on reference answers limits training when annotations are scarce. Existing approaches exploit unlabeled questions through self-generated rewards, and recent semi-supervised methods use limited labeled data to select reliable training examples. However, answer agreement and sample-level selection provide limited guidance for the quality of individual reasoning trajectories: different rollouts that reach the same pseudo-answer receive identical correctness feedback, even when their reasoning differs in reliability. Motivated by this consideration, we propose , a trajectory-guided semi-supervised reinforcement learning framework for LLM reasoning. maintains a dynamic reasoning graph anchored by trajectories whose answers are verified on labeled examples, and uses related questions to provide auxiliary support for each unlabeled rollout. It separates pseudo-label confidence from trajectory guidance, using the former to weight each prompt's policy update and the latter to shape rollout-level group-relative advantages. A separate admission rule conservatively expands the graph with pseudo-labeled examples while allowing broader participation in training. We design an evaluation on six mathematical reasoning benchmarks under three labeled–unlabeled data allocations to assess reasoning performance and label efficiency.

Primary area:
reinforcement learning
Keywords:
LLM, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_R1R3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEER studies reasoning RL with a mixture of labeled and unlabeled mathematics problems. It assigns pseudo-answers using student consensus, weights updates by confidence, and maintains a dynamic memory of accepted trajectories. TF-IDF retrieval over questions and trajectories supplies an additional reward for unlabeled examples. The experiments compare two model families and several label budgets on five mathematics benchmarks, including comparisons with TraPO and an ablation of the graph-reward weight.

Critical strengths
  1. The semi-supervised setting is useful, and TraPO is a relevant comparison. Reporting several labeled/unlabeled budgets makes it possible to assess whether the gains persist beyond a single label allocation.

  2. The graph-weight ablation provides a direct check of the proposed additional reward. In the Qwen3 1k+1k setting, the score increases from 51.151.1 at λ=0\lambda=0 to 53.253.2 with the full method, so there is evidence that the graph term contributes within this training pipeline.

  3. The memory reuses student-generated trajectories without a separate process verifier. Question retrieval followed by trajectory comparison is a relatively accessible design, and the paper evaluates it with both Qwen2.5-Math-7B and Qwen3-4B.

Critical weaknesses
  1. The empirical gains over TraPO are mostly modest: three of the four reported model/data configurations improve the average score by less than one point. Without uncertainty estimates, it is difficult to assess how reliable these small gains are. The evaluation is also limited to 4B/7B models and mathematics benchmarks, and the lack of reward training curves makes the learning dynamics difficult to assess. Overall, the evidence for a consistent advantage is not yet convincing.

  2. The consensus-based pseudo-labeling reinforces the model’s majority answers, which may improve performance largely by concentrating the answer distribution, similarly to entropy minimization. Although the paper includes entropy-based baselines, it remains unclear how much of the gain comes from this effect rather than useful reasoning information in the trajectory graph. An analysis of answer entropy, pseudo-label accuracy, and successful-trajectory diversity during training would help distinguish improved reasoning from stronger self-consistency.

  3. The graph ablation is informative, but it also limits how much of the improvement can be attributed to the graph. In the Qwen3 1k+1k setting, λ=0\lambda=0 already scores 51.151.1, compared with 53.253.2 for STEER and 47.547.5 for TraPO. The other reported gains over TraPO are mostly below one point. Thus, the 5.7-point gap to TraPO cannot all be credited to trajectory guidance. Can the authors separate the effects of confidence weighting and sample admission from the graph reward?

Questions for authors

Please see Critical Weaknesses.

Reviewer_aC4mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes STEER, a method for semi-supervised RLVR. Their idea combines majority voting/confidence filtering for learning from unlabelled data with rollout similarity search. The core idea is that rollouts that are lexically similar to rollouts with labels or with consensus answers sampled from lexically similar problems should be granted higher advantages during RL. The authors show that STEER yields performance gains on math reasoning tasks when compared with existing unsupervised and semi-supervised approaches.

Critical strengths
  • Semi-supervised learning with RL is an interesting and underexplored domain.
Critical weaknesses
  • STEER fundamentally relies on a combination of (a) majority vote and confidence filtering and (b) rollout similarity to assign rewards. The main issues with majority vote and confidence filtering are that (1) learning may primarily be a result of the model "sharpening" its existing solutions rather than learning to scale test time compute/explore via long horizon CoT, as the model is incentivized to upweight trajectories that converge to the most common answer; and (2) learning only progresses when the consensus decision is generally the correct one under the selected model and dataset, which may be true for easy datasets but not true when RL is needed for learning to solve difficult problems. See [1], [2]. For this method to be useful for current reasoning systems, I believe we need evidence to show that RL + STEER can teach the model to solve problems that are difficult for the base model, and that pass@k rates are generally maintained or even improved through training. Otherwise, I do not believe this is method will likely be meaningful useful.
  • While the rollout similarity idea is interesting, I am not confidence that lexical similarity of problems and rollouts as measured with TF-IDF is a meaningful measure of problem or rollout similarity, given the complexity of reasoning rollouts and math problems in general. This is important because the main mechanism the authors rely on is that similarity of rollouts (as measured using TF-IDF) is a good predictor of answer correctness. Indeed, looking at the results in Figure 3, it seems that using the best setting of λ=10e−2\lambda = 10e-2 only yields a ~1 point improvement vs. using λ=0\lambda = 0 (I would appreciate confidence intervals for such claims), which suggests that the approach is likely highly noisy at best, and ineffective at worse.
  • Many gains are very small, and without confidence intervals or statistical tests it is difficult to draw meaningful conclusions e.g. 52.3 vs. 52.8 for TRAPO vs. STEER with Qwen3-4B-Base, or 52.2 for 51.3 for TRAPO vs. STEER with Qwen2.5-Math-7B.
  • It is surprising that RL results in the semi-supervised setting with Qwen2.5 are better than the fully supervised setting e.g. "Fully Supervised w/ 4K Labels" gives an average of 48.8 while STEER gives 49.1. Why does learning from the actual labels do worse here? My concern is that this indicates that there are underlying problems in training or evaluation, perhaps due to the choice of data and model: see [3] for how RL with this class of model can go wrong and lead to misleading results. More generally, I would like to see results on newer reasoning models (e.g. Qwen3.5) that can properly scale test time compute and that more closely reflect the behaviour of modern reasoning LLMs.

[1] He et al., How Far Can Unsupervised RLVR Scale LLM Training? https://arxiv.org/pdf/2603.08660 [2] Huang et al., Self-Improvement in Language Models: The Sharpening Mechanism. https://arxiv.org/pdf/2412.01951 [3] Shao et al., Spurious Rewards: Rethinking Training Signals in RLVR. https://arxiv.org/pdf/2506.10947

Questions for authors

To raise my score, I will need to see:

  • Empirical evidence that selecting rollouts using TF-IDF similarity scores correctness well. This should be easy to measure.
  • Empirical evidence that STEER improves performance beyond sharpening.
  • Some results using a reasoning model that somewhat reflects the behaviors of current SOTA open-weight reasoning models (can reason and scale test time compute, e.g. a small Qwen3.5 model, or at least Qwen3-4B with thinking enabled, and not the base model), on suitable datasets.
  • An explanation for why "Fully Supervised w/ 4K Labels" trails the semi-supervised setting.
Reviewer_UGhFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies RLVR for mathematical reasoning under limited access to verified answers. It introduces STEER, a semi-supervised framework that aims to make better use of the reasoning trajectories associated with labeled examples.

For each unlabeled prompt, STEER first uses answer agreement across sampled rollouts to construct a pseudo-label and a self-confidence score. It then retrieves related questions from a dynamic reasoning graph, where nodes store trusted prompts, answers, and accepted reasoning trajectories. Each rollout receives a graph-based score according to its similarity to trajectories associated with retrieved neighbors. The resulting graph advantage is combined with the answer-based GRPO advantage.

High-confidence pseudo-labeled examples are subsequently added to the graph. The authors evaluate STEER on OpenR1-Math-220k using Qwen2.5-Math-7B and Qwen3-4B-Base, reporting results on AIME, AMC, MATH-500, Minerva, and OlympiadBench. STEER outperforms the evaluated baselines in the reported averages across several labeled/unlabeled allocations.

Critical strengths

S1. The paper introduces a well-motivated way to reuse labeled reasoning trajectories. STEER goes beyond pseudo-labeling at the answer level and attempts to distinguish the quality of individual rollouts within the same answer group. This is a meaningful direction for low-label RLVR, where reliable intermediate supervision is scarce.

S2. The evaluation is broader than a single benchmark or backbone. The experiments include two model backbones, multiple labeled/unlabeled allocations, and five mathematical reasoning benchmarks. The results provide encouraging initial evidence, particularly in the low-label Qwen3-4B setting, that the proposed signal may be useful.

Critical weaknesses

W1. The graph initialization and trajectory-scoring procedure are not fully defined under the reported configuration. The combination of empty trajectory sets at initialization, Equation (10), and Tw=0T_w=0 is difficult to reconcile. Since graph-based trajectory scoring is the central component of STEER, this is not a minor implementation detail. It affects both the validity of the optimization procedure and the reproducibility of the reported results. The paper also omits the values of τs\tau_s, τ\tau, total training steps, and the policy for filtering or updating graph nodes.

W2. The experiments do not isolate the claimed trajectory-level mechanism, and the reported gains lack uncertainty estimates. The λ=0\lambda=0 ablation only shows that removing the graph reward hurts performance in one setting. It does not distinguish the effect of trajectory matching from question-level lexical retrieval, graph expansion, or pseudo-label dynamics. More targeted controls are needed, such as shuffled trajectories, random neighbors, question-only retrieval, or disabling pseudo-node insertion. Since the strongest-baseline improvements are often below one point, multi-seed results are also necessary before concluding that the method consistently outperforms TraPO.

Questions for authors

Q1: Algorithm 1 initializes all labeled nodes with Tj=∅\mathcal{T}_j=\varnothing, while Table 3 reports Tw=0T_w=0. How is Equation (10) computed when an unlabeled sample retrieves an empty node during early training? Are empty nodes excluded, or are verified labeled trajectories collected before the first optimization step? This clarification would directly affect my assessment of the method’s correctness.

Q2: Please report the values of τs\tau_s, τ\tau, the total number of training steps or epochs, the graph-size policy, and the exact rule for adding, refreshing, removing, or filtering graph nodes.

Q3: Can the authors provide ablations that isolate the contribution of trajectory-level supervision? In particular, I would like to see comparisons with shuffled neighbor trajectories, random neighbors, question-similarity-only weighting, and a variant without pseudo-labeled node insertion.

Q4: Can the authors report results over multiple independent training seeds, including mean and standard deviation or confidence intervals, for STEER, TraPO, and the λ=0\lambda=0 variant? This is particularly important for AIME and AMC, as well as for the small average improvements over TraPO.

Q5: Are STEER and TraPO run under matched total optimization, rollout, and compute budgets? The appendix states that TraPO uses an 8/108/10-epoch warm-up and different KL/entropy settings, whereas STEER uses Tw=0T_w=0. Please report total optimizer updates, rollouts, and GPU time for each method. It would also be helpful to clarify whether exact or near-duplicate overlap between the training data and evaluation benchmarks was audited.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.