acceptodds
Under review as a conference paper at ICLR 2027

When All Rollouts Fail, Critique Provides Direction

Abstract

Group-relative reinforcement learning is widely used to improve reasoning in large language models. However, it cannot learn from a problem when all sampled answers are wrong, because their relative advantages collapse to zero. Critique-guided revision can continue exploration beyond these failed answers. Yet when all revisions are also wrong, the learning signal disappears again because outcome rewards cannot distinguish useful critiques from unhelpful ones. We propose Signpost, a training framework based on Critique as Direction. A critique can be useful even if it does not immediately produce a correct answer, as long as it points the model toward one. Signpost trains the same language model to generate both answers and critiques. Its directional reward measures how much a critique increases the likelihood of a target solution under a frozen reference model. This provides a learning signal when all revisions fail, without rewarding incorrect answers. Once a critique helps discover a verified solution, Signpost trains the model to produce that solution directly from the original problem using a bounded-likelihood objective. This transfers critique-guided discoveries into question-only solving and closes the gap between training and inference. Experiments on mathematical reasoning show that Signpost outperforms GRPO, DAPO, Scaf-GRPO, and Critique-GRPO. On Qwen3-8B, it improves pass@8 over GRPO by 6.6 to 10.0 percentage points across AIME 2024–2026. Critiques are used only during training, while inference requires no critiques, revisions, or additional computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM reasoning, reinforcement learning, natural language feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KVkTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose an approach to extracting a learning signal for LLMs trained through RL when rollouts fail. They introduce a two-stage training pipeline that teaches the model to generate helpful critiques that guide it toward a correct solution. In the second stage, the model is trained to produce verified corrected solutions directly, without needing the critique.

Critical strengths
  • The proposed approach shows strong improvements over the compared baselines.
  • The overall direction of improving how critiques are evaluated, beyond the rewards obtained by responses conditioned on them, makes a lot of sense.
  • I think this work could have positive implications for settings where successful rollouts are extremely rare.
Critical weaknesses

Most of my concerns are around the design decisions made in Signpost. I feel there may be opportunities to simplify several of these choices, and I would be happy to raise my score if the authors could provide more intuition or evidence to support them.

Experimental Design Decisions

  1. In Stage I, when at least one revision succeeds, the critique reward combines four terms: departure from the initial answer, revision correctness, critique–revision alignment, and an answer-leakage penalty. How necessary are the additional terms beyond revision correctness? Would be great to have an ablation on that.

  2. Why does Stage I only consider initially incorrect responses? Have the authors considered also generating critiques for initially correct responses and penalizing critiques whose revisions turn those responses incorrect?

  3. I’m not very familiar with LUFFY, but from the perspective of simplifying the paper’s design choices, would SFT on the corrected responses in Stage II work just as well? Is there a specific reason for adopting the LUFFY-style objective instead?

Baselines

  1. Could the authors clarify how they implemented the Scaf-GRPO baseline? In particular, were hints generated offline using DeepSeek-R1 with access to ground-truth solution traces, as in the original paper?

  2. Could the authors clarify how the proposed approach is different from [1]

Nits

  1. L209 missing equation ref

[1]: Song, Yuda, et al. "Expanding the capabilities of reinforcement learning via text feedback." arXiv preprint arXiv:2602.02482 (2026).

Questions for authors

Please refer to the weaknesses section

7 comments

Response to Experimental Design Decision 1 (Ablation of Critique Reward Terms)

We sincerely thank the reviewer for the constructive feedback and for suggesting an ablation on the four terms of the Stage-I critique reward in Equation 4:

rkC=[λmodedkmode+λsuccrkR+λalignrkRak−λleakℓk]+r_k^{\mathrm C} = \left[\lambda_{\mathrm{mode}} d_k^{\mathrm{mode}} + \lambda_{\mathrm{succ}} r_k^{\mathrm R} + \lambda_{\mathrm{align}} r_k^{\mathrm R} a_k - \lambda_{\mathrm{leak}} \ell_k\right]_+

where our configuration sets λsucc=0.4\lambda_{\mathrm{succ}} = 0.4 and λmode=λalign=λleak=0.1\lambda_{\mathrm{mode}} = \lambda_{\mathrm{align}} = \lambda_{\mathrm{leak}} = 0.1.

  • Design intuition. Revision correctness (λsuccrkR\lambda_{\mathrm{succ}} r_k^{\mathrm R}) carries 4×4\times the weight of any other term and serves as the primary learning signal. However, using binary revision correctness alone leaves three practical vulnerabilities in online critique RL: (1) Answer leakage (ℓk\ell_k): without −λleakℓk-\lambda_{\mathrm{leak}}\ell_k, the critic can bypass diagnostic reasoning by stating the final answer directly inside the critique tags, allowing the refiner to copy the answer rather than learn from diagnostic guidance; (2) By-chance corrections (aka_k): a strong solver may occasionally sample a correct revision while ignoring a vague or irrelevant critique; gating notion alignment by revision success (λalignrkRak\lambda_{\mathrm{align}} r_k^{\mathrm R} a_k) rewards critiques whose suggested mathematical concepts are actually adopted by the successful revision; and (3) Modal-answer anchoring (dkmoded_k^{\mathrm{mode}}): on hard problems where initial rollouts repeatedly collapse onto the same wrong consensus answer, λmodedkmode\lambda_{\mathrm{mode}} d_k^{\mathrm{mode}} encourages critiques that break the model out of this false attractor.
  • Ablation results. We conduct an ablation on Qwen3-8B comparing the full reward in Equation 4 against simplified variants:
Stage-I Critique Reward Variant (Qwen3-8B)Refine (%)Leakage (%)AIME24 p@8AIME25 p@8AIME26 p@8
Revision Correctness Only (λsucc\lambda_{\mathrm{succ}})8.615.376.760.070.0
+ Answer-Leakage Penalty (λsucc,λleak\lambda_{\mathrm{succ}}, \lambda_{\mathrm{leak}})9.33.280.063.370.0
+ Notion Alignment (λsucc,λleak,λalign\lambda_{\mathrm{succ}}, \lambda_{\mathrm{leak}}, \lambda_{\mathrm{align}})9.13.880.060.070.0
Full Eq. 4 (All 4 Terms, Ours)10.83.683.366.773.3

Compared with revision correctness alone, the full reward improves refinement accuracy, reduces answer leakage, and strengthens downstream solving. We will add this ablation to the revision, including the intermediate variants to clarify the individual contributions of leakage prevention, notion alignment, and mode departure.

Response to Experimental Design Decision 2 (Critiquing Initially Correct Responses)

We thank the reviewer for this insightful question regarding why Stage I focuses on initially incorrect responses and whether also critiquing initially correct responses (while penalizing critiques that flip them to incorrect) would be beneficial.

  • Why Stage I defaults to initially incorrect responses. Because Signpost operates as a direct single-turn solver at inference without generating critiques, the primary role of Stage I is to serve as a training-time exploration engine on hard problems where direct sampling fails, discovering verified solutions for Stage II to internalize. Since generating K=8K = 8 critique–revision pairs per selected response is the main computational cost of Stage I, allocating this rollout budget to already-correct responses reduces the exploration budget available for unresolved problems. Moreover, our default training prompt explicitly instructs the critic that "The initial answer is incorrect. Analyze why it is wrong..."—applying this prompt unchanged to a correct answer would presuppose an error and encourage the model to invent flaws.
  • Additional experiment with initially correct responses. Following the reviewer's suggestion, we implemented a Correct-Inclusive Stage I variant using a neutral review prompt (where the model is not told whether the initial answer is correct) and sampling both initially incorrect and initially correct responses, applying a regression penalty (−1.0-1.0) whenever a critique causes an initially correct answer to be revised into an incorrect one:
Stage-I Selection Variant (Qwen3-8B)AIME24 (m@8 / p@8)AIME25 (m@8 / p@8)AIME26 (m@8 / p@8)
Incorrect-Only (Default Signpost)58.8 / 83.351.6 / 66.752.1 / 73.3
Incorrect + Initially Correct (w/ Penalty)54.3 / 76.748.2 / 60.046.7 / 70.0

For direct unassisted solving, the default incorrect-only allocation is slightly stronger because more training compute is devoted to resolving hard failures.

Response to Experimental Design Decision 3 (LUFFY Objective vs. SFT in Stage II)

We appreciate the reviewer raising the question of whether standard SFT on verified revisions in Stage II could replace the LUFFY-style bounded-likelihood objective ψ(p)=p/(p+β)\psi(p) = p/(p+\beta), simplifying the framework.

  • Why bounded internalization differs from standard SFT. In Stage II, a verified solution τ+\tau^+ is discovered under the critique-guided context (q,τ−,c)(q, \tau^-, c) and recontextualized under the problem-only prompt qq, replacing one failed rollout in the direct group G(q)\mathcal{G}(q) alongside G−1G - 1 on-policy rollouts. Under the critique-free policy πθS(⋅∣q)\pi_\theta^{\mathrm S}(\cdot \mid q), pivotal tokens in τ+\tau^+ that redirect the reasoning away from the initial mistake can have very low probability pt(θ)≪1p_t(\theta) \ll 1. Standard SFT applies a constant learning weight with respect to each target token's log-probability, ∂log⁡pt/∂log⁡pt=1\partial \log p_t / \partial \log p_t = 1, regardless of this context mismatch. In contrast, the LUFFY policy-shaping transform has derivative
= \frac{\beta p_t}{(p_t+\beta)^2} = \psi_t(1-\psi_t) \le \frac{1}{4}.$$ This weight tends to zero as $p_t\to0$, smoothly down-weighting extreme off-context tokens when internalizing verified revisions alongside the on-policy GRPO update. - **Ablation: LUFFY surrogate vs. SFT in Stage II.** Starting from the exact same Stage-I checkpoint (step 120) on Qwen3-8B, we keep the on-policy GRPO update on direct responses unchanged and replace only the Stage-II objective on inserted verified revisions with standard SFT (unweighted NLL): | Stage-II Objective on Verified Revisions | Token Weight w.r.t. $\log p_t$ (Magnitude, Before Other Weights) | AIME24 (m@8 / p@8) | AIME25 (m@8 / p@8) | AIME26 (m@8 / p@8) | | :--- | :---: | :---: | :---: | :---: | | Stage I Checkpoint (Before Stage II) | — | 44.2 / 73.3 | 40.0 / 56.7 | 45.0 / 60.0 | | Stage II with Standard SFT ($-\log p_t$) | Constant ($1$) | 54.2 / 76.6 | 47.3 / 60.0 | 50.1 / 66.6 | | **Stage II with Bounded $\psi(p_t)$ (Ours)** | **Adaptive ($\le 1/4$)** | **58.8 / 83.3** | **51.6 / 66.7** | **52.1 / 73.3** | These results support the effectiveness of the bounded-likelihood objective for integrating critique-guided discoveries into on-policy RL: it outperforms standard SFT on all three AIME benchmarks.

Response to Baselines & Nits (Scaf-GRPO Implementation and Comparison with [1])

We thank the reviewer for the questions on our baseline implementation and related literature, as well as catching the missing reference on Line 209.

  • Implementation of Scaf-GRPO. We followed Scaf-GRPO’s official implementation and generated hierarchical hints offline using a DeepSeek-R1 teacher model with access to verified reference solution traces. We used the existing verified reasoning traces in DeepScaleR whenever available. For samples without step-by-step reasoning, we generated candidate reasoning traces using DeepSeek-V3.2 with rejection sampling, retaining only those whose final answers matched the ground truth under Math-Verify. These retained traces served as reference solutions for offline hint generation. During RL training, we progressively injected the pre-generated hints into all-wrong rollout groups until a hinted rollout succeeded, then used it to replace a failed trajectory. The baseline used the same 8K DeepScaleR training set and Qwen3-8B/14B backbones as Signpost.

  • Differences from Song et al. (2026) [1] ("Expanding the capabilities of reinforcement learning via text feedback", arXiv:2602.02482). While both works leverage textual critiques during training to improve single-turn inference, Signpost differs from [1] (RLTF) in four fundamental ways:

    1. Self-generated critiques vs. external teacher judge: RLTF relies on a much larger external judge model (Qwen3-235B-A22B-Instruct) to generate feedback at every training turn. Signpost uses a single self-improving policy πθ\pi_\theta as both solver and critic without querying any external teacher for critiques.
    2. Handling all-wrong refinement rollouts: In RLTF-SD, obtaining a positive success-based learning signal on an all-wrong first-turn prompt still requires the second-turn revision to succeed (R(x0,y1)=1R(x_0, y_1) = 1); when all revisions also fail, this signal is absent. RLTF-FM extracts signal on failures by training the student with cross-entropy loss to imitate the external judge's critique tokens. Because Signpost does not rely on an external judge, we instead introduce the directional reward rkDr_k^{\mathrm D} (Equation 3) to score self-generated critiques in all-wrong groups by their induced change in a frozen reference model's target log-likelihood.
    3. Bounded internalization vs. advantage-weighted regression: RLTF-SD drops the off-policy importance ratio and applies an advantage-weighted regression update (A(y1)∇θlog⁡πθ(y1∣x0)A(y_1)\nabla_\theta \log \pi_\theta(y_1 \mid x_0)). Signpost reconstructs the direct rollout group and applies the bounded LUFFY transform ψ(p)=p/(p+β)\psi(p)=p/(p+\beta) with min-clip alongside on-policy GRPO, adaptively weighting off-context tokens rather than applying the same sequence-level advantage to every token.
    4. Two-stage decoupled schedule vs. joint optimization: RLTF jointly optimizes multi-turn RL with self-distillation or feedback modeling. We show both in our main schedule ablation (Table 5) and our QA–CA task-interference analysis (Appendix G) that joint optimization suffers from a multi-task seesaw effect, and decoupling directional critique learning (Stage I) from solution internalization (Stage II) improves AIME pass@8 by +10.0 to +16.6 points over joint training.

    To further isolate the third difference, we plan to add an AWR-style ablation, replacing only the Stage-II objective on inserted verified revisions with advantage-weighted log-likelihood while keeping the Stage-I checkpoint, revision-selection protocol, on-policy GRPO component, and training budget fixed. We will compare downstream accuracy and training stability with bounded internalization.

  • Nit (L209): We have fixed the missing equation reference on Line 209 (referencing Equation 1, Equation~\ref{eq:method-grpo-advantage}) and will cite and discuss [1] in Section 2 (Related Work).

Thank you again for your thoughtful review and helpful suggestions. We hope our responses and additional ablations clarify the critique reward design, the choice of Stage-I training inputs, the Stage-II internalization objective, and the relationship to prior work. If these clarifications and results resolve your main concerns, we would be grateful if you could reconsider your assessment and rating. We welcome any remaining questions.

Reviewer_qRWyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Signpost trains a model to generate useful critiques even when all sampled revisions remain incorrect. Critiques are scored by their effect on a frozen model’s likelihood of the reference target. A second stage trains on successful revisions under the original question, transferring the benefit to direct solving. Experiments on Qwen3-8B and Qwen3-14B report improvements over several RL baselines without using critiques at inference.

Critical strengths
  1. The all-wrong refinement case is a useful problem to address. A critique can identify a helpful correction without producing a successful revision immediately. Scoring the critique separately gives the model a way to learn from these cases without directly reinforcing the incorrect revision.
  2. The gains are measured in direct solving and appear at both model sizes. I also found Table 5 helpful: the comparison with Stage I → GRPO tests whether continued outcome-based training can substitute for the proposed internalization stage.
Critical weaknesses
  1. I could not reconcile some results with the evaluation description. Appendix I.2 says the multiple-choice prompts and demonstration counts “remain to be finalized,” although Table 2 already reports the results. The AIME aggregation also needs clarification. Under Eq. 25, a single evaluation on 30 problems gives pass@8 in increments of about 3.33 points, whereas Table 1 includes 81.7 and 77.3. Averaging across runs could explain this, but the paper does not specify it. These details matter for assessing the reported gains.
  2. The “shuffled directional rewards” ablation in Table 3 actually multiplies rewards by −1. This tests reversed supervision, rather than whether arbitrary reward variation would give similar benefits. The claim in Sec. 4.3 therefore goes beyond what this control establishes. Removing the directional reward is useful evidence, but also removes its training updates. A within-group reward permutation would test the stated explanation more directly.
Questions for authors
  1. What are the exact problem counts, sampling pools, and averaging procedures behind the AIME results in Tables 1 and 5? How do the reported values follow from Eq. 25?
  2. Which prompts and demonstration counts produced Table 2? Does “remain to be finalized” refer only to documenting settings that were already fixed during evaluation?
  3. Is reward negation the actual implementation of the “shuffled” condition? Please clarify the conclusion it supports. For Table 6, how many problems and fresh revision trials were evaluated, and how many inputs were excluded from the AUC calculation?
3 comments

Response to Weakness 1 & Question 1

Thank you for identifying the inconsistencies between the reported scores and the evaluation description. We have recomputed the relevant results from the original per-problem evaluation logs. The scores 81.7 and 77.3 are consistent with averaging ten evaluation repetitions; however, our check also identified other entries requiring correction, with most discrepancies arising from post-aggregation rounding and numerical reporting.

Evaluation protocol. Each AIME benchmark contains 30 problems. For each evaluated checkpoint, we generate eight responses per problem in each of R=10R=10 evaluation repetitions. Within each repetition, a problem counts as solved if at least one of its eight responses is correct. We then average the resulting pass@8 scores:

\frac{100}{300} \sum_{r=1}^{10}\sum_{i=1}^{30} \mathbf{1}\{\text{at least one correct response among eight}\}.$$ This applies Eq. 25 separately to each repetition and then averages the ten scores. Each repetition evaluates the same 30 problems; the repetitions measure sampling variability and do not increase the number of distinct test problems. Consequently, the aggregate score must be an integer multiple of $1/3$ percentage point before rounding. In particular, 81.7 and 77.3 are compatible with this rule: $100\times245/300=81.666\ldots$ and $100\times232/300=77.333\ldots$, respectively. **Corrections based on the original logs.** The reported values 76.8, 59.1, and the two 73.1 entries are incompatible with this aggregation rule. The following corrections were obtained by recalculating the scores from the saved correctness records, rather than by selecting the nearest protocol-compatible values. | Entry | Originally reported | Corrected score | Log-verified solved problem–repetition pairs | |---|---:|---:|---:| | Table 5, Stage I → GRPO, AIME24 | 76.8 | 76.7 | 230/300 | | Table 5, Stage I → GRPO, AIME25 | 59.1 | 59.0 | 177/300 | | Table 1, Qwen3-14B GRPO, AIME24 | 73.1 | 73.0 | 219/300 | | Table 1, Qwen3-14B Critique-GRPO, AIME24 | 73.1 | 73.0 | 219/300 | | Tables 3–5, Qwen3-8B Signpost, AIME25 | 66.6 | 66.7 | 200/300 | We have completed the log-based verification of the affected scores and standardized rounding across tables. All scores in the revision follow the same rule: aggregate over all evaluation repetitions first, then round once to one decimal place. **Effect on the comparisons.** Calculated from the recomputed, unrounded aggregate scores, the Signpost-minus-Stage-I→GRPO differences on AIME24/25/26 are **6.7/7.7/13.3 percentage points**, respectively. Signpost remains higher on all three benchmarks after correction, so the main conclusion of this comparison is unchanged. We will update the corresponding tables and differences reported in the text. The GPQA Diamond evaluation also averages accuracy over ten repetitions, with one response per problem per repetition; the rationale and protocol are detailed in our response to Weakness 1 & Question 2. ### Response to Weakness 1 & Question 2 Thank you for requesting clarification of the OOD evaluation protocol.The three OOD benchmarks use a shared zero-shot multiple-choice template: a common system instruction, followed by the question stem and labeled options, with the final option letter in `\boxed{X}`. They do not use different reasoning instructions or demonstration counts. The benchmark-specific components are the input-field adapters, option sets, and answer mappings. Within each benchmark, the same protocol is applied across compared models. The phrase "remain to be finalized" in Appendix I.2 is a stale editing sentence, which we will remove. The key evaluation fields are: | Benchmark | Split & Problem Count | Demos | Responses per Problem | | :--- | :---: | :---: | :---: | | ARC-C | Challenge Test (1,172) | 0-shot | $n=1$ | | MMLU-Pro | Test (12,032) | 0-shot | $n=1$ | | GPQA Diamond | Diamond Test (198) | 0-shot | $n=1$ | All three use temperature 0.6, top-$p$ 1.0, and exact matching of the final boxed choice. Because GPQA Diamond contains substantially fewer problems than ARC-C and MMLU-Pro, we conduct ten evaluation repetitions, sampling one response per problem in each repetition ($n=1$), and report the mean accuracy to reduce sampling variability. To keep this response concise, the full shared prompt, benchmark-specific input/option construction and answer mapping, decoding limits, and seed settings have been specified together in **Appendix I.2 (Evaluation Protocol)** in the revision.

Response to Weakness 2 & Question 3 (Part 1: Shuffled vs. Negated Rewards)

The reviewer is correct: the original "Shuffled directional rewards" row actually negates the reward rather than permuting it within a group. We will correct the label and add the genuine within-group permutation control reported below.

  • Clarification of the original row. The reviewer is completely correct: in the submitted version of Table 3, the row labeled "Shuffled directional rewards" was implemented by multiplying the directional rewards by −1-1 (i.e., negated / reversed directional supervision, −rkD-r_k^{\mathrm D}). This setting tests whether inverting the directional preference harms learning—and indeed it reduces refinement accuracy from 10.8% to 6.0% and performs worse than removing the directional reward altogether (8.7%). However, we fully agree with the reviewer that negation tests reversed supervision rather than whether arbitrary reward variation within a group could yield similar benefits.
  • New within-group reward permutation ablation. Following the reviewer's exact suggestion, we conducted a true within-group directional reward permutation experiment on Qwen3-8B: for each eligible all-wrong refinement group of K=8K = 8 critiques, we compute the exact directional rewards {r1D,…,r8D}\{r_1^{\mathrm D}, \ldots, r_8^{\mathrm D}\} and randomly permute them across the K=8K = 8 critiques within the same group. This preserves the exact per-group directional-reward distribution, mean, and variance while breaking the alignment between each critique and its directional score. It does not guarantee the same realized parameter-gradient magnitude:
Directional Supervision Variant (Qwen3-8B)Refine (%)AIME24 p@8AIME25 p@8AIME26 p@8
Signpost (Unmodified rkDr_k^{\mathrm D})10.883.366.773.3
Without directional reward (rkD=0r_k^{\mathrm D} = 0)8.780.060.066.7
Within-group permuted rkDr_k^{\mathrm D} (New)6.470.060.056.7
Negated directional reward (−rkD-r_k^{\mathrm D}, prev. "Shuffled")6.073.360.063.3

This new control supports the importance of aligning directional rewards with individual critiques, beyond preserving within-group reward variation. We will rename the negated row accurately and add the within-group permutation results to Table 3 in the revision.

Response to Question 3 (Part 2: Table 6 Sample Counts and Exclusions)

Table 6 uses two denominators: all screened inputs with complete fresh trials contribute to the B1–B5 correction rates, whereas within-input ROC-AUC includes only inputs containing both fresh successes and failures. The counts and exclusion rule are specified below.

  • Evaluated problems and fresh revision trials. The diagnostic starts from a held-out pool of 256 failed initial responses on hard DeepScaleR problems. For each failed input, the model independently generates Kd=10K_{\mathrm d} = 10 critiques (2 critiques per rank bin across B1–B5) and 1 screening revision per critique. Retaining only complete groups where all Kd=10K_{\mathrm d} = 10 screening revisions fail yields 120 screened all-wrong inputs (1200 critiques). For each critique, we then sample M=8M = \text{8} fresh independent revisions, totaling ∣I∣×Kd×M=9,600|\mathcal{I}| \times K_{\mathrm d} \times M = \text{9,600} fresh revision trials (1920 trials per bin). All ∣I∣|\mathcal{I}| screened inputs are included in the B1–B5 correction rate estimates in the top half of Table 6.
  • ROC-AUC exclusions. Because within-input ROC-AUC is only defined when an input's Kd×MK_{\mathrm d} \times M fresh trials contain both at least one success (uikm=1u_{ikm}=1) and at least one failure (uikm=0u_{ikm}=0), inputs with zero fresh successes across all trials cannot produce a within-input ROC curve. Out of the ∣I∣=120|\mathcal{I}| = \text{120} screened all-wrong inputs, 34 inputs contained both outcome classes and were used to compute the 0.563 within-input ROC-AUC, while 86 inputs remained all-wrong across all fresh trials and were excluded from the AUC denominator.

We will report these exact counts in the caption and protocol description of Table 6 to make the diagnostic fully self-contained.

Thank you again for your careful review and constructive suggestions on evaluation rigor. We hope the responses above clarify the evaluation protocol, sample counts and exclusion rules, and the role of the new within-group reward permutation control. If these clarifications and additional results resolve your main concerns, we would be grateful if you could reconsider your assessment and rating. We welcome any remaining questions.

Dear qRWy,

Thank you again for your time and feedback. We would be grateful to hear whether our responses and additional results address your concerns. Please let us know of any remaining questions so that we can clarify them before the discussion period ends.

Reviewer_FMisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Signpost, a two-stage reinforcement learning framework that addresses the vanishing learning signal when both initial rollouts and critique-guided revisions fail. Stage I introduces a directional reward based on the change in a frozen reference model's likelihood of a ground-truth target, enabling critique learning even when all revisions are incorrect. Stage II transfers verified critique-guided solutions to direct question-answering through a bounded-likelihood objective. Experiments on Qwen3-8B and Qwen3-14B show consistent improvements over several RL baselines on mathematical reasoning benchmarks, with additional gains on out-of-distribution tasks. Critiques are used only during training, with no additional inference-time computation.

Critical strengths
  • Evaluating critiques by their directional utility rather than immediate correction success is a meaningful idea that addresses a limitation of outcome-based critique optimization.
  • The method transfers successful critique-guided discoveries into direct question-to-answer behavior, avoiding additional inference-time critique or revision overhead.
Critical weaknesses
  • Increasing a frozen reference model's likelihood of the target does not necessarily imply that a critique provides valid or generalizable reasoning guidance, particularly when the target contains only the final answer. The paper would benefit from stronger evidence that the reward reliably identifies genuinely useful critiques rather than exploiting biases of the reference model.
  • Signpost requires additional critique generation, reference-model scoring, and a two-stage training pipeline. The reported Qwen3-8B training cost is 912.6 H100 GPU-hours, compared with 616 GPU-hours for GRPO. More comprehensive compute-matched comparisons and evidence of stability across independent runs would strengthen the practical case for the method.
  • The main evaluation focuses on mathematical reasoning and a limited set of model scales. Further evidence is needed to establish whether the gains generalize to other reasoning domains, model families, and more diverse settings where intermediate progress is difficult to verify.
Questions for authors
  • How strongly does the directional reward correlate with the actual usefulness of a critique across different models and problem types? Does the correlation remain strong when the target contains only the final answer rather than a full derivation?
  • Can the authors provide compute-matched comparisons with the strongest baselines, including the cost of reference-model scoring and critique generation, to clarify the method's efficiency?
  • How robust are the reported improvements across random seeds and different backbone families? In particular, does the method generalize beyond the two Qwen3 backbones and mathematical reasoning tasks?
  • How much of the improvement comes from directional critique training versus training on verified revisions using the bounded-likelihood objective? Can the authors isolate the contribution of each component under matched training budgets?
  • How well does Signpost work beyond mathematical reasoning, and does it retain its advantages across different model families and scales?
5 comments

Response to Weakness 1 & Question 1

The directional reward is effective for training: it improves critique-guided correction and downstream direct-solving performance. This conclusion is supported directly by training ablations, not solely by the correlation between likelihood gain and correction outcomes.

Final-answer-only targets. All directional-reward targets in our experiments contain only the ground-truth final answer, without a full derivation. This includes Table 6 and the additional cross-reviser diagnostic below. Section 3.2 (p. 4, lines 170–174) describes the target as “the serialized ground-truth field” that “may contain only the final answer rather than a complete derivation.” We will make this wording explicit: final-answer-only scoring is the setting used throughout our experiments, not merely a possible configuration. Thus, the reported diagnostic results directly address the final-answer-only setting raised by the reviewer.

Direct evidence from training ablations. Under the same 420-step training budget, removing the directional reward reduces refinement accuracy from 10.8% to 8.7% and lowers direct-solving pass@8 by 3.3, 6.7, and 6.7 percentage points on AIME24/25/26, respectively:

Training variant (Qwen3-8B)Refine (%)AIME24 pass@8 (%)AIME25 pass@8 (%)AIME26 pass@8 (%)
Without directional reward (rD=0r^{\mathrm D}=0)8.780.060.066.7
Full Signpost10.883.366.773.3

Percentage-point differences are calculated from aggregate scores before rounding.

These results establish that the reward contributes to actual correction and final task performance, rather than merely increasing a proxy score. The gains persist when the trained model operates as a direct solver without critiques at inference.

Independent validation of the reward signal. Table 6 evaluates directional scores against independently sampled fresh revision outcomes on inputs whose screening revisions all fail. The diagnostic includes 120 screened inputs, 10 critiques per input, and 8 fresh revisions per critique, totaling 9,600 fresh revision trials. All 120 inputs contribute to the correction-rate estimates. Within-input ROC-AUC includes the 34 inputs containing both fresh successes and failures; the remaining 86 inputs have no fresh successes and are excluded from the AUC calculation.

The within-input ROC-AUC is 0.563 (95% CI: 0.510–0.681). Thus, even when screening yields no successful correction, the score retains predictive information about which critiques are more useful. We interpret this as modest positive discrimination, not strong correlation or verification of intermediate reasoning.

Additional evidence across revisers. We keep the questions, incorrect initial answers, Qwen-generated critiques, and frozen-reference scores fixed, changing only the fresh reviser. Both models independently generate M=4M=4 fresh revisions per critique under matched sampling settings and generation-length limits. Consequently, B1–B5 contain exactly the same critiques for both models. The frozen-reference scores continue to use only the final-answer target.

Fresh reviserB1 correction rate (%)B2 (%)B3 (%)B4 (%)B5 (%)Within-input ROC-AUC
Qwen3-8B1.824.206.055.006.500.574 (95% CI: 0.516–0.692)
Llama-3.1-8B-Instruct0.352.502.001.652.500.554 (95% CI: 0.501–0.637)

ROC-AUC confidence intervals are calculated using question-level bootstrap.

Despite different absolute correction rates, both revisers exhibit above-chance discrimination without regenerating critiques or recomputing scores, providing additional evidence that the score’s predictive value extends beyond the original reviser.

Our claim is therefore training effectiveness, not perfect critique verification. These results do not rule out all reference-model biases, but show that final-answer-only directional scores provide useful training supervision and predictive information about independently sampled corrections.

Response to Weakness 2 & Question 2

The reported 912.6 H100 GPU-hours already covers Signpost's full training pipeline, including critique generation, revision rollouts, and frozen-reference scoring. Signpost also outperforms the compute-matched baselines and GRPO trained with 35% more compute, as detailed below.

  • Lightweight overhead of reference scoring. We clarify that the reported 912.6 H100 GPU-hours for Signpost on Qwen3-8B (120 steps in Stage I + 300 steps in Stage II) already includes all critique generation, revision rollouts, and frozen reference-model scoring, with no hidden offline preprocessing cost. In particular, frozen reference scoring requires only two teacher-forced forward passes (without autoregressive decoding) on the capped subset of all-wrong groups (η=0.5\eta = 0.5), accounting for only 0.6% of total step time. This percentage refers specifically to reference scoring, not to the combined cost of critique generation and revision rollouts.
  • Comparison against extended and compute-matched baselines. Building on the extended-GRPO comparison in Appendix E (Table 7), we report the following expanded comparison. We doubled the training horizon of standard GRPO from 300 steps (616 GPU-hours) to 600 steps (approximately 1,232 GPU-hours), exceeding Signpost's total compute budget by 35%, and additionally evaluate DAPO and Critique-GRPO under a matched compute budget (approximately 913 GPU-hours):
Method (Qwen3-8B)StepsH100 GPU-HoursAIME24 (m@8 / p@8)AIME25 (m@8 / p@8)AIME26 (m@8 / p@8)
GRPO (Standard)300616.057.5 / 76.740.0 / 56.746.2 / 63.3
GRPO (Extended, 1.35×1.35\times Signpost's budget)600~1,232.056.4 / 80.049.1 / 63.347.1 / 66.7
DAPO (Compute-Matched)460~913.055.3 / 76.744.6 / 63.347.1 / 66.7
Critique-GRPO (Compute-Matched)400~913.054.3 / 73.335.6 / 53.343.4 / 60.0
Signpost (Ours)420912.658.8 / 83.351.6 / 66.752.1 / 73.3

Even with 35% more compute, 600-step GRPO remains below Signpost on all three AIME benchmarks, while leaving 90 more all-wrong training groups per epoch (1,308 vs. 1,218). Moreover, critiques are used only during training; at inference, Signpost operates as a direct solver without additional critique or revision calls.

For Scaf-GRPO and Critique-GRPO, any additional cost of teacher-generated trajectories or prefixes—including offline preparation and closed-source teacher calls where used—should also be included in an end-to-end comparison. Signpost does not require an additional offline teacher-generation stage or closed-source teacher calls for critiques.

Response to Question 4

Both stages contribute: Stage I improves correction and supplies verified discoveries, while Stage II internalizes them into direct-solving behavior. Removing the directional reward reduces refinement accuracy from 10.8% to 8.7%; omitting Stage II or substituting standard GRPO also lowers direct-solving performance. Tables 3 and 5, together with the additional SFT comparison below, examine these contributions under the same 420-step training budget on Qwen3-8B:

Variant (Matched 420-Step Budget on Qwen3-8B)Refine (%)AIME24 (m@8 / p@8)AIME25 (m@8 / p@8)AIME26 (m@8 / p@8)
Stage I Only (w/o Stage II internalization)10.844.2 / 73.340.0 / 56.745.0 / 60.0
Stage II Only (w/o Stage I critique learning)—38.7 / 66.727.5 / 50.030.4 / 56.7
Stage I →\to Standard GRPO10.855.4 / 76.742.0 / 59.046.3 / 60.0
Signpost w/o Directional Reward (rD=0r^{\mathrm D}=0)8.756.7 / 80.049.4 / 60.048.6 / 66.7
Stage I →\to Stage II w/ SFT (instead of ψ\psi)10.854.2 / 76.747.3 / 60.050.1 / 66.7
Full Signpost (Stage I →\to Stage II)10.858.8 / 83.351.6 / 66.752.1 / 73.3

These component comparisons match optimization steps; the GPU-hour-matched baseline comparisons are reported above.

  • Contribution of directional critique training. Removing the directional reward in Stage I drops refinement accuracy from 10.8% to 8.7% and lowers direct-solving pass@8 by 3.3, 6.7, and 6.7 points on AIME24/25/26. Skipping Stage I altogether ("Stage II Only") also causes substantial degradation, consistent with the importance of learning useful critiques before internalizing their discoveries.
  • Contribution of bounded-likelihood internalization. Replacing our bounded transform ψ(p)=p/(p+β)\psi(p)=p/(p+\beta) with unweighted SFT also degrades performance, supporting the effectiveness of the bounded-likelihood internalization objective relative to this alternative. This comparison does not isolate gradient boundedness as the sole cause, since the objectives also differ in their weighting and optimization dynamics.

Response to Weakness 3, Question 3 & Question 5

For broader-domain generalization, Table 2 already reports gains over GRPO on ARC-C, MMLU-Pro, and GPQA Diamond at both model sizes, and the WebShop/ALFWorld comparison below extends the evaluation to training in interactive environments. For cross-family generalization, we provide the Llama-3.1-8B comparison below; we will report the mean and standard deviation across three independent Qwen3-8B training seeds separately.

  • Out-of-distribution generalization to scientific and multidisciplinary reasoning (Table 2). Without any target-domain fine-tuning, Signpost transfers from mathematical training to general reasoning benchmarks: ARC-C (science), MMLU-Pro (12,032 questions across 14 disciplines including law, engineering, humanities, and social sciences), and GPQA Diamond (graduate-level biology, physics, and chemistry):
MethodQwen3-8B ARC-CQwen3-8B MMLU-ProQwen3-8B GPQA-DQwen3-14B ARC-CQwen3-14B MMLU-ProQwen3-14B GPQA-D
Base model86.627.247.488.330.750.8
GRPO90.736.960.091.440.064.0
DAPO91.442.464.695.246.570.1
Scaf-GRPO92.345.166.792.843.466.3
Critique-GRPO91.446.065.492.044.665.7
Signpost (Ours)94.152.568.694.853.069.5

Signpost outperforms GRPO on every benchmark at both model sizes (by +15.6 / +13.0 points on MMLU-Pro and +8.6 / +5.5 points on GPQA Diamond) and surpasses the strongest baseline on MMLU-Pro by +6.5 points on both Qwen3-8B and Qwen3-14B.

  • Generalization to interactive agentic decision-making (WebShop and ALFWorld). To further evaluate Signpost in domains where intermediate progress is difficult to verify and rewards are sparse across multi-step interactions, we compare Signpost with GRPO on two interactive agent benchmarks: WebShop (online shopping navigation and product selection) and ALFWorld (embodied household task execution).

Both methods are trained within each environment; these are not zero-shot evaluations of the mathematics-trained checkpoints. Directional-reward targets again contain only the final answer, without a full derivation or reference interaction trajectory. These experiments therefore test whether the training framework can be applied beyond mathematics, complementing the zero-shot OOD transfer results in Table 2.

Method (Qwen3-8B)WebShopALFWorldAvg.
GRPO0.820.890.86
Signpost (Ours)0.860.930.90

Avg. is the arithmetic mean of the two displayed task scores, rounded to two decimal places.

Signpost improves over GRPO by 0.04 on both WebShop and ALFWorld, providing additional evidence that directional critique learning and solution internalization are useful beyond mathematical reasoning in these two interactive environments. We will document the task counts, evaluation metrics, training budgets, and final-answer target construction alongside these results in the revision.

  • Generalization to another backbone family (Llama-3.1-8B) and stability across random seeds. To evaluate Signpost beyond the Qwen3 family, we conduct additional experiments comparing GRPO and Signpost on Llama-3.1-8B:
Method (Llama-3.1-8B)AIME24 (m@8 / p@8)AIME25 (m@8 / p@8)AIME26 (m@8 / p@8)
GRPO6.7 / 23.36.7 / 10.03.3 / 10.0
Signpost (Ours)10.0 / 30.06.7 / 13.36.7 / 16.7

Signpost improves pass@8 on all three benchmarks with this additional backbone, providing cross-family evidence beyond the two Qwen3 models. Absolute performance remains lower than on Qwen3, so we interpret this as evidence of applicability to another family rather than comprehensive validation across all families and scales.

Thank you again for your time and constructive feedback. We hope the responses above clarify the empirical support and limitations of the directional reward, the complete training cost, and the scope of our generalization evidence. If these clarifications and additional results resolve your main concerns, we would be grateful if you could reconsider your assessment and rating. We welcome any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.