- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose an approach to extracting a learning signal for LLMs trained through RL when rollouts fail. They introduce a two-stage training pipeline that teaches the model to generate helpful critiques that guide it toward a correct solution. In the second stage, the model is trained to produce verified corrected solutions directly, without needing the critique.
Critical strengths
- The proposed approach shows strong improvements over the compared baselines.
- The overall direction of improving how critiques are evaluated, beyond the rewards obtained by responses conditioned on them, makes a lot of sense.
- I think this work could have positive implications for settings where successful rollouts are extremely rare.
Critical weaknesses
Most of my concerns are around the design decisions made in Signpost. I feel there may be opportunities to simplify several of these choices, and I would be happy to raise my score if the authors could provide more intuition or evidence to support them.
Experimental Design Decisions
-
In Stage I, when at least one revision succeeds, the critique reward combines four terms: departure from the initial answer, revision correctness, critique–revision alignment, and an answer-leakage penalty. How necessary are the additional terms beyond revision correctness? Would be great to have an ablation on that.
-
Why does Stage I only consider initially incorrect responses? Have the authors considered also generating critiques for initially correct responses and penalizing critiques whose revisions turn those responses incorrect?
-
I’m not very familiar with LUFFY, but from the perspective of simplifying the paper’s design choices, would SFT on the corrected responses in Stage II work just as well? Is there a specific reason for adopting the LUFFY-style objective instead?
Baselines
-
Could the authors clarify how they implemented the Scaf-GRPO baseline? In particular, were hints generated offline using DeepSeek-R1 with access to ground-truth solution traces, as in the original paper?
-
Could the authors clarify how the proposed approach is different from [1]
Nits
- L209 missing equation ref
[1]: Song, Yuda, et al. "Expanding the capabilities of reinforcement learning via text feedback." arXiv preprint arXiv:2602.02482 (2026).
Questions for authors
Please refer to the weaknesses section
7 comments
Response to Experimental Design Decision 3 (LUFFY Objective vs. SFT in Stage II)
We appreciate the reviewer raising the question of whether standard SFT on verified revisions in Stage II could replace the LUFFY-style bounded-likelihood objective , simplifying the framework.
- Why bounded internalization differs from standard SFT. In Stage II, a verified solution is discovered under the critique-guided context and recontextualized under the problem-only prompt , replacing one failed rollout in the direct group alongside on-policy rollouts. Under the critique-free policy , pivotal tokens in that redirect the reasoning away from the initial mistake can have very low probability . Standard SFT applies a constant learning weight with respect to each target token's log-probability, , regardless of this context mismatch. In contrast, the LUFFY policy-shaping transform has derivative
Response to Baselines & Nits (Scaf-GRPO Implementation and Comparison with [1])
We thank the reviewer for the questions on our baseline implementation and related literature, as well as catching the missing reference on Line 209.
-
Implementation of Scaf-GRPO. We followed Scaf-GRPO’s official implementation and generated hierarchical hints offline using a DeepSeek-R1 teacher model with access to verified reference solution traces. We used the existing verified reasoning traces in DeepScaleR whenever available. For samples without step-by-step reasoning, we generated candidate reasoning traces using DeepSeek-V3.2 with rejection sampling, retaining only those whose final answers matched the ground truth under Math-Verify. These retained traces served as reference solutions for offline hint generation. During RL training, we progressively injected the pre-generated hints into all-wrong rollout groups until a hinted rollout succeeded, then used it to replace a failed trajectory. The baseline used the same 8K DeepScaleR training set and Qwen3-8B/14B backbones as Signpost.
-
Differences from Song et al. (2026) [1] ("Expanding the capabilities of reinforcement learning via text feedback", arXiv:2602.02482). While both works leverage textual critiques during training to improve single-turn inference, Signpost differs from [1] (RLTF) in four fundamental ways:
- Self-generated critiques vs. external teacher judge: RLTF relies on a much larger external judge model (
Qwen3-235B-A22B-Instruct) to generate feedback at every training turn. Signpost uses a single self-improving policy as both solver and critic without querying any external teacher for critiques. - Handling all-wrong refinement rollouts: In RLTF-SD, obtaining a positive success-based learning signal on an all-wrong first-turn prompt still requires the second-turn revision to succeed (); when all revisions also fail, this signal is absent. RLTF-FM extracts signal on failures by training the student with cross-entropy loss to imitate the external judge's critique tokens. Because Signpost does not rely on an external judge, we instead introduce the directional reward (Equation 3) to score self-generated critiques in all-wrong groups by their induced change in a frozen reference model's target log-likelihood.
- Bounded internalization vs. advantage-weighted regression: RLTF-SD drops the off-policy importance ratio and applies an advantage-weighted regression update (). Signpost reconstructs the direct rollout group and applies the bounded LUFFY transform with min-clip alongside on-policy GRPO, adaptively weighting off-context tokens rather than applying the same sequence-level advantage to every token.
- Two-stage decoupled schedule vs. joint optimization: RLTF jointly optimizes multi-turn RL with self-distillation or feedback modeling. We show both in our main schedule ablation (Table 5) and our QA–CA task-interference analysis (Appendix G) that joint optimization suffers from a multi-task seesaw effect, and decoupling directional critique learning (Stage I) from solution internalization (Stage II) improves AIME pass@8 by +10.0 to +16.6 points over joint training.
To further isolate the third difference, we plan to add an AWR-style ablation, replacing only the Stage-II objective on inserted verified revisions with advantage-weighted log-likelihood while keeping the Stage-I checkpoint, revision-selection protocol, on-policy GRPO component, and training budget fixed. We will compare downstream accuracy and training stability with bounded internalization.
- Self-generated critiques vs. external teacher judge: RLTF relies on a much larger external judge model (
-
Nit (L209): We have fixed the missing equation reference on Line 209 (referencing Equation 1,
Equation~\ref{eq:method-grpo-advantage}) and will cite and discuss [1] in Section 2 (Related Work).
Thank you again for your thoughtful review and helpful suggestions. We hope our responses and additional ablations clarify the critique reward design, the choice of Stage-I training inputs, the Stage-II internalization objective, and the relationship to prior work. If these clarifications and results resolve your main concerns, we would be grateful if you could reconsider your assessment and rating. We welcome any remaining questions.
Response to Experimental Design Decision 1 (Ablation of Critique Reward Terms)
We sincerely thank the reviewer for the constructive feedback and for suggesting an ablation on the four terms of the Stage-I critique reward in Equation 4:
rkC=[λmodedkmode+λsuccrkR+λalignrkRak−λleakℓk]+
where our configuration sets λsucc=0.4 and λmode=λalign=λleak=0.1.
Compared with revision correctness alone, the full reward improves refinement accuracy, reduces answer leakage, and strengthens downstream solving. We will add this ablation to the revision, including the intermediate variants to clarify the individual contributions of leakage prevention, notion alignment, and mode departure.
Response to Experimental Design Decision 2 (Critiquing Initially Correct Responses)
We thank the reviewer for this insightful question regarding why Stage I focuses on initially incorrect responses and whether also critiquing initially correct responses (while penalizing critiques that flip them to incorrect) would be beneficial.
For direct unassisted solving, the default incorrect-only allocation is slightly stronger because more training compute is devoted to resolving hard failures.