- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AskR, a question generator trained to make a frozen reasoning model’s self-verification more useful for test-time search. AskR generates sub-questions, which the reasoning model answers and checks. Only the question generator is trained, using GRPO with a reward that encourages high self-verification scores on correct solutions and penalizes them on incorrect ones. At test time, the scores guide Best-of-N, TPO, or TMPC without a separately trained reward model. Experiments on text and visual reasoning benchmarks report improvements over PRM-guided baselines. Further experiments compare training objectives and evaluate transfer across reasoner sizes.
Critical strengths
-
The paper offers an interesting alternative to training a separate verifier: it learns which sub-questions make a fixed reasoner’s self-verification more predictive of final-answer correctness. The approach requires only final-answer supervision, without step-level labels, and trains the question generator with LoRA on 5K examples.
-
The evaluation covers text and vision-language benchmarks, three test-time optimizers, and multiple model families. Comparisons with the untrained question generator under the same inference pipeline (Table 3) help establish that learning the questioning policy contributes beyond introducing the SubQA format alone.
-
The model-scale experiments show that a single trained Qwen3-4B question generator improves performance across Qwen3 reasoners from 1.7B to 32B. This suggests that the learned questioning strategy can be reused across reasoner sizes without retraining AskR for each model.
Critical weaknesses
-
The comparison with RLVR uses different inference procedures and budgets. AskR uses best-of-N selection or iterative optimization, whereas the RLVR baseline generates answers directly. The reported gains therefore do not isolate the benefit of learning to ask from additional test-time computation. A single-sample AskR comparison and compute-matched search baselines for RLVR would strengthen this claim.
-
The benefit of decomposition and SubQA-level verification over direct self-verification is not established. Table 3 and Table 8 retain sub-question decomposition at inference, while the PRM baselines use an external verifier. A missing control is standard CoT followed by the same frozen reasoner assessing whether the complete solution reaches the correct answer, without access to ground truth. Using the same optimizers and comparable total inference budgets would clarify whether AskR’s decomposition and verification pipeline offers an advantage over this simpler alternative.
-
The role of the self-verification score in training remains unclear. Table 8 shows that including r improves performance, but does not distinguish informative feedback from changes in reward weighting. Given the paper’s own observation that self-verification need not indicate progress toward the correct final answer (Sections 1 and 4.1), a control that shuffles r across candidates within each problem would be useful. This would preserve the score distribution while breaking its association with each solution, helping determine whether that association is essential to the gains.
Questions for authors
-
How does AskR perform with a single generated solution, without candidate selection or iterative search? Does its advantage over RLVR persist when both systems receive comparable inference budgets and search opportunities?
-
With the same AskR-generated candidates and search procedure, how does SubQA-level scoring compare with directly judging the correctness of the complete solution?
-
Does the training benefit of depend on its association with the corresponding solution? A control that shuffles these scores across candidates within each problem would help distinguish informative feedback from reward reweighting.
3 comments
Response to Reviewer EX2G [2/2]
Q2. SubQA-level scoring vs. direct self-verification on identical candidates
[Q2] With the same AskR-generated candidates and search procedure, how does SubQA-level scoring compare with directly judging the correctness of the complete solution?
We thank the reviewer for suggesting this controlled comparison.
We isolate the contribution of SubQA-level verification through a controlled extension of Table 2 using Qwen2.5-7B (BoN, ). Both verification methods evaluate identical AskR-generated candidates using the same frozen reasoner, differing only in their scoring prompt:
- Direct Self-Verification: The reasoner judges the correctness of the complete solution using a holistic correctness prompt, without access to ground truth.
- SubQA Verification (AskR): The reasoner verifies individual SubQA pairs, whose judgments are aggregated into the intrinsic reward.
| Scoring Method | MMLU-Pro | LAB-Bench | SciEval |
|---|---|---|---|
| Direct Self-Verification | 59.04 | 33.38 | 60.04 |
| SubQA Verification (AskR) | 61.23 | 35.05 | 65.43 |
SubQA verification achieves higher average accuracy than direct self-verification on identical candidates. This distinction is central to AskR: the questioning policy is explicitly trained to elicit trajectories that the frozen reasoner can reliably verify, rather than relying on generic holistic judgments.
Together with the response to W2, this provides complementary evidence for AskR's design: the complete reward-elicitation pipeline improves test-time optimization, while this fixed-candidate comparison directly evaluates the contribution of its SubQA-level scoring.
Revision: We added the fixed-candidate selection and verification comparisons and corresponding analysis in Lines 1452–1490.
W3 / Q3. Does the association between intrinsic reward and its trajectory matter?
[W3] The role of the self-verification score in training remains unclear. Table 8 shows that including r improves performance, but does not distinguish informative feedback from changes in reward weighting. Given the paper’s own observation that self-verification need not indicate progress toward the correct final answer (Sections 1 and 4.1), a control that shuffles r across candidates within each problem would be useful. This would preserve the score distribution while breaking its association with each solution, helping determine whether that association is essential to the gains.
[Q3] Does the training benefit of (r) depend on its association with the corresponding solution? A control that shuffles these scores across candidates within each problem would help distinguish informative feedback from reward reweighting.
We thank the reviewer for this insightful suggestion. To distinguish informative feedback from reward reweighting, we introduce a shuffled-reward training ablation.
For each training problem, we randomly permute the intrinsic rewards across candidates within the GRPO group, replacing the original reward with . This preserves the per-problem distribution of intrinsic scores while breaking their correspondence with the originating trajectories. All other training and inference settings remain unchanged.
We show the effect of shuffling intrinsic rewards during AskR training (Qwen3-4B, TMPC):
| Training Reward | MMLU-Pro | LAB-Bench | SciEval | Avg. |
|---|---|---|---|---|
| Shuffled | 70.32 | 33.06 | 72.36 | 58.58 |
| Original (Eq. 5) | 78.42 | 44.83 | 76.46 | 66.57 |
Shuffling reduces average accuracy by 7.99 points, with particularly large drops on MMLU-Pro (−8.10) and LAB-Bench (−11.77).
Since the intrinsic reward values are preserved within each problem but their trajectory associations are disrupted, this result provides strong evidence that the benefit of depends on informative, trajectory-specific feedback rather than merely changing reward magnitudes.
We hope these additional controlled experiments clarify the respective contributions of reward elicitation, test-time optimization, and correctness-aligned training.
Revision: We added the shuffled-reward training ablation and corresponding analysis in Lines 1492–1503.
Dear Reviewer EX2G,
As the discussion period will end in about two days, we wanted to kindly follow up in case you have not yet had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be very happy to provide further clarification.
We briefly summarize the main updates addressing your concerns:
- Compute-controlled RLVR comparisons: We added BoN scaling experiments from N=1 to N=5, showing that AskR approaches RLVR performance (76.79% vs. 76.91% at N=5) with nearly identical latency, while keeping the reasoner frozen.
- Direct self-verification baselines: Under the same TMPC optimizer and nearly identical reasoner output-token budgets, AskR outperforms CoT with direct self-verification by +9.78 points. On identical AskR-generated candidates, SubQA-level verification further improves accuracy by +3.08 points over holistic self-verification.
- Shuffled-reward ablation: We added the suggested experiment that shuffles intrinsic rewards across candidates during training. This reduces average accuracy by 7.99 points, demonstrating the importance of trajectory-specific reward information rather than reward reweighting alone.
We hope these controlled experiments provide stronger evidence for AskR's contributions and address your concerns. If you find our responses satisfactory, we would be grateful if you could take them into consideration when reassessing your evaluation.
We sincerely appreciate your time and valuable suggestions, and look forward to hearing from you.
Best regards,
Authors of Paper 13725
Response to Reviewer EX2G [1/2]
We thank the reviewer for the constructive feedback and the valuable suggestions for additional experiments. We address each concern below.
W1 / Q1. Single-sample AskR and inference-budget comparison
We thank the reviewer for suggesting a more controlled comparison of training-based and test-time optimization.
We conduct additional Qwen3-4B experiments on MMLU-Pro, scaling BoN from N=1 to 5. Following Reviewer's W2, CoT and RLVR use direct self-verification of complete solutions for candidate selection, while AskR uses its SubQA-based intrinsic reward.
To assess inference costs fairly, we report both generated output tokens and end-to-end wall-clock latency. Token counts include question generation, reasoning, and verification for AskR, and reasoning and direct self-verification for CoT/RLVR:
1. Informative rewards enable more effective test-time scaling. As N increases to 5, AskR gains +4.99 points, compared with +3.75 points for CoT using direct self-verification. This highlights the value of learning to elicit informative rewards, rather than relying on direct self-verification alone.
**2. AskR approaches RLVR under comparable inference budgets. RLVR benefits from training the reasoning backbone, whereas AskR keeps it frozen and learns a questioning policy. Nevertheless, reward-guided selection narrows the accuracy gap from 2.18 points at N=1 to just 0.12 points at N=5 (76.79% vs. 76.91%), with nearly identical end-to-end latency (3.21 vs. 3.16 seconds). At N=4, AskR already reaches 76.15% using fewer output tokens than RLVR at N=5 (21,996 vs. 23,155).
These results clarify the complementary contributions of learned reward elicitation. Rather than requiring additional training of the reasoner, AskR enables a frozen backbone to approach RLVR-level performance through effective intrinsic-reward-guided test-time optimization.
W2. SubQA verification vs. direct self-verification
We thank the reviewer for suggesting this important control experiment.
Building on W1/Q1, where AskR benefits more from reward-guided selection than CoT with direct self-verification, we further investigate whether this advantage persists under a stronger test-time optimizer.
As suggested by the reviewer, we introduce CoT + TMPC with Direct Self-Verification, where the same frozen Qwen3-4B reasoner evaluates complete CoT solutions without access to ground-truth answers. We compare it against AskR using the same TMPC optimizer and comparable output-token budgets:
With the same optimizer and nearly identical output-token consumption (~1% difference), AskR improves average accuracy by +9.78 points.
Together with W1/Q1, these results show that the benefit of AskR cannot be attributed solely to additional test-time search. Even when direct self-verification is combined with the stronger TMPC optimizer, it does not match AskR's performance.
This supports the effectiveness of learning to elicit informative intrinsic rewards through SubQA-based reasoning, beyond simply applying an optimizer to holistic self-verification.