acceptodds
Under review as a conference paper at ICLR 2027

RLVR from Within: Learning to Ask to Elicit Intrinsic Rewards for Test-Time Reasoning

Abstract

Test-time reasoning for large language models (LLMs) often relies on external process reward models (PRMs) or value estimators to guide search over intermediate reasoning states. However, learning such rewards from final outcomes poses a challenging credit-assignment problem, and the reasoner’s own self-verification may not reliably predict final-answer correctness. To address both challenges, we propose AskR, which learns what to ask to make the reasoner’s self-verification more predictive of final correctness and thus more useful as an intrinsic reward. Rather than learning an external PRM, AskR learns a sub-question policy that structures reasoning into sequences of sub-questions and sub-answers (SubQAs), exposing dense self-verification signals at each SubQA. By shaping which intermediate reasoning states are exposed, AskR makes the induced intrinsic reward more predictive of final-answer correctness and therefore more useful for test-time optimization. Across our benchmarks, AskR consistently improves over PRM-guided reasoning under the same optimizer, by 7.12 points on average and up to 9.18 points depending on the optimizer. In its strongest setting, AskR further outperforms Reinforcement Learning with Verifiable Rewards (RLVR) on average, while using only 5% and 17% as much training data as RLVR on text and visual reasoning tasks, respectively. Our results suggest a new route to test-time reasoning: instead of learning how to evaluate arbitrary reasoning states, learn which states to expose so that the reasoner's existing verification signal becomes useful.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time reasoning, Large language models, Inference-time scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EX2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AskR, a question generator trained to make a frozen reasoning model’s self-verification more useful for test-time search. AskR generates sub-questions, which the reasoning model answers and checks. Only the question generator is trained, using GRPO with a reward that encourages high self-verification scores on correct solutions and penalizes them on incorrect ones. At test time, the scores guide Best-of-N, TPO, or TMPC without a separately trained reward model. Experiments on text and visual reasoning benchmarks report improvements over PRM-guided baselines. Further experiments compare training objectives and evaluate transfer across reasoner sizes.

Critical strengths
  1. The paper offers an interesting alternative to training a separate verifier: it learns which sub-questions make a fixed reasoner’s self-verification more predictive of final-answer correctness. The approach requires only final-answer supervision, without step-level labels, and trains the question generator with LoRA on 5K examples.

  2. The evaluation covers text and vision-language benchmarks, three test-time optimizers, and multiple model families. Comparisons with the untrained question generator under the same inference pipeline (Table 3) help establish that learning the questioning policy contributes beyond introducing the SubQA format alone.

  3. The model-scale experiments show that a single trained Qwen3-4B question generator improves performance across Qwen3 reasoners from 1.7B to 32B. This suggests that the learned questioning strategy can be reused across reasoner sizes without retraining AskR for each model.

Critical weaknesses
  1. The comparison with RLVR uses different inference procedures and budgets. AskR uses best-of-N selection or iterative optimization, whereas the RLVR baseline generates answers directly. The reported gains therefore do not isolate the benefit of learning to ask from additional test-time computation. A single-sample AskR comparison and compute-matched search baselines for RLVR would strengthen this claim.

  2. The benefit of decomposition and SubQA-level verification over direct self-verification is not established. Table 3 and Table 8 retain sub-question decomposition at inference, while the PRM baselines use an external verifier. A missing control is standard CoT followed by the same frozen reasoner assessing whether the complete solution reaches the correct answer, without access to ground truth. Using the same optimizers and comparable total inference budgets would clarify whether AskR’s decomposition and verification pipeline offers an advantage over this simpler alternative.

  3. The role of the self-verification score in training remains unclear. Table 8 shows that including r improves performance, but does not distinguish informative feedback from changes in reward weighting. Given the paper’s own observation that self-verification need not indicate progress toward the correct final answer (Sections 1 and 4.1), a control that shuffles r across candidates within each problem would be useful. This would preserve the score distribution while breaking its association with each solution, helping determine whether that association is essential to the gains.

Questions for authors
  1. How does AskR perform with a single generated solution, without candidate selection or iterative search? Does its advantage over RLVR persist when both systems receive comparable inference budgets and search opportunities?

  2. With the same AskR-generated candidates and search procedure, how does SubQA-level scoring compare with directly judging the correctness of the complete solution?

  3. Does the training benefit of rr depend on its association with the corresponding solution? A control that shuffles these scores across candidates within each problem would help distinguish informative feedback from reward reweighting.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EX2G [1/2]

We thank the reviewer for the constructive feedback and the valuable suggestions for additional experiments. We address each concern below.

W1 / Q1. Single-sample AskR and inference-budget comparison

[W1] The reported gains do not isolate the benefit of learning to ask from additional test-time computation. A single-sample AskR comparison and compute-matched search baselines for RLVR would strengthen this claim.

[Q1] How does AskR perform with a single generated solution, without candidate selection or iterative search? Does its advantage over RLVR persist when both systems receive comparable inference budgets and search opportunities?

We thank the reviewer for suggesting a more controlled comparison of training-based and test-time optimization.

We conduct additional Qwen3-4B experiments on MMLU-Pro, scaling BoN from N=1N=1 to 55. Following Reviewer's W2, CoT and RLVR use direct self-verification of complete solutions for candidate selection, while AskR uses its SubQA-based intrinsic reward.

To assess inference costs fairly, we report both generated output tokens and end-to-end wall-clock latency. Token counts include question generation, reasoning, and verification for AskR, and reasoning and direct self-verification for CoT/RLVR:

NNMethodAccuracy (%)Output TokensLatency (s)
1CoT + Self-Verify69.604,3980.75
RLVR + Self-Verify73.985,1961.06
AskR71.805,9921.10
2CoT + Self-Verify71.538,7971.40
RLVR + Self-Verify74.829,9401.58
AskR73.2411,3391.66
3CoT + Self-Verify72.4413,1952.03
RLVR + Self-Verify75.9915,7442.11
AskR75.4118,3252.12
4CoT + Self-Verify72.9617,5942.59
RLVR + Self-Verify76.3519,0682.64
AskR76.1521,9962.76
5CoT + Self-Verify73.3521,9923.21
RLVR + Self-Verify76.9123,1553.16
AskR76.7926,6543.21

1. Informative rewards enable more effective test-time scaling. As NN increases to 5, AskR gains +4.99 points, compared with +3.75 points for CoT using direct self-verification. This highlights the value of learning to elicit informative rewards, rather than relying on direct self-verification alone.

**2. AskR approaches RLVR under comparable inference budgets. RLVR benefits from training the reasoning backbone, whereas AskR keeps it frozen and learns a questioning policy. Nevertheless, reward-guided selection narrows the accuracy gap from 2.18 points at N=1N=1 to just 0.12 points at N=5N=5 (76.79% vs. 76.91%), with nearly identical end-to-end latency (3.21 vs. 3.16 seconds). At N=4N=4, AskR already reaches 76.15% using fewer output tokens than RLVR at N=5N=5 (21,996 vs. 23,155).

These results clarify the complementary contributions of learned reward elicitation. Rather than requiring additional training of the reasoner, AskR enables a frozen backbone to approach RLVR-level performance through effective intrinsic-reward-guided test-time optimization.

W2. SubQA verification vs. direct self-verification

[W2] A missing control is standard CoT followed by the same frozen reasoner assessing whether the complete solution reaches the correct answer, without access to ground truth. Using the same optimizers and comparable total inference budgets would clarify whether AskR’s decomposition and verification pipeline offers an advantage over this simpler alternative.

We thank the reviewer for suggesting this important control experiment.

Building on W1/Q1, where AskR benefits more from reward-guided selection than CoT with direct self-verification, we further investigate whether this advantage persists under a stronger test-time optimizer.

As suggested by the reviewer, we introduce CoT + TMPC with Direct Self-Verification, where the same frozen Qwen3-4B reasoner evaluates complete CoT solutions without access to ground-truth answers. We compare it against AskR using the same TMPC optimizer and comparable output-token budgets:

MethodMMLU-ProLAB-BenchSciEvalAvg.Reasoner Output Tokens
CoT + TMPC (Direct Verification)71.4924.4374.4556.79133,209
AskR + TMPC (SubQA Verification)78.4244.8376.4666.57134,512

With the same optimizer and nearly identical output-token consumption (~1% difference), AskR improves average accuracy by +9.78 points.

Together with W1/Q1, these results show that the benefit of AskR cannot be attributed solely to additional test-time search. Even when direct self-verification is combined with the stronger TMPC optimizer, it does not match AskR's performance.

This supports the effectiveness of learning to elicit informative intrinsic rewards through SubQA-based reasoning, beyond simply applying an optimizer to holistic self-verification.

AuthorsOpenReview ↗

Response to Reviewer EX2G [2/2]

Q2. SubQA-level scoring vs. direct self-verification on identical candidates

[Q2] With the same AskR-generated candidates and search procedure, how does SubQA-level scoring compare with directly judging the correctness of the complete solution?

We thank the reviewer for suggesting this controlled comparison.

We isolate the contribution of SubQA-level verification through a controlled extension of Table 2 using Qwen2.5-7B (BoN, N=8N=8). Both verification methods evaluate identical AskR-generated candidates using the same frozen reasoner, differing only in their scoring prompt:

  • Direct Self-Verification: The reasoner judges the correctness of the complete solution using a holistic correctness prompt, without access to ground truth.
  • SubQA Verification (AskR): The reasoner verifies individual SubQA pairs, whose judgments are aggregated into the intrinsic reward.
Scoring MethodMMLU-ProLAB-BenchSciEval
Direct Self-Verification59.0433.3860.04
SubQA Verification (AskR)61.2335.0565.43

SubQA verification achieves higher average accuracy than direct self-verification on identical candidates. This distinction is central to AskR: the questioning policy is explicitly trained to elicit trajectories that the frozen reasoner can reliably verify, rather than relying on generic holistic judgments.

Together with the response to W2, this provides complementary evidence for AskR's design: the complete reward-elicitation pipeline improves test-time optimization, while this fixed-candidate comparison directly evaluates the contribution of its SubQA-level scoring.

Revision: We added the fixed-candidate selection and verification comparisons and corresponding analysis in Lines 1452–1490.

W3 / Q3. Does the association between intrinsic reward and its trajectory matter?

[W3] The role of the self-verification score in training remains unclear. Table 8 shows that including r improves performance, but does not distinguish informative feedback from changes in reward weighting. Given the paper’s own observation that self-verification need not indicate progress toward the correct final answer (Sections 1 and 4.1), a control that shuffles r across candidates within each problem would be useful. This would preserve the score distribution while breaking its association with each solution, helping determine whether that association is essential to the gains.

[Q3] Does the training benefit of (r) depend on its association with the corresponding solution? A control that shuffles these scores across candidates within each problem would help distinguish informative feedback from reward reweighting.

We thank the reviewer for this insightful suggestion. To distinguish informative feedback from reward reweighting, we introduce a shuffled-reward training ablation.

For each training problem, we randomly permute the intrinsic rewards across candidates within the GRPO group, replacing the original reward Ri=ciriR_i=c_i r_i with Rishuffle=cirπ(i)R_i^{\mathrm{shuffle}}=c_i r_{\pi(i)}. This preserves the per-problem distribution of intrinsic scores while breaking their correspondence with the originating trajectories. All other training and inference settings remain unchanged.

We show the effect of shuffling intrinsic rewards during AskR training (Qwen3-4B, TMPC):

Training RewardMMLU-ProLAB-BenchSciEvalAvg.
Shuffled70.3233.0672.3658.58
Original (Eq. 5)78.4244.8376.4666.57

Shuffling reduces average accuracy by 7.99 points, with particularly large drops on MMLU-Pro (−8.10) and LAB-Bench (−11.77).

Since the intrinsic reward values are preserved within each problem but their trajectory associations are disrupted, this result provides strong evidence that the benefit of rr depends on informative, trajectory-specific feedback rather than merely changing reward magnitudes.

We hope these additional controlled experiments clarify the respective contributions of reward elicitation, test-time optimization, and correctness-aligned training.

Revision: We added the shuffled-reward training ablation and corresponding analysis in Lines 1492–1503.

Dear Reviewer EX2G,

As the discussion period will end in about two days, we wanted to kindly follow up in case you have not yet had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be very happy to provide further clarification.

We briefly summarize the main updates addressing your concerns:

  • Compute-controlled RLVR comparisons: We added BoN scaling experiments from N=1 to N=5, showing that AskR approaches RLVR performance (76.79% vs. 76.91% at N=5) with nearly identical latency, while keeping the reasoner frozen.
  • Direct self-verification baselines: Under the same TMPC optimizer and nearly identical reasoner output-token budgets, AskR outperforms CoT with direct self-verification by +9.78 points. On identical AskR-generated candidates, SubQA-level verification further improves accuracy by +3.08 points over holistic self-verification.
  • Shuffled-reward ablation: We added the suggested experiment that shuffles intrinsic rewards across candidates during training. This reduces average accuracy by 7.99 points, demonstrating the importance of trajectory-specific reward information rather than reward reweighting alone.

We hope these controlled experiments provide stronger evidence for AskR's contributions and address your concerns. If you find our responses satisfactory, we would be grateful if you could take them into consideration when reassessing your evaluation.

We sincerely appreciate your time and valuable suggestions, and look forward to hearing from you.

Best regards,

Authors of Paper 13725

Reviewer_5EsMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AskR, which trains a question generator to elicit useful self-verification rewards from a frozen reasoner. The reasoner answers the generated sub-questions and jointly verifies the resulting QA pairs. AskR is trained with a reward combining this verification score and signed final-answer correctness. At inference, the verification score guides BoN, TPO, or TMPC. Experiments cover language and vision-language benchmarks, with additional comparisons across model families, reasoner scales, and training objectives.

Critical strengths
  1. Learning questions that make a fixed reasoner's verification more useful is an interesting alternative to training an external reward model. The resulting reward can be used with several test-time optimizers.
  2. Tables 3 and 8 provide meaningful evidence for the training objective, comparing against both the untrained asker and an asker trained only on answer correctness. The fixed-asker experiments across reasoner scales further support the usefulness of the learned questioning policy.
Critical weaknesses
  1. The practical advantage under matched inference compute remains unclear. Appendix C reports token costs, but matching candidate counts does not match computation when AskR adds question generation and generative verification. Table 5 also does not explain how repeated prompts, verification inputs, and reward-model scoring are counted. Without a common compute budget or measured latency, the comparisons with PRM guidance and simpler test-time methods do not establish which approach uses inference resources more effectively.
  2. The evidence for improved reward selection is less direct than the end-to-end results suggest. Table 8 supports the proposed objective, but changing the asker also changes the candidate trajectories. Likewise, the benchmark-level correlations in Table 7 do not establish that the reward ranks competing answers to the same problem more reliably. A comparison of selectors on identical SubQA candidates would help distinguish better candidate generation from better reward-guided selection, which is central to the paper's interpretation.
Questions for authors
  1. Does Table 5 sum all input and output tokens across question generation, candidate generation, self-verification, and PRM scoring? For example, PRM + BoN reports the same 291 input tokens as CoT despite scoring eight candidates. Please clarify the accounting, including prefix caching, and report accuracy under matched total inference budgets; wall-clock latency would provide useful complementary evidence.
  2. On the same candidates generated using the trained asker, how does intrinsic-reward selection compare with uniform selection and majority voting? Reporting these comparisons for both training objectives in Table 8 would clarify whether correctness-aligned training primarily improves candidate quality or the usefulness of verification for selection.
  3. Are the correlations and AUROC in Table 7 pooled across problems? Could the authors report within-problem ranking performance on candidate sets containing both correct and incorrect answers, before and after training?
  4. Which exact checkpoints and thinking settings are used in Tables 4 and 6? In particular, how are images processed for the vision-language results labeled Qwen3-4B in Table 4: is a Qwen3-VL model or a separate visual component used?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5EsM [1/2]

We thank the reviewer for the constructive feedback and insightful suggestions. We address each concern below and have revised the manuscript accordingly.

W1 & Q1. Inference efficiency and token accounting

[W1] Without a common compute budget or measured latency, the comparisons with PRM guidance and simpler test-time methods do not establish which approach uses inference resources more effectively.

[Q1] Does Table 5 sum all input and output tokens across question generation, candidate generation, self-verification, and PRM scoring? For example, PRM + BoN reports the same 291 input tokens as CoT despite scoring eight candidates. Please clarify the accounting, including prefix caching, and report accuracy under matched total inference budgets; wall-clock latency would provide useful complementary evidence.

We thank the reviewer for emphasizing the importance of accurate inference-cost accounting and fair comparisons.

Clarification of Table 5. The original "Input" column reports initial prompt length rather than cumulative input tokens, explaining why CoT and PRM + BoN both report 291 tokens. The original output-token counts cover reasoner-side generation but exclude AskR's question generation and external PRM scoring. To avoid ambiguity, we replace Table 5 with a new comparison reporting generated output tokens and wall-clock latency, with explicit accounting of the included costs.

Controlled inference-budget comparisons. We focus on Best-of-N as the cleanest shared-optimizer setting, where candidates are generated independently and reward-guided selection does not affect subsequent generations, unlike iterative optimizers such as TPO and TMPC. We conduct additional experiments using Qwen3-4B on MMLU-Pro, varying candidate counts to compare AskR and PRM under two complementary budgets: generated output tokens and end-to-end wall-clock latency. For AskR, output tokens include question generation, reasoning, and intrinsic verification; for PRM, they include reasoner generation, as external PRM scoring is not directly reflected in generated-token counts.

BudgetMethodNAccuracy (%)Cost
Output tokensAskR + BoN877.3642,640 tokens
PRM + BoN2475.2942,765 tokens
Wall-clock latencyAskR + BoN877.364.69 s
PRM + BoN1675.164.92 s

AskR consistently achieves higher accuracy under both budget criteria: +2.07 points at nearly identical reported output-token counts and +2.20 points at comparable end-to-end latency. Notably, AskR achieves these gains with 8 candidates, compared with 16–24 for PRM, demonstrating the effectiveness of its learned intrinsic reward beyond simply generating more candidates.

Understanding the inference overhead: AskR generates questions only once per problem (755 output tokens), rather than once per candidate. Its intrinsic verification takes 0.72 seconds per candidate, compared with 0.40 seconds for PRM scoring. While generated-token counts cannot fully capture external reward-model computation, end-to-end latency accounts for the practical runtime of both pipelines, including questioning and scoring overhead.

Overall, these complementary comparisons demonstrate that AskR's accuracy advantage persists under comparable generated-token and wall-clock budgets, even when accounting for its additional questioning and verification costs. We hope these measurements provide convincing evidence of AskR's effectiveness and clarify its accuracy-efficiency trade-offs.

Revision: We replaced the original Table 5 with the new inference-budget comparison in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer 5EsM [2/2]

W2 / Q2. Candidate quality vs. reward-guided selection

[W2] A comparison of selectors on identical SubQA candidates would help distinguish better candidate generation from better reward-guided selection, which is central to the paper's interpretation.

[Q2] On the same candidates generated using the trained asker, how does intrinsic-reward selection compare with uniform selection and majority voting?

We appreciate this important suggestion.

While trajectory generation and reward quality are coupled by design, we can still characterize their reward-correctness alignment. Accordingly, we compare AskR and PRM using both pooled and within-problem metrics and demonstrate that learning what to ask indeed elicits more informative intrinsic rewards for test-time reasoning without training an external verifier (cf. our responses to Reviewer Ht3S, W1 and W2).

We agree that the contribution of reward-guided selection should be evaluated separately. Extending Table 2 under the same Qwen2.5-7B setting (BoN, N=8N=8), we compare intrinsic-reward selection, uniform selection, and majority voting under both training objectives:

Training ObjectiveSelectionMMLU-ProLAB-BenchSciEvalAvg.
Correctness-alignedIntrinsic Reward61.2335.0565.4353.90
Uniform56.5331.3156.5348.12
Majority Voting58.7633.4862.2451.49
Answer correctness onlyIntrinsic Reward58.0933.3262.0151.14
Uniform55.8330.7359.6648.74
Majority Voting56.4731.0060.5149.33

Under uniform selection, the two training objectives achieve comparable accuracy (48.12% vs. 48.74%), suggesting similar average candidate quality. However, with intrinsic-reward selection, correctness-aligned training achieves 53.90% vs. 51.14%, increasing the selection gain over uniform from +2.40 to +5.78 points.

Since all selectors operate on identical candidates within each training objective, these results provide direct evidence that correctness-aligned training makes self-verification substantially more effective for selecting correct solutions, beyond differences in average candidate quality.

Revision: We added the fixed-candidate selection comparisons in Lines 1453–1493.

Q3. Within-problem reward-correctness alignment

Are the correlations and AUROC in Table 7 pooled across problems? Could the authors report within-problem ranking performance on candidate sets containing both correct and incorrect answers, before and after training?

Table 7 reports pooled reward-correctness alignment within each benchmark. To directly assess whether the intrinsic reward can distinguish competing solutions to the same problem, we evaluate per-problem AUROC before and after AskR training, using N=8N=8 candidates per problem.

For each problem containing both correct and incorrect candidates, we compute AUROC using only candidates from that problem (ties count as 0.5), then macro-average. No cross-problem candidate comparisons are involved.

AskRMMLU-ProLAB-BenchSciEvalAvg.
Before Training0.55340.59620.55390.568
After Training0.75630.62150.75260.710

Training increases the average per-problem AUROC from 0.568 to 0.710, with particularly substantial improvements on MMLU-Pro (0.553 → 0.756) and SciEval (0.554 → 0.753).

These results show that the trained AskR elicits intrinsic rewards that more reliably rank correct above incorrect candidates within individual problems, rather than merely reflecting differences in problem difficulty.

Revision: We added the within-problem AUROC evaluation for PRM and AskR (before and after training) in Lines 481–499.

Q4. Model checkpoints and thinking settings

Which exact checkpoints and thinking settings are used in Tables 4 and 6? How are images processed for the vision-language results labeled Qwen3-4B in Table 4: is a Qwen3-VL model or a separate visual component used?

The exact checkpoints used in our experiments are listed below:

Model / SettingHugging Face Checkpoint
Qwen2.5-7B (Language)Qwen/Qwen2.5-7B-Instruct
Qwen2.5-7B (Vision-Language)Qwen/Qwen2.5-VL-7B-Instruct
Qwen3-1.7BQwen/Qwen3-1.7B
Qwen3-4B (Language)Qwen/Qwen3-4B-Instruct-2507
Qwen3-4B (Vision-Language)Qwen/Qwen3-VL-4B-Instruct
Qwen3-8BQwen/Qwen3-8B
Qwen3-14BQwen/Qwen3-14B
Qwen3-32BQwen/Qwen3-32B
Gemma3-4Bgoogle/gemma-3-4b-it

All experiments use non-thinking inference. For switchable checkpoints, thinking is disabled using enable_thinking=False. Vision-language inputs are processed natively by the corresponding multimodal backbone, without a separate visual component.

Revision: We added the model checkpoints and clarified the configurations in Lines 1232–1253.

Dear Reviewer 5EsM,

As the discussion period will end in about two days, we wanted to kindly follow up in case you have not yet had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be very happy to provide further clarification.

We briefly summarize the main updates addressing your concerns:

  • Compute-matched comparisons: We added output-token- and latency-matched experiments, showing that AskR outperforms PRM by +2.07 and +2.20 points, respectively, under comparable inference budgets.
  • Fixed-candidate selection experiments: We compared intrinsic-reward selection against uniform selection and majority voting. Correctness-aligned training increases the selection gain over uniform selection from +2.40 to +5.78 points, providing direct evidence for improved reward-guided selection.
  • Within-problem reward alignment: We added per-problem AUROC analyses, showing an improvement from 0.568 before training to 0.710 after training, confirming that the learned rewards better distinguish correct from incorrect candidates.
  • Model configuration clarification: We specified the exact model checkpoints, non-thinking settings, and native vision-language processing configurations to improve reproducibility.

We hope these additional experiments provide stronger evidence for AskR's effectiveness and address your concerns regarding inference efficiency and reward quality. If our responses have addressed your concerns, we would be grateful if you could reconsider your evaluation.

We sincerely appreciate your time and constructive suggestions, and look forward to hearing from you.

Best regards,

Authors of Paper 13725

Reviewer_Ht3SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Trains an asker to generate subquestions for a frozen reasoner, an interesting alternative to PRM.
  • Uses the reasoner's self-verification to guide search without answer labels at inference.
  • Approach gets gains across text and vision tasks. Authors do ablations and transfer across reasoner sizes.
Critical strengths
  • Learning an interface for self-verification is a promising area and this paper makes progress on it. The objective ablations and reuse of a fixed asker across reasoner sizes support its value.
Critical weaknesses

Main:

  • The PRM comparison changes both reasoning format and scoring, so it does not isolate reward quality.
  • The reported reward-alignment analysis pools problems of different difficulty. Show within problem variation. Second:
  • Reconcile the paper's top-k selection and candidate counts with the code's threshold selection and unused count flags.
  • Fix the optional STEM verifier's score-ordering bug, which can assign scores to the wrong requests. State whether this path affected any reported results.
Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ht3S [1/2]

We thank the reviewer for the constructive feedback. We address each concern below and have revised the manuscript accordingly.

W1. PRM comparison and reward quality

The PRM comparison changes both reasoning format and scoring, so it does not isolate reward quality.

AskR aims to elicit informative intrinsic rewards, rather than improve a standalone reward model. Its learned questioning policy jointly shapes the reasoning trajectories and the resulting verification signals. For reward-guided test-time optimization, the key question is whether these signals align with final-answer correctness and support effective candidate selection.

We therefore compare AskR and PRM in terms of reward-correctness alignment across the three benchmarks:

MethodMetricMMLU-ProLAB-BenchSciEval
PRMPearson0.07430.07120.0789
Spearman0.08820.05510.0385
AUROC0.55100.56070.5227
AskRPearson0.47310.38320.4512
Spearman0.43100.28980.4244
AUROC0.76010.61420.7591

AskR consistently exhibits stronger reward-correctness alignment. This comparison characterizes the resulting reward-elicitation pipelines, rather than independently isolating scoring on identical trajectories.

To further substantiate the result, our response to W2 reports within-problem AUROC (0.710 for AskR vs. 0.524 for PRM), directly evaluating reward discrimination among competing solutions to the same problem. Moreover, our fixed-candidate control (Reviewer EX2G, Q2) shows that SubQA verification outperforms direct self-verification by +3.08 accuracy points on identical AskR-generated candidates.

Together, these complementary analyses support our central contribution: learning what to ask elicits more informative intrinsic rewards for test-time reasoning without training an external verifier.

W2. Within-problem reward-correctness alignment

The reported reward-alignment analysis pools problems of different difficulty. Show within problem variation.

To directly evaluate whether intrinsic rewards distinguish competing solutions to the same problem, we additionally report per-problem AUROC using N=8N=8 candidates per problem.

For each problem containing both correct and incorrect candidates, we compute AUROC using only within-problem candidate comparisons (ties count as 0.5), and then macro-average. No candidates from different problems are compared.

MethodMMLU-ProLAB-BenchSciEvalAvg.
PRM0.56110.49450.51740.524
AskR (Trained)0.75630.62150.75260.710

AskR achieves an average per-problem AUROC of 0.710, substantially exceeding PRM's 0.524. This analysis directly demonstrates the effectiveness of AskR's intrinsic reward for within-problem candidate ranking.

Revision: We added the within-problem AUROC evaluation for PRM and AskR (before and after training), together with the corresponding analysis, in Lines 481–499.

AuthorsOpenReview ↗

Response to Reviewer Ht3S [2/2]

W3. Clarification of top-k selection and candidate counts

Reconcile the paper's top-k selection and candidate counts with the code's threshold selection and unused count flags.

We thank the reviewer for pointing out this. We believe this refers to the TMPC implementation (cf. Appendix E.4.2), and we’d like to clarify the description of trajectory selection:

Specifically, trajectory admission is threshold-based rather than top-k-based: trajectories with rϕ(x,τ)≥0.8r_\phi(x,\tau)\geq 0.8 contribute SubQA pairs to the SegmentedBuffer. These pairs are grouped into three positional segments (opening, middle, ending), where duplicates are removed and at most three pairs per segment are retained, prioritized by their originating trajectory's reward across iterations. One pair is then sampled from each non-empty segment to guide subsequent generation.

Thus, the threshold controls trajectory admission, while the top-3 limit controls buffer capacity, not the number of generated candidates. The count-related flags noted by the reviewer are not used in this selection procedure. Candidate generation follows the configuration described in Appendix E.4.2.

Notably, these are implementation details of the TMPC-based test-time optimizer, rather than requirements of AskR itself.

Revision: We clarified the trajectory selection and buffer management procedures in Appendix E.4.2.

W4. STEM verifier ordering issue

Fix the optional STEM verifier's score-ordering bug, which can assign scores to the wrong requests. State whether this path affected any reported results.

We thank the reviewer for catching this issue. The concurrent verify_batch implementation could return scores in completion order rather than request order. We fixed it by tagging results with request indices and restoring their original order, and added a regression test with randomized completion latencies.

Notably, this bug did not affect any reported results. The concurrent verify_batch implementation is only used in LLM-judge verifiers for STEM answers. The affected concurrent verification path was not used:

  1. Evaluation: All six benchmarks use exact-match grading rather than the LLM verifier. Therefore, the affected verification path is not involved in evaluation.

  2. Training: During the reported training runs, each STEM answer was sent to the judge as an independent call, and its score was written back to that sample immediately. The verify_batch path was not used, so no score could be assigned to the wrong sample.

Consequently, all reported results remain unchanged. We will include the fix and regression test in the updated code release. We thank the reviewer again for helping us improve the clarity and reproducibility of our work.

Dear Reviewer Ht3S,

We sincerely appreciate your positive evaluation and encouraging feedback on our work.

As the discussion period will end in about two days, we wanted to kindly follow up in case you have not yet had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be very happy to provide further clarification.

We briefly summarize the main updates addressing your concerns:

  • Within-problem reward alignment: We added per-problem AUROC evaluations, showing that AskR achieves 0.710 compared with 0.524 for PRM, providing stronger evidence that its intrinsic rewards can distinguish correct from incorrect candidates within the same problem.
  • Implementation and reproducibility: We clarified the threshold-based TMPC selection and buffer management procedures, and fixed the STEM verifier ordering issue, confirming that the reported results remain unaffected.

We hope these additional experiments and clarifications have addressed your concerns and would be grateful if you could consider them when reassessing your evaluation of our work.

Thank you again, and we look forward to hearing from you.

Best regards, Authors of Paper 13725

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We really thank all reviewers for their constructive feedback. We have conducted additional controlled experiments and strengthened the manuscript accordingly, with all revisions highlighted in blue in the updated manuscript.

We emphasize that AskR learns to elicit informative intrinsic rewards from a frozen reasoner, rather than retraining the backbone, training an external verifier, or proposing a new test-time optimizer. This distinction is central to understanding its contribution and accuracy-efficiency trade-offs.

Below, we summarize the key additions:

(1) Stronger evidence for intrinsic reward quality

  • The within-problem analyses show that trained AskR achieves an average AUROC of 0.710 vs. 0.524 for PRM, demonstrating informative reward-correctness alignment beyond problem-difficulty effects (Reviewers Ht3S W1-W2; 5EsM Q3).

  • Controlled ablations further show that correctness-aligned training increases the gain from reward-guided selection over uniform selection from +2.40 to +5.78 points (Reviewer 5EsM W2/Q2), while shuffling trajectory-specific rewards reduces accuracy by 7.99 points (Reviewer EX2G W3/Q3).

(2) Stronger controlled comparisons for test-time optimization

  • Our BoN scaling experiments compare AskR, CoT, and RLVR using direct self-verification baselines. At N=5, AskR achieves 76.79%, outperforming CoT (73.35%) and approaching RLVR (76.91%) at comparable latency, while keeping the reasoner frozen (Reviewer EX2G W1/Q1).

  • Under TMPC with nearly identical reasoner output-token budgets, AskR outperforms CoT with direct self-verification by +9.78 points on average, supporting the effectiveness of its learned SubQA-based reasoning pipeline (Reviewer EX2G W2).

  • Furthermore, on identical AskR-generated candidates, SubQA-level verification outperforms holistic self-verification by +3.08 percentage points on average, directly demonstrating the benefit of SubQA-level scoring for candidate selection (Reviewer EX2G Q2).

(3) Clearer inference-cost accounting and reproducibility

  • We conducted output-token- and latency-matched comparisons using the same BoN optimizer. AskR achieves +2.07 points over PRM at nearly identical generated output-token counts and +2.20 points at comparable wall-clock latency, even when PRM uses more candidates (Reviewer 5EsM W1/Q1).

  • Finally, we clarified model configurations and TMPC implementation details and fixed the verifier ordering issue, confirming that the reported results remain unaffected (Reviewers Ht3S W3-W4; 5EsM Q4).


We believe these additional experiments provide further evidence for AskR's contributions. We sincerely thank the reviewers for their constructive feedback and hope the strengthened manuscript merits their continued consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.