acceptodds
Withdrawn from ICLR 2027

Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time reinforcement learning adapts a model on its own unlabeled test set using majority-vote pseudo-labels and has shown strong results in mathematics. We show that this recipe collapses on medical multiple-choice QA: accuracy stagnates while output diversity rapidly declines. Through a controlled experiment that keeps the questions, model, and optimizer fixed while changing only the answer space, we trace this failure to answer-space structure rather than domain difficulty. In small answer spaces, incorrect rollouts often collide on the same wrong pseudo-label and reinforce it; in large answer spaces, they disperse and receive little reward. This diagnosis motivates PROSE, Process Reward Guided Self-Training, which rewards reasoning quality instead of answer agreement. PROSE scores each reasoning step with a medical process reward model, assigns the trajectory reward as the minimum score across steps, and enforces answer-format constraints. Without labels, PROSE substantially improves a general Llama model, surpassing purpose-built medical models and matching much larger systems. Because the process signal is internalized into the policy, the adapted model requires no reward model at inference and transfers its gains to unseen datasets. We further show that the minimum aggregation is essential: mean aggregation can be exploited, saturating the proxy reward while degrading accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical question answering, LLM reasoning, test-time reinforcement learning, process reward model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x6QEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper observes that test-time reinforcement learning, which rewards trajectories that match the majority vote, can fail when the answer space is narrow, such as multi-choice questions. Therefore, they propose PROSE, process-reward-guided self-training. It rewards the trajectory that has their reasoning process correct instead of the answer, with an auxiliary answer format constraints. More specifically, it uses the lowest process score as the overall reward of the whole reasoning trajectory. By adopting another process reward model, PROSE improves the performance of LLMs on multi-choice medical questions.

Critical strengths
  1. The paper is well motivated: It observes the failure modes of existing test-time RL, and then propose a solution that targets this issue.
  2. The method outperforms the baseline methods: A clear performance gain is shown in the experiment section.
  3. The authors compare the design choices of using mean or minimum process rewards as the training signal, and demonstrate that min consistently outperforms the mean solution. This can be an useful strategy for future process reward reinforcement learning.
Critical weaknesses
  1. This paper adopts an existing process reward model and applied it to their own pipeline. The process reward model, Med-PRM, already shows it superior performance on various QA benchmarks. Although Med-PRM was designed for inference time selection, it still remains a question if the performance gain of PROSE comes from its own design or the performance of Med-PRM.
  2. The paper works only with multi-choice questions, which are known to be prone to spurious signals in reinforcement learning. More scenarios, such as free form medical question answer, should be studied to fully verify the hypothesis in this paper.
  3. The results of the paper is based on the evaluation performance. More details, such as the process reward scores and response lengths, should be shared.
Questions for authors
  1. The improvement of PROSE over baseline is not observed at the scale of 1.7B, and it starts to show only at and after 4B model sizes. Do you think this can be related to reasoning lengths or quality?
Reviewer_RLFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies test-time reinforcement learning for medical multiple-choice question answering. It argues that majority-vote pseudo-label rewards become unreliable when a small answer space causes incorrect responses to concentrate on the same answer. It proposes PROSE, which replaces majority-vote rewards with the minimum step-level correctness score assigned by a frozen medical process reward model, together with a guard against unparseable final answers.

The authors evaluate PROSE on four medical QA benchmarks using Llama and Qwen models. They report improvements over majority-vote test-time RL and process-reward-based inference selection, alongside experiments on transfer to other datasets, reward aggregation, model size, and output-format enforcement. The results suggest that process-reward-guided adaptation can improve medical QA performance, although its benefits depend on policy capacity.

Critical strengths

The empirical gains are substantial and supported by relevant comparisons. For Llama-3.1-8B, PROSE improves average accuracy across the four benchmarks from 0.635 before adaptation and 0.671 with majority-vote test-time RL to 0.740. Comparisons against best-of-N, reward-weighted selection, and a PRM-derived hard-answer-label training baseline help establish that the proposed adaptation has value beyond simply accessing a stronger verifier. The three-seed MedQA evaluation provides additional support for the main improvement.

The paper investigates useful practical limitations. Transfer to questions outside the adaptation dataset provides evidence that the gains are not exclusively tied to the adapted question set. The aggregation and format-guard experiments identify consequential implementation choices, while the weaker results for the 1.7B model appropriately qualify the method’s applicability. These findings could be useful to researchers developing verifier-guided adaptation systems.

Critical weaknesses

The experiment does not isolate answer-space size as the cause of the reported failure. The open-ended and multiple-choice AMC conditions differ in their prompts, output formats, and available answer information. Moreover, the distractors are selected from the same base model’s most frequent incorrect answers, making them particularly relevant to that model’s existing errors. Consequently, the observed difference cannot be attributed specifically to answer-space cardinality. The “protection gap” also counts correctly unrewarded wrong answers: a large gap can coexist with an incorrect positively rewarded answer and an unrewarded correct answer. It therefore does not, by itself, establish that the resulting policy update is beneficial. This matters because the proposed explanation is a central contribution.

The methodological novelty and necessary comparisons are insufficiently established. Prior work such as PURE already investigates minimum-based process-reward optimization to mitigate reward exploitation, although its credit-assignment formulation differs from PROSE’s scalar trajectory reward. Med-PRM also uses minimum-step scoring and explores verifier-guided policy improvement through rejection-sampling fine-tuning. PROSE’s medical transductive setting and empirical findings may constitute a useful contribution, but the paper should explain precisely what is new and include a PRM-filtered fine-tuning baseline using the same adaptation questions and verifier. Without this comparison, it remains unclear how much benefit specifically requires the proposed RL procedure.

Questions for authors

Does the open-ended versus multiple-choice result persist with matched prompting and distractors that are not selected from the evaluated model’s own frequent errors? Can you report the rates of rewarding incorrect answers and withholding reward from correct answers, rather than relying primarily on overall reward-label agreement?

How does PROSE differ technically from existing minimum-based process-reward optimization? How does it compare with PRM-filtered fine-tuning using the same target questions, reward model, and a comparable adaptation budget?

Please provide the exact dataset sizes and split construction, especially for the disjoint MedQA four- and five-option subsets. How were the final training checkpoint and stopping point selected, and were these choices independent of target-set gold-label accuracy?

Reviewer_S2VFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the failure of majority-vote-based test-time reinforcement learning (TTRL) on medical multiple-choice QA, attributing it to incorrect answers being reinforced in small answer spaces. The authors propose PROSE, which replaces majority-vote rewards with rewards from a medical process reward model (PRM). The method shows improvements over standard TTRL and PRM-based inference-time selection.

Critical strengths
  • The analysis of why majority-vote TTRL fails in small answer spaces is interesting, particularly the controlled experiment comparing open-ended and multiple-choice answering.

  • The empirical gains over TTRL are substantial and consistent across the evaluated datasets.

  • The comparison between mean and minimum aggregation of process rewards has interesting insights.

Critical weaknesses
  • Limited novelty and unclear TTRL motivation: The proposed method is essentially standard RL against a pretrained PRM, similar to conventional RLHF. It is unclear why this requires a test-time RL framing rather than standard post-training. A comparison against PRM-based RL on a separate training set would help justify this choice. The assumption that a PRM is available to train against makes this a bad fit for a TTRL framing and I do not get the motivation behind that.

  • Missing strong baselines and compute matching. The PRM inference baselines only rerank generated trajectories whereas the proposed method gets to do significant RL training. Stronger methods such as PRM-guided beam search or tree search are missing, along with training baselines such as SFT and iterative rejection-sampling fine-tuning. Comparisons should also account for the substantial additional training compute used by PROSE.

  • Narrow empirical scope The method and motivation are not specific to medicine, yet experiments are restricted to medical QA with a specialized medical PRM. At least one additional reasoning domain is needed to demonstrate generality.

  • Insufficient analysis of minimum reward aggregation Minimum aggregation may incentivize shorter reasoning traces to avoid low-scoring steps. Changes in reasoning length and step count should be reported for min/mean aggregation training.

Questions for authors
  • How does PROSE compare against standard PRM-based RL trained on a separate set of unlabeled questions? What is the specific advantage of the test-time training paradigm?

  • Can the authors compare against stronger PRM-guided inference methods, such as beam search, and additional training baselines such as rejection sampling fine-tuning?

  • Including one more domain (non-medical) would increase the strength of the results. It would be interesting to demonstrate a similar small answer space effect in another domain.

  • How does minimum aggregation affect reasoning length and the number of steps?

  • Why is it assumed that a PRM is available, but not an outcome level reward model? The assumption of having a PRM available needs more justification.

  • Med-PRM is a specialized medical model with retrieval access to clinical literature. It is unclear whether PROSE improves reasoning or simply distills medical knowledge from a stronger, domain-specialized model. This is why I have an issue with the TTRL framing, as specialized knowledge is now available at test-time and the performance gains simply might be due to the transfer of knowledge from the PRM to the model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.