- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
MedPRMBench is a step-level benchmark for verifying medical reasoning with an error taxonomy adapted from PRMBench. It draws on seven medical QA sources containing different kinds of medical reasoning tasks and formats. Each reasoning trace is distilled into a clinical reasoning blueprint to ensure comparable error injection. The authors define 14 error types, extending PRMBench’s with five new and two re-scoped ones. After automatic filtering, 14,982 candidate instances are validated by one of 86 physicians and checked by a second with 1,000 samples being double-annotated. The best of 20 prompted critics reaches 75.4 PRMScore. A Qwen3-8B PRM trained on the split reaches 86.6, replicates across backbones, and transfers to ProcessBench and PRMBench without further training. Ablations cover physician review, leave-one-type-out training, Class A vs. B traces, and the evaluation protocol.
Critical strengths
- As far as I know there is no comparable existing medical PRM benchmark targeting clinical failure modes, so this resource fills a real gap. Train/eval split are consistent and clean and the authors provide an unusual amount of quality control.
- Every sample is annotated and checked by two clinicians (of 86) rather than spot-checked on a subset, which makes physician involvement unusually extensive. This gives the resource a level of grounding that synthetic benchmarks usually lack.
Critical weaknesses
- The annotation is circular; physician corrections are only adopted if 2/3 LLMs support them (l.245f.). These three models are Gemini-3.1-Pro, Claude-Sonnet-4.6, GPT-5.4 (l.1765 f.), yet GPT-5.4 also seems to be the best critic (l. 309).
- The papers motivation is that existing benchmarks mainly cover mathematics and not patient specific clinical errors (l.13f.). The paper’s own probes seem to suggest that MedPRMBench doesn’t address this: replacing a question with an unrelated one leaves 96.7% of the trained PRM’s step labels unchanged and even on the Contextual Applicability (R-3) and Safety Awareness (R-5) error steps, which depend on patient context, labels are unchanged 96.6% and 92.7% of the time (p.23).
Questions for authors
- Lines 255f. state that 1,600 physician-authored corrections failed the LLM majority vote, how many corrections did physicians suggest in total?
- On the double-annotated 1,000 samples, how often does the LLM vote agree with the independent second clinician?
- Appendix P2 (p. 38) reports the “hard subset” only for the frontier models, how does the trained PRM compare?
5 comments
Response to Reviewer vrFt (2/2): questions
Response to Reviewer vrFt (2/2)
Q1. How many corrections did physicians suggest in total? 13,189, in 11,614 of the 14,981 completed annotations (77.5%): 11,372 to the step–type map and 1,817 to the reference reasoning. The vote adopted 11,525 (87.4%); the 1,664 rejected corrections fall in the 1,600 excluded candidates.
Q2. How often does the vote agree with the independent second clinician? On the 923 double-annotated candidates with a map vote (which sets the evaluated step labels), the vote and the independent physician reach the same keep-or-correct decision in 83.1% of cases. Both call for a correction in about 90% of candidates, so chance agreement alone is 82.4% and this raw rate carries little information. The informative breakdown is adoption conditional on what the independent physician did:
- the other physician produced exactly the same erroneous steps (346 cases): the vote adopted the correction in 93.1%;
- they corrected the map differently (490): adopted in 88.4%;
- they accepted the original labels (87): adopted in 86.2%.
Adoption is highest when the independent physician had marked the same erroneous steps. The two procedures also answer different questions: the vote judges whether a specific physician correction is justified, while the second physician annotates the same system labels independently. The vote never replaces a physician label with a model label.
Q3. How does the trained PRM do on the Hard Subset? 87.8 under the same protocol (86.6 overall), with error-step recall 74.2% and correct-step acceptance 97.2%. The subset is harder for eight of the nine API critics but not, by PRMScore, for the trained PRM, consistent with App. P.2's description of it as a model-dependent stress test. The revised Figure 7 adds this result.
Thank you for your response. I don't find the rebuttal very convincing and will keep my scores.
Response to Reviewer vrFt (1/2): weaknesses
Response to Reviewer vrFt (1/2)
Thank you for the specific questions.
W1. Is the annotation circular?
We agree that GPT-5.4 is both a voter and the top critic. We separate label writing from candidate selection:
We measured the remaining selection channel directly:
The revised Table 3 reports scores on questions with no decisive voter and marks the critics that served as voters.
W2. Does MedPRMBench test patient-specific errors?
Physicians labeled every step with the full vignette in view.
We then tested whether detecting the benchmark's errors draws on the question (vignette and options). A strong non-voter critic, qwen3.8-max (74.1 PRMScore; top critics 73.7–75.4), scored all 6,500 corrupted chains with the question intact, swapped (App. D.11) and withheld. Type groups were fixed before scoring: patient-context (R-3, R-5, R-6, E-1) and chain-internal (S-1, S-2, R-2).
The top Table 3 critic shows the same pattern. GPT-5.4 (Table 3 snapshot, same prompt) re-scored all 6,500 chains with the question withheld: against its Table 3 outputs, recall drops by 8.6 [7.8, 9.4] on patient-context and 4.7 [3.6, 5.6] on chain-internal types, a difference of 3.9 [2.7, 5.2], and the swapped question changes 33% of its step decisions.
The open-weight Qwen3-32B, which anyone can rerun, replicates both results. In non-thinking mode, withholding the question lowers its recall by 15.2 [14.0, 16.4] on patient-context and 7.5 [6.3, 8.6] on chain-internal types, a difference of 7.7 [6.2, 9.3], and the swap changes 49% of its step decisions; in thinking mode the difference is 7.2 [5.6, 8.8] and the swap changes 57%.
A post-hoc check on the log-odds scale gives the same contrast for all four critic settings, with every 95% CI excluding zero (App. D.11). The prompted critics we tested are thus affected most on the pre-specified patient-context types.
Restatement in the chain does not explain the baseline's overall invariance: steps before an erroneous step restate a median 15% of the vignette's content words, and on the third of R-3 steps with the least restatement, 96.9% of decisions survive the swap. Its decisions change most on E-1 (91.4% agreement, App. D.11), R-2 (92.6%) and R-5 (92.7%), against 95.2% on all physician-labeled erroneous steps, and these are among its weakest types (Table 13).
Applied to the trained 8B baseline itself, the same pre-specified contrast gives −1.33 [−2.05, −0.63]: withholding the question does not reduce its recall on the patient-context types. That is the invariance you identified, measured on our own pre-specified endpoint, and we report it as a property of this baseline rather than of the labels. The near-invariance in App. D.11 therefore describes how the trained baseline verifies, and MedPRMBench exposes the failure you describe: a PRM trained on it reaches 86.6 overall yet misses a third of safety errors and half of prerequisite errors, both patient-context types. The revised Section 4.2 states this limitation, and App. D.11 reports the full ablation.