acceptodds
Under review as a conference paper at ICLR 2027

MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

Abstract

Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce MedPRMBench, a step-level medical reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6,500 evaluation questions, 13,000 reasoning chains, and 113,910 step labels, plus a leakage-clean 6,870-question training split. The best of 20 prompted critics reaches only 75.4% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6% PRMScore and 89.0% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6% to 2.1%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.

Primary area:
datasets and benchmarks
Keywords:
process reward models, medical reasoning, benchmark, step-level evaluation, error detection, clinical AI safety, reward modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vrFtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MedPRMBench is a step-level benchmark for verifying medical reasoning with an error taxonomy adapted from PRMBench. It draws on seven medical QA sources containing different kinds of medical reasoning tasks and formats. Each reasoning trace is distilled into a clinical reasoning blueprint to ensure comparable error injection. The authors define 14 error types, extending PRMBench’s with five new and two re-scoped ones. After automatic filtering, 14,982 candidate instances are validated by one of 86 physicians and checked by a second with 1,000 samples being double-annotated. The best of 20 prompted critics reaches 75.4 PRMScore. A Qwen3-8B PRM trained on the split reaches 86.6, replicates across backbones, and transfers to ProcessBench and PRMBench without further training. Ablations cover physician review, leave-one-type-out training, Class A vs. B traces, and the evaluation protocol.

Critical strengths
  1. As far as I know there is no comparable existing medical PRM benchmark targeting clinical failure modes, so this resource fills a real gap. Train/eval split are consistent and clean and the authors provide an unusual amount of quality control.
  2. Every sample is annotated and checked by two clinicians (of 86) rather than spot-checked on a subset, which makes physician involvement unusually extensive. This gives the resource a level of grounding that synthetic benchmarks usually lack.
Critical weaknesses
  1. The annotation is circular; physician corrections are only adopted if 2/3 LLMs support them (l.245f.). These three models are Gemini-3.1-Pro, Claude-Sonnet-4.6, GPT-5.4 (l.1765 f.), yet GPT-5.4 also seems to be the best critic (l. 309).
  2. The papers motivation is that existing benchmarks mainly cover mathematics and not patient specific clinical errors (l.13f.). The paper’s own probes seem to suggest that MedPRMBench doesn’t address this: replacing a question with an unrelated one leaves 96.7% of the trained PRM’s step labels unchanged and even on the Contextual Applicability (R-3) and Safety Awareness (R-5) error steps, which depend on patient context, labels are unchanged 96.6% and 92.7% of the time (p.23).
Questions for authors
  1. Lines 255f. state that 1,600 physician-authored corrections failed the LLM majority vote, how many corrections did physicians suggest in total?
  2. On the double-annotated 1,000 samples, how often does the LLM vote agree with the independent second clinician?
  3. Appendix P2 (p. 38) reports the “hard subset” only for the frontier models, how does the trained PRM compare?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vrFt (1/2): weaknesses

Response to Reviewer vrFt (1/2)

Thank you for the specific questions.

W1. Is the annotation circular?

We agree that GPT-5.4 is both a voter and the top critic. We separate label writing from candidate selection:

  1. The vote cannot write a label. Every retained step label was written or explicitly approved by physicians (annotated by one, checked by a different one in 98.6% of records), so a voter cannot make a label agree with its own critic prediction; it can only influence which candidates are removed.
  2. The roles are different tasks with different inputs. A voter sees the physician's verdict, correction and comment and judges whether the correction is justified; a critic labels each step blind.
  3. No model decides alone. A correction needs two of three models from different vendors, so GPT-5.4 was decisive only when the other two split. The reasoning vote uses Claude-Sonnet-4.6 and the map vote, which sets the evaluated step labels, Claude-Opus-4.5; GPT-5.4 and Gemini-3.1-Pro vote in both.

We measured the remaining selection channel directly:

  • Without the 1,284 questions on which GPT-5.4's vote was decisive, it scores 75.3 (75.4 overall) and the order of all nine API critics is identical (τ = 1.00).
  • The effect on its margin over GPT-5.2 is small: +0.63 [0.37, 0.88] where its vote was not decisive vs. +0.75 overall.
  • Adding back the 611 exclusions confirmed by two physicians, scored against their correction, keeps the order identical (τ = 1.00), also when restricted to the 344 that all nine critics scored.
  • On the 539 scorable candidates excluded by GPT-5.4's own decisive reject vote, GPT-5.4 is still the top critic (74.3 vs. 73.3 for GPT-5.2), so its votes and its critic judgments do not act as one filter.

The revised Table 3 reports scores on questions with no decisive voter and marks the critics that served as voters.

W2. Does MedPRMBench test patient-specific errors?

Physicians labeled every step with the full vignette in view.

We then tested whether detecting the benchmark's errors draws on the question (vignette and options). A strong non-voter critic, qwen3.8-max (74.1 PRMScore; top critics 73.7–75.4), scored all 6,500 corrupted chains with the question intact, swapped (App. D.11) and withheld. Type groups were fixed before scoring: patient-context (R-3, R-5, R-6, E-1) and chain-internal (S-1, S-2, R-2).

  • Withholding the question lowers error recall by 11.2 points [10.2, 12.2] on patient-context types and by 4.5 [3.5, 5.5] on chain-internal types, a difference of 6.7 [5.3, 8.0] (question-cluster bootstrap).
  • With the swapped question, it changes 40% of its step decisions (agreement 60.2%), mostly by flagging the now-inconsistent chain, while the trained PRM changes 4.6% on the same corrupted chains (agreement 95.4%).

The top Table 3 critic shows the same pattern. GPT-5.4 (Table 3 snapshot, same prompt) re-scored all 6,500 chains with the question withheld: against its Table 3 outputs, recall drops by 8.6 [7.8, 9.4] on patient-context and 4.7 [3.6, 5.6] on chain-internal types, a difference of 3.9 [2.7, 5.2], and the swapped question changes 33% of its step decisions.

The open-weight Qwen3-32B, which anyone can rerun, replicates both results. In non-thinking mode, withholding the question lowers its recall by 15.2 [14.0, 16.4] on patient-context and 7.5 [6.3, 8.6] on chain-internal types, a difference of 7.7 [6.2, 9.3], and the swap changes 49% of its step decisions; in thinking mode the difference is 7.2 [5.6, 8.8] and the swap changes 57%.

A post-hoc check on the log-odds scale gives the same contrast for all four critic settings, with every 95% CI excluding zero (App. D.11). The prompted critics we tested are thus affected most on the pre-specified patient-context types.

Restatement in the chain does not explain the baseline's overall invariance: steps before an erroneous step restate a median 15% of the vignette's content words, and on the third of R-3 steps with the least restatement, 96.9% of decisions survive the swap. Its decisions change most on E-1 (91.4% agreement, App. D.11), R-2 (92.6%) and R-5 (92.7%), against 95.2% on all physician-labeled erroneous steps, and these are among its weakest types (Table 13).

Applied to the trained 8B baseline itself, the same pre-specified contrast gives −1.33 [−2.05, −0.63]: withholding the question does not reduce its recall on the patient-context types. That is the invariance you identified, measured on our own pre-specified endpoint, and we report it as a property of this baseline rather than of the labels. The near-invariance in App. D.11 therefore describes how the trained baseline verifies, and MedPRMBench exposes the failure you describe: a PRM trained on it reaches 86.6 overall yet misses a third of safety errors and half of prerequisite errors, both patient-context types. The revised Section 4.2 states this limitation, and App. D.11 reports the full ablation.

AuthorsOpenReview ↗

Response to Reviewer vrFt (2/2): questions

Response to Reviewer vrFt (2/2)

Q1. How many corrections did physicians suggest in total? 13,189, in 11,614 of the 14,981 completed annotations (77.5%): 11,372 to the step–type map and 1,817 to the reference reasoning. The vote adopted 11,525 (87.4%); the 1,664 rejected corrections fall in the 1,600 excluded candidates.

Q2. How often does the vote agree with the independent second clinician? On the 923 double-annotated candidates with a map vote (which sets the evaluated step labels), the vote and the independent physician reach the same keep-or-correct decision in 83.1% of cases. Both call for a correction in about 90% of candidates, so chance agreement alone is 82.4% and this raw rate carries little information. The informative breakdown is adoption conditional on what the independent physician did:

  • the other physician produced exactly the same erroneous steps (346 cases): the vote adopted the correction in 93.1%;
  • they corrected the map differently (490): adopted in 88.4%;
  • they accepted the original labels (87): adopted in 86.2%.

Adoption is highest when the independent physician had marked the same erroneous steps. The two procedures also answer different questions: the vote judges whether a specific physician correction is justified, while the second physician annotates the same system labels independently. The vote never replaces a physician label with a model label.

Q3. How does the trained PRM do on the Hard Subset? 87.8 under the same protocol (86.6 overall), with error-step recall 74.2% and correct-step acceptance 97.2%. The subset is harder for eight of the nine API critics but not, by PRMScore, for the trained PRM, consistent with App. P.2's description of it as a model-dependent stress test. The revised Figure 7 adds this result.

Thank you for your response. I don't find the rebuttal very convincing and will keep my scores.

Reviewer_iM9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks whether process reward models and prompted critics can identify erroneous steps in medical reasoning, including cases where the final answer is correct. MedPRMBench pairs correct and corrupted reasoning chains from existing medical QA datasets, using controlled error injection, physician review, and model-consistency filtering. The experiments measure step-level error detection and clean-chain false alarms, examine differences across 14 clinical error types, and include a PRM trained on the released training split.

Critical strengths
  1. The benchmark tests for intermediate medical reasoning errors that final-answer evaluation can overlook. Figure 1 illustrates this distinction with a reasoning chain that introduces unsupported clinical evidence yet still reaches the correct final answer.

  2. Breaking down performance across 14 clinical error types helps identify weaknesses that aggregate scores can obscure. Reporting error recall alongside clean-chain false alarms further distinguishes missed errors from over-flagging of correct reasoning.

Critical weaknesses
  1. The annotation-agreement evidence does not yet establish the reliability of the fine-grained error types. The agreement reported in Appendix D.12 is measured on 1,000 Class A candidates and concerns binary judgments of whether a step is erroneous, rather than assignments to the 14 error types. Two physicians may agree that a step is erroneous while disagreeing on its error type. Since the paper uses these type labels to identify specific gaps in model capabilities, the reported agreement statistics do not establish the reliability of these fine-grained diagnostics. In addition, Class B data undergo the same review workflow but are not included in this independent agreement evaluation.
  2. The existing ablation does not separately examine the effect of model-vote filtering on evaluation results. Table 4 shows that physician review improves the training signal, but Section 3.4 and Appendix J.2 state that physician corrections must receive support from at least two models to be adopted, resulting in the exclusion of 1,600 candidates. Some of the voting models also appear in the evaluation, so this filtering process may bias the final evaluation set toward judgments these models endorse. The existing physician-review ablation does not rule out this potential selection bias, and its effect on model scores and rankings remains unclear.
Questions for authors
  1. Among the 1,600 candidates excluded by model voting, how many had corrections proposed by the initial physician and confirmed by the second-stage physician checker? Which error types are most represented among these cases?
  2. The annotation-agreement analysis in Appendix D.12 concerns binary error labels. Did the authors also measure agreement on error-type labels and independent annotation agreement for Class B examples?
4 comments
AuthorsOpenReview ↗

Response to Reviewer iM9X (1/2): error-type and Class B agreement

Response to Reviewer iM9X (1/2): error-type and Class B agreement

W1 / Q2. Agreement on error types.

You are right that binary agreement does not establish type reliability, and we now report both levels. On the 1,000 independently double-annotated Class A candidates the pooled step-by-type κ is 0.52; per type: E-4 0.89, S-1 0.84, S-2 0.84, E-5 0.63, R-3 0.59, E-3 0.52, R-5 0.50, E-2 0.49, E-1 0.47, R-6 0.39, R-1 0.36, R-2 0.32, R-7 0.32, R-4 0.31. On the 2,755 steps both physicians marked erroneous, they assign the identical type set in 39.5%, share at least one type in 77.6%, and agree on the error category (Simplicity/Soundness/Sensitivity) in 72.2% (κ = 0.68). Type labels are thus reliable at the category level and coarser within it: 53% of type disagreements stay inside one category, and the five most frequent confusions all pair a specific type with the broad Evidence-Based Soundness label (R-1 with R-6, R-7, R-2, R-4 or R-3). The revised Table 13 marks the Soundness types with Class A κ below 0.40 (R-1, R-2, R-4, R-6, R-7) as coarser diagnostics, and App. D.12 reports every type.

What this changes for the diagnostics is testable. On the 444 Class A double-annotated test questions, the trained PRM's per-type recall under the released types and under the other physician's independent types correlates at Spearman ρ = 0.90 (13 types with at least 20 steps), with the same four weakest types (E-1, R-1, R-2, R-5). On this subset the identification of the weakest types in Section 4.2 is robust to the choice of physician, while individual per-type recalls move by several points; the revised text states that scope.

Q2, Class B. We have now measured it directly, and we report both pairings. Two further physicians (attendings at Grade-A tertiary hospitals, none among the 86 annotators and checkers) independently re-annotated 280 Class B test candidates (3,619 steps) under the Class A protocol, blind to the released labels, to critic predictions and to each other's annotations; each reviewed all 280. Between themselves they agree at binary step-level κ = 0.73 (95% CI 0.70–0.76), comparable to 0.69 on Class A. Against the released Class B labels they agree at κ = 0.54 and 0.57, below the 0.70 that an independent physician reaches against released Class A labels.

That gap is one-directional and has a measurable cause. 90.9% and 79.6% of the disagreement consists of steps the physicians flag and the release leaves clean; they confirm 92.7% and 85.8% of the steps the release does mark erroneous, and place the first error at the same step in 77.5% and 82.9% of candidates. A deterministic text alignment against the clean reference chain, using no clinical or model judgment, locates the extra flags: 89.6% of them fall on steps that the error construction rewrote, against 33.0% of all released-clean steps, and the physicians flag 50.1% of rewritten-but-clean steps versus 2.9% of steps identical to the reference. The two labellings therefore agree on where each error starts and differ on whether the steps rewritten to follow from it are counted again, which is the convention stated in Section 4.1: each error is labelled once, at the step where it is introduced, and that step is the target of strict first-error localization. Both labellings remain recoverable in the release.

Per-type profiles also agree across provenance: ρ = 0.82 for the trained PRM, with the same four weakest types, and 0.80–0.94 (median 0.91) for the nine API critics.

AuthorsOpenReview ↗

Response to Reviewer iM9X (2/2): the model vote and the evaluation set

Response to Reviewer iM9X (2/2): the model vote and the evaluation set

W2 / Q1. Does the vote filter bias the evaluation set?

Q1. Of the 1,600 excluded candidates, 1,571 carried a correction by the initial physician; the checker left 1,241 of these unchanged and flagged or edited 330, and 29 corrections came from the checking stage. We report the 1,241 as corrections the checker did not alter, which is weaker than an explicit second endorsement. Among them the most represented constructed types are R-2 (372 cases), E-5 (342), R-6 (340), R-1 (323) and S-1 (294); relative to all annotated candidates, each type's share is 0.78–1.38 times its base rate, so the exclusions are spread across all 14 types. The failing vote was the map vote for 1,286, the reasoning vote for 250, and both for 64. The most frequent rejected change adds R-1 to a step (619 of 1,350); 40% of rejected map corrections only re-type steps, and half change types only within one category.

Effect on scores and rankings, in both directions:

  • Remove what the voters let in. Without the 1,284 questions on which GPT-5.4 was decisive, every API critic moves by at most 0.46 and the ranking is identical (τ = 1.00); without every question on which any voter was decisive (4,844 remain), it is also identical (at most 0.53).
  • Add back what the voters kept out. 611 of the 1,241 exclusions have archived critic outputs on the identical chain; pooled with the test set and scored against the physicians' correction, they leave the ranking identical (τ = 1.00; at most 0.23), as do all 785 scorable exclusions (at most 0.33). Restricted to the exclusions that all nine critics scored (344 and 448), so that every critic is compared on identical items, the ranking is again identical (τ = 1.00; at most 0.12 and 0.14). These archived outputs come from our earlier full-corpus run on the same chains rather than from the Table 3 run.

The effect on GPT-5.4's margin over GPT-5.2 is small: +1.23 where its vote was decisive and +0.63 [0.37, 0.88] elsewhere. The revised PDF reports scores on questions with no decisive voter beside Table 3 (full breakdown in Table 24).

These tests bound selection on the populations we can score; they do not establish that the retained pool is unbiased, and we state that scope in the revision.

AuthorsOpenReview ↗

Follow-up to Reviewer iM9X

Thank you again for your review. As the discussion period is closing, we wanted to check whether our response addressed your concerns.

Both points you raised are now measured directly. On error types we report agreement at every level, including the exact type-set agreement of 39.5%, category agreement of 72.2% and per-type κ for all 14 types, and we mark the five coarser Soundness types in the revised Table 13. On Class B we ran the blind double annotation you asked for: two further physicians agree with each other at κ = 0.73 and with the released labels at κ = 0.54 and 0.57, and we trace that gap to a labelling convention rather than to label noise, with both labellings recoverable in the release.

If any point remains open, or if a specific number from these analyses would help, we would be glad to provide it before the discussion closes.

AuthorsOpenReview ↗

Final note for Reviewer iM9X

As the discussion period closes today, we wanted to flag one thing in case it is useful for your assessment.

The Class B blind double annotation you asked for is a new result that was not in the submission: two further physicians, neither among the original annotators, re-annotated 280 Class B candidates independently and blind, and we report their agreement both with each other and with the released labels, together with a deterministic analysis of where the two labellings differ. It is in the revised Appendix D.12 and Table 23, and summarised in our first comment to you.

If that result, or the type-level agreement we now report for all 14 types, changes your view on the weaknesses you raised, we would be grateful if you could say so before the discussion closes. If it does not, a brief note on what is still missing would help us for the revision.

Reviewer_YUwkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MedPRMBench, a benchmark specifically designed to evaluate the reasoning process of medical large language models. It covers 14 categories of medical reasoning errors and provides step-level annotations for identifying specific errors within reasoning chains. The authors construct the data based on 7 medical question answering datasets and, through physician review, form a test set containing 6,500 evaluation questions. Furthermore, the authors fine-tune Qwen3-8B using the training set of this benchmark to train a medical process reward model (PRM) for identifying erroneous steps in the reasoning process, rather than merely judging whether the final answer is correct.

Critical strengths
  1. There is support from a very large number of physician resources.
  2. The paper also proposed a reward model which facilitates subsequent evaluation work.
Critical weaknesses
  1. In Section 3.4, the authors mention that "Physician corrections are adopted only if a >= 2/3 three-model consistency vote supports them." I am quite confused. Since there is support from so many professional physicians, why not conduct cross-validation among physicians instead of relying on the model's majority voting to judge correctness?
  2. In the processing of Category B questions (samples without a reasoning ground truth source), the authors used multiple models to generate trajectories and invited physicians to verify them. Although the authors used multiple models, if a model is evaluating trajectories generated by itself, would this lead to higher scores for itself? This may require some experimental analysis to ensure that the results on such samples are not biased toward the model that generated the sample.
  3. Even in tertiary hospitals, there may be many very young physicians who are not sufficiently qualified. If the minimum, maximum, and average years of clinical practice of the physicians involved in annotation could be reported, I think this would greatly enhance the credibility of the paper. I am also curious whether there was any standardized rating scale used during physician evaluation to make physicians' judgments as consistent as possible.
Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YUwk

Response to Reviewer YUwk

Thank you for the constructive questions.

W1. Why a model vote rather than cross-validation among physicians?

Physician cross-validation is the primary mechanism: every record received a second-stage check, by a different physician in 98.6% of records, and checkers flagged or edited 17.1% of records. In addition, 1,000 Class A candidates were annotated twice, independently and blind (κ = 0.69 on step labels; App. D.12), and we have now run a similar blind double annotation on 280 Class B candidates with two further physicians (κ = 0.73).

Physicians control every label. A vote is held only on a physician's correction; voters see it with the physician's verdict and comment and decide only whether it is kept. The vote cannot write, edit or add a label, and a failed correction removes the candidate.

A third physician for all 11,614 corrected records was beyond our budget. The vote, which needs two of three models from different vendors, is a screen on physician corrections rather than a substitute for physicians: it adopted 87.4% of corrections, and 93.1% of those for which the second physician had independently marked the same erroneous steps on the double-annotated subset. Rejections are enriched for type-boundary changes: 40% of rejected map corrections only re-type steps (31% of adopted ones), most often by adding R-1.

In both selection tests we ran (removing the questions on which any voter was decisive, and adding back exclusions that two physicians confirmed), the ranking of the nine API critics is unchanged (τ = 1.00; general response, point 2).

W2. Could a model score its own Class B traces higher?

We find no evidence of self-preference. First, a correction, now made in Appendix F: every retained Class B chain comes from Gemini-3-Pro-preview or GPT-5.2 (2,376 and 413 test chains), none from DeepSeek-R1. Second, critics never see a raw trace: each chain is rebuilt from the clinical reasoning blueprint and then corrupted (median word-level Jaccard 0.43 to the source). Third, a direct test: a critic's PRMScore on its own chains minus the other generator's, minus the same contrast averaged over the other Table 3 API critics (removing pool difficulty). With 95% bootstrap intervals:

  • GPT-5.2: −0.08 [−1.01, 0.86]; after the same adjustment, its error recall on its own chains is 2.9 points higher, not lower.
  • Gemini-3-Pro-preview, the generator itself (archived outputs, 1,113 own vs. 217 other chains): −0.23 [−2.79, 3.02].
  • Gemini-3.1-Pro, its successor: −2.00 [−4.45, 0.25].

This is consistent with Table 5, where the trained PRM and four of six critics find generated traces harder.

W3. Physician qualifications and a standardized protocol.

Seniority was controlled at recruitment, which required at least three years of clinical practice; all 86 annotators and checkers had between three and eight years (now in Section 3.4 and Appendix D.12). Years were not recorded per reviewer, so we report this range rather than a mean. All practise at Grade-A tertiary hospitals in China, the highest accreditation tier. Beyond seniority, label quality is measured directly: every record received a second-stage check (by a different physician in 98.6% of records), per-annotator κ on step labels ranges from 0.60 to 0.74 across the 19 physicians with at least 200 paired steps in the Class A double annotation, and on the new blind Class B double annotation two further attending physicians agree with each other at κ = 0.73.

Review followed a fixed protocol. Reviewers worked from a written guide that defines each of the 14 error types, which our physician team built from the clinical literature on PRMBench's three-category scaffold (seven retained, two re-scoped, five new; Table 1), and lists the questions to answer per record. The interface recorded categorical decisions, not free scores: (1) whether the reference reasoning is correct, with erroneous step indices and corrected text; (2) whether the step–type map is accurate, with the corrected map; (3) a mandatory written rationale. The checker reviewed the same fields. Because these are categorical judgments against written definitions, we report κ rather than a rating-scale reliability; the revised Appendix J.2 now includes the guide and the interface fields.

AuthorsOpenReview ↗

Follow-up to Reviewer YUwk

Thank you again for your review. As the discussion period is closing, we wanted to check whether our response addressed your concerns.

In response to your questions we added evidence rather than argument: a blind double annotation of 280 Class B candidates by two further physicians (binary step-level κ = 0.73, against 0.69 on Class A), a direct test for generator self-preference on the Class B traces (all three contrasts within noise, and a correction to Appendix F), and the recruitment criterion, per-annotator agreement (κ 0.60–0.74) and the full annotation protocol, now in Section 3.4 and Appendix J.2.

If any point remains open, or if you would like a specific detail from these analyses, we would be glad to provide it before the discussion closes.

Thanks for the response, the authors have solved most problems, so I will raise the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General response: physician-controlled labels, the model vote, and new analyses

We thank the reviewers. All three ask about the three-model consistency vote. Below we state what it does and summarize the new evidence: re-analyses of the archived review records and critic outputs, a new blind double annotation of Class B, and a context ablation (critic scores under the Table 3 protocol).

1. Physicians write or approve every label; the vote only screens their corrections. Each of the 14,981 completed candidates was annotated by one physician and checked at a second stage, by a different physician in 98.6% of records; checkers flagged or edited 17.1% of records. Physicians made 13,189 corrections in 11,614 records, and 63% of map corrections added erroneous steps the constructed labels had missed. A vote is held only on a physician's correction (the 3,367 records without a correction were never voted on) and only decides whether it is kept; it cannot write, edit or add a label, and a rejected correction removes the candidate rather than reverting it. The vote adopted 87.4% of corrections.

2. Removing or restoring vote-affected questions leaves the API-critic ranking unchanged. The three map-vote models (GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.5) are also Table 3 critics, so we tested both directions:

  • Removing every question on which any voter was decisive (4,844 remain) leaves the order of all nine API critics identical (Kendall τ = 1.00).
  • Adding back 611 exclusions that one physician corrected and a second confirmed, scored against that correction, also gives τ = 1.00 (no critic moves more than 0.23), as does the subset of 344 that all nine critics scored, so that every critic is compared on identical items.

The effect on GPT-5.4's margin over GPT-5.2 is small: +0.63 [0.37, 0.88] on questions where its vote was not decisive vs. +0.75 overall.

3. Type labels and Class B: agreement measured directly. On 1,000 independently double-annotated Class A candidates, pooled step-by-type κ is 0.52 (0.68 at the category level) and the error category agrees on 72.2% of jointly flagged steps; the trained PRM's per-type recall ranking correlates at ρ = 0.90 across the two labelings, with the same four weakest types. Class B now has its own blind double annotation: on 280 Class B candidates two further physicians agree with each other at binary step-level κ = 0.73 (Class A pairs: 0.69), and with the released Class B labels at κ = 0.54 and 0.57 against 0.70 on Class A. Their disagreement is almost entirely extra flags on steps the error construction rewrote; the comment to iM9X characterises it. Class B per-type profiles also match Class A (ρ = 0.82 for the trained PRM, 0.80–0.94 for the nine API critics), and an acceptance-stage physician (not among the 86) audited 504 Class B records without flagging any.

4. Patient context: the prompted critics we tested are affected most on patient-context types. Withholding the question costs a strong non-voter critic (qwen3.8-max, 74.1 PRMScore) 11.2 points of error recall on pre-specified patient-context types vs. 4.5 on chain-internal types; the top critic GPT-5.4 (8.6 vs. 4.7) and the open-weight Qwen3-32B (15.2 vs. 7.5) show the same pattern. A post-hoc check on the log-odds scale gives the same contrast for every critic. Under the App. D.11 swap, qwen3.8-max changes 40% of its step decisions and GPT-5.4 33%, vs. 4.6% for the trained PRM, so the invariance in App. D.11 describes the trained baseline.

5. Further details. All 86 physicians practise at Grade-A tertiary hospitals, recruited with at least three years of clinical practice (all within three to eight years), and agreement is consistent across physicians (per-annotator κ 0.60–0.74 across the 19 physicians with at least 200 paired Class A steps). We find no evidence of self-preference on generated Class B traces. The trained PRM scores 87.8 on the Hard Subset (86.6 overall).

Revision. We have uploaded a revised PDF with all changes in blue. It adds a main-text summary of label reliability and scope (Section 5.4) and these analyses to Appendices D.6, D.11, D.12 and J.2, including the Class B double annotation (D.12). It also adds critic scores on questions with no decisive voter beside Table 3, the trained PRM to Figure 7, the corrected Class B generator list (Appendix F), and the physicians' recruitment requirement and seniority range (Section 3.4, Appendix D.12). As stated in Section 3.4, we will release the vote records, initial annotations and checker outcomes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.