acceptodds
Under review as a conference paper at ICLR 2027

CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

Abstract

The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction. To ensure high clinical fidelity at scale, we construct the dataset using a semi-automated generation pipeline paired with review by six medical experts. CheXpercept contains 12,000 QA items derived from 2,400 CXRs, covering eight clinically critical pulmonary and cardiac lesions. To demonstrate the current landscape of VLM perception, we benchmark 18 general and medical VLMs on CheXpercept. The models achieve adequate performance only at the coarse level, with accuracy degrading precipitously on deeper visual tasks. Notably, recent general models outperform recent medical VLMs overall. While medical models show marginal improvements over their specific backbones at certain perception levels, the inherent capability of the backbone itself ultimately dictates performance far more than current domain adaptation strategies.

Primary area:
datasets and benchmarks
Keywords:
Chest X-ray, Medical Benchmark, Lesion Perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mKGROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CheXpercept, a chest X-ray benchmark designed to assess visual perception beyond disease-presence classification. Its 12,000 question–answer items, derived from 2,400 images and reviewed by six medical experts, cover lesion detection, contour evaluation and revision, and attribute extraction for eight pulmonary and cardiac lesions. An evaluation of 18 general and medical vision-language models finds that performance declines substantially beyond coarse-level detection. Recent general models outperform the evaluated medical models overall, with backbone capability appearing to have a greater effect than domain adaptation.

Critical strengths

This paper is well written and easy to follow. The motivation is quite interenting and important.

Critical weaknesses

In this work, the authors introduced a medical benchmark, named CheXpercept. They firstly collected a related dataset and evaluated 18 general and medical VLMs. Yet the above contributions might be insufficient for an acceptance of top-tier conferences with the following concerns. The first one is the gap or mismatch between motivation and their finished work. The motivation of expert-level lesion perception is quite interesting and important, yet their collected dataset aims to find the regions of lesion step by step, which seems like an active segmentation mask annotation tool or framework. Even the generated questions mainly about the locations of the lesions, in other words, just like the segmentation masks. The indeed clinical impact mainly focuses on the diagnosis as well as their interpretability. If the generated QA pairs are almost related to the locations of lesions, why not just using SAM-like segmentation models?

Further, if the authors could step forward, adding more usages of lesions, such as the improvement for Chain-of-Thought of MLLMs in Medical Image Understanding, it would be much more important and promising.

The claims or findings about the experimental comparisons might be unfair, such as ‘Notably, recent general models outperform recent medical VLMs overall.’ In abstract. The general models are usually 27B, 31B, even 122B, while the medical models are range from 4B, 7B 32B. The authors need to consider the model size in the final claims.

Questions for authors

see weakness

Reviewer_ZGWxOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CheXpercept, a benchmark for evaluating fine-grained lesion perception in chest X-rays through sequential tasks covering lesion detection, contour evaluation and revision, and semantic attribute extraction. It contains 12,000 QA items from 2,400 CXRs and evaluates 18 VLMs.

Critical strengths
  1. The motivation is important, as existing benchmarks provide limited evaluation of fine-grained lesion perception.

  2. The multi-stage design and lesion-level visual grounding enable more detailed analysis of model capabilities. The authors also conduct extensive analyses, providing useful insights for the research community.

  3. The extensive evaluation reveals substantial limitations in current VLMs and the limited benefits of medical domain adaptation.

Critical weaknesses
  1. The human baseline is based on only 24 questions per physician, limiting the reliability of comparisons with expert performance.

  2. The ground-truth masks are largely model-generated and expert-filtered rather than independently annotated. The reliability of the ground truth and potential selection bias remain unclear.

  3. Raw accuracy is used despite class imbalance in some tasks. Majority-class baselines and class-balanced metrics are needed to better assess actual perceptual capabilities.

Questions for authors
  1. How representative are the synthetic mask perturbations and multiple-choice revision tasks of real-world lesion perception and contour correction?

  2. What clinical evidence supports the area-based severity thresholds and other rule-based semantic attribute definitions?

6 comments

If we have misunderstood the intent of your feedback or questions in any way, please let us know. We are more than happy to engage in further discussion promptly.

W: Weakness, Q: Question

[W1] Limited human evaluation

Recognizing the practical burden of asking individual medical experts to manually evaluate a large volume of questions, we limited the number of evaluation items per physician. However, we carefully constructed the questions so that all lesion types and evaluation paths were uniformly included. Furthermore, to enhance the reliability of the human baseline, we expanded the pool of evaluators from 3 to 5 by recruiting two additional experts (who evaluated 24 questions each). The updated results, including the 95% confidence intervals, are presented in the table below:

S1 E2ES2 E2ES3 E2ES4 E2ES2 OPS3 OPS4 OPRR (0-4)RF (0-3)LF (0-1)
98.3 [95.8, 100.0]93.8 [87.5, 98.8]67.5 [52.5, 80.1]45.7 [35.3, 56.6]96.2 (+2.4) [92.5, 100.0]72.5 (+5.0) [60.0, 85.0]54.3 (+8.6) [44.2, 65.2]3.05 [2.75, 3.33]2.50 [2.27, 2.70]1.00 [1.00, 1.00]
S3 Exp.S3 Con.S3 Res.S4 Dist.S4 Loc.S4 Sev.S4 Comp.
85.0 [75.0, 95.0]85.0 [75.0, 95.0]95.0 [87.5, 100.0]94.3 [88.6, 98.6]74.3 [64.2, 83.8]94.3 [88.9, 98.6]85.7 [78.6, 92.5]

[W2] Reliability of ground-truth masks and potential selection bias

Reliability of the ground truth

Given the practical limitations of manually annotating every lesion mask for a large-scale dataset, we adopted a model-assisted approach where physicians selected the highest-quality mask from model-generated candidates. During this process, physicians implicitly envision an ideal contour for the lesion and choose only the candidate that best aligns with this standard. Therefore, the selected masks are of comparable quality to those drawn entirely by hand, providing sufficient reliability to serve as ground truth for evaluation.

Potential Selection Bias

To address concerns that certain types of cases (e.g., difficult-to-judge lesions) might have been systematically excluded during the selection process, we compared our dataset (CheXpercept) with the source population (MIMIC-ILS). We used Qwen3.8-27B to structure the MIMIC-CXR radiology reports for both groups, extracting information on lesion severity and extent to compare their distributions. The analysis revealed no significant differences between the two groups. Notably, "mild" or "small" lesions, which are often harder to assess due to ambiguous boundaries, still account for a substantial proportion in CheXpercept. Pneumothorax cases were excluded from this specific comparison because their radiology reports were unavailable. The detailed results (including the macro average for each set and the detailed distribution across CheXpert lesions) are presented in the table below:

Dataset / FindingMild / SmallMild-to-ModerateModerateModerate-to-SevereSevere / Large
MIMIC-ILS (macro average)44.7%3.9%13.7%2.1%35.6%
CheXpercept (macro average)43.1%4.3%13.9%3.3%35.4%
Cardiomegaly56.4%6.1%29.1%4.2%4.2%
Edema45.9%10.8%21.6%11.5%10.1%
Effusion47.2%13.2%30.6%3.5%5.6%
Atelectasis54.0%0%16.0%4.0%26.0%
Opacity44.9%0%0%0%55.1%
Consolidation33.3%0%0%0%66.7%
Pneumonia20.0%0%0%0%80.0%

[W3] Absence of class-balanced metrics and majority-class baselines

We appreciate the reviewer for this valuable suggestion. Some tasks in Stages 3 and 4 are structured with a varying number of choices per question or formulated as multiple-answer questions (with several ground truths). Consequently, it is difficult to uniformly apply majority-class baselines or class-balanced metrics across all tasks. Nevertheless, following your suggestion, we will report the majority-class baselines and class-balanced metrics for those tasks that feature a fixed number of choices and a single ground-truth answer.

TaskMajority Class Acc
S166.7
S250.0
S3 Res (revision result)28.2
S4 Dist (distribution)39.6
S4 Sev (severity)54.6
S4 Comp (comparison)70.6
ModelS1S2S3 ResS4 DistS4 SevS4 Comp
GPT-5.482.755.571.072.168.477.9
GPT-5.4-nano74.752.229.767.058.357.1
Gemini-3.1-pro85.756.670.575.678.081.0
Gemini-3.1-flash85.152.732.757.461.866.9
Qwen3.6-27B88.750.938.262.258.565.7
Qwen3.5-122B87.850.037.865.564.764.8
GLM-4.6V69.350.133.264.061.259.2
InternVL3.581.832.833.266.763.271.0
Gemma4-31B81.054.255.062.673.164.7
MedGemma-27B76.550.824.545.040.842.3
MedGemma1.5-4B50.249.825.231.233.828.9
HuatuoGPT-V-7B65.949.620.737.137.730.8
Lingshu-32B89.147.924.148.960.665.8
Hulu-Med-32B88.949.924.452.648.753.2

[Q1] Synthetic mask perturbations and multiple-choice revision tasks

All tasks were carefully formulated through discussions with clinical experts during the design phase. First, regarding mask deformation, we generated suboptimal masks using SAM. Rather than merely simulating arbitrary errors, our specific goal was to produce masks that closely resemble the ground-truth but exhibit distinct inaccuracies at the contour level. The deformation was confined exclusively to the contour regions, and its magnitude was carefully adjusted in consultation with medical experts to achieve this subtle balance. Furthermore, all generated suboptimal masks were manually reviewed by these experts to ensure that the errors were at a level that a physician would actually notice and correct in real-world clinical practice.

Second, as mentioned in lines 177–178, the revision task was inspired by the clinical training process, where an attending physician provides feedback to correct a resident’s initial findings. While simulating this process as a multi-turn dialogue would be ideal, it is impractical for large-scale evaluation due to constraints such as context length limitations. Therefore, we designed it as a three-turn multiple-choice format (expansion, contraction, and revision result selection). The core objective of this task is to evaluate whether the model accurately perceives the precise contour of a lesion. Since selecting the correct revision direction in a multiple-choice format inherently requires this contour-level perceptual capability, the task effectively fulfills its evaluation purpose.

[Q2] Clinical evidence for the area-based thresholds and rule-based attribute definitions

All rule-based attribute definitions in our benchmark were established through consultation with clinical experts.

Geometric extent (severity)

For severity, we evaluated lesion severity through the model’s ability to perceive the geometric extent of a lesion, for the following reasons.

First, from a clinical perspective, geometric extent is a primary determinant when radiologists assess lesion severity. For instance, an existing clinical study on edema [1] has directly quantified severity based on the geometric extent of the lesion. We therefore considered the ability to judge geometric extent as reflecting the perceptual basis of severity assessment.

Second, quantifying severity across lesion types is inherently difficult. Radiologists exhibit differing tendencies in how they describe severity, and even if lesion-specific severity thresholds were defined, the resulting questions would often be impractically fine-grained. For instance, a small pleural effusion transitions to a more severe category within a very narrow range, so a question such as “Does the effusion occupy more than 1/5 but less than 2/5 of the lower lung?” would be difficult to answer reliably even for board-certified radiologists.

For these reasons, we evaluated the geometric extent of all lesions over a broader and uniform range (using 1/3, 2/3). However, we acknowledge that directly linking this capability to “classifying severity” might overstate what the task actually measures. Therefore, we will revise the manuscript to describe this task in terms of lesion size and geometric extent rather than severity.

Distribution

Radiologists typically evaluate lesion spread with reference to anatomical lung lobes or zones. Because our evaluation is based on 2D chest radiographs, we established our criteria based on lung zones rather than lobes. Similarly, a previous medical study on COVID-19 pneumonia [2] has assessed disease spread by dividing the lungs into six distinct zones. Thus, our distribution attribute counts the number of involved lung zones, directly reflecting standard clinical reporting.

Comparison

Relative size comparisons (e.g., "left larger than right") are common in reports. To ensure reliable visual discernibility by humans, we set a 1.5-fold difference threshold, as differences as small as 1.01 are practically indistinguishable without precise measurement tools.

Location

The location attribute refers to the specific anatomical regions where a lesion is situated, following standard radiological terminology. Building on the lung zones (upper, middle, and lower) used as the basic frame of reference, we additionally included finer-grained regions commonly used in radiology reports (medial, lateral, peripheral, and the costophrenic angle).

Reference

[1] Warren, Melissa A., et al. "Severity scoring of lung oedema on the chest radiograph is associated with clinical outcomes in ARDS." Thorax 73.9 (2018): 840-846.

[2] Borghesi, Andrea, and Roberto Maroldi. "COVID-19 outbreak in Italy: experimental chest X-ray scoring system for quantifying and monitoring disease progression." La radiologia medica 125.5 (2020): 509-513.

Reviewer_es4rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CheXpercept is a chest X-ray benchmark designed to evaluate lesion perception beyond disease-presence classification. It contains 12,000 question–answer items derived from 2,400 CXRs covering eight findings. Evaluation proceeds through lesion detection, candidate-contour assessment, contour revision, and attribute extraction, with different paths for lesion-free cases and masks requiring or not requiring revision.

The benchmark combines existing segmentation resources, controlled mask perturbations, rule-based question generation, and expert review. Eighteen VLMs are evaluated under end-to-end scoring and an Oracle-Passed setting that replaces incorrect previous answers with ground truth. The results reveal substantial difficulties in contour assessment and point-wise revision. Comparisons between medical models and their respective backbones suggest that the evaluated medical adaptations improve coarse detection more consistently than the benchmark’s finer tasks. Subtask analyses and expert agreement measurements provide useful supporting evidence, although some broader claims about intrinsic clinical perception exceed what the current evaluation isolates.

Critical strengths
  1. A useful decomposition of lesion-related visual skills. The benchmark separates disease presence, contour assessment, local revision decisions, and attribute extraction instead of treating diagnostic accuracy as sufficient evidence of visual understanding. The combination of end-to-end and Oracle-Passed evaluation, together with subtask and response-bias analyses, helps identify distinct failure modes. This makes the benchmark potentially valuable for diagnosing weaknesses that conventional classification evaluations overlook.

  2. Substantial construction and validation effort. The paper documents how candidate masks are curated, perturbed, and converted into questions, with six medical experts participating in review and correction. Appendix F.4 reports cross-review of 1,207 masks and additional agreement checks on attribute and contour-revision judgments. These are meaningful quality-control measures. The comparisons between medical models and their own foundation backbones also strengthen the empirical analysis beyond comparisons between unrelated model generations.

Critical weaknesses
  1. The connection between the operational tasks and “expert-level clinical perception” is not fully established. Stage 4 supplies the optimal lesion mask, and its answers are largely determined by geometric quantities: occupied zones, region overlap, area fractions, and bilateral area ratios (§3.1.4; Appendix C.4). This tests interpretation of an existing annotation, but does not establish how much performance depends on perceiving lesion evidence in the underlying radiograph. Similarly, the common one-third and two-thirds area thresholds operationalize geometric burden, without demonstrating that they represent clinical severity across the different findings. These tasks are useful, but controls retaining the lesion mask and lung outlines while removing radiographic texture are needed to characterize what they measure. This distinction directly affects the interpretation of the reported semantic-perception failures and conclusions about medical adaptation.

  2. The headline performance degradation mixes perceptual errors with answer-format requirements and scoring structure. Stages 3 and 4 require all three or four subanswers to be correct, respectively, whereas earlier stages are binary decisions. End-to-end scoring additionally forces all subsequent stages to be incorrect after any earlier error. The paper helpfully reports subtask scores, but the strongest models achieve approximately 70–89% on several individual later-stage tasks, making the blanket characterization of deeper perception as collapsing too broad. There is also a concrete formatting confound: Table 11 includes textual responses such as “Answer: No” in the “Etc” category, which accounts for approximately one-third of InternVL’s Stage 2 outputs. Such responses can be semantically valid despite violating the requested numeric format. Finally, E2E-to-OP gains combine removal of the forced-error scoring rule with changes to conversation history; they do not independently establish sensitivity to actual upstream error propagation. These issues matter because the paper attributes the observed failures primarily to intrinsic perception.

Questions for authors
  1. How does Stage 4 performance change when radiographic texture is removed while retaining the lesion mask and the anatomical outlines needed for the questions? What evidence supports interpreting the shared area thresholds as clinical severity rather than geometric extent?

  2. How much do scores and model rankings change after normalizing unambiguous textual answers such as “Yes” and “No” into the corresponding options? Please distinguish format failures from incorrect decisions. To isolate actual error propagation, how does per-turn accuracy with uncorrected conversation history—but without forcing later answers to zero—compare with Oracle-Passed accuracy?

  3. Figure 12 includes the instruction to ignore lesions overlapping the heart in a cardiomegaly example, although Appendix C states that this restriction applies only to non-cardiomegaly cases. Is this an illustration error, or was the instruction present in the actual cardiomegaly evaluation? If the latter, how does correcting it affect the results?

  4. Were the three baseline physicians independent of benchmark construction, and were human–model comparisons evaluated on the same sampled cases? Given the small human evaluation set, confidence intervals and the physicians’ relevant expertise would help establish how reliably this baseline calibrates the claimed expert-level gap.

  5. Given the paper’s broad conclusions about the perceptual limitations of current VLMs, could the authors extend the evaluation beyond GPT-5.4 to include newer frontier models such as GPT-6 Astra and Claude Opus 5.5? Evaluating them under the same E2E and Oracle-Passed protocols, with clearly reported inference settings, would help establish whether the observed contour-revision and attribute-extraction limitations persist. Please also clarify the model-selection cutoff date and any availability constraints that prevented their inclusion.

5 comments

If we have misunderstood the intent of your feedback or questions in any way, please let us know. We are more than happy to engage in further discussion promptly.

W: Weakness, Q: Question

[W1, Q1] The connection between the operational tasks and expert-level clinical perception

Annotation dependence and image perception in Stage 4

CheXpercept’s evaluation is fundamentally divided into two phases: visually perceiving the presence and contour of a lesion (Stages 1–3), and extracting attributes based on the perceived lesion (Stage 4). Therefore, it is by our intended design that a model interprets a given annotation (lesion mask) in Stage 4. In the default End-to-End setting, the model cannot even reach Stage 4 without successfully recognizing the lesion evidence in the radiograph during Stages 1–3. According to medical experts, this design directly mirrors the cognitive process of them, who first perceive a lesion’s region and then judge its location, distribution, and extent based on the spatial relationship between that implicitly annotated lesion and the lung area.

Furthermore, evaluating a given annotation in Stage 4 does not mean the task can be solved without perceiving the original image. In Stage 4, the model is provided only with the lesion mask; it must directly perceive the lung area and anatomical zones from the raw image to judge attributes (excluding comparison). Notably, the control setting you suggested would require explicitly providing lung contours, which underscores that this anatomical information is otherwise expected to be perceived directly from the original image. In short, Stage 4 evaluates the ability to concurrently perceive the lesion mask and the anatomical structures in the image and understand their spatial relationship.

Evaluation of severity

For severity, we evaluated lesion severity through the model’s ability to perceive the geometric extent of a lesion, for the following reasons.

First, from a clinical perspective, geometric extent is a primary determinant when radiologists assess lesion severity. For instance, an existing clinical study on edema [1] has directly quantified severity based on the geometric extent of the lesion. We therefore considered the ability to judge geometric extent as reflecting the perceptual basis of severity assessment.

Second, quantifying severity criteria for each lesion type is inherently challenging because standards vary among specialists. Furthermore, even if strict quantitative criteria were established, the evaluation itself would become impractical, as it would require distinguishing excessively minute differences. For instance, a small pleural effusion transitions to a more severe category within a very narrow range, so a question such as “Does the effusion occupy more than 1/5 but less than 2/5 of the lower lung?” would be difficult to answer reliably even for board-certified radiologists.

For these reasons, we evaluated the severity of all lesions over a broader and uniform range (using 1/3, 2/3). However, we acknowledge that directly linking this capability to “classifying severity” might overstate what the task actually measures. Therefore, we will revise the manuscript to describe this task in terms of lesion size and geometric extent rather than severity.

Requested experiment

We understood your suggestion for a texture-removal control experiment to stem from a concern that Stage 4 might solely evaluate the ability to interpret annotations without image perception. However, as explained above, the explicit purpose of Stage 4 is to evaluate the model's ability to interpret the given lesion mask in conjunction with the anatomical structures within the original image, mirroring post-localization clinical reasoning.

Because this directly aligns with our evaluation objective, we did not perform the texture-removal experiment for this rebuttal. Nevertheless, if you still believe that observing the results under a condition where only the lesion mask and lung contours are provided is necessary for interpreting Stage 4, please let us know. Additionally, if you have any specific "lesion evidence" in mind, we would appreciate it if you could clarify. We would be more than happy to conduct this experiment and share the results during the discussion period.

Reference

[1] Warren, Melissa A., et al. "Severity scoring of lung oedema on the chest radiograph is associated with clinical outcomes in ARDS." Thorax 73.9 (2018): 840-846.

[W2, Q2] Errors in evaluation

Interpretation of performance

We agree with your observation that some strong models achieved 70–89% accuracy on individual sub-tasks in the later stages. Consequently, we agree that our statement claiming "deeper levels of perception collapse" is an overstatement. We will tone down this expression in the revised manuscript to more accurately reflect the models' partial successes.

Formatting confounders (performance error due to strict scoring)

We acknowledge that our initial strict number-matching evaluation penalized InternVL, which uniquely generated text responses (e.g., "Answer: No") rather than the option numbers alone. We have rescored the results for InternVL, the only model affected, by normalizing these semantically clear text responses to their corresponding multiple-choice options. In our revision, we will report the performance with the impact of this strict scoring format removed. The updated results are presented below:

MetricBeforeAfterDelta
S1 E2E89.589.5+0.0
S2 E2E37.147.9+10.8
S3 E2E0.00.0+0.0
S4 E2E6.89.1+2.3
S2 OP37.149.3+12.2
S3 OP0.10.1+0.0
S4 OP16.816.8+0.0
Depth RR1.041.04+0.00
Depth RF1.852.12+0.27
Depth LF0.750.75+0.00

Clarification on the E2E evaluation setting

We would like to clarify the mechanics of the End-to-End (E2E) evaluation. In the E2E setting, the moment a model provides an incorrect answer at a specific stage, the conversation is immediately terminated, and all subsequent stages are automatically scored as incorrect. In other words, subsequent stages never proceed based on a conversation history containing an incorrect answer.

In contrast, the Oracle-Passed (OP) setting replaces any incorrect answer with the ground truth before proceeding to the next stage. The E2E score is equivalent to taking the OP results and simply marking everything after the first incorrect answer as wrong. Therefore, the difference between the E2E and OP settings reflects purely a difference in scoring rules, not a mix of history changes and scoring rules. However, we realize that our use of the word "propagation" in the manuscript could be misunderstood as implying that a history of wrong answers influences subsequent stages. We will revise this wording to prevent any confusion.

Requested experiment

We understood your suggested experiment to be based on the premise that the E2E setting involves histories with incorrect answers. Since such histories do not exist in our actual E2E setup, we did not perform this specific experiment for the rebuttal. However, if you are specifically interested in observing the effect of a history containing incorrect answers on subsequent stages, please let us know. We would be happy to conduct this experiment and share the results during the discussion period.

[Q3] Illustration error in figure 12

We appreciate your careful attention to detail. This was an error made during the illustration process; the actual prompt used for evaluating cardiomegaly cases did not include that specific instruction (“ignore lesions overlapping the heart”). Therefore, our reported results remain completely unaffected. In the revised manuscript, we will correct Figure 12 to accurately reflect the exact prompt used during the actual evaluation.

[Q4] Human annotation

The physicians who participated in the human baseline evaluation are a subset of the experts involved in the benchmark construction process. Due to practical resource constraints in recruiting additional experts, assembling a completely independent group of evaluators was difficult. However, to mitigate any potential bias from prior exposure, we carefully assigned the evaluation cases so that no physician evaluated cases they had personally annotated or reviewed during the construction phase.

The human evaluation was conducted on a sampled subset of the full dataset used for the models. Detailed information about the participating physicians is provided in Appendix F.5; all possess experience in reading chest X-rays. Our original human evaluation was conducted by three radiation oncologists with 6, 4, and 8 years of clinical experience. To increase the reliability of our baseline, we expanded this pool to five evaluators by adding two more physicians with 11 and 8 years of experience (each evaluating 24 questions). The updated results, including 95% confidence intervals, are presented alongside the frontier model evaluation in the consolidated table under [Q5] below.

[Q5] Evaluation of frontier models

We appreciate the suggestion. The original model evaluation for this paper was conducted in May 2026, meaning that GPT-6 Astra and Claude Opus 5.5 were not yet released. Following the reviewer's request, we have now evaluated Claude Opus 5.5. This evaluation was conducted with a temperature of 1.0, taking approximately 1 day and costing around $230. However, we could not include GPT-6 Astra due to time, budget, and resource constraints, as its evaluation is estimated to take over 4 days. The consolidated results comparing the human baseline and Claude Opus 5.5 are presented in the tables below:

EvaluatorS1 E2ES2 E2ES3 E2ES4 E2ES2 OPS3 OPS4 OPRR (0-4)RF (0-3)LF (0-1)
Human98.3 [95.8, 100.0]93.8 [87.5, 98.8]67.5 [52.5, 80.1]45.7 [35.3, 56.6]96.2 (+2.4) [92.5, 100.0]72.5 (+5.0) [60.0, 85.0]54.3 (+8.6) [44.2, 65.2]3.05 [2.75, 3.33]2.50 [2.27, 2.70]1.00 [1.00, 1.00]
Opus 5.588.648.62.127.652.0 (+3.4)16.0 (+13.9)59.5 (+31.9)1.082.380.79
EvaluatorS3 Exp.S3 Con.S3 Res.S4 Dist.S4 Loc.S4 Sev.S4 Comp.
Human85.0 [75.0, 95.0]85.0 [75.0, 95.0]95.0 [87.5, 100.0]94.3 [88.6, 98.6]74.3 [64.2, 83.8]94.3 [88.9, 98.6]85.7 [78.6, 92.5]
Opus 5.531.851.299.586.675.488.996.8

Notably, while the human baseline generally outperforms the model across most E2E metrics, Claude Opus 5.5 demonstrated superior performance in specific sub-attributes under the OP setting. For instance, Opus 5.5 outperformed the human baseline in S3 Revision Result Selection (99.5 vs. 95.0), S4 Location (75.4 vs. 74.3), and S4 Comparison (96.8 vs. 85.7). These findings suggest that while frontier models still lag behind radiologists in sequentially delineating lesion contours, they exhibit remarkable geometric perception of the lesions themselves once the lesion masks are provided. We will incorporate a discussion of this partial success into the revised manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

[Mid-Discussion Reminder] Request for feedback on our responses

Dear Area Chair and Reviewers,

With the discussion period halfway over, we are writing to politely remind you that our responses to all reviews have been posted.

We have carefully addressed every weakness and question, and we hope your concerns have been fully resolved. If there are still any unresolved issues, we kindly ask that you let us know as early as possible. Given the strict time constraints of the remaining discussion period, the sooner we receive your feedback, the better equipped we will be to conduct any additional experiments you might request.

We are ready to engage in further discussion immediately to clear up any remaining questions. Thank you again for your time and effort in evaluating our work.

Best regards,

Authors

AuthorsOpenReview ↗

[Second Reminder] Update on discussions and request for feedback

Dear Area Chair and Reviewers,

As we draw closer to the end of the discussion period on November 18, we would like to provide a brief update and send another gentle reminder regarding our rebuttals.

We are very encouraged to share that we have had a productive exchange with one of the reviewers (es4r), who confirmed that our response successfully resolved their main concerns and subsequently raised their score. We deeply appreciate this active engagement.

We would be incredibly grateful if the remaining reviewers could also take a moment to review our responses. At this stage, we believe we have fully addressed all the concerns raised by the other two reviewers as well. However, if our rebuttals have not fully resolved your concerns, or if you need any further clarification, please let us know. We are on standby and ready to provide additional explanations to clear up any remaining questions before the deadline.

Thank you once again for your continued time and effort.

Best regards,

Authors

AuthorsOpenReview ↗

[Final Reminder] Entering the final phase of the discussion period (Nov 18 AoE)

Dear Area Chair and Reviewers,

As we approach November 18 (AoE), which marks the conclusion of the discussion period, we are writing to send our final gentle reminder.

We would like to reiterate that we firmly believe our rebuttals have resolved all the concerns and points raised in your reviews.

However, if you feel that any of our responses have not fully addressed your concerns, or if there are any lingering issues, we highly encourage you to participate in the discussion and let us know. We are closely monitoring the system and are fully prepared to provide immediate explanations before the phase officially closes.

Thank you once again for your dedication and valuable time in evaluating our work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.