- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a benchmark that evaluates whether a model can generate a radiology report based on the patient's prior EHR history. The authors compare two report-generation approaches, an end-to-end model and a structured-state pipeline. To account for omissions in the reference radiology reports, which can unfairly penalize otherwise correct outputs, they report omission-tolerant performance alongside standard evaluation metrics. Under standard metrics, the structured pipelines plateau at a micro-F1 of about 0.50, slightly below end-to-end LLMs (about 0.55). Under the omission-tolerant metric, however, all systems improve by 0.10–0.12 and perform comparably. The authors argue that the evaluation convention, not only the model, determines the conclusion.
Critical strengths
None
Critical weaknesses
Weakness 1: Clarity and organization
The paper is hard to follow, and in my view, it is difficult to see how its components connect because the paper switches between topics too frequently. (1) Beyond the evaluation, the contributions extend to loosely related components such as auto-research, which are mixed throughout the abstract, introduction, and related work. (2) Many components lack detail. For example, although the paper mentions some of the inputs in passing, it does not provide a complete specification of the benchmark, such as the full list of modalities and EHR history components used (e.g., whether prior images are included). It is also unclear what the 73 union concepts in the structured pipeline refer to. (3) Em dashes are overused (about 100 times). Overall, the paper reads as a mix of loosely connected parts, which makes its potentially valuable core finding difficult to evaluate.
Weakness 2: Limited contribution of the evaluation itself
In my view, the main contribution of this paper is the finding that evaluation should account for errors inherent in the reference radiology reports. However, this finding is relatively straightforward, and there are already prior attempts to fill in missing information by referencing other reports for evaluation. As a result, the novelty of the proposed evaluation is limited. In addition, the task itself requires generating a radiology report without seeing the actual image (Lines 198-201), which raises concerns about its validity. Since the reference reports are written by radiologists based on findings observed in the images, many findings cannot be inferred from the EHR history alone. For example, if the reference report describes lung cancer, a history of coughing alone is not enough for a model to infer that finding. Therefore, errors caused by the absence of images may have a larger effect on the results than the report omissions the paper focuses on.
Questions for authors
- What components make up the EHR history, and from which MIMIC source (e.g., module or table) was each component extracted?
- What is the specific structure of the structured pipeline? What do the 73 union concepts refer to?
- What prompts are used for each model, including the system prompt and the input prompt? How is the EHR history formatted and presented to the model within the prompt?
We sincerely thank all reviewers and the AC for their careful and constructive feedback. The reviews identified important issues—including the validation of the omission-tolerant reading, patient-clustered statistics and the clarity of the benchmark specification—that we believe require substantial revision and additional experiments beyond the scope of this discussion period. We have therefore decided to withdraw the submission and will incorporate these suggestions into a substantially revised version.