acceptodds
Withdrawn from ICLR 2027

VESP: VISUAL EVIDENCE and semantic consistency preservation IN AUTOREGRESSIVE MRI REPORT GENERATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automated radiology report generation from 3D MRI remains challenging because visual evidence must remain associated with the appropriate clinical findings and influential throughout autoregressive generation. Existing approaches typically introduce visual information during encoding, but may suffer from insufficient field-level evidence organization, semantic inconsistency between visual and textual findings, and diminishing visual influence as generation progresses. We propose **VESP**, a visual evidence preservation framework for structured MRI report generation that maintains image-grounded evidence from visual encoding to final token prediction. First, **Schema-guided Visual Evidence Linking (SVEL)** organizes shared MRI representations into field-specific visual memories and propagates information among related clinical fields using schema-derived relations. Second, **Visual-Text Evidence Alignment (VTEA)** aligns image-side and report-side predictions within a shared structured finding space, encouraging generated findings to remain consistent with visual evidence. Third, **Evidence-Retained Adaptive Decoding (ERAD)** selectively reuses learned field-level visual predictions at schema-defined candidate decisions during inference, preserving visual influence without additional training or learnable parameters. Experiments on a private shoulder MRI dataset and the public KneeCoT knee MRI dataset show consistent improvements, achieving 75.65 micro-F1 on shoulder MRI and 96.69 micro-F1 on knee MRI. These results demonstrate that preserving evidence identity, semantics, and influence throughout autoregressive generation improves the accuracy and image consistency of structured MRI reports.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Report Generation, 3D MRI, Vision-Language Models, Visual Evidence Preservation, Structured Report Generation, Multimodal Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CxMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VESP combines schema-guided field-specific visual retrieval, visual-text finding alignment, and inference-time candidate-token refinement for structured MRI report generation. The reported gains on shoulder MRI and KneeCoT are promising. However, the experiments do not adequately separate the proposed evidence-preservation explanation from simpler effects of auxiliary classification and score fusion. Important direct baselines and implementation details are missing, and a numerical inconsistency affects the results discussion.

Critical strengths
  1. The reporting schema connects visual representations, auxiliary supervision, and generation-time decisions.
  2. The shoulder and knee datasets provide evidence across different anatomy, imaging configurations, and schemas.
  3. The main comparison tables include confidence intervals, and the progressive ablation provides an initial breakdown of gains.
Critical weaknesses
  1. Diagnosis fields differ from earlier Findings fields in task, class distribution, difficulty, and graph connections. The section analysis therefore does not isolate position or prefix length. Reordering sections, placing the same field at different positions, and testing image removal or replacement would more directly assess the mechanism.
  2. A direct classifier-to-template baseline is missing. The model already predicts field-level visual labels from a predefined option set. Directly rendering these predictions into the schema, or applying simple classifier-language-model score fusion, would test whether the full proposed pathway is needed.
  3. The ablations do not establish that all components are necessary. In Table 3, SVEL changes shoulder micro-F1 from 73.36 to 73.48, only 0.12 points without an uncertainty estimate. Sequential additions cannot isolate each component. Controls for graph propagation, auxiliary classification, KL alignment, and a simplified ERAD design are needed, preferably on both datasets.
  4. Reproducibility details are incomplete. Figure 2 depicts ERAD gates and thresholds that are not fully defined in Section 3.4. The complete training objective, loss weights, extraction point of H_f, volumetric input handling, schemas, and annotation validation protocol need clarification. In particular, the paper should state whether H_f has observed the target field value during training.
  5. Results reporting requires correction. The page 7 text reports shoulder Med3DVLM F1 and recall as 35.58 and 24.48, while Table 1 gives 57.22 and 56.46. The associated explanation depends on the former values. KneeCoT BLEU-4 values near 0.99 also warrant a precise account of template normalization and metric computation, together with rare-condition and default-option baselines.
Questions for authors

See weakness.

Reviewer_R7BuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VESP, a visual evidence preservation framework for structured autoregressive MRI report generation. The framework consists of three components: Schema-guided Visual Evidence Linking (SVEL), which organizes shared visual representations into field-specific visual memories; Visual-Text Evidence Alignment (VTEA), which aligns image-side and report-side predictions within a shared structured finding space; and Evidence-Retained Adaptive Decoding (ERAD), which reuses visual predictions to refine candidate token logits at schema-defined decision points during inference.The method is evaluated on a private shoulder MRI dataset and the public KneeCoT knee MRI dataset. VESP achieves 75.65 micro-F1 on shoulder MRI and 96.69 micro-F1 on knee MRI, outperforming the strongest reported baselines by 2.13 and 2.28 percentage points, respectively. Ablation studies suggest that VTEA and ERAD contribute more substantially to the overall improvement than SVEL. The paper further reports that the improvements are particularly pronounced in later diagnosis sections of generated reports.

Critical strengths

1.Important and practical research problem. The paper addresses the challenge of maintaining visual evidence throughout autoregressive medical report generation, which is relevant to the reliability of medical vision-language systems. 2.Coherent framework design. SVEL, VTEA, and ERAD target different stages of the generation process and form a relatively clear pipeline from visual evidence organization to decoding-time refinement. 3.Evaluation across two MRI domains. Experiments on shoulder MRI and KneeCoT provide evidence that the method can be applied to different anatomical structures and reporting schemas.

Critical weaknesses

1.Insufficient evidence for field-specific visual evidence organization. SVEL uses learnable field queries to retrieve representations from shared visual tokens, but the paper does not demonstrate that the resulting field-level memories contain distinguishable and clinically relevant visual evidence. The current results mainly show a small performance improvement from adding SVEL, rather than directly validating the proposed evidence-linking mechanism. 2.Visual-text agreement is not equivalent to image-grounded clinical consistency. VTEA uses structured supervision for both visual and textual branches and encourages their predictions to agree. However, agreement between two branches trained with the same labels does not necessarily mean that the generated findings are supported by the actual MRI evidence. The paper lacks an independent evaluation of image-report consistency. 3.The mechanism behind ERAD's improvement is insufficiently isolated. ERAD achieves a relatively large gain in the ablation study, but it is unclear whether this improvement specifically results from preserving visual evidence or simply from reranking a restricted set of candidate tokens using an auxiliary classifier. More controlled comparisons are needed to support the claimed mechanism.

Questions for authors

1.Could the authors clarify how SVEL ensures that the retrieved visual memories contain field-specific evidence rather than general image features? It would be helpful to provide additional visualizations or experiments to demonstrate the effectiveness of field-conditioned retrieval. 2.Could the authors explain how VTEA distinguishes actual image-text consistency from agreement induced by shared supervision? Since both branches are trained using the same structured annotations, their agreement may not necessarily indicate that the generated findings are grounded in the MRI images. 3.Could the authors provide further evaluation of the clinical correctness of generated reports? In particular, radiologist assessments and additional analysis of annotation agreement would help validate the reliability of the generated findings.

Reviewer_Vc1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript proposes the framework of VESP, which uses the structured field keywords as the special token to learn structural clinical knowledge as the visual evidence, Visual and textual finding predictions are supervised within a shared label space, with an additional consistency objective encouraging agreement between the two branches. During the inference stage, the evidence-retained adaptive decoding uses the visual classifier to adjust candidate tokens. Experiments on private shoulder MRI and public KneeCoT data report improvements in structured micro-F1 over six comparison methods.

Critical strengths

1.The ERAD module supports the potential value of resuing the visual classifier during the report generation. ERAD applies bounded adjustments at schema defined decisions without introducing additional learnable parameters or a separate training stage.

2.The proposed framework was validated across two different MRI settings, Knee and Shoulder, although these experiments do not establish transfer to unseen institutions or schemas, they provide evidence that the combined method is applicable beyond a single reporting task on a specific anatomy.

Critical weaknesses

The methodological novelty is limited and insufficiently demonstated. SVEL combines learned-query visual retrieval with relation-aware aggregation over a predefined clinical graph. VTEA combines supervised finding classification with a conventional KL-divergence consistency objective. ERAD introduces a task-specific decoding rule that applies bounded visual-classifier biases to candidate token logits. Although the schema-specific implementation differs from previous approaches, the manuscript does not clearly present a methodological advance beyond their adaptation and integration. Furthermore, In experimental results, no complete example fully shows how a schema field is annotated, associated with visual evidence, aligned with the corresponding text representation, and used to guide decoding. This makes it difficult to assess how the proposed components works.

Questions for authors

1.The schemas both in private shoulder testset and the KneeCoT dataset are unclear, it is highly recommended to provide the complete schemas and specific example tracing one field from reference-report annotation through visual retrieval, text encoding and decoding. This is helpful to figure out the field-level memories aggregated into section-level evidence tokens and the corresponding text representation identified during generation. 2.The complete ERAD decision rule is unclear in this manuscript. Fig.2 shows visual-confidence, margin, disagreement, and language-uncertainty gates that are absent from Section 3.4. It is recommended to define these quantities and explain which text or token distribution they use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.