acceptodds
Withdrawn from ICLR 2027

Holistic Diagnostic Semantics from Decomposed Imaging for Endoscopic Report Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Endoscopic Report Generation (ERG) aims to automatically generate clinical reports from ear, nose, and throat (ENT) endoscopic images to assist clinical diagnosis. Unlike chest X-ray imaging, where a single image often provides relatively complete visual evidence, ENT endoscopy involves decomposed imaging, where multiple local views jointly support the overall diagnosis. Each view contains only partial anatomical and pathological information, while the clinical report summarizes findings from the entire examination. This discrepancy makes it challenging to derive holistic diagnostic semantics from distributed visual evidence and establish reliable correspondence between visual and textual representations. To address this challenge, we propose Holistic Semantic Learning for Endoscopic Report Generation (HSL-ERG), which learns holistic diagnostic semantics from distributed visual evidence under decomposed imaging. Specifically, the Holistic Semantic Representation Module (HSRM) aggregates distributed local visual evidence according to learned view contributions to construct a holistic examination-level semantic representation. Furthermore, the Reliable Correspondence Alignment Loss (RCAL) constructs adaptive bidirectional contrastive constraints between visual and textual representations to improve their correspondence reliability. Experiments on ENT-Endo, OCASD, and CTRG-Brain demonstrate the effectiveness of HSL-ERG. On ENT-Endo, HSL-ERG outperforms existing methods by 0.010 in BLEU-4 and 0.020 in clinical F1.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Endoscopic Report Generation, Decomposed Imaging, Holistic Diagnostic Semantics, Vision-Language Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lm97OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HSL-ERG for generating examination-level reports from multiple ENT endoscopic images containing partial and complementary evidence. Its two main components are HSRM, which combines frozen CLIP image representations with local visual features and performs hierarchical weighting within and across frames, and RCAL, a bidirectional contrastive loss that downweights potential false negatives based on examination-level visual similarity. Validation covers multi-view ENT-Endo, multi-slice CTRG-Brain, and single-image OCASD, using language-generation and clinical consistency metrics. On ENT-Endo, the method reports improvements of 0.010 in BLEU-4 and 0.020 in clinical F1 over the strongest reported baselines for each metric. Additional experiments include component ablations, input-view and loss-weight sensitivity analyses, report-pairing perturbations, and qualitative examples.

Critical strengths

1: The paper addresses a meaningful examination-level report generation problem. ENT examinations contain multiple local views with partial and complementary evidence, while supervision is provided through a single report. The dataset analysis grounds this challenge in a concrete clinical setting, making the problem formulation a substantive motivation for studying multi-view aggregation.

2: The main benchmark provides encouraging evidence of practical utility. On ENT-Endo, the method improves BLEU-4 from 0.224 to 0.234 and clinical F1 from 0.518 to 0.538 over the strongest reported results for each metric. Component ablations also show gains from both HSRM and RCAL individually. These findings support the usefulness of the combined approach.

Critical weaknesses

1: The central claim of diagnostic semantic enhancement is insufficiently substantiated. The HSRM ablation jointly changes CLIP feature fusion, local weighting, and frame weighting, leaving unclear whether gains arise from the proposed aggregation mechanism or simply from additional pretrained features. Figure 7 shows CLIP similarities rather than learned frame weights, without independently validating their diagnostic relevance. Consequently, the evidence does not establish that HSRM identifies and prioritizes complementary diagnostic views, which is central to the claimed contribution.

2: The contribution of the frozen CLIP semantic prior is not established. The paper motivates CLIP features as high-level semantic context for medical images, but the HSRM ablation jointly changes CLIP feature fusion and attention-based aggregation. It therefore does not establish whether CLIP contributes useful diagnostic information or whether the gains arise from other components. This matters because semantic enhancement is central to the paper’s explanation of its contribution.

Questions for authors

Clarification on CLIP semantics: Is the frozen CLIP representation used in HSRM the CLS token before CLIP’s output projection or the final projected image embedding? What evidence, including visualizations or analyses of anatomical structures or pathological findings, supports its interpretation as a diagnostic semantic prior for ENT endoscopic images?

Clarification on frame importance: Are the CLIP similarities in Figure 7 used to compute the learned frame weights, or are they shown only for illustration? What evidence links the actual aggregation weights to independently identified diagnostic frames and their corresponding generated findings? This would help establish whether HSRM prioritizes clinically important evidence.

Reviewer_EhxCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ENT endoscopic report generation. ENT endoscopy is decomposed imaging: a single view only gives partial information, and multiple local views together support the examination-level report. The authors propose HSL-ERG. HSRM aggregates local visual evidence by learned view contributions to build an examination-level representation. RCAL adds adaptive bidirectional contrastive constraints between visual and text representations. Experiments are on ENT-Endo, OCASD and CTRG-Brain. On ENT-Endo they report +0.010 BLEU-4 and +0.020 clinical F1 over existing methods.

Critical strengths
  1. The problem is real. ENT endoscopy does have partial views and case-level reports, so holistic aggregation and alignment make sense.
  2. The two modules are coherent with the motivation, and the paper includes ablations and qualitative attention maps.
Critical weaknesses
  1. Experimental workload is thin. The main decomposed multi-view evidence is only ENT-Endo (1,106 exams). CTRG-Brain is multi-slice and OCASD is single-image, so they do not directly validate the decomposed multi-view claim. The 16-page paper still feels short on experiments.
  2. Comparisons are insufficient. Baselines are mostly general report generation models. It is unclear how they are adapted to multiple views, and there is no dedicated multi-view or multiple-instance aggregation baseline.
  3. Numbers are inconsistent. Abstract says +0.020 clinical F1, the text says different; OCASD F1 is 0.575 in text but 0.561 in Table 1. Table 1 and Table 3 training settings are unclear. No mean/std or significance, and +0.010 BLEU-4 is a small margin.
  4. RCAL text encoder is not clearly specified (frozen CLIP text encoder or trained?). Clinical metric extraction from free-text reports is not described. No sensitivity analysis for tau_r.
Questions for authors
  1. Please reconcile the numbers in abstract, text and tables, clarify Table 1 vs Table 3 training, and report mean/std over at least 3 seeds.
  2. How are baselines adapted to multiple views? Can you add a multi-view or multiple-instance baseline?
  3. What exactly is the text encoder in RCAL, and is it frozen? Please add sensitivity analysis for tau_r.
  4. How are clinical F1 and other CE metrics extracted from free-text reports, and is the extraction validated?
Reviewer_RwnWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the HSL-ERG method to tackle the challenges under the decomposed imaging paradigm for otorhinolaryngology endoscopy. Multiple local views contain only partial anatomical and pathological information, holistic diagnostic semantics are hard to derive from distributed visual evidence, and the correspondence between visual and textual representations is unreliable. The method leverages HSRM to dynamically aggregate distributed local visual evidence according to the diagnostic contribution of each view and construct an examination-level holistic representation. It further devises RCAL to suppress potential false-negative pairs via adaptive bidirectional contrastive constraints for improved cross-modal correspondence. Experiments on the ENT-Endo, OCASD and CTRG-Brain datasets yield performance gains, and ablation studies validate the synergistic benefits of the two modules, with the model showing strong robustness against varying view budgets and image-text correspondence perturbations.

Critical strengths
  1. Reasonable empirical scope. The evaluation spans three input regimes, namely multi-view ENT-Endo, multi-slice CTRG-Brain, and single-image OCASD, and reports module-level ablations, a view-budget sweep, a loss-weight sweep, a label-perturbation robustness test, and qualitative comparisons. This provides a fuller picture of the method's behavior than a single-dataset evaluation would.

  2. Clinically relevant and under-explored setting. Case-level, multi-region ENT reporting is genuinely different from single-image CXR reporting, and the dataset statistics (6.54 frames/exam, 53.3% multi-region cases, long-tailed findings) document this concretely.

Critical weaknesses
  1. Table 6 region rows sum to 1,094 cases (98.9%), not the stated 1,106 (100.0%). The ablation baseline A0 is also undefined, reproducing the R2Gen row of Table 1 exactly on OCASD yet diverging on ENT-Endo, so it is unclear which model the headline gains are measured against.

  2. CXR report generation can rely on established labelers (CheXbert, RadGraph). Endoscopic report generation has no such standard, so the protocol has to be specified by the authors. It is not. The paper says only that CE is computed over "pathological finding categories annotated in the dataset" and "disease diagnosis categories defined in the dataset", but never specifies how a generated free-text report is mapped to those categories, i.e., how negation, synonyms, laterality, and severity are handled.

  3. The role of CLIP injection in HSRM is unclear. Since the same CLIP vector is added to every local token its contribution to the weighted aggregation reduces to a global term. CLIP can only affect token weighting through the nonlinear scoring MLP. With a linear scorer its effect cancels under softmax normalization. The current ablation does not isolate this mechanism.

  4. The relation to prior work is not made explicit. The paper argues that methods specific to ENT endoscopic report generation remain limited, while related work on multi-scene and nasal endoscopic report generation is not discussed or used as a baseline. Attention-weighted pooling of local and multi-view features is likewise a common design in multi-view and multi-slice report generation, including several methods used as baselines here (Chen et al., 2021, Shi et al., 2023, Zhang et al., 2025). Clarifying how the proposed modules differ from these approaches would help establish the contribution.

[1] Zhihong Chen, et al. Cross-modal memory networks for radiology report generation. ACL-IJCNLP, pp.5904–5914,2021.

[2] Yanzhao Shi, et al. Granularity matters: pathological graph-driven cross-modal alignment for brain CT report generation. EMNLP, pp. 6617–6630, 2023.

[3] Xiaodan Zhang, et al. MEPNet: Medical entity-balanced prompting network for brain CT report generation. AAAI, pp. 25940–25948, 2025

Questions for authors
  1. Please state how the generated free-text report is mapped to the CE categories.

  2. The authors are requested to elaborate on the data issues in Weakness 1.

  3. Clarifying how the proposed modules differ from these approaches would help establish the contribution.

  4. How were the 6.54 frames per examination selected? If clinicians chose them as representative, how would the method handle raw examination video with far more redundant or blurred frames, and how much missing information does Figure 1(d) actually reflect?

  5. Since the token weights sum to one, the CLIP feature acts as a constant offset. Could you show, analytically or by ablating the CLIP injection, the token attention, and the view attention separately, that it actually modulates the weighting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.