acceptodds
Under review as a conference paper at ICLR 2027

Contextualize to Generate: Hierarchical Semantic Organization for Vision–Language Generation

Abstract

Image captioning translates visual content into natural language and has advanced considerably with visual representation learning and language modeling. However, different generation tasks may rely on different visual evidence. While conventional captions often focus on salient objects and their relationships, specialized reports can require detailed observations distributed across multiple regions. Simply aggregating these observations into a global representation may therefore weaken the local evidence required for generation. To better organize such evidence, here we study how local visual observations and shared context can be used together while preserving the original visual sequence, and propose Hierarchical Semantic Contextualization (HSC). Specifically, HSC first uses learnable semantic anchors to extract semantic information relevant to the task from visual features under paired text guidance. Then, HSC integrates the extracted context with the original visual sequence and leverages Hierarchical Semantic Memory to progressively organize local, structural, and semantic information. In the full model, a variational residual with KL regularization further refines the contextualized representation while preserving its direct path to the generator. As a result, local observations remain available together with information collected across the input. We carefully evaluate the proposed model on whole-slide image (WSI) report generation and MS COCO captioning, covering both specialized report generation and general image captioning. The results show that the proposed model consistently improves the matched baselines, with the full configuration providing additional gains overall. Component and placement analyses further examine the contributions of the contextualization design and residual refinement. These results support the effectiveness of the proposed framework across different generation settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Captioning, Medical Report Generation, Multimodal Learning, Vision–Language Generation, Information Bottleneck

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jXWuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes hierarchical semantic contextualization for vision–language generation. Learnable anchors extract shared context, and progressively queried local, structural, and semantic memories enrich the retained visual sequence. An optional variational residual follows this contextualization. Experiments on whole-slide pathology report generation and independently trained COCO captioning models compare the method with matched baselines, component controls, and alternative residual placement.

Critical strengths
  1. The contextualization gain is supported by targeted comparisons: WSI BLEU-4 rises from 0.158 to 0.175 with HSC, while parameter-matched flat contextualization reaches 0.159 and removing text alignment gives 0.160 (Tables 2–3). Together, these controls make a stronger case for the proposed organization and supervision than a comparison with the unmodified backbone alone.
  2. The evaluation distinguishes the larger contextualization gain from the smaller residual contribution, reports variability across three seeds, and measures memory and inference overhead. Improvements in independently trained COCO captioning models provide further evidence of applicability beyond pathology reporting, although the gains there are smaller.
Critical weaknesses
  1. The alignment objective is important but not defined precisely enough to reproduce. Equation 7 writes it as A(Sv, St), and the expanded method description still does not specify the agreement function or how visual and text sequences are aggregated. Removing alignment substantially reduces the reported HSC gain, making this a consequential reporting gap.
  2. The reported residual improvement does not isolate the benefit of KL regularization from adding the residual architecture. The paper correctly distinguishes a zero-KL residual from removing VIB, yet the coefficient sweep contains only positive weights and the no-VIB baseline (Table B7). The sweep shows sensitivity to regularization, but the missing zero-KL result leaves a KL-specific attribution unresolved. This limits the secondary residual interpretation rather than the main HSC result.
Questions for authors
  1. What is the exact definition of A(Sv, St), including similarity normalization, aggregation over unequal sequence lengths, and the final loss reduction? This is necessary to reproduce a component with a substantial measured effect.
  2. If evaluated, what result does the residual branch obtain with its KL weight set to zero under the same selection protocol? This would distinguish regularization from the residual architecture. Without that control, the evidence supports the combined branch and regularizer rather than a KL-specific gain.
5 comments
AuthorsOpenReview ↗

1) The alignment objective uses normalized anchor similarities, independent maximum pooling, and cross entropy supervision.

We sincerely thank the reviewer for pointing out that Eq. (7) and Appendix A.1 did not fully define the alignment loss. We agree that the exact computation is important for reproducing our method.

During training, visual tokens, text embeddings, and semantic anchor responses are projected into a shared feature space and normalized using L2 normalization. We compute cosine similarities for visual features and anchors and for text features and anchors. Independent maximum pooling produces the following scores:

\qquad m^t_{bk}=\max_t\frac{\langle\hat T_{bt},\hat A_{bk}\rangle}{\tau}.$$ The alignment loss is: $$\mathcal L_{\mathrm{align}}=\frac12\left[\mathrm{CE}(m^v,y)+\mathrm{CE}(m^t,y)\right], \qquad y_b=b.$$ Here, $\tau$ denotes the temperature. Both $m^v$ and $m^t$ have shape $B\times K$, and $y_b=b$ defines the target anchor index. Each cross entropy term is averaged over the batch before the two terms are averaged. **Separate pooling over visual and text tokens handles unequal sequence lengths without requiring direct token correspondence.** The loss provides auxiliary supervision during training, while inference does not require reference text. We will correct Eq. (7) and Appendix A.1 to describe the implemented cross entropy objective rather than the previously stated symmetric KL formulation.
AuthorsOpenReview ↗

2) The β = 0 control distinguishes the contribution of the variational residual from that of KL regularization.

We sincerely thank the reviewer for this helpful suggestion. We agree that testing only positive values of β alongside a No VIB baseline does not isolate the contribution of KL regularization. We have therefore added a β = 0 control that retains the variational posterior, sampling process, and residual connection while removing only the KL penalty.

Table 1. Contribution of the variational residual and KL regularization on the WSI test set.

ConfigurationBLEU-1BLEU-4METEORROUGE-L
A1: No VIB0.4058530.1751860.1738610.337849
A2: Residual VIB, β = 00.4061530.1752150.1744740.338475
A3: Ours (Residual VIB, β = 0.003)0.4095630.1781530.1783070.338756
A2 − A1: Residual effect+0.000300+0.000029+0.000613+0.000626
A3 − A2: KL contribution+0.003410+0.002938+0.003833+0.000281

Key finding. The additional improvement is primarily associated with KL regularization in this experiment. The residual architecture alone increases BLEU-4 by 0.000029 compared with No VIB. Adding KL regularization at β = 0.003 provides a further 0.002938 improvement. BLEU-1, METEOR, and ROUGE-L also increase. These results were obtained with one random seed. We will include the β = 0 control and this analysis in the revised manuscript.

Dear Reviewer jXWu

Sincere gratitude for your comment. We believe that we have addressed all the comments raised by you. Item by item responses to the your comments are listed above this response. We are looking forward to your feedback.

Best wishes!

Authors.

Reviewer_sx1yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HSC, which adds semantic anchors and hierarchical memory before text generation. It also includes an optional variational residual. Experiments cover pathology report generation and COCO captioning.

Critical strengths

The method improves over HistGen under the same training setup. The paper includes component ablations and results averaged over three seeds.

Critical weaknesses

The experiments use outdated comparisons and do not evaluate medical accuracy. The generated reports are not examined in enough detail to show what the method actually improves.

  1. The paper does not compare with recent methods. The newest baseline is HistGen from 2024. PathGenIC, which evaluates on the same benchmark, is missing. The COCO experiment also compares only HistGen variants.

  2. Medical accuracy is not evaluated. The WSI results use only BLEU, METEOR, and ROUGE. The paper does not check whether diagnoses, tumor grades, and other findings are correct. No clinical metric or expert assessment is reported.

  3. Full generated reports are not shown. Figure 2 only contains truncated snippets. There is no analysis of what HistGen and the proposed model get right or wrong. The paper needs full report comparisons with incorrect statements and missing findings marked.

  4. The alignment loss in Equation 7 is not defined. Appendix A.1 does not give the formula either. Removing this loss drops BLEU-4 from 0.175 to 0.160, so its definition is an important missing detail.

  5. Table B7 omits the β = 0 result mentioned in the text. This comparison is needed to check whether KL regularization helps beyond adding the residual branch.

Questions for authors
  1. What are the results against PathGenIC and on medical accuracy metrics?

  2. Can the authors provide full generated reports alongside the references and HistGen outputs, with errors marked?

  3. What is the exact alignment loss, and what is the result with β = 0?

9 comments
AuthorsOpenReview ↗

1) Ours achieves the highest scores across all six reported metrics in the comparison with HistGen and the released PathGenIC checkpoints.

We sincerely thank the reviewer for this insightful comment and for bringing Pan et al. (2026) to our attention. We agree that the distinction between HSC and existing hierarchical methods should be made clearer. We will include Pan et al.'s method in the revised manuscript as an additional comparison to better position our contribution relative to recent work.

To address this concern, we additionally evaluated the released PathGenIC-base and PathGenIC-NN checkpoints on the same 769 WSI test cases, using common reference reports and the same local evaluation pipeline. We report these results alongside HistGen and our submitted HSC results.

Table 1. Comparison with HistGen and released PathGenIC checkpoints on the WSI benchmark.

Model and result sourceBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-L
HistGen0.3750.2610.1990.1580.1630.314
PathGenIC-NN0.4060.2870.2200.1760.1750.336
PathGenIC-base0.4070.2850.2170.1720.1730.333
Ours0.4080.2890.2220.1780.1760.338

Key finding. Ours achieves the highest scores across all six reported metrics, including a BLEU-4 score of 0.178, compared with 0.176 for PathGenIC-NN and 0.172 for PathGenIC-base.

These additional comparisons strengthen our evaluation against recent pathology report generation methods and provide further evidence of the effectiveness of HSC on the WSI benchmark.

Reference Pan, L., Song, C., Gan, X., Xu, K., & Deng, M. (2026). Hierarchical encoder-decoder for image captioning. Neurocomputing, 660, 131833. https://doi.org/10.1016/j.neucom.2025.131833

AuthorsOpenReview ↗

2)Additional medical entity evaluations demonstrate improved factual consistency and competitive performance against PathGenIC

We sincerely thank the reviewer for highlighting the importance of medical factuality evaluation beyond conventional text similarity metrics. We agree that BLEU, METEOR, and ROUGE alone cannot adequately assess the correctness of diagnoses, tumor grades, and other clinically important findings.

To address this concern, we conducted additional medical entity agreement evaluations on all 769 WSI test cases. Following the entity matching approach used in factENT, we evaluated generated reports using a pathology lexicon and a publicly available BioBERT disease entity extractor. We also extended the comparison to the released PathGenIC-base and PathGenIC-NN checkpoints using a common evaluation protocol.

Table 2. Medical entity agreement between generated and reference reports on the WSI test set.

MetricHistGenOursDifference
Pathology lexicon, macro exact-entity F10.51660.5170+0.0004
Pathology lexicon, micro entity F10.55610.5628+0.0067
BioBERT NCBI-Disease, exact-entity F10.16860.1701+0.0015

Table 3. Entity agreement comparison with PathGenIC on the same 769 WSI test cases.

ModelRadiology entity F1Clinical i2b2 entity F1
HistGen0.3834450.131968
PathGenIC-base0.3733550.126510
PathGenIC-NN0.3754430.133350
Ours0.3981650.139983

Key finding. Ours achieves higher medical entity F1 scores than HistGen across all three evaluations in Table 2 and outperforms the evaluated PathGenIC checkpoints under both entity extractors in Table 3. In particular, Ours improves radiology entity F1 over PathGenIC-NN by 0.022722, with a paired 95% bootstrap confidence interval of [0.007503, 0.037778].

The PathGenIC comparison uses its publicly released base and nearest-neighbor checkpoints rather than the complete guideline and feedback system. Its published BioBERT factENT scores are based on a different evaluation procedure and are therefore not directly compared with our results.

To further examine clinical correctness, Appendix C presents detailed case studies comparing diagnoses, tumor grades, and key pathological findings. These analyses explicitly identify correct predictions, incorrect statements, and missing clinical information in the generated reports.

AuthorsOpenReview ↗

3)Complete report comparisons and clinical case studies have been added to Appendix C to assess diagnostic correctness and identify remaining errors

We sincerely thank the reviewer for emphasizing the importance of presenting complete generated reports and examining clinically significant findings. We agree that qualitative comparisons are valuable for understanding the diagnostic accuracy and limitations of generated pathology reports.

To address this concern, we have added two detailed clinical case studies in Appendix C (Figures C1 and C2), including the original WSI images, reference diagnosis excerpts, and complete saved reports generated by HistGen and Ours. We also provide clinical finding tables that explicitly identify correct diagnoses, incorrect statements, and missing pathological information.

Table 4. Clinical comparison of HistGen and Ours on representative WSI cases.

CaseHistGenOurs
TCGA-08-0344Predicts anaplastic astrocytoma, WHO grade IIICorrectly identifies glioblastoma, WHO grade IV
TCGA-3A-A9IHOmits the primary pancreatic malignancyIdentifies pancreatic ductal adenocarcinoma, moderate differentiation, and perineural invasion

Key finding. Ours recovers clinically important diagnoses and pathological findings that are incorrectly predicted or omitted by HistGen. In the first case, our method correctly identifies both the tumor diagnosis and WHO grade. In the second case, it captures the primary malignancy and key pathological characteristics, including differentiation and perineural invasion.

Appendix C also documents incorrect anatomical sites and omitted findings, providing a more detailed assessment of both the improvements and remaining errors in generated reports.

Reviewer_LxAmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to enrich visual features for generation without replacing the original sequence of local visual tokens. It proposes Hierarchical Semantic Contextualization, combining text-guided semantic anchors with a hierarchy of memory reads, followed by a variational residual refinement. The method is evaluated on WSI report generation and MS COCO captioning with controlled ablations. Experimental results show the proposed method is effective.

Critical strengths
  • The controlled WSI evaluation shows the proposed representation from the generation backbone. HSC improves BLEU-4 from 0.158 to 0.175 and the full model reaches 0.178. This is the evidence that the reported WSI improvement is attributed to the representation module.

  • The experiemtns are conducted on two generative datasets, WSI and COCO.

Critical weaknesses
  • This paper does not explain how the proosed HSC differs from recent similar approaches that organize global, regional, and local evidence.

  • The paper compares WSI methods through HistGen and evaluates COCO only through a HistGen-based matched backbone. Related works are missing. Table 1 and 2 has identical baselines and could be merged into one table.

  • The baselines on VLLM results are missing. I suggest these VLLM results should be provided.

Questions for authors

What precisely is the COCO “HistGen” backbone?

6 comments
AuthorsOpenReview ↗

1)HSC differs from existing hierarchical methods by preserving the original visual token sequence and progressively constructing semantic context

We thank the reviewer for this insightful comment and agree that the distinction between HSC and existing hierarchical or multiscale methods should be made clearer.

Recent studies already model visual evidence at multiple levels. Hu et al. (2025) combine multilevel regional feature selection with knowledge retrieval for WSI report generation. MLLM-HWSI (Alawode et al., 2026) aligns cell, patch, region, and slide features with pathology language, then fuses the multiscale tokens for generation. HierCap (Pan et al., 2026) extracts global, regional, and grid features using separate encoders and integrates them through a hierarchical fusion module in the decoder. These approaches emphasize region selection, explicit spatial scales, or fusion of representations extracted at different levels.

HSC instead focuses on organizing context within a retained visual-token sequence. Text-guided anchors first extract shared context and add it to the original tokens. Each token queries local memory. The pooled local response queries structural memory, and the structural response queries semantic memory. The three responses are combined and returned to the original sequence before generation.

Key finding. HSC differs from approaches based on region selection or multiscale feature fusion by retaining the original visual tokens and enriching them with progressively constructed semantic context. This preserves token level visual evidence while allowing local, structural, and semantic information to contribute to generation.

References

[1] Dingyi Hu, Zhiguo Jiang, Jun Shi, Fengying Xie, Kun Wu, Kunming Tang, Ming Cao, Jianguo Huai, and Yushan Zheng. Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection. Computer Methods and Programs in Biomedicine, 263:108677, 2025. https://doi.org/10.1016/j.cmpb.2025.108677

[2] Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, and Sajid Javed. MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13732–13743, 2026. https://openaccess.thecvf.com/content/CVPR2026/html/Alawode_MLLM-HWSI_A_Multimodal_Large_Language_Model_for_Hierarchical_Whole_Slide_CVPR_2026_paper.html

[3] Lizhi Pan, Chengtian Song, Xiaozheng Gan, Keyu Xu, and Mengqian Deng. Hierarchical encoder-decoder for image captioning. Neurocomputing, 660:131833, 2026. https://doi.org/10.1016/j.neucom.2025.131833

AuthorsOpenReview ↗

2)The WSI comparison tables have been consolidated, and the COCO evaluation has been extended with a standard Transformer baseline.

We sincerely thank the reviewer for this constructive suggestion. Following the reviewer's suggestion, we have merged Tables 1 and 2 in the revised manuscript, removing duplicated baseline entries while retaining comparisons with the six existing WSI methods. To further strengthen the COCO evaluation, we conducted additional controlled experiments using a standard Transformer encoder-decoder as the baseline. We then incorporated HSC into the same architecture to assess its contribution independently of the original HistGen-based design.

Specifically, we evaluated five configurations: H0 represents the standard Transformer baseline; H1 includes the complete HSC framework with Anchor, HDM, and VIB; H2 removes Anchor; H3 removes HDM; and H4 disables the auxiliary alignment objective.

Evidence — MS COCO 2014 Karpathy split. The following table presents the complete H0–H4 controlled experimental results.

Table 1. Controlled ablation results of Ours and its components on the MS COCO 2014 Karpathy test split using a standard Transformer encoder-decoder backbone

Configuration (seed 456789, n=1)BLEU-4METEORROUGE-LCIDErSPICE
H0: Standard Transformer0.28820.24110.51460.88910.1729
H1: Ours (Full Model)0.30580.25010.52680.96230.1816
H2: w/o Anchor0.30320.24910.52410.94920.1792
H3: w/o HDM0.29670.24520.52010.93230.1791
H4: w/o Alignment0.29900.24520.52090.92730.1772

Key finding. HSC improves caption generation within the same standard Transformer architecture. Compared with H0, Ours increases BLEU-4 from 0.2882 to 0.3058 and CIDEr from 0.8891 to 0.9623. The controlled comparison shows that these gains are not restricted to the original HistGen generator.

AuthorsOpenReview ↗

3) Ours achieves higher BLEU-4, ROUGE-L, and CIDEr scores than Qwen3-VL-2B-Instruct on MS COCO.

We sincerely thank the reviewer for suggesting the inclusion of modern VLM baselines. Following this suggestion, we additionally evaluated Qwen3-VL-2B-Instruct on 5,000 images from the MS COCO Karpathy test split, using the same evaluation metrics as our captioning model.

Table 2. Comparison with Qwen3-VL-2B-Instruct on the MS COCO Karpathy test split.

ModelBLEU-4METEORROUGE-LCIDEr
Qwen3-VL-2B-Instruct0.17100.24810.47530.5024
Ours0.30840.25010.52690.9664

Key finding. Ours achieves higher BLEU-4, ROUGE-L, and CIDEr scores than Qwen3-VL-2B-Instruct in the reported MS COCO evaluation. BLEU-4 increases from 0.1710 to 0.3084, while CIDEr increases from 0.5024 to 0.9664. This provides an additional comparison with a modern pretrained VLM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.