- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes hierarchical semantic contextualization for vision–language generation. Learnable anchors extract shared context, and progressively queried local, structural, and semantic memories enrich the retained visual sequence. An optional variational residual follows this contextualization. Experiments on whole-slide pathology report generation and independently trained COCO captioning models compare the method with matched baselines, component controls, and alternative residual placement.
Critical strengths
- The contextualization gain is supported by targeted comparisons: WSI BLEU-4 rises from 0.158 to 0.175 with HSC, while parameter-matched flat contextualization reaches 0.159 and removing text alignment gives 0.160 (Tables 2–3). Together, these controls make a stronger case for the proposed organization and supervision than a comparison with the unmodified backbone alone.
- The evaluation distinguishes the larger contextualization gain from the smaller residual contribution, reports variability across three seeds, and measures memory and inference overhead. Improvements in independently trained COCO captioning models provide further evidence of applicability beyond pathology reporting, although the gains there are smaller.
Critical weaknesses
- The alignment objective is important but not defined precisely enough to reproduce. Equation 7 writes it as A(Sv, St), and the expanded method description still does not specify the agreement function or how visual and text sequences are aggregated. Removing alignment substantially reduces the reported HSC gain, making this a consequential reporting gap.
- The reported residual improvement does not isolate the benefit of KL regularization from adding the residual architecture. The paper correctly distinguishes a zero-KL residual from removing VIB, yet the coefficient sweep contains only positive weights and the no-VIB baseline (Table B7). The sweep shows sensitivity to regularization, but the missing zero-KL result leaves a KL-specific attribution unresolved. This limits the secondary residual interpretation rather than the main HSC result.
Questions for authors
- What is the exact definition of A(Sv, St), including similarity normalization, aggregation over unequal sequence lengths, and the final loss reduction? This is necessary to reproduce a component with a substantial measured effect.
- If evaluated, what result does the residual branch obtain with its KL weight set to zero under the same selection protocol? This would distinguish regularization from the residual architecture. Without that control, the evidence supports the combined branch and regularizer rather than a KL-specific gain.
5 comments
2) The β = 0 control distinguishes the contribution of the variational residual from that of KL regularization.
We sincerely thank the reviewer for this helpful suggestion. We agree that testing only positive values of β alongside a No VIB baseline does not isolate the contribution of KL regularization. We have therefore added a β = 0 control that retains the variational posterior, sampling process, and residual connection while removing only the KL penalty.
Table 1. Contribution of the variational residual and KL regularization on the WSI test set.
| Configuration | BLEU-1 | BLEU-4 | METEOR | ROUGE-L |
|---|---|---|---|---|
| A1: No VIB | 0.405853 | 0.175186 | 0.173861 | 0.337849 |
| A2: Residual VIB, β = 0 | 0.406153 | 0.175215 | 0.174474 | 0.338475 |
| A3: Ours (Residual VIB, β = 0.003) | 0.409563 | 0.178153 | 0.178307 | 0.338756 |
| A2 − A1: Residual effect | +0.000300 | +0.000029 | +0.000613 | +0.000626 |
| A3 − A2: KL contribution | +0.003410 | +0.002938 | +0.003833 | +0.000281 |
Key finding. The additional improvement is primarily associated with KL regularization in this experiment. The residual architecture alone increases BLEU-4 by 0.000029 compared with No VIB. Adding KL regularization at β = 0.003 provides a further 0.002938 improvement. BLEU-1, METEOR, and ROUGE-L also increase. These results were obtained with one random seed. We will include the β = 0 control and this analysis in the revised manuscript.
Dear Reviewer jXWu
Sincere gratitude for your comment. We believe that we have addressed all the comments raised by you. Item by item responses to the your comments are listed above this response. We are looking forward to your feedback.
Best wishes!
Authors.
1) The alignment objective uses normalized anchor similarities, independent maximum pooling, and cross entropy supervision.
We sincerely thank the reviewer for pointing out that Eq. (7) and Appendix A.1 did not fully define the alignment loss. We agree that the exact computation is important for reproducing our method.
During training, visual tokens, text embeddings, and semantic anchor responses are projected into a shared feature space and normalized using L2 normalization. We compute cosine similarities for visual features and anchors and for text features and anchors. Independent maximum pooling produces the following scores:
\qquad m^t_{bk}=\max_t\frac{\langle\hat T_{bt},\hat A_{bk}\rangle}{\tau}.$$ The alignment loss is: $$\mathcal L_{\mathrm{align}}=\frac12\left[\mathrm{CE}(m^v,y)+\mathrm{CE}(m^t,y)\right], \qquad y_b=b.$$ Here, $\tau$ denotes the temperature. Both $m^v$ and $m^t$ have shape $B\times K$, and $y_b=b$ defines the target anchor index. Each cross entropy term is averaged over the batch before the two terms are averaged. **Separate pooling over visual and text tokens handles unequal sequence lengths without requiring direct token correspondence.** The loss provides auxiliary supervision during training, while inference does not require reference text. We will correct Eq. (7) and Appendix A.1 to describe the implemented cross entropy objective rather than the previously stated symmetric KL formulation.