- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces OTTER (Optimal Transport for Token-level Expression-to-Region alignment) to address the what-to-which problem: grounding generated language to a specific object instance beyond its category. It extracts contextual token states and visual states from a frozen MLLM and trains a lightweight probe using referring-instance segmentation supervision. The probe combines semantic, attention, and spatial costs to compute an entropy-regularized transport plan, whose transported mass is used to construct grounding maps. The reported results suggest distributed instance-specific grounding across token states and transfer to free-generation and cross-dataset settings.
Critical strengths
-
An interesting instance-grounding question. The paper examines whether generated modifiers and nouns encode information distinguishing specific object instances beyond category identity.
-
Multiple evaluation settings. The study includes counterfactual referent pairs, context perturbations, free generation, and cross-dataset transfer.
Critical weaknesses
The core method is insufficiently specified and contains inconsistencies. Figure 1 depicts visual patch states as vision-encoder outputs, but Section 4.2 refers to intermediate MLLM-layer features without clarifying whether the visual states are encoder outputs or visual-token hidden states within the LLM. Section 4.3 also contains an unresolved "Appendix ??" reference for refinement ablations, but no corresponding experiments appear in the appendix. Section 4.3 states that both textual and visual masses are computed using Eq. (4), yet that equation defines only textual scoring s(h_i), with no visual scoring function or patch-mass formula provided. Moreover, A_ij in Eq. (7) is not defined, and its attention layer, head aggregation, and correspondence to generated tokens and visual patches are not explained. These unresolved definitions and missing details prevent readers from reconstructing the complete method or reliably assessing the reported results.
Questions for authors
The authors need to resolve the inconsistencies above and provide a complete, reproducible specification of OTTER, including feature and attention extraction, visual scoring, transport marginals, and implementation details.
We thank the reviewer for the careful reading and helpful feedback. We agree that Section 4 is incomplete. Below we give the full specification as implemented; and we will fix the paper, especially Sections 4.2 and 4.3, accordingly.
Visual states. The patch states v_j are the hidden states of the image tokens inside the frozen language model, taken at the feature layer: layer 27 for Qwen2.5-VL, as in Appendix A. They are not vision-encoder outputs. The "vision encoder" block in Figure 1 was meant as an overview of where the patches originate, but it is misleading, and we will redraw the figure.
Attention Aij (Eqs. 7 and 8). A is the self-attention of the same decoder block, averaged over heads. Row i is the attention from text token i to the image tokens. Each row is clipped at zero, normalized to sum to one, smoothed with a Gaussian on the patch grid (sigma 0.8) and normalized again. When a word spans several sub-tokens, their states and attention rows are aggregated.
Cost (Eqs. 5 to 9). Token and patch states are projected by a LayerNorm and a linear layer to 128 dimensions, then L2-normalized, giving z_i and z_j. The cost is
Cij=ws(1−<zi,zj>)+wagiaCijatt+wpgipCijspa,
where:
Marginals. The source and target marginals concatenate the token masses and the visual masses with one dustbin entry each. Token masses come from the scoring module of Eq. 4. Visual masses, instead, are uniform (1/N for each of the N patches). We will correct this, and apologize for the oversight. Each dustbin has mass 1. The costs of the token and patch dustbins are two learned scalars, and the cost between the two dustbins is zero.
Transport and maps (Eqs. 10 and 11). We solve the entropic OT problem (epsilon 0.05) with 80 log-domain Sinkhorn iterations. The map is Oj, encoding the share of patch j's capacity filled by text rather than by the dustbin. During training, one joint problem is solved over the expression's tokens, and the loss of Eq. 13 is applied to its map. Token-level maps solve one problem per word.
Refinement ("Appendix ??"). We apologize with the reviewer as the reference points to a table we had commented out and overlooked. It compares four inference-time operators, applied after training without retraining or recalibrating thresholds:
Each value is the change in RefCOCO val mIoU with respect to the unrefined map, at the threshold calibrated on held-out training images.