acceptodds
Under review as a conference paper at ICLR 2027

From What to Which: Decoding Modifier Grounding in Frozen MLLMs

Abstract

As Multimodal Large Language Models (MLLMs) can describe increasingly complex visual scenes, token-level grounding becomes crucial. Yet, when an MLLM generates "the yellow banana on the left", established grounding approaches focus on _what_ is in the image ("banana"), overlooking tokens that help describe _which_ instance is meant ("yellow", "left"). In this work, we ask whether frozen MLLM representations contain decodable grounding information about the referred instance across generated tokens, extending to modifiers such as attributes, spatial expressions, and relational/action terms. To address this question, we introduce OTTER, a lightweight supervised probe over frozen MLLM representations that uses Optimal Transport (OT) to align generated tokens with visual regions and produce compact grounding maps. Our results show that (i) instance-discriminative visual information can be decoded from modifier tokens, with the clearest evidence for spatial terms, but (ii) is not confined to them, as contextualized object nouns also carry referential information; (iii) the recovered grounding remains informative under context perturbations, while selected regions remain relevant to generation; and (iv) the learned OT-based grounding extends beyond the controlled setting to free generation and cross-dataset transfer.

Primary area:
interpretability and explainable AI
Keywords:
Multimodal Large Language Models, Model Interpretability, Representation Probing, Visual Grounding, Multimodal Representation Learning, Token-Level Grounding, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gJzvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces OTTER (Optimal Transport for Token-level Expression-to-Region alignment) to address the what-to-which problem: grounding generated language to a specific object instance beyond its category. It extracts contextual token states and visual states from a frozen MLLM and trains a lightweight probe using referring-instance segmentation supervision. The probe combines semantic, attention, and spatial costs to compute an entropy-regularized transport plan, whose transported mass is used to construct grounding maps. The reported results suggest distributed instance-specific grounding across token states and transfer to free-generation and cross-dataset settings.

Critical strengths
  1. An interesting instance-grounding question. The paper examines whether generated modifiers and nouns encode information distinguishing specific object instances beyond category identity.

  2. Multiple evaluation settings. The study includes counterfactual referent pairs, context perturbations, free generation, and cross-dataset transfer.

Critical weaknesses

The core method is insufficiently specified and contains inconsistencies. Figure 1 depicts visual patch states as vision-encoder outputs, but Section 4.2 refers to intermediate MLLM-layer features without clarifying whether the visual states are encoder outputs or visual-token hidden states within the LLM. Section 4.3 also contains an unresolved "Appendix ??" reference for refinement ablations, but no corresponding experiments appear in the appendix. Section 4.3 states that both textual and visual masses are computed using Eq. (4), yet that equation defines only textual scoring s(h_i), with no visual scoring function or patch-mass formula provided. Moreover, A_ij in Eq. (7) is not defined, and its attention layer, head aggregation, and correspondence to generated tokens and visual patches are not explained. These unresolved definitions and missing details prevent readers from reconstructing the complete method or reliably assessing the reported results.

Questions for authors

The authors need to resolve the inconsistencies above and provide a complete, reproducible specification of OTTER, including feature and attention extraction, visual scoring, transport marginals, and implementation details.

1 comment

We thank the reviewer for the careful reading and helpful feedback. We agree that Section 4 is incomplete. Below we give the full specification as implemented; and we will fix the paper, especially Sections 4.2 and 4.3, accordingly.

Visual states. The patch states v_j are the hidden states of the image tokens inside the frozen language model, taken at the feature layer: layer 27 for Qwen2.5-VL, as in Appendix A. They are not vision-encoder outputs. The "vision encoder" block in Figure 1 was meant as an overview of where the patches originate, but it is misleading, and we will redraw the figure.

Attention AijA_ij (Eqs. 7 and 8). AA is the self-attention of the same decoder block, averaged over heads. Row ii is the attention from text token ii to the image tokens. Each row is clipped at zero, normalized to sum to one, smoothed with a Gaussian on the patch grid (sigma 0.8) and normalized again. When a word spans several sub-tokens, their states and attention rows are aggregated.

Cost (Eqs. 5 to 9). Token and patch states are projected by a LayerNorm and a linear layer to 128 dimensions, then L2-normalized, giving z_i and z_j. The cost is

Cij=ws(1−<zi,zj>)+wagiaCijatt+wpgipCijspa,C_ij = w_s (1 - <z_i, z_j>) + w_a g_i^a C_ij^{att} + w_p g_i^p C_ij^{spa},

where:

  • Catt=−logC^{att} = -log of the normalized attention, divided by its mean over patches;
  • CspaC^{spa} is the squared distance between patch j and the attention centroid of token i, divided by its maximum;
  • the weights w are positive (softplus) and learned;
  • the gates gig_i are per-token sigmoids of a linear layer on hih_i.

Marginals. The source and target marginals concatenate the token masses and the visual masses with one dustbin entry each. Token masses come from the scoring module of Eq. 4. Visual masses, instead, are uniform (1/N for each of the N patches). We will correct this, and apologize for the oversight. Each dustbin has mass 1. The costs of the token and patch dustbins are two learned scalars, and the cost between the two dustbins is zero.

Transport and maps (Eqs. 10 and 11). We solve the entropic OT problem (epsilon 0.05) with 80 log-domain Sinkhorn iterations. The map is OjO_j, encoding the share of patch jj's capacity filled by text rather than by the dustbin. During training, one joint problem is solved over the expression's tokens, and the loss of Eq. 13 is applied to its map. Token-level maps solve one problem per word.

Refinement ("Appendix ??"). We apologize with the reviewer as the reference points to a table we had commented out and overlooked. It compares four inference-time operators, applied after training without retraining or recalibrating thresholds:

  • the raw thresholded map;
  • a morphological preset (two closing iterations, hole filling, and removal of components below 0.05% of the image);
  • a grid CRF;
  • the CRF followed by the same morphology.
mIoU gain over the raw mapMorph.CRFCRF + Morph.
OTTER adaptive gated + learned weights+0.74+0.36+0.51
OTTER semantic only+0.62-1.76-1.64
OTTER adaptive gated+0.60-3.27-3.18
OTTER full + learned weights+0.58-0.63-0.51
OTTER semantic + attention+0.79-0.95-0.81
OTTER full + fixed dustbins+0.74-1.08-0.97
OTTER semantic + spatial+0.58-1.20-1.10
OTTER full+0.74-1.11-0.99
Softmax no OT+0.76+0.15+0.27
Softmax semantic only+0.74-0.39-0.27
Softmax full+0.74-0.45-0.33

Each value is the change in RefCOCO val mIoU with respect to the unrefined map, at the threshold calibrated on held-out training images.

  • Morphology adds 0.6 to 0.8 mIoU to every head.
  • CRF helps only the main head and the softmax baseline, and hurts all other heads.
  • The mIoU values reported in the paper (e.g. 45.83 in Table 10) use the raw map. Morphology is an optional operating point when a binary mask is needed; the unrefined transport map remains the grounding signal.
Reviewer_gYC2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether frozen MLLM token representations encode instance-specific grounding beyond object identity. It introduces OTTER, a supervised optimal-transport probe trained on referring-expression masks. Counterfactual referent pairs, context perturbations, and free-generation experiments suggest that referential information is distributed across contextualized nouns and modifiers, with strong evidence for spatial expressions.

Critical strengths
  1. The paper studies a meaningful interpretability question and uses same-image, same-category counterfactual pairs to examine instance discrimination. The finding that contextualized nouns also carry referential information provides a useful qualification of the noun–modifier distinction.
  2. Evaluation includes deletion tests, training-scale ablations, and cross-backbone screening. The authors appropriately acknowledge the distinction between supervised decodability and causal explanation, and report weaker results on non-Qwen backbones.
Critical weaknesses
  1. Insufficient probe controls. Raw-cosine and shuffled-projection baselines do not separate learned grounding from information in contextual token states. Equally trained visual-only and non-contextual text probes are missing.
  2. Confounded OT comparison. Only OT receives dustbin abstention. The claim that softmax cannot abstain is incorrect.
  3. Weaker instance discrimination. Table 11 shows that OT reduces target–distractor gaps for spatial and attribute tokens, despite improving mask overlap.
  4. Limited additional faithfulness. Table 4 shows nearly identical deletion effects for OTTER and the semantic branch, leaving OT’s added explanatory value unclear.
  5. Ambiguous token-level formulation. Equation 11 aggregates transport across tokens, whereas Algorithm 1 appears to solve OT separately per token. This ambiguity obscures how token-specific maps are constructed.
  6. Limited cross-backbone support. Under Appendix A’s screening protocol, average token grounding drops to 11.88 on InternVL3.5 and 10.74 on Gemma, limiting conclusions beyond Qwen models.
  7. Mixed free-generation results. On COCO, raw OTTER achieves F1-IoU of 0.320 versus TAM’s 0.409. Improvements depend on semantic residual expansion, weakening the case for OT alone.
  8. Insufficient evidence for anticipatory grounding. The claim that nouns identify instances before modifiers appear lacks a dedicated quantitative table, sample count, and uncertainty estimates.
  9. Unclear transfer checkpoint selection. RefCOCOg uses a different checkpoint from the primary experiments without a clear selection criterion, making the transfer result harder to interpret.
Questions for authors
  1. How does OTTER compare with equally trained probes using visual features alone, or non-contextual token embeddings paired with the same visual features?
  2. Does OT retain its advantage over a softmax baseline with a learned null category, matched cost weights, and identical supervision?
  3. Is transport solved jointly over the full expression or independently for each token? Equation 11 aggregates across tokens, whereas Algorithm 1 appears to solve OT separately per token; please clarify the marginals and map extraction.
  4. How should the lower target–distractor gaps under OT be reconciled with the instance-discrimination objective? Which conclusions require OT beyond the learned semantic branch?
  5. Are transfer evaluations explicitly image-disjoint from probe training, and how was the different checkpoint used for RefCOCOg selected?
2 comments

We thank the reviewer for the careful feedback.

First, we apologize and clarify that the submitted results used block 35, not layer 27 as stated (best in Table 8). At layer 27, RefCOCO val calibrated mIoU is 52.5 ± 0.1 over 3 seeds (45.8 reported); we have fixed the following tables accordingly.

W1. To separate learned grounding from feature content, we trained controls identical to OTTER except for the text input (semantic cost only). L/R pref. is the share of the 217 left/right pairs of Table 3 where the left/right word's map favours the referred instance (chance 0.5).

ProbeText inputmIoUL/R pref.
B1constant query24.70.500
B2input embeddings32.80.552
B3 (OTTER)layer-27 states52.30.961

The large gap between B3 and B2 (+0.409 ± 0.034, positive in every seed) shows that context is crucial for spatial tokens. A second control re-runs Table 3 with token states from a grey or unrelated image (the map still uses the real one): left/right barely changes (0.96 to 0.92 and 0.94), while attributes drop from 0.95 to 0.67 and 0.69.

So spatial information hardly depends on the image content; attribute information largely does.

W2/W3/W4. Our softmax could already abstain, so we separated transport from abstention with probes that share OTTER's cost, supervision and seeds and change only the operator:

OperatormIoUMod. pref.Spatial Soft-IoU
C1 softmax0.5310.9220.363
C2 softmax + null0.5260.9250.355
C3 OTTER0.5250.9220.339

The lower target-distractor gap of OT (Table 11) comes from transport (C3 minus C2: -0.044), not from abstention (C2 minus C1: +0.008).

Recomputed Table 4, phrase deletion AUC: OTTER 0.560, semantic 0.558, attention 0.480, random 0.278. This confirms the reviewer's reading.

We agree that OT itself gives no meaningful advantage: rather than improving interpretability, it is the operative principle to present our claim.

W5. Eq. 11 only defines OjO_j, which leaves OiO_i (Eq. 12, Algorithm 1) ambiguous. In training, OT is solved jointly over the expression's tokens, and Eq. 11 gives the map used in the loss. Token-level maps OiO_i are used at inference, where OT is solved separately for each word, as in Algorithm 1. We will make this explicit and rename EE as OO (Eqs. 3, 12).

W6. First, we applied each probe zero-shot to the other backbones (150 expressions of Table 9, Soft-IoU averaged over token types):

Trained on / tested onQwen2.5-VLQwen3-VLInternVL3.5Gemma-3
Qwen2.5-VL-3B36.810.79.89.9
Qwen3-VL-2B8.832.625.68.8
InternVL3.5-2B7.718.732.710.2
Gemma-3-4B8.710.88.722.2
TAM10.410.910.79.3

Off the diagonal, a head performs at or below TAM. Only Qwen3-VL and InternVL3.5 (25.6, 18.7) transfer.

Second, the diagonal (each probe on its own backbone, Table 9 layers) is two to three times above TAM. It required fixing an embedding dimensional error, only in Gemma and InternVL; submitted Table 9: 37.30, 32.66, 11.88, 10.74. Overall: the instance information is present in all four MLLMs, but the probe that reads it must be trained per backbone.

W7. We recomputed Table 6 (3 seeds):

MethodCOCO F1 (Obj)OpenPSG F1 (Obj)
OTTER, submitted0.320 (0.193)0.343 (0.210)
OTTER, layer 270.475 (0.337)0.523 (0.384)
TAM0.408 (0.277)0.385 (0.255)

Plain OTTER beats TAM in every seed without the residual, and matches it on function words.

W8. Given only the category, Qwen described one of two same-category instances; on 199 descriptions with the noun before left/right, we checked whether the noun map prefers the named instance when it is not salient:

Target preference (3 seeds)ItemsValue
Noun, all items1990.663 ± 0.013
Noun, target not the more central960.632 ± 0.016
Noun, target not the larger720.560 ± 0.035
Modifier, target not the larger720.866 ± 0.056

Nouns anticipate the instance above chance, also when it is not the most central one; when it is not the larger one, their preference is no longer distinguishable from chance, while the modifier stays high. We will therefore state: nouns carry a partial, salience-driven anticipation, and the instance-specific signal is concentrated at the modifier.

W9. The submitted Table 7 used ot_full, the simplest full-cost variant (fixed weights, no per-token gates), to minimize additional architectural contributions, unlike the adaptive-gated head used elsewhere. With the main head, phrase IoU is 0.373 ± 0.017 both on all of RefCOCOg val and on the images unseen in training, vs. 0.355 for ot_full and 0.186 for TAM.

Thanks the authors for their detailed response. However, they changed a large amount of results in the original paper. The novelty and necessity of their purposed method is also limited. Meanwhile, the reproduction of the results is still an issue.

Overall, I keep my score.

Reviewer_5UA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies "what-to-which" problem in visual grounding which goes beyond identifying the objects but also identifying the specific instance based on the modifiers. It introduces OTTER, a lightweight optimal-transport-based probe on frozen VLM representations to learn the correspondence between tokens and patches. Trained with RefCOCO instance masks, the paper presents an extensive analysis on VLM grounding behavior such as phrase-level vs. noun-only grounding, and concludes that information is distributed across modifiers and other contextualized tokens.

Critical strengths
  1. The research question is valuable, moving focus from noun-only attribution to contextualized instance identification, which addresses a blind spot of previous VLM attribution methods.
  2. The evaluation protocol is systematic and self-critical. Each step states what the previous result does not establish and designs and experiment to address it.
Critical weaknesses
  1. The measurements do not answer the question the paper poses. The probe is supervised only at the phrase level, yet the analyses focus on token maps separately. The paper's scores attribute much weight to non-semantic tokens, which shows modifier-specific grounding cannot be reliably separated, departing from its own motivation.
  2. The novelty of the method is limited. Previous work such as F-LMM has explored training a lightweight supervised head to perform grounding/segmentation based on frozen VLMs. The paper also conducts no comparison with such previous methods.
Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the careful reading and constructive comments.

W1. We agree that the original analysis may be insufficient to rule out an "every token points at the referent" effect. However, two details matter, and we will emphasize them in the paper:

  • Training excludes a stopword list (left, right, top, bottom, front, back, side and most function words) from the OT sources, so the maps of these words are zero-shot.
  • Tables 2, 3, 5a and 7 read a teacher-forced copy of the expression, placed after it had already appeared in the prompt, so every token had already seen the modifier.

New setting. All new results use a probe retrained at layer 27, the layer stated in the paper. Each word is now read where the expression first appears in the prompt, so it sees only the image and the preceding words.

We use the 2,971 pairs of Table 3. Each pair has two same-category instances in one image, with expressions differing by a modifier (only left/right in 217 pairs). Each expression's instance is the target, and the other is the distractor. Target preference is the fraction of maps whose mean value is higher on the target, so a rule that ignores the text scores 0.5. Gap is the mean map value on the target minus that on the distractor. We report mean ± sd over seeds, with seed-0 bootstrap intervals over images in brackets.

(a) Function-word baseline, a new column for Table 3. Function words do not specify the instance, so they show how much referent information is spread over all tokens. Each expression gets one map from its modifiers, one from its nouns and one from its function words (955 pairs with function words on both sides):

Joint mapExpressionsTarget pref.Gap
All modifiers1,9100.906 ± 0.0050.480 ± 0.026
Spatial modifiers1,6270.903 ± 0.0040.463 ± 0.021
Attributes7090.879 ± 0.0040.447 ± 0.031
Nouns1,9100.786 ± 0.0030.330 ± 0.010
Function words1,9100.744 ± 0.0020.268 ± 0.010

Compared with the function words of the same expressions, target preference is higher by 0.171 [0.145, 0.205] for spatial modifiers, 0.063 [0.014, 0.127] for attributes, and only 0.042 [0.023, 0.068] for nouns. On left/right pairs, modifiers reach 0.974 and function words 0.621. This shows that every token carries some referent information, but modifiers carry far more than the baseline set by function words.

(b) Position analysis on the Table 3 pairs. Each word is mapped on its own and grouped by its position relative to the first modifier that differs between the two expressions:

Target pref.All pairsLeft/right pairs
Before the modifier0.649 ± 0.0060.611 ± 0.007
At the modifier0.879 ± 0.0050.959 ± 0.012
After the modifier0.866 ± 0.0030.934 ± 0.003
  • Words with identical preceding text on both sides score 0.50, as expected.
  • Target preference rises sharply at the modifier (+0.230 ± 0.010) and stays high afterwards. Nouns go from 0.654 before the modifier to 0.885 after it.
  • Earlier information comes from salience and one-sided words (man/woman).

This shows that the instance information appears at the modifier and is then passed on to later words. This is why function words looked informative in Table 3.

(c) Source of the information.

  • In the blank-image condition of Table 3, token states are computed with a grey image, while the map uses the real one. Left/right keeps 0.923 (0.959 with the real image), while attributes drop from 0.946 to 0.674.
  • A probe trained like OTTER, but on the non-contextual input embeddings of the same words, reaches 0.552 on left/right against 0.961 for OTTER (difference 0.409 ± 0.034).

This shows that spatial information does not come from the image content, nor from the word alone: the model computes it from the word in context. Attribute information, in contrast, needs the image.

(d) Table 5a, recomputed with symmetric isolation. Every word is now isolated both as the bare word and as "{word} object". Retention relative to nouns (spatial, attribute, function) is 0.879, 0.693, 0.594 for the bare word, and 1.107, 1.141, 0.403 for "{word} object". This shows that function words depend on context more than spatial words in both variants, so the conclusion of Table 5a holds.

We will state the claim as: modifiers carry most, but not all, of the instance-specific information, which then carries over to later tokens.

W2 (F-LMM). We agree that a light supervised module on frozen features is not new in itself. However, F-LMM and our work have different goals, and their results are not directly comparable. F-LMM adds grounding for conversation, grounds only words picked by a keyword selector (so modifiers are not studied) and decodes with a U-Net and a SAM refiner. We use masks only to supervise a 0.55M-parameter probe that maps every token. A segmentation benchmark would mostly measure F-LMM's decoder.

Reviewer_5UA9OpenReview ↗

Response to authors

I thank the authors for the detailed response and the new analyses. The function-word baseline and the position analysis address my main concern: they show that modifiers carry more instance information than other tokens, and the restated claim is more consistent with the paper's motivation.

The response, however, also shows that the token-level analyses in the submitted paper were computed in a setting where every token had already seen the modifier. These analyses are therefore confounded, and the new setting changes some of the central findings. On novelty, the response agrees that a lightweight supervised head on frozen features is not new, so the contribution rests mainly on the analysis.

Overall, the new analyses are informative, but they substantially change the main results of the submitted paper, and the methodological contribution remains limited. I therefore keep my rating at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.