acceptodds
Withdrawn from ICLR 2027

Revealing Invisible Defects in Visual Representations: A Psychometric Instrument for Binding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Task accuracy can conceal differences in how visual representations respond to changes in attribute–object pairings. We introduce BInD, the Binding Intervention Difference, a diagnostic protocol for frozen visual encoders. BInD compares cosine-distance responses to pairing-preserving and pairing-breaking image edits, without human annotation or a trained probe at scoring time. It measures readout-specific sensitivity, not the total binding information recoverable by an arbitrary decoder. Synthetic and real-photo suites cover color, load, arrangement, and a size control at pooled and patch interfaces. Across 32 encoders, we report reliability, axis-specific associations, and comparisons with eleven intrinsic metrics under training-objective and capability controls. Within-recipe interventions change several metrics substantially without consistently improving binding readouts. Our strongest application is compression: two DINOv2 merge configurations have nearly identical ImageNet accuracy but retain 86% versus 34% of the reference color score. Pruning can preserve that score while degrading segmentation. Distillation and vision-language case studies further show why the interface and evaluation task matter. We provide suite specifications, a scoring pipeline, and an interpretation guide with explicit limits on causal and real-image claims.

Primary area:
datasets and benchmarks
Keywords:
binding, dataset, phychometric

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_55bDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes BInD, a diagnostic for measuring how sensitive frozen vision encoders are to feature binding. For a given image, BInD compares two cosine distances: the distance between the original embedding and the embedding of a pairing-breaking edit, and the distance between the original embedding and the embedding of a pairing-preserving edit. A larger value indicates that the representation is more sensitive to changes in attribute–object pairing. The metric is computed at a global interface (pooled embeddings) and a dense interface (patch-wise embeddings). The authors construct synthetic and real-photo suites covering color, load, arrangement, and a size control, and evaluate 32 encoders. They compare BInD against eleven intrinsic metrics and apply it to token compression, distillation, and VLM case studies. The central result is that two DINOv2 token-merging configurations with nearly identical ImageNet kNN accuracy retain 86% versus 34% of the reference color score.

Critical strengths

S1. The motivation is compelling. Downstream task accuracy (e.g., ImageNet kNN) cannot distinguish encoders that differ in binding sensitivity. The proposed preserved/broken contrast is a sensible way to probe this property directly from a frozen encoder.

S2. The evaluation is broad, covering 32 encoders across six training-objective groups.

Critical weaknesses

W1. Clarity and presentation. Many terms and procedures are used before they are defined, or are never defined, which made the paper very hard to follow. Examples:

  • L111–112: The sentence on matching pixel distances appears without explaining what matching pixel distances means or how it is done. This is only clarified later in Sec. 3.3 and App. E.
  • L146: "A master seed fixes generation." The generation process itself is not described: which factors are sampled, and from what distributions.
  • Sec. 3.3, undefined terms: The following paragraph refers to "the four curation rules" (hygiene, balance, deduplication, stratification) and to "control examinees," but neither term was introduced; "hygiene" in particular must be inferred. As a result, I could not determine from the paper how the benchmark is constructed. Please name and define each rule once, together with its concrete criterion, and report how many items each rule removes.
  • Sec. 3.4 (e.g., L177, "Read the four axes separately (Figure 2)"): The section is written as instructions to the reader, but relies on concepts that have not been explained at that point. For example, it does not say why the margin is "capability-sensitive," or what "reference-normalized accuracy" is.
  • Sec. 3.5: Experimental results are reported without describing the experiments. "Synthetic–photo rank transfer" is not defined: which correlation, computed over what, with which controls. The "hybrid stimuli" are described in only a single clause.
  • Sec. 4.3, 4.4, and 5.3: These sections draw conclusions from results that appear only in the appendix (e.g., Tables 9, 10, and 13; App. E and O). If these results support claims in the main text, they should be presented there.

W2. Whether the cosine readout measures binding. BInD relies entirely on cosine distances, and it remains unclear whether this readout faithfully reflects binding sensitivity. As the authors acknowledge, a low BInD score does not imply that binding information is absent; the information may simply be inaccessible to cosine geometry. A natural check would be:

  • train a lightweight decoder (e.g., a linear probe) to predict the attribute–object pairing from the embeddings, and
  • compare its accuracy with BInD across encoders.

W3. Alternative explanation for the color drop in Table 2. The color axis uses near-hue colors. Token merging averages patch features, so the drop could reflect degraded discrimination between the two similar hues rather than a loss of color–shape binding. The one-object hue-discrimination control (Sec. 3.3) is reported only for uncompressed encoders, and the size control does not address hue fidelity. Could the authors re-evaluate the two ToMe configurations with the distinct-color variant and with the one-object hue control?

W4. Asymmetric interpretation of external evidence. (i) MMVP (Sec. 5.5, App. N). The MMVP Color category is unrelated to the BInD color axis (−.07) but strongly associated with the size axis (+.83). The paper reads this as showing that "a benchmark category and a diagnostic construct need not coincide." An equally plausible reading is that the synthetic color axis does not capture color-related discrimination in natural photographs. This reading is also consistent with the non-significant synthetic-to-photo rank transfer for color (r = +.30, Sec. 3.5). The paper should discuss this alternative explicitly, rather than locating the mismatch on the benchmark side. (ii) Dense–depth association (Sec. 4.1, 5.3, App. J). Sec. 4.1 cites the dense color margin's association with NYUv2 depth (+.64) as the one piece of evidence that does not share the intervention principle. However:

  • the paper offers no rationale for why color–shape binding sensitivity should predict monocular depth;
  • the same color margin read at the pooled interface correlates with depth at −.63, on the same encoders, stimuli, and controls;
  • the dense margin is unrelated to ADE20K mIoU (−.03).
Questions for authors

Q1. Have the authors evaluated the two DINOv2 configurations in Table 2 (ToMe without proportional attention vs. official ToMe) on the real-photo color suite? Does the discrepancy persist on real images? This matters because color is the axis whose synthetic ranking does not transfer to photographs (r = +.30).

Reviewer_ms1YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces BInD (Binding Intervention Difference), a training-free method for evaluating how visual encoders represent four axes: color binding across two or three objects, spatial arrangement, and relative object size. The authors construct a synthetic evaluation suite and conduct several experiments to assess the reliability and effectiveness of BInD. They then demonstrate its diagnostic utility in settings such as pruning, token merging, and knowledge distillation, showing that similar downstream performance can conceal substantial differences in binding sensitivity.

Critical strengths
  1. BInD is a simple and efficient method that requires no additional training and can be applied directly to frozen visual encoders. The authors also provide an extensive evaluation across 32 encoders with different architectures and training objectives.

  2. I found the diagnostics section particularly interesting, especially the analyses of how techniques such as pruning and knowledge distillation affect compositional capabilities.

Critical weaknesses
  1. The authors mention in lines 50-55 that BInD is complementary to existing compositional benchmarks, but I found the distinction between the two approaches unclear. Both BInD and existing compositional benchmarks use structured examples to evaluate specific aspects of compositional understanding. It seems that existing benchmarks targeting attribute-object binding or spatial arrangement could have been used to conduct many of the paper's analyses and diagnostic experiments. I would have liked to see a clearer explanation of what BInD offers beyond these benchmarks and why a new evaluation suite is necessary.

  2. From what I gather, the authors benchmark is relatively limited in diversity, relying on simple synthetic shapes and a narrow range of compositional relationships involving only two or three objects. While this controlled setting is useful, it is unclear whether the findings generalize to more complex and diverse visual scenes.

  3. The paper is difficult to follow and would benefit from a clearer, more cohesive narrative. Many sections read as a collection of results and observations rather than a connected story, making it difficult to identify the main takeaways.

Questions for authors
  1. What does BInD offer beyond existing compositional benchmarks that evaluate specific capabilities, such as attribute-object binding and spatial arrangement?

  2. Could the authors provide more details about the benchmark's diversity? Additionally, have they examined how BInD performance correlates with performance on existing, more diverse compositional benchmarks, such as those involving other attributes, more complex relationships, or natural images?

Reviewer_kTgkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a simple but interesting idea for measuring feature bindings, in which pair-preserving and pair-breaking images are generated for the original image, and the difference in cosine similarity to the original image is measured. A large difference potentially means a certain attribute is associated with a certain object. This work offers a set of test images for evaluating feature bindings.

Critical strengths

None

Critical weaknesses

Although the approach is simple and interesting, I don’t know what BInD can provide compared to other methods like (Huang et al., 2026). BInD only considers sensitivity to feature binding, and I’m not sure if a higher value of BInD is sufficient to say sensitivity to feature binding. This can be partly because I don’t fully understand the paper.

Also, I guess feature binding is domain-specific; if one can say a model is sensitive to feature binding in a certain domain, this is not applicable to other domains. BInD only provides synthetic test suites (realistic images are also offered but I’m not sure if it’s validity is confirmed).

Due to these points, the paper offers limited insights into feature binding, and I have to rate the paper negatively.

Questions for authors

A high value of BInD is necessary and sufficient condition for sensitivity to feature binding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.