- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper conducts a study on how much relational information is encoded in the visual foundation models like DINO. Basically, probing is done on the frozen embeddings from these visual foundation models to serve as proxy of how much relational information is encoded in each model. Experimental results demonstrate that visual foundation models can decently encode some information like the relative object size, but still have limitations on some hard-negative samples on human-object interactions and figuring out which entities actively participate in the human-object relation.
Critical strengths
-
The paper studies an interesting question of how much relational information is encoded in visual foundation models, which is relatively underexplored in existing literature.
-
The experiments are comprehensive and nicely presented in figures which are easy for readers to understand the performance difference between the models.
Above are the two major strengths of the paper. Other strengths include that the evaluation and experimental details are listed in appendix which are beneficial for reproducing the results. Also, confidence intervals are reported for better understanding the performance distribution on the samples in the evaluated dataset.
Critical weaknesses
- I think many of the designed experiments are not persuasive to show that visual foundation models have encoded relational information. Here are the points:
-
For example, the linear probing experiments for relative size is designed as probing whether a certain patch belongs to the largest object in the scene. However, this is basically the way to detect salient objects from the images, as salient subjects may have outstanding feature representations compared to background or minor objects. Recent works like [1, 2] also support this. Therefore, this is not something closely related to relational information in the images. The goal of "odd-one-out" seem to be essentially identifying the semantic difference between the objects in the scene.
-
For the later interaction encoding, the accuracy of 55-67% on hard negatives roughly indicates that the visual foundation models do not encode much relational information, because the hard negative samples mostly have one distractor choice (e.g., riding v.s. standing on), which means a random guess between the correct choice and the distractor choice should have accuracy close to 50%.
Therefore, I think that (1) many of the evaluated aspects in the paper are not essentially relational encoding in visual foundation models; (2) Some of the experiments demonstrate that visual foundation models encode weak relational information.
- I find this paper relatively hard to understand. First, the title of the paper is about the limitation of the relational encoding in visual foundation models. However, most parts of the paper (Sections 3-5) are discussing about how decent the visual foundation models encode the relational information (e.g., there generalizability to OOD scenarios), instead of discussing on the limitations. The only clear parts that discuss about limitations are in Section 6 (in Section 5 there may be some obscure clues of limitations, but from the words it is hard to figure out whether the authors want to present a limitation, or just stating some facts), which uses more than a page for discussion and conclusion, which is basically unreadable. The audience are unclear about what is the takeaway and what on earth are the limitations of relational encoding in these visual foundation models, and what are the ways to fix them.
[1] Rawlekar et al. Finding Distributed Object-Centric Properties in Self-Supervised Transformers. CVPR 2026.
[2] Kupyn et al. S3OD: Towards Generalizable Salient Object Detection with Synthetic Data. ICLR 2026.
Questions for authors
-
Why the title of the paper is the limitation of relational encoding for visual foundation models, but the main parts are mostly demonstrating how good relational encoding information is within these models? What is the goal/purpose of this paper, and what takeaway should the audience have after reading this paper? These are not clear as the writing of the paper is confounding.
-
I think relational encoding should be mostly be something like positional relationship (e.g., one object is on the left/right of another object). However, besides the study on the object interaction in Section 5. Most studies are about object size (which is essentially saliency detection), and even semantic information (the "odd-one-out" experiment) which I found little relation with relational information. What are the reasons or motivations for choosing these tasks for studying "relational encoding"?
5 comments
Answer to paper focus and questions
We thank the reviewer for the thorough read of our paper and for sharing the interest in the fundamental question of “how much relational information is encoded in visual foundation models”.
The audience are unclear about what is the takeaway and what on earth are the limitations of relational encoding in these visual foundation models, and what are the ways to fix them.
To address their general concern of unfocusedness, we have now reorganized the main text of introduction and abstract to show a more focused and coherent narrative. In particular we:
-
Rewrote the abstract, introduction and final section (Discussion and Conclusion) around a central takeaway: frozen ViT embeddings allows us to detect whether an interaction occurs in an image, but all trained probes perform poorly in identifying which entity is actively participating and which instead is a bystander. We also explicitly framed the synthetic experiments as validation of our probing procedure.
-
Added controls showing that the limited accessibility is not an artifact of one probe family, of the labels, or of the pipeline: we include a human reference ceiling, a patch level attention probe (BAN), the validation of the obtained SAM 3 masks on V-COCO ground truths and the inpainting artifact control.
-
Shortened and restructured the discussion section around three main paragraphs: discussion, limitations and conclusions.
Regarding the question of "what are the ways to fix them", our study does not provide a positive answer.
Answer to questions
Question 1.
Our work addresses open research questions on the successes and failures of relational encoding in visual foundation models. The purpose of this paper is to be a cautionary tale on the limits of what a foundation visual model can and cannot be expected to do.
The positive results in Sections 3–4 serve a specific purpose: on controlled synthetic tasks, they show that linear and bilinear probes can solve relational tasks, i.e., recover contextual information when it is readily accessible.
This validates the probing method, so that the worse results on human-object interactions in Section 5, the main focus of the paper, can be attributed to the weaker encodings and not to a generic failure of the method.
The takeaway is threefold.
(i) Whether an interaction is present in an image is decodable from the frozen features: the best performing bilinear probes reach 84%, where a human annotator agrees with the labels for 88% of pairs.
(ii) Single entity's tokens only weakly decode which entities take part in an interaction, at best 6 points above a probe trained on position alone.
(iii) Reading the human and object encodings jointly recovers only part of this information: at best 71%, still 16 points below an human annotator, and this does not change with increased probe complexity, such as MLPs or attention-based probes.
Our main message for the vision community is that these dense feature maps, which work well for segmentation and retrieval, should not be expected to generate, through a light read-out, reliable maps of which entities interact with which.
Question 2.
We believe that “relational” has a wider meaning, and it should involve any property that is defined only by the interaction between different entities, and not by the features of a single one.
For example, the goal of the experiment on object size is to probe whether the local patch of an object encodes the information of the relative size of that object with respect to the rest of the scene.
The experiment of the “odd-one-out” asks whether the local patch of an object encodes the information of the identity of an object with respect to the identities of all other object in the scene.
Both of these are fully “relational” in the sense that they are not defined by the object in its isolation, but by its relation to the external context.
Moreover, the scope of Section 4 can be seen as a preparation to the more general questions presented in Section 5 on HOI detection in natural images. The results show that the probing machinery works as intended, and validates the conclusion that the failing of probing in HOI in natural images is due to the accessibility of relational information in encodings of the patches and not on failures of the method itself.
Dear authors,
I sincerely thank you for your substantial effort in answering my questions and updating the manuscript. Your detailed explanation on the takeaway of this paper definitely makes it much clearer for what is the conclusion of this paper. From these major takeaways, I feel that it is not very surprising for me that the features from vision models encode little information about interaction encoding. In other words, this is something that I would expect before reading the paper. Therefore, although the paper provides comprehensive analysis on the findings, which is undoubtedly valuable, they kind of conform to the community's knowledge/feeling, with little surprise and mediocre insight. Regarding the concept of "relational", I still feel the task like "odd-one-out" is more like a semantic task, with weak relationship to a "relational" task.
I do acknowledge the authors' effort in revising the manuscript, to make the takeaways more clear. However, I still feel that my original rating is proper given the paper kind of proving something that is not very surprising with comprehensive experiments.
Answers to critical weaknesses
Weakness 1.
We thank the reviewer for the references. Both works concern the separation of an image into objects and background. In particular Rawlekar et al. show that patch similarities in DINO separate foreground from background, and Kupyn et al. train a segmentation network on top of DINOv3 to predict a salient-object mask. Both share as a limitation the decomposition of the foreground into individual instances.
Our case falls exactly in their limitation: for the relative-size tasks, every probed patch belongs to an object. Positives and negatives are all "foreground", similarly shaped simple objects on a plain background. Furthermore, each object is a positive in some images and a negative in others. A signal that separates objects from background would not separate positives from the negatives within the same image. Therefore, it cannot solve the relative-size task. We, on the other hand, ask the probes to recover which of several objects is the largest, a comparison that the cited works do not address.
The probes does not even rely on the object's own absolute size alone. To show this, we construct an additional 3D dataset in which an object of a given size is equally likely to be the smallest or the largest in the image, and we report (in the Appendix) the average output of a single-patch linear probe as a function of the sizes of both objects. A probe relying only on the probed object's own size would give the same output whatever the size of the other object. Instead, the probe's output for a patch of object i depends also on the size of object j.
Our description of the “odd-one-out” task was unclear. Here, each image contains five objects: four identical copies of one sprite and a single, different sprite. The single-patch linear probe sees a patch of one object only and predicts whether it belongs to the single sprite (the odd one out) or to the repeated one. The experiment and dataset are designed so that each sprite appears about equally often in the odd and in the repeated role, so its identity alone carries no information about the task label. This task is relational since the labels do not depend on the absolute identity of the object, but on its relation with the larger context of the image. Since the probe sees a single patch, this comparison must already be present in the embedding.
Weakness 2. For the later interaction encoding, the accuracy of 55-67% on hard negatives roughly indicates that the visual foundation models do not encode much relational information, because the hard negative samples mostly have one distractor choice (e.g., riding v.s. standing on), which means a random guess between the correct choice and the distractor choice should have accuracy close to 50%.
The reviewer’s comment reflects our own, and the paper's conclusion: for the human-object interactions we tested, none of our probes (linear, bilinear, MLP or attention over patches) can reliably identify from the frozen encodings which entities participate in the interaction.
We note that hard negatives are not constructed via confounding verbs. For example, a hard negative for "riding" is an image where there is an instance of "riding", but the probes receive as inputs a human and an object that are not the ones involved in it. As a baseline, instead of random chance, we use a probe trained using position alone. Evaluated on the same examples, this positional probe reaches 61%: single-entity probes do not exceed it, while pair probes exceed it by 4–10 points.
Therefore, I think that (1) many of the evaluated aspects in the paper are not essentially relational encoding in visual foundation models; (2) Some of the experiments demonstrate that visual foundation models encode weak relational information.
We do not agree with the reviewer about (1): All our tasks are intrinsically relational, since they are not defined by the absolute properties of single entities, but exist only as a relation between multiple entities. This is true for the synthetic tasks of relative size and the “odd-one-out”, as described in more detail above. This is especially the case for human-object-interactions, where, e.g., a human is always present but the interactions are not. We agree on (2): visual foundation models encode relational information that at best are not readily accessible, and should not be used naively and directly for relational scene understanding.