acceptodds
Under review as a conference paper at ICLR 2027

Limitations of relational encoding in visual foundation models

Abstract

Vision Transformers (ViTs) provide dense, patch-level embeddings that support semantic segmentation, depth estimation, and correspondence. Richer visual understanding, however, requires representations to encode physical and spatial relations between entities. We analyze how much of this relational information can be extracted from frozen embeddings of recent foundation ViT models, pre-trained on static images only (DINOv2, DINOv3), on image-text pairs , PE-Core, PE-Spatial), and natively on videos (LeVJEPA). We use single-entity and pairwise probes, validating them on synthetic tasks and applying them to human-object interaction recognition in HICO-DET. To assess whether performance depends on proxy shortcuts, such as pose detection or spatial proximity, we evaluate out-of-distribution (OOD) generalization, position-related shortcuts, and counterfactual inpainting. We find a split between the two probe families. Probes on single entities can detect whether an interaction occurs, but fail to discern which entities actively participate: their accuracy is comparable to that of the best probe trained on positional encoding only. Pairwise probes retain an accuracy gain of points over this positional baseline, and the OOD study shows a degree of generalization across objects. Increasing probe complexity through non-linearity does not yield consistent improvements, with MLP probes performing comparably on average to their linear or bilinear counterparts. Moreover, we find that models exposed to videos or text during training do not perform better on static images. These results suggest that the successes of ViTs in dense semantic extraction from local patches do not extend to the understanding of physical interactions in static images.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformers, Self-supervised learning, human-object interactions, relational encoding

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_9AbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper conducts a study on how much relational information is encoded in the visual foundation models like DINO. Basically, probing is done on the frozen embeddings from these visual foundation models to serve as proxy of how much relational information is encoded in each model. Experimental results demonstrate that visual foundation models can decently encode some information like the relative object size, but still have limitations on some hard-negative samples on human-object interactions and figuring out which entities actively participate in the human-object relation.

Critical strengths
  • The paper studies an interesting question of how much relational information is encoded in visual foundation models, which is relatively underexplored in existing literature.

  • The experiments are comprehensive and nicely presented in figures which are easy for readers to understand the performance difference between the models.

Above are the two major strengths of the paper. Other strengths include that the evaluation and experimental details are listed in appendix which are beneficial for reproducing the results. Also, confidence intervals are reported for better understanding the performance distribution on the samples in the evaluated dataset.

Critical weaknesses
  • I think many of the designed experiments are not persuasive to show that visual foundation models have encoded relational information. Here are the points:
  1. For example, the linear probing experiments for relative size is designed as probing whether a certain patch belongs to the largest object in the scene. However, this is basically the way to detect salient objects from the images, as salient subjects may have outstanding feature representations compared to background or minor objects. Recent works like [1, 2] also support this. Therefore, this is not something closely related to relational information in the images. The goal of "odd-one-out" seem to be essentially identifying the semantic difference between the objects in the scene.

  2. For the later interaction encoding, the accuracy of 55-67% on hard negatives roughly indicates that the visual foundation models do not encode much relational information, because the hard negative samples mostly have one distractor choice (e.g., riding v.s. standing on), which means a random guess between the correct choice and the distractor choice should have accuracy close to 50%.

Therefore, I think that (1) many of the evaluated aspects in the paper are not essentially relational encoding in visual foundation models; (2) Some of the experiments demonstrate that visual foundation models encode weak relational information.

  • I find this paper relatively hard to understand. First, the title of the paper is about the limitation of the relational encoding in visual foundation models. However, most parts of the paper (Sections 3-5) are discussing about how decent the visual foundation models encode the relational information (e.g., there generalizability to OOD scenarios), instead of discussing on the limitations. The only clear parts that discuss about limitations are in Section 6 (in Section 5 there may be some obscure clues of limitations, but from the words it is hard to figure out whether the authors want to present a limitation, or just stating some facts), which uses more than a page for discussion and conclusion, which is basically unreadable. The audience are unclear about what is the takeaway and what on earth are the limitations of relational encoding in these visual foundation models, and what are the ways to fix them.

[1] Rawlekar et al. Finding Distributed Object-Centric Properties in Self-Supervised Transformers. CVPR 2026.

[2] Kupyn et al. S3OD: Towards Generalizable Salient Object Detection with Synthetic Data. ICLR 2026.

Questions for authors
  • Why the title of the paper is the limitation of relational encoding for visual foundation models, but the main parts are mostly demonstrating how good relational encoding information is within these models? What is the goal/purpose of this paper, and what takeaway should the audience have after reading this paper? These are not clear as the writing of the paper is confounding.

  • I think relational encoding should be mostly be something like positional relationship (e.g., one object is on the left/right of another object). However, besides the study on the object interaction in Section 5. Most studies are about object size (which is essentially saliency detection), and even semantic information (the "odd-one-out" experiment) which I found little relation with relational information. What are the reasons or motivations for choosing these tasks for studying "relational encoding"?

5 comments
AuthorsOpenReview ↗

Answers to critical weaknesses

Weakness 1.

We thank the reviewer for the references. Both works concern the separation of an image into objects and background. In particular Rawlekar et al. show that patch similarities in DINO separate foreground from background, and Kupyn et al. train a segmentation network on top of DINOv3 to predict a salient-object mask. Both share as a limitation the decomposition of the foreground into individual instances.

Our case falls exactly in their limitation: for the relative-size tasks, every probed patch belongs to an object. Positives and negatives are all "foreground", similarly shaped simple objects on a plain background. Furthermore, each object is a positive in some images and a negative in others. A signal that separates objects from background would not separate positives from the negatives within the same image. Therefore, it cannot solve the relative-size task. We, on the other hand, ask the probes to recover which of several objects is the largest, a comparison that the cited works do not address.

The probes does not even rely on the object's own absolute size alone. To show this, we construct an additional 3D dataset in which an object of a given size is equally likely to be the smallest or the largest in the image, and we report (in the Appendix) the average output of a single-patch linear probe as a function of the sizes of both objects. A probe relying only on the probed object's own size would give the same output whatever the size of the other object. Instead, the probe's output for a patch of object ii depends also on the size of object jj.

Our description of the “odd-one-out” task was unclear. Here, each image contains five objects: four identical copies of one sprite and a single, different sprite. The single-patch linear probe sees a patch of one object only and predicts whether it belongs to the single sprite (the odd one out) or to the repeated one. The experiment and dataset are designed so that each sprite appears about equally often in the odd and in the repeated role, so its identity alone carries no information about the task label. This task is relational since the labels do not depend on the absolute identity of the object, but on its relation with the larger context of the image. Since the probe sees a single patch, this comparison must already be present in the embedding.

Weakness 2. For the later interaction encoding, the accuracy of 55-67% on hard negatives roughly indicates that the visual foundation models do not encode much relational information, because the hard negative samples mostly have one distractor choice (e.g., riding v.s. standing on), which means a random guess between the correct choice and the distractor choice should have accuracy close to 50%.

The reviewer’s comment reflects our own, and the paper's conclusion: for the human-object interactions we tested, none of our probes (linear, bilinear, MLP or attention over patches) can reliably identify from the frozen encodings which entities participate in the interaction.

We note that hard negatives are not constructed via confounding verbs. For example, a hard negative for "riding" is an image where there is an instance of "riding", but the probes receive as inputs a human and an object that are not the ones involved in it. As a baseline, instead of random chance, we use a probe trained using position alone. Evaluated on the same examples, this positional probe reaches 61%: single-entity probes do not exceed it, while pair probes exceed it by 4–10 points.

Therefore, I think that (1) many of the evaluated aspects in the paper are not essentially relational encoding in visual foundation models; (2) Some of the experiments demonstrate that visual foundation models encode weak relational information.

We do not agree with the reviewer about (1): All our tasks are intrinsically relational, since they are not defined by the absolute properties of single entities, but exist only as a relation between multiple entities. This is true for the synthetic tasks of relative size and the “odd-one-out”, as described in more detail above. This is especially the case for human-object-interactions, where, e.g., a human is always present but the interactions are not. We agree on (2): visual foundation models encode relational information that at best are not readily accessible, and should not be used naively and directly for relational scene understanding.

AuthorsOpenReview ↗

Answer to paper focus and questions

We thank the reviewer for the thorough read of our paper and for sharing the interest in the fundamental question of “how much relational information is encoded in visual foundation models”.

The audience are unclear about what is the takeaway and what on earth are the limitations of relational encoding in these visual foundation models, and what are the ways to fix them.

To address their general concern of unfocusedness, we have now reorganized the main text of introduction and abstract to show a more focused and coherent narrative. In particular we:

  • Rewrote the abstract, introduction and final section (Discussion and Conclusion) around a central takeaway: frozen ViT embeddings allows us to detect whether an interaction occurs in an image, but all trained probes perform poorly in identifying which entity is actively participating and which instead is a bystander. We also explicitly framed the synthetic experiments as validation of our probing procedure.

  • Added controls showing that the limited accessibility is not an artifact of one probe family, of the labels, or of the pipeline: we include a human reference ceiling, a patch level attention probe (BAN), the validation of the obtained SAM 3 masks on V-COCO ground truths and the inpainting artifact control.

  • Shortened and restructured the discussion section around three main paragraphs: discussion, limitations and conclusions.

Regarding the question of "what are the ways to fix them", our study does not provide a positive answer.

Answer to questions

Question 1.

Our work addresses open research questions on the successes and failures of relational encoding in visual foundation models. The purpose of this paper is to be a cautionary tale on the limits of what a foundation visual model can and cannot be expected to do.

The positive results in Sections 3–4 serve a specific purpose: on controlled synthetic tasks, they show that linear and bilinear probes can solve relational tasks, i.e., recover contextual information when it is readily accessible.

This validates the probing method, so that the worse results on human-object interactions in Section 5, the main focus of the paper, can be attributed to the weaker encodings and not to a generic failure of the method.

The takeaway is threefold.

(i) Whether an interaction is present in an image is decodable from the frozen features: the best performing bilinear probes reach 84%, where a human annotator agrees with the labels for 88% of pairs.

(ii) Single entity's tokens only weakly decode which entities take part in an interaction, at best 6 points above a probe trained on position alone.

(iii) Reading the human and object encodings jointly recovers only part of this information: at best 71%, still 16 points below an human annotator, and this does not change with increased probe complexity, such as MLPs or attention-based probes.

Our main message for the vision community is that these dense feature maps, which work well for segmentation and retrieval, should not be expected to generate, through a light read-out, reliable maps of which entities interact with which.

Question 2.

We believe that “relational” has a wider meaning, and it should involve any property that is defined only by the interaction between different entities, and not by the features of a single one.

For example, the goal of the experiment on object size is to probe whether the local patch of an object encodes the information of the relative size of that object with respect to the rest of the scene.

The experiment of the “odd-one-out” asks whether the local patch of an object encodes the information of the identity of an object with respect to the identities of all other object in the scene.

Both of these are fully “relational” in the sense that they are not defined by the object in its isolation, but by its relation to the external context.

Moreover, the scope of Section 4 can be seen as a preparation to the more general questions presented in Section 5 on HOI detection in natural images. The results show that the probing machinery works as intended, and validates the conclusion that the failing of probing in HOI in natural images is due to the accessibility of relational information in encodings of the patches and not on failures of the method itself.

Dear authors,

I sincerely thank you for your substantial effort in answering my questions and updating the manuscript. Your detailed explanation on the takeaway of this paper definitely makes it much clearer for what is the conclusion of this paper. From these major takeaways, I feel that it is not very surprising for me that the features from vision models encode little information about interaction encoding. In other words, this is something that I would expect before reading the paper. Therefore, although the paper provides comprehensive analysis on the findings, which is undoubtedly valuable, they kind of conform to the community's knowledge/feeling, with little surprise and mediocre insight. Regarding the concept of "relational", I still feel the task like "odd-one-out" is more like a semantic task, with weak relationship to a "relational" task.

I do acknowledge the authors' effort in revising the manuscript, to make the takeaways more clear. However, I still feel that my original rating is proper given the paper kind of proving something that is not very surprising with comprehensive experiments.

Reviewer_iaFHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the extent to which relational information can be extracted from frozen representations of visual foundation models. The authors evaluate six ViT-based models with different pretraining strategies, using linear, bilinear, and MLP probes on synthetic relational tasks and human-object interaction recognition in HICO-DET. The study further introduces hard-negative samples, position-only baselines, out-of-distribution evaluations, and counterfactual inpainting to examine whether the probes rely on relational information or simpler visual shortcuts.

Critical strengths
  1. The paper investigates whether visual foundation models encode relational information beyond object-level semantics, which is an important problem for understanding their representation capabilities.
  2. Diverse probing experiments and analyses. The evaluation considers multiple foundation models and relational tasks. The inclusion of hard negatives, position-only baselines, OOD evaluations, and counterfactual inpainting provides useful perspectives on potential shortcuts in relational probing.
Critical weaknesses
  1. Limited probing capacity weakens the main conclusions. The paper relies primarily on linear, low-rank bilinear (rank 4), and shallow MLP probes. Poor probing performance does not necessarily indicate that relational information is absent from pretrained representations. Moreover, averaging patch embeddings within entity masks may discard fine-grained interaction cues.
  2. Insufficient control in comparisons across pretraining strategies. The evaluated models differ in architecture, scale, and training data, making it difficult to isolate the effects of different pretraining objectives. In particular, LeVJEPA is evaluated using repeated static images, which does not assess its ability to exploit temporal information.
  3. The reliability of hard-negative samples is insufficiently validated. In Section 5, unannotated human-object pairs are treated as hard negatives, but missing HOI annotations could introduce false negatives. Additionally, Appendix B uses SAM 3 and heuristic mask merging without reporting quantitative segmentation or entity-matching accuracy.
  4. Limited experimental evidence for the broader conclusions. The natural-image evaluation covers only six HOI verbs, limiting the generality of the findings. Furthermore, the counterfactual inpainting experiments in Figure 4 may introduce appearance and contextual changes unrelated to the interactions themselves, making the interpretation of relational sensitivity less conclusive.
Questions for authors
  1. Could the authors evaluate more expressive relational probes, such as attention-based or multi-token architectures, to distinguish limitations of the representations from limitations of the probing methods?
  2. How can the authors isolate the effects of different pretraining objectives given the differences between the evaluated models? Would evaluating LeVJEPA on actual video sequences provide additional insights?
  3. How do the authors ensure that the constructed hard negatives are correctly labeled? Could they provide quantitative validation of the SAM 3 segmentation masks and entity-matching procedure?
  4. Could the authors extend the evaluation to more diverse HOI categories and introduce additional counterfactual controls to distinguish genuine relational sensitivity from inpainting-induced artifacts?
2 comments
AuthorsOpenReview ↗

Answers to question 1. and 2.

We thank the reviewer for the thorough read of our work and the constructive comments. We answer all questions below:

1. Could the authors evaluate more expressive relational probes, such as attention-based or multi-token architectures, to distinguish limitations of the representations from limitations of the probing methods?

To address the reviewer’s concern, we both discuss the probes as currently presented in the paper, and we report the results for an additional one.

First, we have verified that the low-rank of the bilinear probe is not a limitation: our experiments, presented in the Appendix, show that performance does not significatively increase with higher rank. To address the reviewer question regarding attention-based multi-token architectures, we now adapt and implement a bilinear attention network, or BAN, (Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang , “Bilinear attention network”, Neurips 2018). The results (and method) described here will be included in the updated paper.

The BAN probe works with an attention-matching mechanism at the level of all individual patches of the human and object masks, without the need to average them first.

The results show that this method does not improve over the bilinear probes on the averaged patches, nor over the MLP probes on the concatenated inputs. The average gain of BAN over the bilinear probes is +0.1%+0.1\% for positives and soft negatives (between −0.4%-0.4\% and +0.5%+0.5\% across models), and +0.2%+0.2\% for hard negatives (between −1.8%-1.8\% and +1.8%+1.8\%), well within the spread across verbs. As a general note we would like to point out that many of the most outstanding successes of, e.g., the DINO family of models is the ease with which its frozen embeddings are used. Works on semantic segmentation, image retrieval, depth estimation show that very strong performance can be achieved with linear probes or even simple PCA decomposition of the embeddings. Our results show that even more complex probes largely fail to find participating entities for human-object-interactions.

2. How can the authors isolate the effects of different pretraining objectives given the differences between the evaluated models? Would evaluating LeVJEPA on actual video sequences provide additional insights?

The reviewer is right that we cannot make detailed quantitative claims on the relative performance of different pretraining strategies. Our main conclusion, however, does not rest on such comparisons. It is a negative result shared by all six models: We believe that our results demonstrate that none of the different pretraining strategies investigated embeds the models with qualitatively stronger relational encoding in the local patch embeddings.

For the LeVJEPA, we have been careful to state that this is not an evaluation of the model as a video-processing model, a regime that our experiments indeed do not study. However, we ask whether the LeVJEPA model, being trained on video, has “percolated” encoding features related to HOI interactions even when it processes static images. Our results show that this is not the case. We have changed the text in the paper to better align to this message, both in the Introduction and in the Discussion. In the Introduction:

We structure the experiments of our work around two central questions:

i) Does relational information emerge from the self-supervised training of models on static images, and can it be readily extracted from the patch embeddings?

ii) Does the answer above change for models trained with objectives beyond those of static images, such as the textual alignment of dino.txt or the PE family and the temporal prediction of LeVJEPA?*

In the Discussion, we point out that:

This limitation is shared by all six models: alignment with text, video pretraining and spatial fine-tuning do not change it qualitatively.

and

Finally, the models differ in size, data and architecture, so we do not rank pretraining objectives, and we evaluated LeVJEPA on static images only, leaving open whether genuine temporal input helps.

We point out that the protocol for processing static-images with the LeVJEPA model is taken from their own repository.

AuthorsOpenReview ↗

Answers to question 3. and 4.

3. How do the authors ensure that the constructed hard negatives are correctly labeled? Could they provide quantitative validation of the SAM 3 segmentation masks and entity-matching procedure?

Regarding the mislabeling of hard negatives, we agree with the reviewer’s concerns. We now provide a “human audit” estimate of the quality of the datasets. We manually inspect at least 200 different instances for each of the evaluation subsets (positives, soft negatives and hard negatives) for each verb. We will provide the resulting data as open data in the project repository.

Within the considered subsets, we find that the human audit agrees with the HICO labels for 88% of positive and soft negatives instances and 87% of hard negatives instances. These “ceilings” strengthen our results:

  • for most models, probes approach these values for positive and soft negative instances (84% for the best model v the 88% manual audit). Thus, probes perform well in the task of finding whether the image depicts an interaction or not.

  • for all models, probes largely fail to solve the task of identifying which entities participate in an interaction, with the best models and probes performing 10pc better than the best positional-probe but still 16pc worse than the human counterpart.

Furthermore, we now perform a quantitative measurement of the segmentation pipeline, using a similar (but independent) HOI dataset, V-COCO (Gupta & Malik, 2015). V-COCO dataset structure is identical to that of HICO-DET, but it additionally contains ground-truth dense masks. Thus, we can compare the results of our segmentation pipeline to the ground-truth (GT) dense masks. The results are now included in the main text, where we show that:

To validate them, we apply the same pipeline to V-COCO, which provides ground-truth instance masks, and compare the patch tokens selected by the SAM~3 and ground-truth masks. Averaged over backbones, patch precision and recall are 96.6%96.6\% and 88.9%88.9\% for humans and 92.5%92.5\% and 87.1%87.1\% for objects

We added a paragraph detailing the V-COCO evaluation to Appendix B.

4. Could the authors extend the evaluation to more diverse HOI categories and introduce additional counterfactual controls to distinguish genuine relational sensitivity from inpainting-induced artifacts?

The natural-image evaluation is limited, unfortunately, by the dataset quality and volume. To train stronger probes one needs a higher quantity of data to avoid overfitting. As an example the non-linear, attention-based BAN probes included now in the experiments have more than 45k trainable parameters, while the most abundant labelled verb has just over 13k images (for holding). With the data available to us, the choice of six verbs is a compromise between variety and dataset quantity. We considered other well known datasets for HOI studies, e.g. V-COCO, but we could not find similarly sized, more diverse HOI categories.

Regarding the inpainting-artifacts, we now perform and report on a counterfactual control for the DINOv3 model. For 1000 positive images containing the verb “riding” (chosen as an example), we inpaint a rectangular part of the image unrelated to the interaction (i.e. no intersection with the human or object mask) of the same size of the largest between human and object. Then, we compute the difference in output for a trained bilinear probe evaluated on the patches’ embeddings of the original images and on those of the modified images. The average variation is μΔP=0.01\mu_{\Delta P} = 0.01, with a standard deviation σΔP=0.04\sigma_{\Delta P}=0.04. This is well below the result for the equivalent inpainting results, which is ΔP=0.15\Delta P = 0.15 when both human and object patches are selected from images where the other entity is inpainted away (the (¬human,¬object)(\neg\texttt{human}, \neg\texttt{object}) case). Thus, the variations found in the inpainting experiment are due to genuine probe sensitivity and not to indirect inpainting artifacts.

Reviewer_1QmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The central question of this paper is whether vision foundation models encode relations between objects in their embeddings instead of embedding only the properties of individual objects. To test this, the authors train various probes, such as linear, bilinear and MLP probes, on the frozen embeddings of foundation models trained on images, image–text pairs and videos. They also use increasingly complex data, starting with synthetic data, where they also test on unseen objects and cross-domain transfer, and finally real human-object interaction data like HICO-DET, including soft and hard negatives. They also use positional baselines, inpainting, and held-out object categories to check for shortcuts. The finding is that the probes can detect whether an interaction occurs, but single-entity probes cannot distinguish objects participating in the interaction from the bystanders better than the positional baseline, and pairwise probes improve over it only by a few points. The paper claims these findings as limitations in relational encoding, although the experiments more directly establish limitations in recovering interaction information with the tested probes.

Critical strengths
  1. The idea of introducing careful controls to catch shortcuts is interesting: The paper checks if the probes rely on shortcuts by introducing hard negatives, a baseline with images replaced with noise (positional baseline), held-out object categories and counterfactual inpainting. These strengthen the analysis of what the probes rely on.

  2. The broad coverage of recent models also adds credibility to the overall evaluation setup.

Critical weaknesses
  1. The claim that ViT embeddings do not capture physical interactions might mean the limitation of the probe rather than missing information. The authors have used masked-average pooled embeddings and this averaging might destroy the relational information spread across patches. Increasing probe capacity after pooling would not recover information lost at this step. So the evidence supports limited accessibility under the tested readouts more directly than a general limitation of relational encoding in ViTs.

  2. There is no upper bound to interpret how best these models can get by fine-tuning the backbone or some other mechanism to show that the hard-negative task is solvable.

Questions for authors
  1. Could the authors test a probe that uses individual patch features, such as attention over human and object patches, while keeping the encoder frozen? This would help check whether mask averaging causes the poor results. Why use single patches for the synthetic tasks but averaged features for HOI?

  2. Could the authors test a supervised HOI model or a fine-tuned encoder on the same hard-negative set? This would help show how well the task can be solved, although it would not establish what information is present in the original frozen features.

I would be open to revising my score based on additional evidence addressing these concerns and how it affects the paper's conclusions.

3 comments

We thank the reviewer for the careful read of our work and the constructive comments and questions. In the following we provide point-by-point answers.

1. Could the authors test a probe that uses individual patch features, such as attention over human and object patches, while keeping the encoder frozen? This would help check whether mask averaging causes the poor results.

We now implement a bilinear attention network, or BAN, (Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang , “Bilinear attention network”, Neurips 2018) on the frozen encodings for the HOI-detection task. The BAN probe works with an attention-matching mechanism at the level of all individual patches of the human and object masks, without the need to average them first.

The results show that this method does not improve over the bilinear models on the average patches, nor over the MLP probes on the concatenated inputs. The average gain of BAN over the bilinear probes is +0.1+0.1% for positives and soft negatives (between −0.4-0.4% and +0.5+0.5% across models), and +0.2+0.2% for hard negatives (between −1.8-1.8% and +1.8+1.8%), well within the spread across verbs.

We agree with the reviewer that our experiments establish limited accessibility rather than complete absence of the information, and we have revised the abstract and discussion to say so. However, we believe that this result is informative in itself. The properties these encoders are known for, such as semantic segmentation and depth estimation, can be recovered from frozen features already with linear readouts. Whether an entity participates in an HOI, the problem studied here, cannot: a linear probe, a bilinear probe, an MLP and now an attention probe over individual patches all remain only weakly stronger than a position-only baseline and well below the manual audit.

Why use single patches for the synthetic tasks but averaged features for HOI?

In the synthetic tasks, the datasets have much more homogeneous entities, so that individual patches of a single entity should be very similar. This is not true in real images, where we used averaging of patches to be sure to include all relevant parts of an object or human.

To give quantitative substance to our claim, we have checked whether the probes trained on individual patches for synthetic tasks degrade significantly when evaluated on mask averaged features. We find that the evaluation performance is almost the same: For the 2D and 3D relative size tasks, the average difference of in-domain performance across all layers (object-averaged minus single-patch) lies between +0.01and+1.2+0.0 1 and +1.2 accuracy points for all models, for both the linear and the bilinear probe. We added a section in Appendix with detailed results. The experiment with BAN probes show that the averaging of patches is not responsible for the poor performances in hard negatives in the HOI detection task.

2. Could the authors test a supervised HOI model or a fine-tuned encoder on the same hard-negative set? This would help show how well the task can be solved, although it would not establish what information is present in the original frozen features.

We understand and share the reviewer’s concern that the HOI detection tasks themselves may not be fully solvable due to the dataset shortfalls and not due to the model itself. As the reviewer points out, however, using a supervised HOI model would not establish what information is present in the original frozen features.

As an alternative method to validate the dataset, we now present an estimated ceiling of performance based on our own evaluation of a subset of images. For each verb, we manually audited a random sample of minimum 200 images for each subset (positive, soft negative and hard negative), blind to model predictions and HICO labels, to check whether the instance has been mislabeled due to wrong or missing annotation. (The data for the manual audit will be contained in the public repository).

This manual audit ceiling is now reported in the main paper, and shows that on average mislabeling is indeed present. Within the considered subsets, we find that the human audit agrees with the HICO labels for 88% of positive and soft negatives instances and 87% of hard negatives instances. These ceilings strengthen our results:

  • for most models, probes perform remarkably well for positive and soft negative instances. For example, the Dino family has an average accuracy of 84%, close to the 88% of the human audit, thus being largely able to determine whether there is an interaction or not in the image.

  • for all models, probes fail to solve the task of identifying which entities participate in an interaction, with the best models and probes performing 10pc better than the best positional probes (71% vs 61%) but still 16pc worse than the manual audit (71% vs 87%).

Thank you for the additional experiments. The BAN probe addresses my first question, although it remains a lightweight readout, so whether stronger readouts could recover this information is still open. This is why my second concern matters: without a trained-model upper bound, it is unclear whether the gap reflects the representation or the difficulty of the task. I therefore maintain my score, though I appreciate the clearer framing in the revision.

We thank the reviewer for their appreciation of our original work and response, and for their willingness to continue the discussion.

Deeper readouts. We agree that we leave open the question of whether stronger readouts may, at some level of complexity, recover this information. Our results show that this does not happen for shallow MLPs or single attention layers: from the bilinear to the BAN probe, an 8x increase in size changes hard-negative accuracy by a mere +0.2 points on average.

For much deeper readouts the question changes, as the reviewer already noted in their original review: a sufficiently powerful head will eventually be able to compute the relation from the tokens. In the extreme limit, however, an identity “encoder” followed by a full HOI model would solve the task, yet we would not say that the pixels themselves “encode relational information”. The success of such readouts say little about what the visual model itself has computed and encoded. This, in our opinion, justifies why we limit our investigations on lightweight readouts only.

Task solvability. Regarding the solvability of the task, we found the reviewer’s initial concern very fair. We believe that fine-tuning the backbone or considering much deeper HOI detection models are both outside the scope of the current investigation. Our solution was instead a manual audit. While it is not a trained model, we believe it still provides an upper bound on how well the task can be solved, with a human in the role of the “trained HOI detector”: the audit agrees with the labels on 87% of hard negatives, against 71% for the best probe. Label noise therefore cannot account for this 16 points gap. We think this performance ceiling has substantially improved our work by closing a real logical gap in the initial manuscript, and we thank the reviewer for pointing it out.

Overall, we thank the reviewer for their constructive criticisms and fair review.

Reviewer_YnggOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the extent to which frozen patch-level representations of modern Vision Transformer (ViT) foundation models encode relational properties and physical human-object interactions beyond single-entity visual attributes. The authors evaluate 6 models spanning 3 pretraining paradigms. The paper is under development and clearly lacking novelty.

Critical strengths

The paper lists down the Limitations of relational encoding in visual foundation models.

Critical weaknesses

The paper has no novelty and the technical aspects are not sound.

Supplementary codebase and videos would have added value wrt clarity.

Extensive review is skipped at the paper needs serious rework.

Questions for authors

The papers seems to be heavily AI generated wrt wordings and experiments.

1 comment

We thank the reviewer for their time. Since the reviewer does not point to specific issues, we summarise the paper contributions and ask for criticism that we can address.

Regarding novelty. The paper tests whether probes can extract relational information of various complexity from frozen patch representations of six foundation models. We develop control experiments such as hard negatives (entities that appear in an image where the interaction occurs but do not participate in it), a position-only baseline, held-out object categories, counterfactual inpainting, and (in the revision) a manual audit of the hard-negative labels. Our main finding is that, for human-object-interactions, frozen encodings provide extractable information on the presence of an interaction in the image but not on the identity of the entities participating in it.

We are very much interested in prior work that provide these results, and be grateful if the reviewer could point them to us.

Soundness. The reviewer remarks that technical aspects of the paper are unsound, but does not point to any specific experiment, claim or analysis. We would be glad to respond to any specific concern or criticism. As we now provide answer to technical points raised by the other reviewers, we hope that those answers can also ease the reviewer's concerns.

Code and videos. Code will be released upon acceptance. All experiments and discussion concern static images only, so we are unsure what video material the reviewer has in mind. If they can clarify, we will be happy to provide it where possible.

AI use. All experiments were designed, implemented and run by the authors. The use of AI tools is described in the “Declaration of AI use” section, in line with the ICLR policy.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Correction to the linear probes on the synthetic datasets.

While preparing the additional experiments, we found two bugs affecting the linear probe on the 3D relative-scale dataset.

First, patch coordinates for DINOv2 were computed assuming 16-pixel instead of 14-pixel patches: object features were pooled from misaligned patches. Second, a subset of the 3D test samples carried incorrect relative-scale labels. We corrected the DINOv2 patch grid and relabelled all 3D test samples.

We have rerun the linear probes with four seeds and updated Fig. 2 and Section 4.1. In-distribution accuracy on the 3D dataset rises from 72–92% to 95–99%. Cross-domain accuracy rises from 48–69% to 54–81% (2D→3D) and from 48–77% to 63–87% (3D→2D).

As a consequence, the advantage of bilinear over linear probes in cross-domain transfer is smaller than what we reported, and DINOv2 performs similarly to the other models of the DINO family instead of at chance level. We have revised all relevant statements.

The conclusion that cross-domain accuracy stays well below in-distribution accuracy for all models is unchanged.

AuthorsOpenReview ↗

Updated manuscript

We have uploaded the revised manuscript. The main changes, described in detail in our responses to the reviewers, are:

  • an additional attention-based probe (BAN)
  • a manual audit of the HICO-DET labels
  • a validation of the segmentation masks
  • a control experiment for inpainting artifacts
  • an additional test of relative versus absolute size for linear probes in the 3D dataset
  • a rewritten abstract, introduction and discussion to reflect the points above and to give a clearer overall message

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.