- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether vision–language image encoders represent which objects occupy agent and patient roles, and whether those representations influence image–text matching. It trains low-dimensional asymmetric bilinear probes on patch representations from four frozen CLIP-family and SigLIP encoders using directed interactions derived from the Panoptic Scene Graph dataset. The authors examine layer-wise decoding, visualize instance-specific interactions, and intervene on the learned subspaces through image-wise mean ablation and exchanges between object regions. Experiments report that role-sensitive benchmark categories generally experience the largest mean degradation under learned-subspace ablation, while object-region exchanges can shift preferences toward captions containing the donor object category. The appendices investigate negative types, random seeds, alternative ablations, subspace dimensions, and cross-seed stability. Together, these results support the presence of compact, behaviorally relevant interaction information, while the interpretation of that information as specifically encoding object-to-role binding remains subject to the qualifications below.
Critical strengths
-
The paper addresses a useful distinction between information being available in an encoder and being used in its final predictions. The progression from probing in Section 3.1 to ablation in Section 3.2 and exchange interventions in Section 3.3 gives the investigation a coherent structure. A failure on a role-sensitive caption comparison does not, by itself, establish that the visual representation lacks the relevant information. Conversely, successful decoding does not establish that the original model uses it. Studying both questions across four encoders is a substantive contribution to understanding compositional failures, independently of whether the work improves benchmark accuracy.
-
The visualizations provide concrete evidence of instance-specific interaction information beyond object-category compatibility alone. Figures 3(b) and 4 examine scenes containing multiple instances of the same object categories, including riders and motorcycles or people and horses. The responses distinguish particular interacting partners within these scenes, rather than merely highlighting every instance of a compatible category. The additional examples in Appendices B.2 and B.3 support this qualitative observation across models or layers. These examples matter because generic category co-occurrence would not be sufficient to explain correct partner selection among repeated categories. They are not a substitute for aggregate controlled evaluation, but they provide meaningful positive evidence that should not be overlooked.
-
The mean-ablation intervention preserves the image-wide patch mean while selectively removing variation in the learned subspace. In Section 3.2, Equation (2) replaces each patch’s projected coordinates with their image-wise mean. Because the rows of are orthonormal, this leaves the orthogonal complement unchanged and preserves the sum of the patch representations at the intervention layer, including after the fixed affine denormalization. The intervention therefore has a useful structural property: it removes spatial variation in selected directions without simply eliminating their image-wide average. Tables 1 and 2 report the largest mean losses on SugarCrepe swap-object and VALSE actant-swap, respectively, for all four encoders. Appendix B.4 also reports five-run variability and an alternative zero-ablation intervention. This provides stronger evidence of behavioral relevance than probe accuracy alone, although it does not establish that every affected direction is specific to binding.
-
The exchange experiment includes an appropriate control for the object-region averaging operation. Section 3.3 compares exchanged object-region subspace means with a no-exchange condition in which each region receives its own mean. This controls for the fact that replacing patch-level coordinates with an object-level mean is itself an intervention. Table 3 additionally reports the original and control accuracies, the number of eligible initially correct examples, and the resulting conditional flip rates, while Figure 5 shows the layer-wise behavior rather than only the selected maxima. For example, CLIP ViT-L has an agent-exchange flip rate of 49.33 ± 3.61%, compared with 1.03 ± 0.29% for the random-subspace control at the reported layer. These design and reporting choices make the demonstrated susceptibility to targeted feature exchange interpretable, even though the precise mechanism of the flips requires further qualification.
-
The dimensionality and stability analyses add information about the compactness and reproducibility of the identified representations. Appendix B.5 examines dimensions from 2 to 32 instead of relying exclusively on the main 16-dimensional configuration. Table 7 reports cross-seed subspace overlap, while Tables 8 and 9 connect dimensionality to downstream intervention effects. A particularly informative example is SigLIP at dimension 2: the reported overlap is 0.962 ± 0.004, and SugarCrepe swap-object accuracy decreases by 3.02 ± 0.22 points under learned-subspace mean ablation, compared with 0.73 ± 0.34 points for random ablation. The corresponding VALSE effect is much weaker, so this does not establish a universally sufficient two-dimensional binding representation. It nevertheless provides useful evidence that some role-sensitive behavior depends on a very compact and reproducible component.
Critical weaknesses
-
The evidence does not yet fully distinguish object-to-role binding from correlated category, identity, and scene-context information. Section 3.1 trains the probe on directed annotated object pairs pooled across interaction predicates, with swapped and replacement negatives. Performance can therefore benefit from information about typical category combinations, directionality, or contextual compatibility, in addition to instance-specific binding. The same-category visualizations are valuable evidence against a purely category-level explanation, but the supplied material does not provide an aggregate evaluation that quantifies their prevalence or controls these alternatives across examples. In Section 3.3, the compared captions also differ in an object category, so an exchange-induced preference change could partly reflect altered category evidence. This limits the specificity of the central interpretation rather than establishing that the representations contain no binding information. A targeted same-category partner-selection analysis and evidence about preservation of object identity under intervention would substantially strengthen the claim.
-
Object-region exchange can alter the image-wide amount of subspace evidence when the two regions contain different numbers of patches. Under the construction in Section 3.3, let the exchanged regions contain and patches, with projected means and . Relative to the own-mean control, exchanging their means changes the summed projected representation by
Thus, unlike the mean-ablation intervention in Section 3.2, this exchange generally changes the global feature mean when region sizes differ. A caption preference could consequently shift partly because the donor category gains aggregate feature evidence, even without a selective reassignment of an otherwise preserved binding representation. This is a confirmed property of the intervention, not a demonstrated explanation of the reported results; the paper does not establish its empirical contribution. A region-size analysis, approximately equal-size exchanges, or a control matching the global mean change would help determine how specifically the exchange results support the proposed mechanism.
-
Equal-dimensional random subspaces do not necessarily provide a matched intervention-strength control. Sections 3.2 and 3.3 compare learned and random subspaces of the same dimension. However, the per-feature normalization described in Appendix A.2 does not whiten the representation. For mean ablation, if and is the within-image covariance of the normalized patch features, the average squared displacement is . This quantity can differ between learned and random subspaces despite identical rank. The reported contrast therefore establishes that the learned directions matter more than typical random directions, but does not fully separate role specificity from differences in removed variation or intervention magnitude. This is a potentially fixable control limitation, not evidence that magnitude explains the findings. Reporting displacement norms and a modest magnitude-matched comparison would clarify how much specificity remains.
-
Configuration selection and the reported uncertainty leave the strength of some comparative claims insufficiently calibrated. Appendix B.5 states that the main dimensionality was chosen with reference to probe accuracy, stability, and downstream ablation behavior; consequently, the complete configuration was not selected solely using an independent probing criterion. Table 3 also reports layers selected for high exchange flip rates, although Figure 5 transparently provides the full curves. These are reasonable exploratory choices, but their selected values should be interpreted accordingly. Moreover, the reported five-run standard deviations characterize training-related variation rather than uncertainty from sampling evaluation images. Some claimed category rankings have relatively small mean separations, such as CLIP ViT-L on VALSE actant-swap versus existence in Table 6. The appendices already provide substantial robustness evidence; the remaining concern is the distinction between an observed largest mean and a reliably larger effect. Clearer selection accounting and paired uncertainty estimates would improve confidence in the stronger comparative statements.
Questions for authors
-
How well does the probe recover interaction partners when object categories alone cannot distinguish the alternatives? Figures 3(b) and 4 provide encouraging same-category examples. Could the authors quantify performance on an available subset containing multiple instances of the relevant categories, ideally including a simple category-and-geometry comparison or a matched distractor analysis? This question concerns the specificity of the binding interpretation in Critical Weakness 1. Consistent partner discrimination beyond those alternatives would substantially strengthen that interpretation. If the effect is limited to a small subset of examples, the conclusion should instead distinguish reliable directed-interaction decoding from more tentative instance-binding evidence.
-
How much of the exchange effect remains after accounting for unequal object-region sizes and the induced global mean change? For the 816 examples used for each exchange type in Section 3.3, please report the distribution of exchanged-region patch-count ratios and examine how the preference change depends on that ratio. A result on approximately equal-size regions, or a control producing the same global mean change without exchanging the two region means, would directly address Critical Weakness 2. Persistence of donor-directed shifts under such a control would strengthen the role-reassignment interpretation. Strong dependence on region imbalance would instead indicate that aggregate feature redistribution contributes materially to the result. This could be examined using the existing examples without requiring a new benchmark.
-
Does the learned-subspace intervention preserve object-identity evidence while changing relational preference? Because the captions in Section 3.3 differ in an object category, it would be useful to distinguish a change in which object fills a role from a change in the availability or strength of the relevant object-category representation. Could the authors provide a targeted object-identity diagnostic before and after intervention, or compare donor-directed relational changes with changes in object-only recognition or matching? Preserved identity evidence together with altered relational preference would support a more specific binding interpretation. Broad identity changes would not invalidate the intervention effect, but would require a narrower explanation of what it demonstrates.
-
Are the learned and random interventions comparable in their actual perturbation magnitudes? Please report the feature displacement or removed within-image variation for the learned and random subspaces, preferably both in normalized coordinates and after denormalization. If these differ substantially, could a matched-magnitude comparison or a small intervention-strength sweep be provided? This question concerns Critical Weakness 3. A stronger role-sensitive effect at comparable displacement would make the specificity argument more convincing. If the difference largely disappears after matching intervention strength, the current random comparison would support importance of the selected directions more clearly than specificity to object-to-role binding.
-
What do the complete paired preference transitions and continuous score changes show in the exchange experiment? The conditional flip rate in Table 3 is valid for asking whether an initially correct example can be redirected; it is not itself a population-level net change. Could the authors additionally report both directions of the preference transition and the paired change in the intended-caption versus donor-caption similarity margin? A comparison on a common initially correct subset would also help because learned and random controls can have different eligible subsets. This would clarify how broadly the exchange produces a donor-directed shift and help interpret the agent–patient differences. A uniformly positive net effect on every evaluated subset is not required, but the full transitions would determine how far the present conditional result can be generalized.
-
Which choices were fixed before inspecting downstream effects, and how stable are the comparative conclusions under the reported alternatives? Please distinguish the selection of probe layers, subspace dimension, and exchange-reporting layers, including the role played by downstream ablation behavior in choosing . The dimensionality and layer analyses already reported are useful; could the authors summarize which central conclusions hold across them and provide paired image-level uncertainty for the principal role-sensitive comparisons? Where multiple examples share an image, the uncertainty analysis should preserve that dependence. This would clarify whether the strongest reported values are exploratory maxima or representative effects, and whether close category rankings should be described as reliable differences or simply differences in reported means.
-
Can a swapped negative also be an annotated positive under another retained directed interaction? Section 3.1 pools multiple predicates without providing the predicate as an input to the probe. Removing symmetric predicate types does not by itself guarantee that both directions of an object pair cannot be positive under different retained interactions. Please clarify whether swapped negatives are checked against the union of annotated positive directed pairs, and report the frequency of any such collisions after filtering. A negligible frequency, or an explicit filtering rule, would resolve the concern. If collisions are substantial, a filtered evaluation would help establish whether the probe’s accuracy and learned subspace are affected. The supplied material does not establish that such conflicts actually occur.
6 comments
[W2, Q2] How much of the exchange effect remains after accounting for unequal object-region sizes and the induced global mean change?
To check that the effect of the subspace exchange does not come from unequal object-region sizes, we evaluated the exchange on approximately equal-size regions (items where one region is at most twice as large as the other). We also added a control producing the same global mean change without exchanging the two region means, and compared it with the learned subspace exchange. In addition, we compared with the random subspace and with the random subspace whose perturbation size is matched to that of the learned subspace (rescaled). Here are our results:
Table C: Preference-flip rates (%) on approximately equal-size regions (Within 2×) and on all items
| Exchange | Model | Items | learned | random | rescaled | global mean |
|---|---|---|---|---|---|---|
| Agent-side | CLIP ViT-B | Within 2× | 24.19 | 2.07 | 7.76 | 0.19 |
| All | 25.49 | 2.36 | 8.17 | 4.18 | ||
| CLIP ViT-L | Within 2× | 49.96 | 0.88 | 5.42 | 1.24 | |
| All | 49.33 | 1.03 | 5.49 | 4.94 | ||
| OpenCLIP ViT-L | Within 2× | 28.69 | 1.03 | 2.83 | 0.00 | |
| All | 25.58 | 1.72 | 3.94 | 1.44 | ||
| SigLIP | Within 2× | 10.70 | 0.27 | 0.67 | 0.42 | |
| All | 11.23 | 0.60 | 1.34 | 1.11 | ||
| Patient-side | CLIP ViT-B | Within 2× | 7.50 | 0.58 | 0.73 | 0.14 |
| All | 11.58 | 0.86 | 1.91 | 1.37 | ||
| CLIP ViT-L | Within 2× | 6.84 | 0.42 | 0.57 | 0.00 | |
| All | 14.58 | 0.35 | 1.60 | 1.09 | ||
| OpenCLIP ViT-L | Within 2× | 5.51 | 0.27 | 0.83 | 0.41 | |
| All | 6.70 | 0.50 | 1.28 | 0.54 | ||
| SigLIP | Within 2× | 1.52 | 0.61 | 0.15 | 0.00 | |
| All | 3.21 | 0.45 | 0.48 | 0.57 |
On the agent side, the flip rates of the learned subspace exchange on approximately equal-size regions stay close to those for all items. On the patient side, they are lower than those for all items, but they remain more than twice as high as those of every control in every model. The control producing the same global mean change gives far lower flip rates than the learned subspace exchange. Thus, the exchange effect does not depend on region imbalance, and the global mean change alone does not reproduce it, which supports a contribution of the learned subspace to binding objects to roles. See Appendix H for details.
[W3, Q4] Are the learned and random interventions comparable in their actual perturbation magnitudes?
We measured the feature displacement of the learned and random interventions for subspace ablation and exchange experiments, both in normalized coordinates, , and after denormalization, , averaged over the intervened patches. In normalized coordinates, the displacements of the two subspaces are similar but after denormalization, the displacement of the learned subspace becomes considerably larger than that of the random subspace for both ablation and exchange in every model.
We therefore provide a matched-magnitude comparison. We scaled the random subspace intervention to match the displacement of the learned subspace intervention after denormalization. We used it as another control in the subspace ablation and subspace exchange experiments. Even at comparable displacement, the random ablation lowers accuracy on the tasks that swap the roles of two objects much less than the learned ablation, and the random exchange flips the caption preference less often than the learned subspace exchange. These results show that the stronger role-sensitive effect of the learned subspace does not come from its larger feature displacement. Appendix F provides the full results.
[W4, Q6] Which choices were fixed before inspecting downstream effects, and how stable are the comparative conclusions under the reported alternatives?
Due to the response length limit, we have posted our full response to [W4, Q6] as a separate follow-up comment below.
[Q3] Does the learned-subspace intervention preserve object-identity evidence while changing relational preference?
To distinguish a change in which object fills a role from a change in the availability or strength of the relevant object-category representation, we evaluated the subspace exchange with two captions that both name all three objects , , and . For example, for agent-side exchange, the captions are "A photo of a {a} {r} a {p}. The photo also shows a {u}." and "A photo of a {u} {r} a {p}. The photo also shows a {a}." Because both captions mention all three objects, a change in object evidence can hardly affect the preference between them, and a change in role assignment is needed to flip it. We computed the preference-flip rate and compared the learned subspace with a random subspace. Here are our results:
Table D: Preference-flip rates (%) when both captions name all three objects
| Exchange | Subspace | CLIP ViT-B | CLIP ViT-L | OpenCLIP ViT-L | SigLIP |
|---|---|---|---|---|---|
| Agent-side | learned | 25.17 | 34.15 | 16.90 | 5.93 |
| random | 2.15 | 1.00 | 1.27 | 0.85 | |
| Patient-side | learned | 12.06 | 11.11 | 4.65 | 2.35 |
| random | 1.95 | 1.71 | 0.97 | 0.86 |
The learned subspace gives higher preference-flip rates than the random subspace in every model and on both sides, most clearly for agent-side exchange. Because both captions name the same objects, these flips come from the exchange of role assignments, not from a change in the availability or strength of the object-category representation. See Appendix G for details.
To check whether object identity remains accessible after the exchange, we trained separate linear classifiers to predict the object categories from the full normalized patch representation , the subspace coordinates , or the orthogonal component , and measured their category classification accuracy. Here are our results:
Table E: Category classification accuracy (%) on PSG val
| Model | Layer | Full representation | Subspace coordinates | Orthogonal component |
|---|---|---|---|---|
| CLIP ViT-B | 8 (agent side) | 65.7 | 50.6 | 64.1 |
| 9 (patient side) | 66.6 | 51.5 | 65.4 | |
| CLIP ViT-L | 14 | 64.8 | 46.0 | 63.4 |
| OpenCLIP ViT-L | 13 | 62.5 | 43.8 | 61.0 |
| SigLIP | 27 (agent side) | 58.3 | 38.8 | 56.9 |
| 12 (patient side) | 67.0 | 46.5 | 65.5 |
In every model, the classifier on the orthogonal component is almost as accurate as the classifier on the full representation. Because the exchange does not change the orthogonal component, this category information stays at the original patch locations. These results show that most of the category information remains at the original locations after the exchange, so object identity remains available after the intervention. Therefore, the learned subspace intervention preserves object-identity evidence while changing relational preference. Appendix G provides the full results.
Dear Reviewer 4HU3,
We thank the reviewer for the careful reading of our paper and for recognizing its contribution to understanding compositionality as well as our experimental results. We also appreciate the many helpful comments. We ran the suggested experiments and are delighted that they improved the paper considerably. Below, we address each comment.
[W1, Q1] How well does the probe recover interaction partners when object categories alone cannot distinguish the alternatives?
To show that our probe binds objects to roles without relying on category shortcuts, we computed probe accuracy on PSG val in which each negative patch has the same object category as the corresponding positive patch. That is, Swap is restricted to objects of the same category, and each replacing object in Agent-Rep, Patient-Rep, and Both-Rep must match the category of the corresponding original object. Because the positive and negative pairs have the same categories, these datasets cannot be solved from the typical combination of categories alone. Here are our results:
Table A: Probe accuracy (%) on PSG val with negatives of the same categories
Our probe remains well above the chance level of 50% in all four models and for every negative type. The probe therefore reads out role information that the combination of object categories cannot provide. The results are reported in Table 11 of Appendix D.
We also performed a simple category-and-geometry comparison. We compared our probe with simple rules that predict the agent from annotations alone. The category rule predicts the agent from the categories of the two objects. For example, for a person and a non-human animal, it predicts the person as the agent. For the geometry comparison, the position rules predict the agent from the vertical position, the horizontal position, or their combination of the two objects, using their centroids. For example, the horizontal rule predicts the object on the right as the agent. Here are our results:
Table B: Accuracy (%) on PSG val
Our probe outperforms all rules in all four models. The probe therefore reads out role–filler binding that category and geometry shortcuts cannot provide. Details are given in Appendix D.