- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses confounds in the way frozen vision models encode spatial edits with regards to compositional generalization. The authors find that factor-wise probes that are commonly used can degenerate to “operation laundering” which is described as multiple operations being assigned to the same “slot” where a slot is a learned position/value in the operation axis. The authors introduce utilizing Hungarian matching with injective alignment and a leave-one-cell out evaluation to detect operation laundering. The author also introduces SO-OPF, a method for learning a factorized readout of a vision encoder latent space. The method learns a mapping from patch to subject and then characterizes changes in patches and subjects to find the operation. SO-OPF results in good performance compared to baseline methods on held-out compositional decoding with the injective evaluation.
Critical strengths
-
The identification of operation laundering and resolution via held-one-cell out evaluation and injective alignment are useful for future work in the field of compositional generalization
-
The proposed method is elegant and clear to understand and performs very well
-
is critical as it identifies a flaw in current evaluation used for determining mapping and performance. They also resolve this flaw with a new evaluation protocol.
-
the proposed method is critical as along with identifying the flaw and exposing it with the new evaluation, the authors are able to derive a method that performs well on the new evaluation protocol meaning it does not have the same flaw as baseline methods.
Critical weaknesses
- the value of the new method/protocol is not explicitly illustrated.
It would be valuable for the authors to illustrate to the reader exactly why operation laundering is a flaw as it relates to the use cases for compositional generalization and other use cases for latent spaces that are composable. This would make the work more relevant to more readers than just those directly familiar with the subject.
Questions for authors
- Is there a particular justification for why certain combinations of backbones, datasets, and assignment semantics were not reported ? for instance compressed identity was only reported for S3D DINOv3 and SIGLIP2 was not evaluated at all on S3D.
Thank you for these questions. Our protocol addresses a distinction central to compositional generalization: success on familiar components must translate into success on their unseen combinations, as formalized by Wiedemer et al. (NeurIPS 2023). Consider a readout that identifies the edited wall but predicts the same operation slot for both hue rotation and inversion. Many-to-one alignment can credit both predictions, although a downstream system using that slot—for example, for operation-conditioned retrieval—cannot distinguish the two changes. The resulting score therefore overstates the usability of the recovered factors.
Injective alignment accommodates arbitrary slot naming while requiring distinct factor identities to occupy distinct slots. Permutation-aware matching has an established precedent in Slot Attention (Locatello et al., NeurIPS 2020), which uses Hungarian matching for set prediction. Our contribution applies this principle to diagnosing support–operation binding under held-out compositions. Its practical impact is substantial: with predictions held fixed, learned-assignment MuJoCo accuracy changes from 0.790 to 0.569 for DINOv3 and from 0.629 to 0.484 for SigLIP2 under many-to-one versus injective alignment. These gaps reflect enforcing distinct slots jointly across both axes. SO-OPF explicitly represents the binding, and the protocol tests whether its recovered slots preserve the distinctions needed for composition.
Regarding experimental coverage, DINOv3 is the primary encoder, and SigLIP2 is used for cross-encoder replication. Compressed identity tests whether assigning consecutive flat-label indices to consecutive grid locations preserves the support–operation structure. On S3D/DINOv3, its INJ is 0.545, compared with 0.874 for factorial Q and 0.769 for learned Q. SigLIP2 is evaluated with factorial and learned Q on COCO and MuJoCo, covering natural-image interventions and an alternative rendering/edit setting.