acceptodds
Under review as a conference paper at ICLR 2027

Predictive Factors as Support×Operation Cells: A Strict Compositional Readout for Frozen Encoders under Controlled Interventions

Abstract

Frozen vision encoders encode responses to spatial image edits, yet factor-wise probes do not test whether an operation remains bound to the region where it occurs. They can therefore credit multiple operations through the same predicted slot, a degeneracy we call operation laundering. We introduce an injectively aligned leave-one-cell-out evaluation that requires distinct ground-truth axis values to occupy distinct slots. Our Support–Operation Factorization (SO-OPF) readout separates support salience from operation identity, making the binding between where and what explicit. We distinguish readout capacity under a known factorial assignment from recovery of that assignment using only flat cell labels. Across controlled synthetic scenes and image-disjoint natural images, SO-OPF composes held-out support–operation pairs, while learned assignments retain much of the performance obtained with known structure. Under the same flat-label objective and a shared fixed training budget, SO-OPF outperforms the tested Dense-MLP and Joint-MLP readouts across encoder families and multiple data constructions. Router controls distinguish position-driven from source-driven routing: source features support generalization to unseen layouts, while edit-space transfer characterizes robustness across intervention realizations. These results establish a controlled framework for evaluating readout capacity, assignment recovery, and alignment separately.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Operation Laundering, SO-OPF Readout

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GcpQOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses confounds in the way frozen vision models encode spatial edits with regards to compositional generalization. The authors find that factor-wise probes that are commonly used can degenerate to “operation laundering” which is described as multiple operations being assigned to the same “slot” where a slot is a learned position/value in the operation axis. The authors introduce utilizing Hungarian matching with injective alignment and a leave-one-cell out evaluation to detect operation laundering. The author also introduces SO-OPF, a method for learning a factorized readout of a vision encoder latent space. The method learns a mapping from patch to subject and then characterizes changes in patches and subjects to find the operation. SO-OPF results in good performance compared to baseline methods on held-out compositional decoding with the injective evaluation.

Critical strengths
  1. The identification of operation laundering and resolution via held-one-cell out evaluation and injective alignment are useful for future work in the field of compositional generalization

  2. The proposed method is elegant and clear to understand and performs very well

  3. is critical as it identifies a flaw in current evaluation used for determining mapping and performance. They also resolve this flaw with a new evaluation protocol.

  4. the proposed method is critical as along with identifying the flaw and exposing it with the new evaluation, the authors are able to derive a method that performs well on the new evaluation protocol meaning it does not have the same flaw as baseline methods.

Critical weaknesses
  • the value of the new method/protocol is not explicitly illustrated.

It would be valuable for the authors to illustrate to the reader exactly why operation laundering is a flaw as it relates to the use cases for compositional generalization and other use cases for latent spaces that are composable. This would make the work more relevant to more readers than just those directly familiar with the subject.

Questions for authors
  • Is there a particular justification for why certain combinations of backbones, datasets, and assignment semantics were not reported ? for instance compressed identity was only reported for S3D DINOv3 and SIGLIP2 was not evaluated at all on S3D.
1 comment

Thank you for these questions. Our protocol addresses a distinction central to compositional generalization: success on familiar components must translate into success on their unseen combinations, as formalized by Wiedemer et al. (NeurIPS 2023). Consider a readout that identifies the edited wall but predicts the same operation slot for both hue rotation and inversion. Many-to-one alignment can credit both predictions, although a downstream system using that slot—for example, for operation-conditioned retrieval—cannot distinguish the two changes. The resulting score therefore overstates the usability of the recovered factors.

Injective alignment accommodates arbitrary slot naming while requiring distinct factor identities to occupy distinct slots. Permutation-aware matching has an established precedent in Slot Attention (Locatello et al., NeurIPS 2020), which uses Hungarian matching for set prediction. Our contribution applies this principle to diagnosing support–operation binding under held-out compositions. Its practical impact is substantial: with predictions held fixed, learned-assignment MuJoCo accuracy changes from 0.790 to 0.569 for DINOv3 and from 0.629 to 0.484 for SigLIP2 under many-to-one versus injective alignment. These gaps reflect enforcing distinct slots jointly across both axes. SO-OPF explicitly represents the binding, and the protocol tests whether its recovered slots preserve the distinctions needed for composition.

Regarding experimental coverage, DINOv3 is the primary encoder, and SigLIP2 is used for cross-encoder replication. Compressed identity tests whether assigning consecutive flat-label indices to consecutive grid locations preserves the support–operation structure. On S3D/DINOv3, its INJ is 0.545, compared with 0.874 for factorial Q and 0.769 for learned Q. SigLIP2 is evaluated with factorial and learned Q on COCO and MuJoCo, covering natural-image interventions and an alternative rendering/edit setting.

Reviewer_FRFuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript tests whether frozen visual models can recognize unseen combinations of changes and their locations. The authors propose separating support salience from operation identity and develop a method that makes the binding between where and what explicit.

Critical strengths

None

Critical weaknesses

W1: As a reader, I find the figures largely uninformative. In particular, the Figure 1 lacks substantive content and shows clear signs of AI assistance. W2: The manuscript shows countless signs of AI assistance. The title "3 SUPPORT×OPERATION FACTORIZATION" is highly characteristic of AI-generated text.

Questions for authors

The AI tools contribute to nearly every stage of the research process. Could the authors clarify which methodological ideas and technical contributions were independently developed by the research team, and how the AI outputs were validated?

2 comments

All core methodological ideas and technical contributions were independently conceived by the research team, including the support–operation factorization, the identification of operation laundering, and the evaluation separating readout capacity from assignment recovery. The team manually validated the scientific content. AI tools were used primarily to assist writing and presentation. The additional assistance listed in our disclosure concerns supporting activities and refinement; it does not mean that AI originated the core contributions.

Our disclosed AI use is consistent with the ICLR 2027 AI Policy for Authors. The AI Use Statement on page 10 explicitly describes the assistance received and confirms that the authors reviewed the AI-assisted work and take responsibility for the submission. The manuscript’s empirical claims are supported by matched-core comparisons, cross-encoder replication, component ablations, and position, layout, and edit controls. These provide concrete scientific evidence against which the contributions should be assessed. We are willing to provide anonymized implementation and evaluation code (https://anonymous.4open.science/r/SO-OPF/) to facilitate independent verification of our methods, evaluation protocol, and reported results, and we welcome rigorous technical scrutiny.

Regarding W1 and W2, “Support×Operation Factorization” directly names the two axes formally defined in Section 3. Figure 1 illustrates two substantive points: the decomposition into support salience and conditional operation identity, and how identical predictions receive different credit under many-to-one and injective alignment. Figure 2 and the experimental tables quantify the corresponding effects. We welcome specific feedback on unclear elements. However, perceived AI-like wording or visual style does not, by itself, establish a methodological flaw or invalidate an experimental result.

A qualified reviewer is expected to read the manuscript carefully, assess its scientific content, and substantiate their judgments with concrete evidence. These are basic responsibilities explicitly reflected in the ICLR Reviewer Guidelines, which require reviewers to “provide supporting arguments for your recommendation.” Speculation about AI-like wording or visual style does not fulfill these responsibilities or establish technical unsoundness. Assigning a soundness score of 1 requires a substantive explanation tied to the manuscript’s claims, assumptions, methods, or experimental evidence. The present review provides no such explanation. Authors are accountable for the accuracy and rigor of their submissions; reviewers are likewise accountable for the accuracy, rigor, and justification of their assessments. We expect this review to meet those standards and request a substantive, evidence-based reassessment.

Dear Reviewer FRFu,

We have just updated the visibility of our 11 Nov response to your question. It had been visible only to the program and senior area chairs; it now appears above for you and the AC. Because this change may not trigger a notification, we are writing to let you know.

Our implementation, evaluation code, and saved per-fit results are available at https://anonymous.4open.science/r/SO-OPF/; the included scripts rebuild the paper's tables from these results on a CPU.

Your review does not identify a specific claim, method, or experimental result that it considers unsound. If you have such concerns, we would be glad to address them before the discussion period ends. Otherwise, we ask that you reassess the soundness and overall ratings on the basis of the paper's technical content.

Sincerely, The authors

Reviewer_zBbXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Support–Operation Factorization (SO-OPF), a structured readout framework for evaluating compositional generalization in frozen vision encoders. SO-OPF separates spatial support (where an intervention occurs) from operation identity (what transformation is applied) and predicts them through dedicated pathways. The paper also identifies operation laundering, a failure mode where multiple distinct operations collapse into the same prediction slot, and introduces an injective alignment metric to prevent such degenerate matching. Experiments on synthetic scenes, MuJoCo, and COCO show that SO-OPF generally outperforms matched MLP-based baselines on unseen support–operation combinations.

Critical strengths

The paper identifies an important limitation of standard factor-wise evaluation, where multiple operations can be incorrectly matched to the same prediction slot. The proposed injective alignment metric addresses this issue and provides a more reliable evaluation of compositional generalization.

Critical weaknesses
  1. Novelty. SO-OPF mainly combines existing techniques, including spatial routing, normalized feature aggregation, and factorized prediction heads. While the injective alignment metric is a useful contribution, the architectural novelty appears relatively limited. The paper could better distinguish its contributions from prior factorized or slot-based approaches.

  2. The experiments are restricted to three predefined spatial supports and three simple image transformations. Even on COCO, the interventions are controlled pixel-level edits rather than natural semantic changes. It remains unclear whether the method can generalize to more complex scenarios, such as irregular or overlapping regions, diverse semantic operations, or multiple simultaneous interventions.

  3. SO-OPF shows a substantial performance drop on MuJoCo, where injective accuracy under learned assignment decreases to 0.569 with DINOv3 and 0.484 with SigLIP2, accompanied by high collapse rates. This raises concerns about the robustness of the proposed factorization. Further analysis of the causes of collapse and potential mitigation strategies would strengthen the paper.

Questions for authors
  1. Can you make novelty more clear?

  2. How is the model generalization to more complex scene?

3 comments
AuthorsOpenReview ↗

Response(2/2) W3

W3. MuJoCo: failure analysis and targeted mitigation

The errors concentrate in one confusion: across 243 saved fits, 93.5–99.3% of operation errors are hue↔inversion confusions.

MuJoCo applies both edits to the material color before rendering. For any color with nonzero saturation, RGB inversion shifts HSV hue by exactly half a turn while also changing saturation and value. The original hue operation rotates by 0.25–0.75 turns, so it often lands close to this half-turn shift; what remains to separate the two edits depends on material appearance and rendering.

We tested this explanation with paired re-renderings that change only the hue-rotation distribution, keeping scenes, materials, the other operations, and the training protocol fixed. Known assignment, ten training seeds, all nine held-out cells:

Distance from half-turn rotationDINOv3 INJSigLIP2 INJ
Original: 0–0.25 turns0.4630.475
0.15–0.40 turns0.7190.879
0.25–0.40 turns0.7960.953

Collapse falls from 66.7% for both encoders to 5.6%/0%, and hue↔inversion swaps from 52.5%/52.1% to 19.0%/4.4% of held-out predictions. The dose–response confirms the half-turn overlap as the cause of the known-assignment failure. These paired controls come from one generator and are reported separately from the original Table 1 cache.

Under learned assignment, the 0.15–0.40 margin raises SigLIP2 INJ from 0.452 to 0.603, while DINOv3 stays near 0.60 (0.599 → 0.623). With material edits, the observed cells constrain the grid only weakly: at the 0.25–0.40 margin, a discrete search over label→slot maps, using flat labels only, finds maps that misplace one to three labels yet validate within one point of the true grid in 18 of 23 fits. Pixel-space edits on the same scenes remove this ambiguity: learned assignment places 0.875/0.963 of observed cells correctly and reaches INJ 0.799/0.900.

The MuJoCo failures therefore trace to the intervention design: hue and inversion nearly coincide on material colors. Separating them restores known-grid composition with both encoders, and pixel-space edits in the same rendering environment also restore learned-assignment composition. The revision adds this analysis and states the design rule it implies: operations in a factorial grid should not coincide on any support, both for the readout to compose and for flat labels to identify the grid.

Dear Reviewer zBbX,

We have just updated the visibility of Part 1 of our response (novelty and more complex settings, posted 11 Nov). It had been visible only to the program and senior area chairs; it now appears above for you and the AC. Because this change may not trigger a notification, we are writing to let you know, and we apologize for the oversight.

A brief map from your points to the evidence:

  • Novelty (W1/Q1). SO-OPF constrains how where binds to what: the router assigns each support its innovation mass, and the operation head redistributes that mass among operation identities while preserving the total. With router, objective, and training budget matched, COCO injective accuracy is 0.754/0.778 (DINOv3/SigLIP2) for SO-OPF versus 0.275/0.269 for a parameter-matched Dense-MLP.
  • More complex settings (W2/Q2). Irregular instance-mask regions on COCO, unseen object layouts, two simultaneous edits (0.843 joint accuracy, chance 1/36), and a new COCO 3×5 grid with five operations, where SO-OPF reaches 0.788/0.817 against 0.426/0.547 for Dense-MLP.
  • MuJoCo collapse (W3). 93.5–99.3% of operation errors are hue↔inversion swaps: inverting a saturated material color shifts its hue by half a turn, close to the sampled hue rotations. Moving the rotation away from the half turn raises known-grid INJ from 0.463/0.475 to 0.796/0.953 and lowers collapse from 66.7% to 5.6%/0%. Pixel-space edits on the same scenes give learned-assignment INJ of 0.799/0.900.

Before the discussion period ends, we would be grateful to know whether these responses address your concerns or whether any point needs further clarification. If they resolve your concerns, we would appreciate your considering this in your final assessment.

Best regards, The authors

Dear Reviewer zBbX,

With the discussion period closing in less than a day, we wanted to check once more whether our two-part response (novelty and more complex settings in Part 1; the MuJoCo collapse in Part 2) addresses your concerns. We would be glad to clarify any remaining point in the time left.

Best regards, The authors

Comments on the paper

1 comment

Thank you for the follow-up and for keeping your score. We answer the three questions in turn.

A note on visibility. Because of an error in our initial settings, our first response to you did not include the area chairs among its readers. We have now corrected this, but your follow-up comment has the same restricted readers, so the AC still cannot see it. If you see this note, we would be grateful if you could add the area chairs to the readers of your comment.

Which tasks need this structure. Any task in which a system must recognise or act on a (region, operation) combination it has not seen together:

  • Checking instruction-guided image edits. An evaluator or reward model must decide whether the requested operation was applied to the requested region ("recolour the background", "invert the person"), including region–operation pairs missing from its training data.
  • Action-conditioned prediction and planning. World models and robot policies encode what an action did to which object and must handle known actions on new objects. A code that merges two actions cannot condition a prediction on either of them.
  • Retrieval and auditing of edited data, such as finding every image whose background was recoloured, which needs an operation code that selects a single operation.

The held-out cell is the minimal form of these settings: every region and every operation appears in training, but not this pair. Compositional zero-shot recognition benchmarks (unseen state–object pairs) use the same split. The difference here is that the readout's slots are learned and must be matched to labels after training, and that matching step is where many-to-one alignment can overstate composition.

Does a higher injective score mean better downstream performance? The injective score is itself the accuracy on such a task: a decoder fixes one label per slot using training combinations only, then names the region and operation of unseen combinations. The two alignments differ only when several operations or regions share a slot. Many-to-one alignment then counts a held-out edit as correct whenever it lands in that shared slot, although no decoder can translate a shared slot back into the one operation that was applied.

An edit-verification test, computed from the saved held-out outputs, shows the consequence (image-disjoint COCO, DINOv3, learned assignment, 27 fits per readout). For each held-out edit, we check whether the readout scores the true instruction (true region, true operation) above an instruction with the true region and a wrong operation. Each readout uses its own many-to-one map, the choice most favourable to the dense readout. The dense readout, at 0.827 under many-to-one alignment, scores exactly 0.50 in every fit: all three operations share one slot, so "hue shift", "inversion" and "stripe pattern" receive identical scores and a verifier reading that slot cannot tell which was applied (with the injective map it scores 0.44). SO-OPF (0.770 under both alignments) scores 0.96. On S3D, the dense readout scores 0.949 under many-to-one alignment and 0.71 on verification, while SO-OPF scores 0.791 and 0.88. In both datasets, the readout with the higher many-to-one score is the weaker verifier, and the injective score ranks them correctly.

The factored readout also supports uses beyond single edits. Trained on single edits, SO-OPF reads two simultaneous edits on different regions of S3D scenes and gets both right in 84.3% of cases, against 2.8% chance (Appendix).

In the revision, we add the task examples above to the introduction and the verification test to the appendix, and we add its script to the anonymized repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.