acceptodds
Withdrawn from ICLR 2027

GAMA: COMPOSITIONAL OBJECT-CONCEPT BINDING FOR CONTROLLABLE MULTIMODAL EMBEDDINGS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frozen multimodal encoders can recognize objects and attributes while still entangling which concept belongs to which object. This limits counterfactual object-level editing: moving a feature direction for one object may unintentionally perturb another object or the surrounding scene. We propose GAMA, a post-hoc structured intervention interface that reconstructs frozen scene embeddings from concept-position tuples while exposing editable object slots. Its core mechanism combines reusable additive concept semantics with gated low-rank multiplicative conjunction residuals. Across controlled text and visual scenes,GAMA preserves frozen-space geometry while supporting targeted replacement and object-concept decomposition. Continuous-factor recovery and a GroundingDINO/CLIP pipeline test a practical relaxation from curated tuples to automatically extracted pseudo-tuples. Because GAMA reorganizes only information retained by a frozen encoder, its scope is compositional recombination within given structured concept spaces, not end-to-end perception or unrestricted openvocabulary recognition of unseen object identities.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Object-Concept Binding, Controllable Multimodal Embeddings, Graph Message Passing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ua3gOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies object-attribute binding in frozen encoders such as CLIP. Given a symbolic description of each scene (per-object concept values and positions), GAMA learns to map these tuples to a vector close to the frozen CLIP embedding of the image or caption. Each object gets an editable slot built from additive concept tables plus a gated low-rank multiplicative residual; slots are summed with context and global branches. Editing is done by vector arithmetic followed by nearest-neighbor retrieval in a gallery of frozen embeddings. On CLEVR, CLEVR-2D, synthetic text, a PUG:SPAR subset and a generated product set, GAMA gives better reconstruction, decomposition and edit retrieval than CLIP average-vector baselines.

Critical strengths
  1. The question is clear and relevant: recognizing concepts is not the same as being able to edit one object without touching the others, and Figure 1 shows this gap well.

  2. The diagnostics are careful in places. The COCO tuple-collision analysis and the manual failure audit (Table A.31) are honest and help explain where errors come from.

Critical weaknesses
  1. The main comparison is close to circular. GAMA takes ground-truth tuples as input, so which attribute belongs to which object is already given; the binding comes from the labels, not the frozen encoder. Its output is also built as a sum of slots, so higher decomposition R² and probe accuracy than CLIP are largely expected by construction. The key missing baseline is a capacity-matched MLP/Transformer trained on the same tuples. Without it, we cannot tell how much of the gain comes from the proposed structure rather than from using labels. The only structural control (Table A.29) is in the appendix and the gains are small (e.g. +0.017 / −0.012 on CLEVR-2D).

  2. The practical value is unclear. The method needs tuples at test time, and on real images the results are weak: exact edit Ret@1 is 0.413 on COCO and 0.093 on the fixed split in Table 4, where ObjEmbed also reconstructs better (MSE 0.106 vs. 0.163). The paper mostly lists what GAMA does not do (perception, open vocabulary, generation) but does not show a downstream task where it helps.

Questions for authors
  1. What concrete downstream use do you see for GAMA, given that it needs structured tuples at inference time? Is there any task where it improves results over using the frozen encoder directly?
  2. Which dataset does Table 3 use, and why do the CLIP numbers differ from Table 1?
Reviewer_3ZySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAMA, a post-hoc interface for editing frozen multimodal representations. Given object–concept–position tuples, GAMA maps a structured scene to a surrogate aligned with a frozen CLIP (or related) embedding, using additive concept tables, low-rank multiplicative interactions, a gate, and object/context/global streams. The paper evaluates reconstruction, decomposition, object replacement, and probing on synthetic scenes, PUG-SPAR, and automatically extracted tuples from real images.

The controlled results suggest that GAMA can improve local replacement when reliable structured tuples are available. However, because the input already specifies the object–attribute assignments, I view the contribution primarily as a structured conditional reconstruction/editing interface rather than a general solution to object–concept binding. The current experiments do not establish that the gated low-rank architecture is necessary, or that it recovers binding information absent from the input; stronger same-input baselines are needed.

Critical strengths
  1. The interface is useful in a well-defined controlled setting. Keeping the backbone frozen and exposing object-level interventions is a reasonable design goal. On PUG-SPAR-33x5, GAMA reaches 1.000 AVG+POS replacement retrieval compared with 0.939 for CLIP (Table A.13), and the synthetic product and occlusion experiments show similar gains under controlled conditions. These results support the narrower claim that a structured surrogate can make certain edits easier to perform.

  2. The paper reports informative limits instead of presenting only positive results. The distinction between exact image retrieval and tuple-exact retrieval is particularly helpful: for the 400 collision cases in Table A.18, the two numbers are 0.155 and 0.670, while exact retrieval on unique tuples is still only 0.585. The paper also reports weak strict real-image editing, the ObjEmbed reconstruction comparison, and the fact that lower dense-scene MSE does not necessarily improve retrieval. This makes the empirical picture more credible.

Critical weaknesses
  1. The task is narrower than the paper’s binding claim, and the comparison set does not match the task. GAMA receives the object–concept–position tuples that specify which attributes belong to which objects. The difficult perception and assignment step is therefore outside the model. The main result is consequently a structured conditional decoder/editor, rather than a method that discovers object–concept binding from an entangled frozen embedding.

This would be a reasonable contribution if presented explicitly as such, but then the key baselines should also receive the same tuples and supervision. The paper does not provide a sufficiently strong comparison to additive-only models, a simple MLP, a lookup-plus-projection model, or variants with the gate, multiplicative residual, context stream, and global stream removed. The existing objective ablation (Table A.27) and dense-scene capacity comparison (Table A.8) do not isolate these components on the actual editing task. As a result, it is difficult to tell whether the gains come from the proposed binding mechanism or simply from giving a supervised structured decoder the information needed to construct the target.

  1. The mechanism behind positional and local binding is not clear from the equations. Equations 12–13 define the object contribution as
hiobj=∑k(bi,k+πpi,k).h_i^{\mathrm{obj}}=\sum_k (b_{i,k}+\pi_{p_{i,k}}).

With this form, swapping two complete objects between two positions leaves the summed object contribution unchanged: the same two object vectors and the same two position vectors are still added. A position-sensitive result could still arise through the ordered context/global paths, but then the result is not being supplied by the object stream alone. Similarly, for a same-position replacement, the position vector cancels in b~(v′,p)−b~(v,p)\tilde b(v',p)-\tilde b(v,p).

The binding-consistency loss in Appendix B raises a related question. If the same concept tuple deterministically produces the same b(v)b(v), the within-group variation in Equations 27–28 appears to reduce largely to variation in the position embeddings. If the implementation contains additional context dependence or another operation not shown in the paper, that detail is important. At present, the equations and the reported ablations do not make it clear what mechanism is responsible for local positional binding.

  1. The real-image evidence is useful as a scope check, but not yet strong evidence of practical object-level editing. The strict fixed-split result in Table A.20 is 0.093 exact edit Ret@1, with 0.483 edited-object match and 0.164 preserved-slot score. The separate 1000-edit experiment reports 0.413 exact Ret@1, but it uses a different configuration and should not be combined with the former result as if they were one headline number. The authors also state that the three-object extension selects checkpoints using the final test_idx subset (Appendix G.1), and that one reconstruction analysis includes training images. These disclosures are welcome, but they limit the strength of the corresponding generalization claims.
Questions for authors
  1. Are Equations 12–13 complete? If so, which branch distinguishes the scene with object A on the left and B on the right from the scene with B on the left and A on the right? Please report a position-swap experiment and, if possible, results after removing the context and global streams.

  2. What is the exact grouping key used in the binding-consistency loss? For a repeated concept tuple with deterministic b(v)b(v), what part of the implementation gives this loss a nonzero gradient on the binding function? If the implementation differs from the equations, please state the difference explicitly.

  3. How are the editing components e(a,o,p)e(a,o,p) in Equation 31 estimated? Are the context and global streams recomputed after an edit, and which data split is used to estimate the components, probes, and retrieval gallery? Please give this information separately for the main controlled experiments, the 1000-edit real-image experiment, and the 140 strict test edits.

  4. Can the authors provide same-input, same-supervision baselines or ablations for additive-only, no multiplicative residual, no gate, no context/global branch, and a simple MLP? These comparisons would determine whether the contribution is the GAMA architecture or the structured input itself.

  5. Can the three-object results be rerun with checkpoint selection on an independent validation set and a held-out test set that is never used for selection? If not, the corresponding claims should be presented explicitly as exploratory diagnostics.

Reviewer_aXTiOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces GAMA, a structured surrogate that maps object-concept-position tuples to a frozen text or image embedding space. Each object slot combines additive concept embeddings with a gated low-rank multiplicative residual. The slots are combined with context and global branches to reconstruct scene embeddings. Experiments evaluate reconstruction, targeted editing, and compositional structure, with additional analyses of practical limitations.

Critical strengths
  1. The paper includes extensive experiments across multiple datasets and settings.

  2. Supplementary code is provided to support inspection of the implementation.

Critical weaknesses

The method section contains substantial inconsistencies between the architecture diagram and the mathematical formulation, alongside unclear descriptions of key components. Figure 2 contains confusing arrows, inconsistent notation, and a layout that obscures the data flow; several connections contradict the equations and textual description. In Eqs. (12)–(13), adding independent position vectors before summing the slots loses object–position associations, making the object stream invariant to swapping objects between fixed positions. Section 3 also leaves important details unclear, including the definition of AVG/AVG+POS and its relationship to the learned slots, the encoding of vec(s_i), and the handling of varying object counts. The authors are responsible for carefully checking the figures and text, rather than submitting AI-generated content without adequate verification.

Questions for authors
  1. Carefully redraw Figure 2 to ensure that its arrows, notation, and data flow match Section 3.

  2. Clarify the methodological and implementation details in Section 3, and distinguish the core original designs from established components and prior work.

Reviewer_sgUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAMA, a post-hoc plugin for frozen multimodal encoders (e.g., CLIP) that disentangles object-attribute bindings. It maps structured concept-position tuples into editable object slots using additive semantics and gated low-rank multiplicative residuals, enabling localized counterfactual feature editing without perturbing non-target objects.

Critical strengths
  • The paper provides a thorough exposition of the methodology, experimental protocols, and ablation studies. The extensive appendices ensures high transparency.
  • The authors delineate what GAMA cannot achieve honestly. They analyze its boundaries, including the inability to perform zero shot recognition of unseen object identities, the heavy reliance on structured concept position tuples as inputs and so on.
Critical weaknesses
  • The paper bypasses comparisons with frontier editing models to justify its motivation. Instead of evaluating against advanced layout-controlled or generative models (e.g., GLIGEN, SlotDiffusion), it relies solely on naive CLIP arithmetic as the baseline, making the problem formulation unconvincing.

  • The limited application scenarios result in marginal contribution to the community. The method depends heavily on perfect structured tuples and synthetic datasets like CLEVR, performing poorly on real images and restricting its utility to narrow representation-level diagnostics rather than practical downstream tasks.

  • The four listed contributions lack substantial originality. First, proposing "a post-hoc structured interface" is not novel, as numerous adapter and concept bottleneck methods already exist for frozen multimodal spaces. Second, combining "additive semantics with gated low-rank conjunction residuals" merely stitches together widely used, mature techniques (e.g., embedding lookups, factorization machines, gating mechanisms) without introducing new mathematical foundations. Third, providing "controlled evidence" is a fundamental requirement for any research paper rather than a standalone contribution. Finally, the "scope analyses" do not yield deep theoretical insights but instead primarily expose the limitations of the proposed method.

  • Poor presentation and excessive defensive writing severely degrade readability. Figure 1 contains unnecessary whitespace and an uncropped LLM figure caption, Figure 2 is overly complex with unreadable text, the main body is cluttered with equations better suited for appendices, and constantly reiterating what the model cannot do disrupts the narrative flow.

Questions for authors

See the weaknesses above.

Reviewer_M8jCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GAMA, a small post-hoc module on top of a frozen encoder (mainly CLIP-B/32). GAMA does not look at the image. It takes a symbolic description of the scene, a list of KK objects, each with its concept values and an optional position, and predicts the frozen embedding of that scene (Eqs. 1–3). The goal is a representation in which one object can be edited without changing the rest of the scene.

  • Model. Each object slot is a sum of per-concept embeddings plus a gated low-rank multiplicative correction for the specific combination of concepts (Sec. 3.3–3.4). The slots are summed and combined with a context branch and a global branch (Sec. 3.5).
  • Training. The output is trained to match the frozen embedding with MSE, cosine, and InfoNCE losses, plus losses that keep each slot's concepts readable and the slots consistent (Sec. 3.8).
  • Main claim. GAMA "preserves frozen-space geometry while supporting targeted replacement and object-concept decomposition".
  • Experiments. The main results use the reconstruction, decomposition, and replacement tests of Uselis et al. (2026) on synthetic text, CLEVR, and CLEVR-2D (Tables 1–3), with a PUG-SPAR subset in the appendix. Additional analyses cover continuous factors, automatically extracted tuples on 2,000 COCO images with an ObjEmbed comparison, a dense 6-object test on real images, and objects never seen in training.
Critical strengths
  1. The proposed method is a post-hoc layer with the encoder frozen and lightweight with about 2.3-2.5M parameters (Table 5), and 2.70 ms per forward pass excluding the encoder. Object replacement is an explicit slot-level operation.
  2. GAMA improves concept accuracy and MSE over the the original sum-multiplicative complexity baseline on the reported datasets (Sec. 4.4, Table A.29). Its visual outputs are much more linearly decomposable than frozen CLIP's based on the results in Table 3.
  3. The real-image study audits its own metric. 400 of 1,000 targets share their full tuple with another gallery image, and on these exact Ret@1 is 0.155 vs 0.670 tuple-exact (Sec. 4.5). It also adds ground-truth-box and background-masking checks and a manual failure audit (Sec. 4.5).
  4. The authors are honest about the scope and negative results. They present it as a deliberate split between perception and binding. The objects come from labels or a separate detector, so a better detector could be plugged in later without retraining CLIP. They state plainly that GAMA reorganizes information already in CLIP's embedding and recombines familiar concepts rather than recognizing new object types. They also report results that do not favor GAMA: (a) On held-out object names, GAMA trails frozen CLIP (0.021 vs 0.039, Sec. 5), which confirms the stated boundary. (b)The ObjEmbed adapter reconstructs the targets better (MSE 0.106 vs 0.163, Table 4). (c) The graph/attention variants lower MSE without improving retrieval (Table 5).
Critical weaknesses

W1. The paper does not clearly state whether GAMA's counterfactual edits on the controlled datasets are scored in the frozen space. The claim that GAMA "preserves frozen-space geometry while supporting targeted replacement" needs evidence that edited GAMA vectors land near the frozen embeddings of the correct scenes. I could not confirm that the reported results provide this evidence for the controlled experiments:

  • (a ) Table 1 is not a direct test of closeness to CLIP. Sec. 4.2 cites Table 1 as showing GAMA "remains close to frozen geometry", and Sec. 4.4 says decomposition asks whether factors "linearly explain the frozen scene space". However, Table 1's R2R^2 behaves like a decomposition score. Frozen CLIP scores below 1, but if Table 1 measured closeness to CLIP, CLIP itself would score exactly 1. The text rows also equal Table 3's scenes-from-objects column. The paper does not say whether the decomposition for the GAMA rows is fitted to GAMA's own outputs or to the frozen targets, so it is unclear how much of this number reflects closeness to CLIP (see Q2).
  • (b) It is unclear what the edits are compared against, and there is little direct evidence that GAMA matches CLIP. For the GAMA rows of Table 2, the paper does not say whether edited vectors are searched among GAMA's own predictions or among frozen CLIP embeddings. The reported distances to CLIP do not have reference points, such as the typical distance between two scenes (Tables A.24, A.25, A.29). The only test against frozen CLIP embeddings I found on a controlled dataset is reconstruction, not editing. On a CLEVR-2D variant, GAMA reaches only 0.275 Ret@1 with binned inputs and 0.754–0.816 with real-valued inputs (Table A.10).
  • (c) In Table 2, frozen CLIP already reaches 0.98–1.00 there, so GAMA can show at most a 0.02 advantage. The appendix does show large gaps where CLIP's edits fail (e.g., Tables A.4b and A.5). These use the same protocol, so it is equally unclear whether they are evaluated in the frozen embedding space. If they are, they would be direct evidence for the claim (see Q1).

This matters because GAMA builds its vectors from labels, largely as a sum of per-object parts. Edits scored in its own space would mostly confirm that design, not show control over CLIP's space.

W2. No experiment isolates the proposed design of additive semantics with gated low-rank conjunction residuals.

  • (a) Only loss terms are ablated. The only ablation of the base model varies loss terms (Sec. 3.8, Table A.27). No variant removes the multiplicative residual or the gate, so the claim in Sec. 3.1 that a purely additive model underfits is never tested. The context/global branches are never removed either.
  • (b) The closest comparison is confounded. The comparison with the original sum-multiplicative baseline (Table A.29) does not isolate any component, because the paper does not describe that baseline's architecture. Its margin also varies by dataset and is small on CLEVR-2D (+0.017 concept accuracy, −0.012 MSE).
  • (c) No generic tuple-input baseline. The frozen encoders embed the raw images or captions, while GAMA receives ground-truth tuples. Apart from the sum-multiplicative baseline, no model takes the same tuples, such as an MLP over concatenated tuples trained with the same objective. Such a baseline would show how much of the gain comes from the tuples alone.

Without those controls, the gains could come from having ground-truth tuples and from the additive slot structure, rather than from the gated low-rank residual that the paper presents as its binding mechanism.

Questions for authors
  1. What are the edits compared against? For the GAMA rows of Table 2 and Fig. 1, which vectors are zz, e(a,o,p)e(a, o, p), and zjz_j in Eqs. 31–32? In particular, is the gallery zjz_j made of GAMA's own predictions z^j\hat{z}_j or of frozen CLIP embeddings? Which steering strength kk does Table 2 use? If the gallery is GAMA's own space, please report replacement Ret@1 with the edited vectors searched among frozen CLIP embeddings. The same question applies to Tables A.4b and A.5. A good result here would largely resolve W1 and raise my score.
  2. What does Table 1's R2R^2 measure? For the GAMA rows, is the decomposition fitted to GAMA's own outputs or to the frozen targets? Table 3's visual rows also match neither visual dataset in Table 1 (for GAMA, 0.959 / 0.968 vs 0.94 / 0.86 on CLEVR-2D and 0.95 / 0.88 on CLEVR), so which dataset do they use? Please also report the distance between z^\hat{z} and zz on held-out scenes, with a reference point such as the typical distance between two different scenes.
  3. Does the gated low-rank residual matter? On one dataset, such as CLEVR-2D, please compare GAMA with an additive-only model (Δ=0\Delta = 0), a model without the gate, a model without the context/global branches, and an MLP over concatenated tuples trained with the same objective. This would directly test contribution (2) (see W2).

Suggestions

  • Define AVG/AVG+POS, e(a,o,p)e(a, o, p), and the steering strength (α\alpha in Eq. 31, kk in the figures) in the main text.
  • Report seed variance for Tables 1–3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.