acceptodds
Under review as a conference paper at ICLR 2027

DeOR: Object-centric Representations from and for Multimodal LLMs

Abstract

Representing objects individually supports scene understanding and object-level manipulation. Slot-based methods decompose scenes into object representations, but many rely on a fixed number of slots despite varying object counts across images. Moreover, these representations often lack explicit object descriptions, making it difficult to identify and select individual objects for manipulation. To address these limitations, we propose DeOR (Description-guided Object Representations), a framework that uses a multimodal large language model (MLLM) to construct a variable number of object representations that can be identified through language. DeOR sequentially generates an object description and a corresponding object token for each object. Each description specifies which object its paired token should represent. Each representation combines the token’s semantic information with visual features from the image. This structure allows objects to be identified through their descriptions and manipulated individually through their representations. DeOR supports compositional generation by recombining object representations across images. Its generated descriptions also allow an external language model to select objects based on user instructions. DeOR then edits the selected representations to remove objects or transfer appearance from another image. On COCO scenes with 4-6 objects, DeOR-L achieves a 26.6% relative improvement in FG-ARI over the strongest evaluated baseline. In zero-shot transfer from COCO to PASCAL VOC, it achieves a 44.2% relative improvement in instance-level mBO over the strongest VOC-trained baseline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
object-centric learning, multimodal large language models, object representations, compositional generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QPq3OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DeOR, an MLLM-based object-centric representation framework. The MLLM sequentially generates object descriptions with corresponding object tokens for objects in the image. Instance masks are then used to supervise correspondence between semantic object states and image regions. Visual memories are extracted for each object and a Reader together with a DiT are used for image reconstruction and editing.

Critical strengths
  1. The authors use an MLLM to dynamically determine the number of objects in an image, while providing explicit descriptions for objects and this work supports grounding, generation and editing.

  2. The work shows a relatively comprehensive and strong system and the experiments are also complete and thorough.

Critical weaknesses
  1. Many modules build on prior works, making this work feel more like a strong system integration or pipeline construction. It lacks some novelty.

  2. AR generating objects one by one may become inefficient in complex scenes.

  3. The overall framework heavily relies on the MLLM's original understanding capability, especially the accuracy and completeness of the generated object descriptions.

Questions for authors
  1. Why DeOR-L has a worse count MAE than DeOR-B?

  2. How to address the token-budget truncation issue in complex scene?

  3. if the generated descriptions miss objects or have errors, how much does this affect the resulting representations or other performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer QPq3 (1/2)

We sincerely appreciate your thoughtful and constructive feedback. We address each of your concerns below.

[W1] Novelty of the object-representation framework

DeOR learns a unified object representation connecting complex-scene decomposition, language-based instance identification, and generative manipulation. Descriptions and object tokens are generated together, jointly determining which objects are represented and how many representations are constructed. These language-identifiable units are then reused directly for compositional generation and editing.

The main experiments demonstrate these connected capabilities. Table 1 and Figure 3 show improved object-region recovery across scene complexities. Table 3 reports 65.73 TargetHit@1 on RefCOCO, with Figure 4 illustrating description–region correspondence. Table 4 reports 17.19 compositional FID for DeOR-L, while Figure 5 demonstrates language-guided object removal and appearance transfer. Together, these results show that the same learned representations support object decomposition, linguistic identification, and visual manipulation. Component ablations and matched FixedQuery comparisons further substantiate the contribution of this representation-learning design beyond simply assembling pretrained components (Tables 5–6; Appendix D).

[W2, Q2] Autoregressive generation cost and generation cutoff

DeOR uses autoregressive description–token generation to construct the object representations required by each scene, jointly determining their number and semantic identities. This enables scene-dependent, language-identifiable object decomposition within a generative representation framework, with crowded-scene results shown in Table 1 and Figure 3. The sequential computation forms the object descriptions and states; the completed object set then conditions image-level generation. Its decoding cost grows with the number and length of generated descriptions, making token allocation an explicit inference-time control. At the default 512-token budget, median per-image runtime is 4.50 seconds for DeOR-B and 7.81 seconds for DeOR-L.

Following your question, we expanded the decoding budget from the default 512 tokens to 1,024 and 2,048 and evaluated the cutoff rate across object-count ranges (Appendix D.2; original Table 19; revised Table 27). On scenes with 11+ objects, increasing the budget to 2,048 reduces DeOR-B’s cutoff from 48.47% to 5.25% (an 89.2% relative reduction), and DeOR-L’s from 85.76% to 69.66%.

Generation cutoff in scenes with 11+ objects

Decoding budgetDeOR-B cutoff ↓DeOR-L cutoff ↓
51248.47%85.76%
1,02411.86%71.53%
2,0485.25%69.66%

These results show that additional inference-time token allocation consistently reduces crowded-scene cutoff for both models, allowing the decoding budget to be adjusted to the demands of complex scenes.

AuthorsOpenReview ↗

Response to Reviewer QPq3 (2/2)

[W3, Q3] Extending MLLM capabilities and evaluating description errors

Extending MLLM capabilities. DeOR learns object-level generative control through its added representation-learning design. This design extends MLLM semantic states into spatially grounded object units for compositional generation and editing (Table 4; Figure 5).

Omitted objects. An object omitted from the generated sequence receives no dedicated object representation. DeOR-B’s generated candidates cover 75.75% of referred RefCOCO objects at IoU ≥ 0.5 (Appendix C.5), quantifying the candidate coverage available for downstream selection.

Description errors. We shuffle selector-input descriptions among same-category DeOR-B candidates while keeping candidate representations and regions fixed. Across all 18,906 evaluation expressions, shuffling reduces E2E [email protected] from 39.74% to 12.31% (−27.43 points) and E2E mIoU from 34.33 to 11.40 (−22.93 points). Shuffled descriptions also perform below category-only inputs (20.61% E2E [email protected]). This control quantifies the effect of incorrect description–object associations on selection (Appendix C.5; original Table 14; revised Table 18).

Contribution of the learned components. Removing description conditioning lowers RefCOCO TargetHit@1 from 65.73 to 30.42 (Table 6). With the same MLLM backbone, removing Reader supervision leaves decomposition performance comparable (Table 5), but reduces donor gain from 0.1156 to 0.0128 (Figure 10; Appendix D.4). These ablations support the contributions of description conditioning to language–region alignment and Reader supervision to object-level generative control.

Object selection with same-category description shuffling

ModelSelector input[email protected] (covered) ↑E2E [email protected] ↑E2E mIoU ↑
DeOR-BCategory only27.2020.6118.27
DeOR-BSame-category shuffled descriptions16.2512.3111.40
DeOR-BGenerated descriptions52.4639.7434.33
DeOR-LGenerated descriptions54.2142.2236.03

[Q1] Count accuracy of DeOR-L DeOR-L’s higher count MAE is associated with a stronger tendency to construct more object representations than the annotated instance count. The existing count analysis shows that its mean count error shifts toward overprediction, with a higher fraction of over-counted images (Appendix D.2; original Table 18; revised Table 26).

Object-count prediction on COCO

COCO count metricDeOR-BDeOR-L
Count MAE ↓1.452.27
Mean count error−0.12+0.99
Over-counted images (%)31.5541.99

We additionally provide qualitative examples of this behavior. In the musician example, the evaluation GT counts only the person, whereas DeOR-L also forms representations for the musical keyboard and microphone, increasing the predicted count from one to three. Similarly, it separately represents the pizza inside the microwave, yielding two representations against one annotated instance. These examples illustrate how additional representations of visible scene elements that are not separately counted in the evaluation annotations contribute to count mismatch. We have added these examples and their interpretation to Appendix D.2.

Reviewer_QPq3OpenReview ↗

Thank you for your rebuttal

Thank you for the clarifications and additional analyses of count errors, decoding budgets, and description errors. These help me understand the method’s contributions and limitations and I maintain my positive assessment and current score of 3.

AuthorsOpenReview ↗

Response to Reviewer QPq3's Follow-up Comment

Dear Reviewer QPq3,

Thank you for carefully considering our clarifications and additional analyses, and for your continued support. Your thoughtful questions have genuinely helped us clarify the presentation of our method and strengthen our experimental evaluation.

Sincerely,
Authors

Reviewer_8KywOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DeOR uses a multimodal language model to generate an image-dependent sequence of object descriptions and tokens. Mask-supervised ownership grounds the tokens, a Writer collects visual memories, and a hierarchical Reader conditions diffusion reconstruction and object manipulation. The paper reports strong region overlap, useful description-based selection, and controlled evidence that Reader supervision enables memory transfer. The integration is technically plausible and informative, although comparisons with supervised mask-caption methods and quantitative validation of conditional composition remain incomplete.

Critical strengths
  • Descriptions participate in forming grounded representations that subsequently support selection and manipulation. The Writer/Reader equations are implementable, and the separation between ownership supervision and memory readout addresses a meaningful modeling problem. Pix2Cap-style mask-caption prediction alone does not provide this generative interface.

  • FixedQuery and description-path ablations support improved region recovery under shared training conditions; the fixed-candidate selection comparison demonstrates a practical benefit of generated descriptions. Reader supervision makes memory transfer substantially more effective. The reported count, truncation, locality, and preservation analyses make the limitations visible.

Critical weaknesses
  • DeOR receives mask-description supervision and large pretrained backbones, whereas most headline OCL baselines have different supervision and scale. These differences are disclosed, and the internal controls partly address them. Nevertheless, the Pix2Cap/X-Decoder mask-caption system (https://arxiv.org/abs/2501.13893v1), whose dataset is used here, deserves substantive discussion and a matched frontend comparison. Its omission limits claims about superior object decomposition or description formation, although it does not remove the novelty of the generative interface.

  • Table 4 cannot measure whether recombined objects preserve their intended identity or appearance. Table 22 demonstrates effective intervention but provides limited evidence about independent donor fidelity and the preservation tradeoff. Generated-description selection succeeds on 39.74% of expressions; DeOR-L reaches the token limit on 85.76% of scenes with 11+ objects. These disclosed limits constrain practical reliability. No code, checkpoints, or raw outputs accompany the submission, so the natural-image results remain independently unverified; this is an artifact limitation, not evidence of fabrication.

Questions for authors
  • How does the proposed frontend differ empirically from Pix2Cap/X-Decoder under comparable mask-caption supervision? Please clarify the FixedQuery matching cost and inference handling of unmatched queries, and distinguish the demonstrated benefit of the combined design from a claim about variable cardinality alone.

  • Can an independent object/attribute metric or human evaluation quantify donor fidelity and non-target preservation, including shuffled or dropped memory controls? A comparison at matched edit strength would help interpret Table 22’s preservation tradeoff.

  • Please provide the exact 16,477-image training manifest or filtering procedure, checkpoint revisions, feature-normalization statistics, seeds, and compute/inference costs, alongside the promised code and weights. The public Pix2Cap training annotations contain 18,212 images, of which 18,013 have a noncrowd countable object with a nonempty description; the stated filter alone did not recover the reported subset.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8Kyw (1/2)

We sincerely appreciate your thoughtful and constructive feedback. We address each of your concerns below.

[W1, Q1-1] Comparison with Pix2Cap/X-Decoder

Pix2Cap/X-Decoder is a segmentation-captioning specialist obtained by fine-tuning segmentation-pretrained X-Decoder on Pix2Cap-COCO. X-Decoder’s pretraining includes COCO2017 segmentation and referring-segmentation annotations, together with image-text corpora (Conceptual Captions, SBU Captions, Visual Genome, and COCO Captions). Its training history therefore differs from DeOR’s.

Nevertheless, our COCO val2017 evaluation shows competitive foreground clustering: DeOR-L achieves 61.5 FG-ARI compared with 63.8 for Pix2Cap/X-Decoder. DeOR additionally supports object-level recombination and editing through the same grounded representations used for decomposition (Table 4; Figure 5). We provide the comparison setting, result, and discussion in Appendix C.7.

Object-centric capabilities across methods

MethodImage-dependent object countPer-object descriptionLanguage-based selectionReconstruction / compositionObject removalAppearance transferInput mask required
Slot-based (CODA)✗✗✗✓△△✗
Pix2Cap / X-Decoder✓✓✗✗✗✗✗
LISA✗✗✓✗✗✗✗
GLaMM△✓✓✗✗✗✗
LaMa / PowerPaint v2 / BrushNetX✗✗✗✗✓✗✓
Paint-by-Example / AnyDoor✗✗✗✗✗✓✓
DeOR✓✓✓✓✓✓✗

[Q1-2] FixedQuery matching, inference, and interpretation

The two controls isolate different factors. FixedQuery replaces both the image-dependent allocation and description conditioning with fixed learnable queries under matched data, objectives, backbones, and schedule; DeOR-B improves instance mBO/mIoU over K=7 by 17.5/16.3 points on COCO (original Table 16; revised Table 22). Table 6 instead retains DeOR's autoregressive, image-dependent allocation and removes only the description paths: removing both description initialization and description attention lowers COCO FG-ARI from 58.87 to 53.63, instance mIoU from 58.02 to 52.15, and RefCOCO TargetHit@1 from 65.73 to 30.42. Because the allocation mechanism is unchanged in this ablation, these drops reflect the contribution of description conditioning beyond variable cardinality alone.

During training, Hungarian assignment minimizes negative soft IoU between query ownership maps and GT masks. Assignment is computed without gradients, followed by area-normalized ownership cross-entropy on matched pairs. Unmatched queries receive no positive mask target or separate no-object label, but remain in ownership competition and the Writer/Reader reconstruction path.

At inference, all K queries participate regardless of the generated description count, competing with the aggregated register background for pixel assignments. No GT-based query selection or confidence, area, or top-k pruning is applied; queries winning no pixels simply produce empty regions. We have clarified these implementation details and the interpretation in Appendix D.1.

[W2-1, Q2] Compositional fidelity and editing preservation

We added three complementary analyses of donor appearance retention and non-target preservation (Appendix D.5, Tables 31–32). Multi-object compositions retain visual information that distinguishes intended donors from same-category distractors. Memory controls support donor-specific appearance transfer, while comparisons at similar foreground change show lower Outside LPIPS with Reader supervision.

Compositional fidelity. We evaluate 100 multi-object compositions generated under the Table 4 protocol, containing 548 intended donor objects. Among 226 Reader-defined active regions with a same-category distractor, 63.72% show higher DINO similarity to the intended donor. This supports retention of donor-specific appearance information beyond category correspondence; Appendix D.5 provides the full analysis.

AuthorsOpenReview ↗

Response to Reviewer 8Kyw (2/2)

[W2-1, Q2] Compositional fidelity and editing preservation (continued)

Donor-memory controls for appearance transfer

Memory conditionDINO-I ↑DINO gain ↑CLIP-I ↑Outside LPIPS ↓
Correct donor, α=10.3136+0.08680.74600.0717
Shuffled donor0.2472+0.02040.71900.0709
Dropped memory0.2217−0.00510.71050.0475

DINO-I and CLIP-I measure similarity between the edited target region and the intended donor. DINO gain measures the increase in DINO-I relative to each model’s same-seed unedited self-reconstruction. Outside LPIPS measures perceptual changes outside the target region relative to the same reference.

Correct donor memory produces a larger increase in similarity to the intended donor than shuffled memory (DINO gain: +0.0868 vs. +0.0204), with similar Outside LPIPS (0.0717 vs. 0.0709). Both DINO-I and CLIP-I are also higher with the correct memory. Dropping memory yields a negative DINO gain (−0.0051). These controls support donor-specific appearance transfer through the visual memories.

Preservation at comparable foreground change

ModelαFG-LPIPSOutside LPIPS ↓
Full0.150.02770.0074
w/o Reader supervision0.750.02780.0093

α controls donor-memory intervention strength. FG-LPIPS and Outside LPIPS measure perceptual changes inside and outside the target region, respectively, relative to each model’s same-seed unedited self-reconstruction. The full set of evaluated strength settings is reported in Appendix D.5.

At nearly identical foreground change (FG-LPIPS: 0.0277 vs. 0.0278), Full reduces Outside LPIPS by 20.4% compared with the model without Reader supervision (0.0074 vs. 0.0093).

[W2-2] Object selection and decoding-budget control

Generated descriptions improve object-selection accuracy, while increasing the decoding budget reduces generation cutoff in crowded scenes. Under identical DeOR-B candidates, descriptions raise E2E selection success from 20.61% to 39.74%, and DeOR-L further reaches 42.22%. Increasing the decoding budget from 512 to 2,048 tokens reduces the cutoff rate before <scene_end> on scenes with 11+ objects for both models, from 48.47% to 5.25% for DeOR-B and from 85.76% to 69.66% for DeOR-L (revised Table 27).

Object selection using generated descriptions (Appendix C.5; original Table 14; revised Table 18).

Selection input[email protected] (covered) ↑E2E [email protected] ↑E2E mIoU ↑
Category only (DeOR-B)27.2020.6118.27
Generated descriptions (DeOR-B)52.4639.7434.33
Generated descriptions (DeOR-L)54.2142.2236.03

Generation cutoff in scenes with 11+ objects (Appendix D.2; original Table 19; revised Table 27).

Decoding budgetDeOR-B cutoff ↓DeOR-L cutoff ↓
51248.47%85.76%
1,02411.86%71.53%
2,0485.25%69.66%

[W2-3, Q3] Training-data specification and reproducibility

Our 16,477-image training set requires valid descriptions for every COCO thing instance remaining after the 512×512 center crop. If any crop-visible thing has an invalid Pix2Cap response, we exclude the entire image. This complete-coverage criterion determines the training subset as follows:

Construction of the training set

Filtering stageImages
Original Pix2Cap training images18,212
Excluded: invalid descriptions for crop-visible things−1,509
Excluded: no valid thing remaining in the crop−226
Final training set16,477

We have added the filtering procedure and exclusion breakdown to Appendix B.1, and expanded Appendices A.7 and B.2–B.3 with pretrained checkpoint sources, normalization-statistics computation, random seeds, training compute, and evaluation time and memory usage. Evaluation times are reported on a single 90-GB NVIDIA B200 GPU partition and include metric computation.

We now provide the exact 16,477-image ID manifest, the filtering script, and the normalization statistics in an anonymized repository: https://anonymous.4open.science/r/ICLR-E5B7/. The repository documents the filtering criteria and exclusion counts used to construct the training subset. Upon acceptance, we will additionally release the training and evaluation code, DeOR-B/L checkpoints, versioned configurations, and raw evaluation outputs, tied to a fixed release commit.

Reviewer_GF1JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeOR uses a multimodal LLM to autoregressively emit a description and a paired object token for each countable object, so the number of representations depends on the image and each one can be named in language. Each token is grounded to image patches by competing with register tokens under mask supervision, then gathers visual memories that a reader turns into conditions for a diffusion transformer. The resulting units can be recombined for compositional generation or edited for removal and appearance transfer. The paper evaluates decomposition on COCO and PASCAL VOC, description–region alignment on RefCOCO, and generation and editing on COCO.

Critical strengths

Casting both the object count and each object's identity as a generated description–token pair is a direct way to get a variable number of language-addressable representations. The two conditioning paths matter: removing both description initialization and description attention drops RefCOCO TargetHit@1 from 65.73 to 30.42.

The loss ablations separate the pieces cleanly. Dropping the grounding loss cuts FG-ARI from 58.87 to 23.82; dropping language modeling raises count MAE from 1.45 to 5.53; Reader supervision barely affects decomposition but drops donor gain from 0.1156 to 0.0128. Decomposition and editability are not bought by the same objective.

The matched FixedQuery control in Appendix D.1 is the right experiment. With the same data, backbones, mask supervision, and schedule, fixed queries can beat DeOR on FG-ARI, while DeOR is clearly better on mBO and mIoU. That is actual evidence that description-conditioned dynamic tokens recover object extent better than a fixed query budget.

Critical weaknesses

The headline gains are not measured against the same problem. The abstract's +26.6% FG-ARI on COCO and +44.2% instance mBO on VOC compare a mask-supervised model (1.5B/7B LLM plus a multi-billion-parameter DiT) with unsupervised slot methods. Under matched supervision, DeOR-B is worse on FG-ARI than FixedQuery with K=7 (58.9 vs. 62.3 on COCO, 44.0 vs. 59.7 on VOC). The VOC "zero-shot" comparison has the same confound: COCO instance-mask training transferred to overlapping categories is not unsupervised object-centric transfer.

The language-grounding baseline is the wrong task. Table 3 feeds a referring expression in as a single description and reads out a mask, which is referring expression segmentation, then compares with CTRL-O, which has no direct mask supervision. Specialist models such as LISA and GLaMM are the relevant baselines and sit much higher than DeOR-B's 48.40 mIoU on RefCOCO val. End-to-end selection with the model's own descriptions reaches only 39.74% [email protected].

The generation claim is not isolated from the backbone. Compositional FID of 17.19 versus CODA's 31.03 uses numbers copied from the CODA paper, on a pretrained Scale-RAE DiT, while DeOR-L's own reconstruction is worse than CODA (rFID 12.93 vs. 10.65, KID×10³ 1.60 vs. 0.35). Object editing is shown qualitatively, with no comparison to inpainting or instruction-editing methods.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer GF1J (1/3)

We sincerely appreciate your thoughtful and constructive feedback. We address each of your concerns below.

[W1-1] Comparison with slot methods

Our comparison with slot-based methods evaluates the shared task of decomposing real-world scenes into individual objects across varying object counts. DeOR is trained with mask–description supervision on 16,477 images (13.9% of COCO train2017); we have made this supervision explicit in the abstract and clarified the comparison setting in Sec. 3.1. This benchmark comparison is not intended to isolate the effect of our design.

That role is served by the matched FixedQuery control, which holds the data, supervision, backbones, and training schedule fixed, and therefore controls for both supervision and model scale. Under these conditions, DeOR-B improves instance-level mBO and mIoU over FixedQuery (K=7) by 17.5 and 16.3 points on COCO (Appendix D.1; original Table 16, revised Table 22).

[W1-2] Interpreting FG-ARI under matched supervision

The matched comparison shows a consistent advantage for DeOR in object-region recovery, reflected in mBO, mIoU, and foreground F1. The existing matched comparison shows that DeOR-B improves instance-level mBO/mIoU over FixedQuery (K=7) by 17.5/16.3 points on COCO and 33.7/29.0 points on VOC, with consistent class-level gains (revised Table 22; originally Table 16). FG-ARI measures clustering agreement over ground-truth foreground pixels, including those assigned to predicted background, rather than explicitly evaluating object-versus-background assignment.

Foreground recovery on COCO

MethodForeground Precision ↑Foreground Recall ↑Foreground F1 ↑
FixedQuery (K=7)78.3057.5166.31
FixedQuery (K=16)82.8157.5367.89
DeOR-B85.2585.3385.29

Our additional analysis shows that DeOR-B improves foreground recall from 57.51% to 85.33% and F1 from 66.31% to 85.29%, while also improving precision; the same trend holds against K=16. These complementary results demonstrate stronger foreground recovery and more accurate object extent under identical training conditions. We have added this diagnostic and its interpretation to Appendix D.1.

[W1-3] Interpretation of zero-shot transfer

Our zero-shot evaluation measures cross-dataset transfer without target-dataset training or adaptation. To extend the evidence beyond COCO–VOC category overlap, we additionally evaluate COCO-trained DeOR-B and CODA on COCO-category-disjoint target classes in FSS-1000 [1] and on cluttered scenes in OCID [2]. DeOR-B improves mBO/mIoU by 5.03/12.27 points on FSS-1000 and FG-ARI by 15.34 points on OCID, without adaptation to either dataset.

Transfer to FSS-1000 and OCID

MethodFG-ARI ↑mBO ↑mIoU ↑
FSS-1000CODA-59.7959.46
DeOR-B-64.8271.73
OCIDCODA45.0841.0432.05
DeOR-B60.4242.6132.33

† FG-ARI is omitted for FSS-1000 because our evaluation uses a single binary target mask per image rather than separate instance labels.

These results provide complementary evidence of target-region recovery beyond COCO-supervised categories and decomposition in cluttered scenes. We have clarified the evaluation protocol in Sec. 3.3 and added the results and evaluation details to Appendix C.2.

[1] Xiang Li et al. FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation. CVPR, 2020.

[2] Markus Suchi et al. EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets. ICRA, 2019.

AuthorsOpenReview ↗

Response to Reviewer GF1J (2/3)

[W2-1] Comparison with referring-segmentation models

The DeOR-B result in Table 3 is obtained without RefCOCO fine-tuning. We additionally fine-tune DeOR-L using only the official RefCOCO training split and compare it with LISA and GLaMM on the same original referring expressions using mIoU. DeOR-L performs comparably to LISA (66.7 vs. 67.3 mIoU on RefCOCO val) and exceeds it on five of eight splits, including all three RefCOCO+ splits (+4.5–8.9 points), while GLaMM, a dedicated referring-segmentation model, remains higher on all splits. DeOR-L is fine-tuned for only 5,000 steps on the RefCOCO training split; our aim is not to surpass referring-segmentation specialists, but to show that the same object representations used for generation and editing can be adapted to language-conditioned region prediction. The full split-wise results and evaluation protocol are provided in Appendix C.4 (revised Table 17).

DeOR connects language-based object identification with object-level image generation and editing. As summarized in the capability table, users can identify an object through its description, remove it or transfer another object’s appearance to it, and combine objects from different images to form new scenes. These functions operate on the same grounded object representations without user-provided pixel masks (Table 4; Figure 5).

Object-centric capabilities across methods

MethodImage-dependent object countPer-object descriptionLanguage-based selectionReconstruction / compositionObject removalAppearance transferInput mask required
Slot-based (CODA)✗✗✗✓△△✗
Pix2Cap / X-Decoder✓✓✗✗✗✗✗
LISA✗✗✓✗✗✗✗
GLaMM△✓✓✗✗✗✗
LaMa / PowerPaint v2 / BrushNetX✗✗✗✗✓✗✓
Paint-by-Example / AnyDoor✗✗✗✗✗✓✓
DeOR✓✓✓✓✓✓✗

[W2-2] Selection using generated descriptions

DeOR-L achieves higher E2E selection success than DeOR-B (42.22% vs. 39.74%), suggesting that the description-based selection interface can benefit from stronger MLLMs. This experiment evaluates whether generated descriptions improve instance-level selection beyond category labels. With DeOR-B’s candidate representations and predicted regions held fixed, generated descriptions increase E2E [email protected] from 20.61% to 39.74% (+19.13 percentage points). This controlled comparison demonstrates that the descriptions provide useful information for identifying individual object representations beyond category membership. We have extended the existing selection table with DeOR-L in Appendix C.5 (original Table 14; revised Table 18).

Object selection using generated descriptions

Method[email protected] (covered) ↑E2E [email protected] ↑E2E mIoU ↑
Category only (DeOR-B)27.2020.6118.27
Generated description (DeOR-B)52.4639.7434.33
Generated description (DeOR-L)54.2142.2236.03
AuthorsOpenReview ↗

Response to Reviewer GF1J (3/3)

[W3-1] Attribution of generation gains and reconstruction quality

To evaluate the contribution of our object-representation design with the generation backbone held fixed, we additionally compare DeOR-B with the matched FixedQuery baselines. These models share the training data, supervision, backbones, Writer/Reader, reconstruction pathway, and training schedule.

Generation under matched training conditions

MethodReconstruction rFID ↓Compositional FID ↓
FixedQuery K=714.2222.90
FixedQuery K=1614.1121.06
DeOR-B13.4321.04

DeOR-B improves reconstruction rFID over both FixedQuery baselines and reduces compositional FID from 22.90 to 21.04 compared with K=7 (Appendix D.1, Table 25). This controlled comparison supports a contribution from description-conditioned representation formation beyond the choice of generation backbone.

For reconstruction, the reported CODA and DeOR results use different training schedules and native generation resolutions. CODA is trained for 500K steps on COCO and downsamples its 512×512 outputs to 256×256 for evaluation. Our submitted DeOR-L uses 5K training steps and generates 224×224 images, which are resized to 256×256 for evaluation. We therefore additionally examine reconstruction within DeOR-L by extending training:

Reconstruction with extended training

Metric ↓5k10k15k
rFID12.930411.750311.5099
KID ×10001.60051.36421.3558

Extending training from 5K to 15K steps reduces rFID from 12.9304 to 11.5099, an improvement of 1.4205 (11.0%), while KID ×10³ decreases from 1.6005 to 1.3558. Reconstruction continues to improve beyond 10K steps. The matched comparison and extended-training analysis are provided in Appendices D.1 (Table 25) and C.3 (Table 15), respectively.

[W3-2] Quantitative comparisons of object editing

We compare DeOR with external methods for object removal and editing. To match the mask-guided target specification of the external baselines, DeOR also receives oracle target masks in both comparisons.

Object removal. We compare with BrushNet [1] and Blended Latent Diffusion (BLD) [2] on 343 image pairs from the RORD [6] validation split, with one pair per scene to avoid repeated backgrounds. Each removal output is evaluated against its paired object-free ground truth using FID, CMMD [7], and DINO distance. DINO distance is one minus the cosine similarity between whole-image DINOv2-base CLS features of the output and ground truth; lower values indicate closer feature-space agreement.

Object removal on RORD-Val (343 image pairs).

MethodFID ↓CMMD ↓DINO distance ↓
BrushNet [1]139.380.8880.506
BLD [2]103.940.7050.460
DeOR-L (10k)99.840.6830.383

DeOR-L achieves the lowest values on all three reported metrics, supporting closer distributional and feature-space agreement with the object-free ground truth than the compared baselines.

Object editing. We compare with TF-ICON [3], AnyDoor [4], and ControlCom [5] on the existing 1,000 COCO val2017 target–donor pairs. We report FID and KID for whole-image distributions. Donor-region CMMD [7] uses CLIP features and a Gaussian RBF kernel to measure distributional similarity between the edited target and donor object region sets.

Object editing on COCO val2017 (1,000 target–donor pairs).

MethodFID ↓Donor-region CMMD ×10³ ↓KID ×10³ ↓
TF-ICON [3]62.000.4097.06
AnyDoor [4]53.200.3863.43
ControlCom [5]55.600.4035.26
DeOR-L (10k)44.010.3520.55

DeOR-L achieves the lowest FID, donor-region CMMD, and KID among the compared methods. Its donor-region CMMD is 0.352, compared with 0.386 for AnyDoor and 0.403 for ControlCom, supporting closer agreement with the donor-region appearance distribution. Detailed evaluation protocols and results are provided in Appendix C.6.

[1] Xuan Ju et al. BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion. ECCV, 2024.

[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended Latent Diffusion. ACM Transactions on Graphics (SIGGRAPH), 2023.

[3] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Composition. ICCV, 2023.

[4] Xi Chen et al. AnyDoor: Zero-shot Object-level Image Customization. CVPR, 2024.

[5] Bo Zhang et al. ControlCom: Controllable Image Composition using Diffusion Model. arXiv:2308.10040, 2023.

[6] Min-Cheol Sagong, Yoon-Jae Yeo, Seung-Won Jung, and Sung-Jea Ko. RORD: A Real-world Object Removal Dataset. BMVC, 2022.

[7] Sadeep Jayasumana et al. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

We sincerely appreciate all reviewers’ thoughtful and constructive feedback on DeOR. Their comments have helped us clarify the contributions of our object-representation framework and strengthen its experimental evaluation.

Revisions are highlighted in blue in the revised PDF. Table numbers have changed as a result of the additions and reorganization; where applicable, our responses provide both the original and revised table numbers for clarity.

We summarize the major updates below:

1. Matched comparisons and cross-dataset evaluation

We expand the matched FixedQuery analysis with foreground recovery and generation results, complementing the existing decomposition comparisons under identical training conditions [GF1J, 8Kyw] - Appendix D.1.

We clarify the interpretation of transfer without target-dataset training and add evaluations on COCO-category-disjoint FSS-1000 classes and cluttered OCID scenes [GF1J] - Section 3.3, Appendix C.2.

2. Comparisons with specialist models

We add comparisons with LISA and GLaMM using a common mIoU evaluation protocol, with the RefCOCO-adapted DeOR-L reported separately from the original model [GF1J] - Appendix C.4.

We compare foreground clustering with Pix2Cap/X-Decoder on COCO and discuss differences in pretraining and learning objectives. A capability comparison further clarifies the scope of DeOR’s decomposition, selection, generation, and editing functions [GF1J, 8Kyw] - Appendices C.4 and C.7.

3. Quantitative generation and editing analyses

We add matched generation comparisons and extend DeOR-L training to 15K steps, improving reconstruction rFID by 11.0% relative to the default 5K checkpoint [GF1J] - Appendices C.3 and D.1.

We add external comparisons for object removal on 343 RORD validation pairs and object editing on 1,000 COCO target–donor pairs, together with analyses of multi-object donor fidelity, donor-memory controls, and non-target preservation at comparable foreground change [GF1J, 8Kyw] - Appendices C.6 and D.5.

4. Object-selection reliability and generation cutoff

We evaluate same-category description shuffling, analyze object-count behavior, and expand decoding budgets to quantify reductions in generation cutoff in crowded scenes [GF1J, 8Kyw, QPq3] - Appendices C.5 and D.2.

5. Training-data specification and reproducibility

We document the filtering procedure yielding 16,477 training images, normalization-statistics computation, pretrained sources, seeds, and computational costs. The exact image-ID manifest, filtering script, and normalization statistics are available in the anonymized repository linked in our responses [8Kyw] - Appendices A.7 and B.1–B.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.