- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DeOR, an MLLM-based object-centric representation framework. The MLLM sequentially generates object descriptions with corresponding object tokens for objects in the image. Instance masks are then used to supervise correspondence between semantic object states and image regions. Visual memories are extracted for each object and a Reader together with a DiT are used for image reconstruction and editing.
Critical strengths
-
The authors use an MLLM to dynamically determine the number of objects in an image, while providing explicit descriptions for objects and this work supports grounding, generation and editing.
-
The work shows a relatively comprehensive and strong system and the experiments are also complete and thorough.
Critical weaknesses
-
Many modules build on prior works, making this work feel more like a strong system integration or pipeline construction. It lacks some novelty.
-
AR generating objects one by one may become inefficient in complex scenes.
-
The overall framework heavily relies on the MLLM's original understanding capability, especially the accuracy and completeness of the generated object descriptions.
Questions for authors
-
Why DeOR-L has a worse count MAE than DeOR-B?
-
How to address the token-budget truncation issue in complex scene?
-
if the generated descriptions miss objects or have errors, how much does this affect the resulting representations or other performance?
4 comments
Response to Reviewer QPq3 (2/2)
[W3, Q3] Extending MLLM capabilities and evaluating description errors
Extending MLLM capabilities. DeOR learns object-level generative control through its added representation-learning design. This design extends MLLM semantic states into spatially grounded object units for compositional generation and editing (Table 4; Figure 5).
Omitted objects. An object omitted from the generated sequence receives no dedicated object representation. DeOR-B’s generated candidates cover 75.75% of referred RefCOCO objects at IoU ≥ 0.5 (Appendix C.5), quantifying the candidate coverage available for downstream selection.
Description errors. We shuffle selector-input descriptions among same-category DeOR-B candidates while keeping candidate representations and regions fixed. Across all 18,906 evaluation expressions, shuffling reduces E2E [email protected] from 39.74% to 12.31% (−27.43 points) and E2E mIoU from 34.33 to 11.40 (−22.93 points). Shuffled descriptions also perform below category-only inputs (20.61% E2E [email protected]). This control quantifies the effect of incorrect description–object associations on selection (Appendix C.5; original Table 14; revised Table 18).
Contribution of the learned components. Removing description conditioning lowers RefCOCO TargetHit@1 from 65.73 to 30.42 (Table 6). With the same MLLM backbone, removing Reader supervision leaves decomposition performance comparable (Table 5), but reduces donor gain from 0.1156 to 0.0128 (Figure 10; Appendix D.4). These ablations support the contributions of description conditioning to language–region alignment and Reader supervision to object-level generative control.
Object selection with same-category description shuffling
| Model | Selector input | [email protected] (covered) ↑ | E2E [email protected] ↑ | E2E mIoU ↑ |
|---|---|---|---|---|
| DeOR-B | Category only | 27.20 | 20.61 | 18.27 |
| DeOR-B | Same-category shuffled descriptions | 16.25 | 12.31 | 11.40 |
| DeOR-B | Generated descriptions | 52.46 | 39.74 | 34.33 |
| DeOR-L | Generated descriptions | 54.21 | 42.22 | 36.03 |
[Q1] Count accuracy of DeOR-L DeOR-L’s higher count MAE is associated with a stronger tendency to construct more object representations than the annotated instance count. The existing count analysis shows that its mean count error shifts toward overprediction, with a higher fraction of over-counted images (Appendix D.2; original Table 18; revised Table 26).
Object-count prediction on COCO
| COCO count metric | DeOR-B | DeOR-L |
|---|---|---|
| Count MAE ↓ | 1.45 | 2.27 |
| Mean count error | −0.12 | +0.99 |
| Over-counted images (%) | 31.55 | 41.99 |
We additionally provide qualitative examples of this behavior. In the musician example, the evaluation GT counts only the person, whereas DeOR-L also forms representations for the musical keyboard and microphone, increasing the predicted count from one to three. Similarly, it separately represents the pizza inside the microwave, yielding two representations against one annotated instance. These examples illustrate how additional representations of visible scene elements that are not separately counted in the evaluation annotations contribute to count mismatch. We have added these examples and their interpretation to Appendix D.2.
Thank you for your rebuttal
Thank you for the clarifications and additional analyses of count errors, decoding budgets, and description errors. These help me understand the method’s contributions and limitations and I maintain my positive assessment and current score of 3.
Response to Reviewer QPq3's Follow-up Comment
Dear Reviewer QPq3,
Thank you for carefully considering our clarifications and additional analyses, and for your continued support. Your thoughtful questions have genuinely helped us clarify the presentation of our method and strengthen our experimental evaluation.
Sincerely,
Authors
Response to Reviewer QPq3 (1/2)
We sincerely appreciate your thoughtful and constructive feedback. We address each of your concerns below.
[W1] Novelty of the object-representation framework
DeOR learns a unified object representation connecting complex-scene decomposition, language-based instance identification, and generative manipulation. Descriptions and object tokens are generated together, jointly determining which objects are represented and how many representations are constructed. These language-identifiable units are then reused directly for compositional generation and editing.
The main experiments demonstrate these connected capabilities. Table 1 and Figure 3 show improved object-region recovery across scene complexities. Table 3 reports 65.73 TargetHit@1 on RefCOCO, with Figure 4 illustrating description–region correspondence. Table 4 reports 17.19 compositional FID for DeOR-L, while Figure 5 demonstrates language-guided object removal and appearance transfer. Together, these results show that the same learned representations support object decomposition, linguistic identification, and visual manipulation. Component ablations and matched FixedQuery comparisons further substantiate the contribution of this representation-learning design beyond simply assembling pretrained components (Tables 5–6; Appendix D).
[W2, Q2] Autoregressive generation cost and generation cutoff
DeOR uses autoregressive description–token generation to construct the object representations required by each scene, jointly determining their number and semantic identities. This enables scene-dependent, language-identifiable object decomposition within a generative representation framework, with crowded-scene results shown in Table 1 and Figure 3. The sequential computation forms the object descriptions and states; the completed object set then conditions image-level generation. Its decoding cost grows with the number and length of generated descriptions, making token allocation an explicit inference-time control. At the default 512-token budget, median per-image runtime is 4.50 seconds for DeOR-B and 7.81 seconds for DeOR-L.
Following your question, we expanded the decoding budget from the default 512 tokens to 1,024 and 2,048 and evaluated the cutoff rate across object-count ranges (Appendix D.2; original Table 19; revised Table 27). On scenes with 11+ objects, increasing the budget to 2,048 reduces DeOR-B’s cutoff from 48.47% to 5.25% (an 89.2% relative reduction), and DeOR-L’s from 85.76% to 69.66%.
Generation cutoff in scenes with 11+ objects
These results show that additional inference-time token allocation consistently reduces crowded-scene cutoff for both models, allowing the decoding budget to be adjusted to the demands of complex scenes.