- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GOLP combines frozen SAM3 proposals and DINOv2 regional descriptors to learn a shared object-prototype bank, alongside separate state, scene, and background variables. Compositional feature reconstruction and a frozen RAE decoder support recognition, reconstruction, and scene manipulation.
Critical strengths
- Cross-scene recognition improves substantially over the evaluated baselines on GSO, OCTB, and YCB (Table 1).
- The shared latent interface supports several useful operations, with qualitative examples of object insertion, removal, and state manipulation.
Critical weaknesses
- The contribution beyond foundation-model priors is insufficiently established. A direct SAM3-region + DINOv2-feature clustering baseline is missing. Without this comparison or matched prior access for competing methods, the recognition gains cannot be clearly attributed to the proposed global representation learning.
- The central mechanism lacks controlled validation. There are no decisive ablations isolating prototype retrieval, compositional reconstruction, and auxiliary semantic guidance. Appendix A.7 explicitly acknowledges unmatched protocols and training budgets, limiting the conclusions from its mask-supervision comparisons.
- The evaluation provides limited evidence of generalization. OCTB and COCO evaluation partitions coincide with validation partitions used for model selection; natural-image results use internal protocols. Moreover, GOLP trails MUFASA on every COCO discovery metric (Table 6), and the collapsed YCB baselines weaken some comparisons.
- Content–state separation remains weakly demonstrated. The evidence mainly consists of selected qualitative interventions. Since the scene encoder receives full-image features and conditions object decoding, object information could bypass the prototype bank. Quantitative leakage and intervention tests are needed.
Questions for authors
- How does GOLP compare with clustering the same SAM3-region DINOv2 descriptors, and with baselines given equivalent priors?
- Can matched ablations establish whether compositional reconstruction improves recognition beyond semantic prototype learning alone?
- Can you report independent held-out results and quantify identity preservation, state control, and scene-code leakage?
3 comments
2. Does joint reconstruction lead to improvements in recognition performance?
Our purpose in introducing the reconstruction task is to learn decodable object content, rather than to use it as an auxiliary objective for classification. While identity learning associates observations with shared representations, it does not inherently dictate how these representations—combined with object states—should be used to reconstruct appearance and shape. Through a joint reconstruction task, we utilize an object feature decoder and a mask decoder to learn this mapping. This is precisely the function of the "reconstruction path" in our design.
Regarding your question about recognition performance, we compared the full GOLP model against a "content-gradient-stop" variant. In this variant, we truncate the gradients of the extracted content before it reaches the object decoder. This prevents the reconstruction process from updating the "shared content" branch, while still allowing reconstruction-based training for other factors and the decoders themselves.
Table R2: Recognition accuracy (%) with and without reconstruction feedback. The ACC metric counts unmatched ground-truth (GT) objects as errors. These results are based on newly trained models (using a random seed of 0) and differ from the original checkpoints and "matched-object" metrics reported in Table R1.
| Dataset | Full GOLP | Content-Gradient-Stop Variant |
|---|---|---|
| OCTB | 48.16 | 49.74 |
| GSO | 72.90 | 82.80 |
| YCB | 45.38 | 47.23 |
In these experiments, we did not observe any improvement in recognition performance resulting from reconstruction feedback. Consequently, we do not characterize "joint reconstruction" as a method that further enhances recognition capabilities beyond semantic identity learning. However, it is important to distinguish between the "feedback mechanism" tested here and the "reconstruction task" itself: even with gradient truncation, the variant still employs the reconstruction task to train its decoders. The relatively high ACC (accuracy) results actually raise a question: should the "shared content" branch also be updated during the reconstruction process? This does not imply that visual decoding relationships can be learned solely through identity assignments.
Why reconstruction remains part of our approach. The global representations we require serve a purpose that extends beyond merely identifying observed objects. In Figure 3, we leverage these representations—in conjunction with an object decoder—to remove or insert specific objects. In Figure 4, by extracting and instantiating these representations, we generate scenes containing objects of specific categories and quantities. To perform such operations, the model must learn how the selected content and its state contribute object features and masks to the scene. The reconstruction process serves as explicit training for this mapping relationship.
Consequently, our core logic lies in learning a representation that simultaneously supports identification, decoding, and reuse within a single framework. We do not interpret a "lack of improvement in ACC" as an "improvement in generation quality" (nor was the latter measured in Reviewer 2’s assessment); rather, we distinguish between the role of reconstruction feedback in "identification" and its role in training "compositional decoding"—the mechanism that enables editing and generation operations.
3. Does identity information originate from a prototype or from other encodings?
We employ a "global prototype" to represent the shared content of an object, while using "state/scene codes" to modulate how that object appears within a specific observation. To investigate this division of labor, we replaced the "soft mixture" with a single predicted prototype, (where ). Subsequently, while keeping this prototype fixed, we swapped the state and scene codes across different scenes.
Table R3: Identity consistency (%) relative to predictions from the original input. We compare the identity predicted based on the "reconstructed object" against the identity predicted based on the "original observation." The 100% consistency for the input data itself arises from self-comparison and does not reflect actual recognition accuracy.
| Condition | OCTB | GSO | YCB |
|---|---|---|---|
| Original Input Baseline | 100.00 | 100.00 | 100.00 |
| Soft Reconstruction | 67.87 | 81.29 | 44.84 |
| Single-Prototype Reconstruction | 67.57 | 81.65 | 46.10 |
| Single-Prototype + Cross-Scene State/Scene Codes | 47.15 | 67.99 | 42.07 |
Substantial identity consistency is maintained even without the use of a soft mixture. Replacing the mixture mechanism with a single prototype results in consistency metric changes of only -0.30, +0.36, and +1.26 percentage points on the OCTB, GSO, and YCB datasets, respectively. In this setup, all observations categorized under the same identity are assigned an identical content vector; in other words, we no longer encode observation-specific variations through distinct mixing weights. Nevertheless, we maintain a largely consistent overall identity. This directly addresses concerns regarding Equation (4): the observed reconstruction consistency does not rely on the specific combination of prototypes used for a given observation. Even if this disentanglement is not absolute, we are still able to reuse the fixed content when other encoding components vary. After swapping "state encodings" and "scene encodings" across different scenarios, we find that identity agreement reaches 47.15% on the OCTB dataset, 67.99% on the GSO dataset, and 42.07% on the YCB dataset. We designed this intervention experiment to test whether the same content representation could be maintained without retaining the original state and scene conditions. The consistency observed in these experiments indicates that cross-scenario content reuse is feasible to a certain extent—meaning that individual prototypes need not be strictly confined to their original encoding combinations.
The decline in consistency following these swaps is also noteworthy; it implies that the degree to which identity information is preserved varies depending on the conditions. Consequently, we argue that these results support the conclusion that "shared object content" and "observation-dependent attributes" have been, to some degree, successfully disentangled.
4. Model Selection, Data Partitioning, and Failures in Benchmarking.
Model selection is performed based on identification accuracy (ACC) on the validation set. In the "Stage-1" control experiments, we select the result that yields the highest ACC on a "fixed-candidate" set within the validation data. Dictionary decoders are selected based on their reconstruction loss during the calibration phase, rather than their performance on intervention tasks.
Regarding the OCTB/COCO benchmark results, the original "test partition" was divided into non-overlapping validation and evaluation subsets—the former used for checkpoint selection, and the latter for reporting final results. In Appendix A.5, these subsets were conflated with the original test set (which contains the same collection of images); we will clarify this distinction in a subsequent revision and provide specific details regarding the partitioning. For the GSO and YCB datasets, the original test partitions were maintained. Furthermore, the validation images used in the OCTB RGB intervention experiments consist of historical data excluded from the evaluator calibration process; consequently, these experiments serve as mechanism diagnostics rather than as entirely independent, novel tests. The internal processing pipeline for COCO/VOC also deviates from the standard procedures prescribed by the official benchmark protocols.
Finally, regarding the "collapsed" runs observed for GOLD/CSGO on the YCB dataset, we do not attribute these to inherent failures of the methods themselves. The aforementioned "same-input control" experiments provide a basis for comparison that does not rely on these specific reproduction outcomes. Additionally, we have retained the results for MUFASA, which demonstrate superior performance on the COCO segmentation task and the VOC overlap metrics. These results constrain the scope of our conclusions concerning segmentation capabilities, while the separate "identity" and "intervention" tests are designed to investigate whether shared content can be recognized and reused across different contexts.
Thank you for your detailed review comments. Below, we clarify our contributions and present the experiments you requested.
1. What do pretrained priors provide?
Thank you for requesting this comparison. We incorporate pretrained priors because they offer useful spatial and category-related information. Our contribution lies in leveraging this information to link object observations to a content bank derived from the entire dataset, and then training on this retrieved content to enable compositional decoding. Instead of using local appearance descriptors directly, we feed the retrieved content into the object decoder; this allows the same representation to generalize across different observations and scene configurations.
Comparison with direct clustering. We have added a K-means-based baseline, using DINOv2 region descriptors, teacher model descriptors cached during the GOLP training process, and a concatenated feature vector (formed by normalizing and combining both). The K-means model is fitted solely on the observations from the training set. In these experiments, we use the same SAM3 object proposals and the same number of categories, with one-to-one matching between proposals and ground truth (GT) objects defined by an IoU threshold of ≥ 0.3.
Table R1. Object matching accuracy (%), with label alignment performed via the Hungarian algorithm during evaluation. “Coverage” refers to the proportion of ground truth (GT) objects that are successfully matched to a shared proposal. For the clustering methods, a random seed of 0 is used; for GOLP, the original reference checkpoint is employed. | Dataset | DINOv2 | Teacher | Joint Features | GOLP | Coverage | | :--- | ---: | ---: | ---: | ---: | ---: | | OCTB | 52.69 | 74.33 | 72.13 | 60.88 | 86.01 | | GSO | 82.16 | 91.08 | 90.38 | 96.24 | 91.61 | | YCB | 48.03 | 60.59 | 65.27 | 54.68 | 83.37 | | COCO | 17.80 | 16.17 | 19.40 | 19.72 | 65.16 | | VOC | 30.53 | 38.38 | 37.02 | 40.53 | 59.17 | |
What can we learn from this comparison? Across all five reported comparisons, our method outperforms direct clustering of DINOv2 features in terms of "object matching accuracy." "Teacher clustering" performs best on the OCTB dataset, while "joint feature clustering" yields the best results on the YCB dataset. GOLP achieves the highest scores on the GSO, COCO, and VOC datasets; however, the 96.40% accuracy observed on the GSO dataset is comparable to the 96.24% achieved by GOLP. Therefore, we do not claim to consistently outperform the direct clustering of the same pre-trained descriptors in terms of recognition performance.
These strong clustering results justify our use of such prior information: our goal is to leverage the category-related knowledge already embedded within these priors, rather than learning everything from scratch. We utilize SAM3 regions to organize object observations, leveraging pre-trained regional semantics to guide the association of these observations with shared content. It should be clarified that the performance gains we achieve—relative to methods employing different pre-trained inputs—reflect the collective advantages of our entire "prior-guided framework," rather than being attributable solely to the "bank learning" component.
Why our objective extends beyond mere clustering. Our goal is to learn reusable object content that facilitates cross-scene recognition, object discovery, reconstruction, and the editing or generation of specific objects. While cluster assignment addresses the question of "which observations belong to the same object," we go further by training the model to decode the associated content based on varying object states. We do not dismiss the utility of clustering as a baseline, nor do we suggest that fixed clustering results cannot be integrated with a decoder. Instead, we distinguish between "identity grouping" and "content learning" geared toward downstream compositional applications. Consequently, when evaluating our global representations, we assess not only recognition accuracy but also their practical utility across a range of scene-level operations.