acceptodds
Under review as a conference paper at ICLR 2027

GOLP: Global Object-Centric Learning via Spatial and Semantic Priors

Abstract

Grouping objects within an image does not automatically yield object knowledge that is reusable across scenes. This paper introduces GOLP, which uses spatial and category-related visual priors to learn global object representations for cross-scene recognition and compositional scene synthesis. A frozen SAM3 generates candidate masks with fixed dataset-level prompts, organizing object observations through their spatial extent and geometry. A frozen DINOv2 provides regional visual descriptors whose category-related structure guides shared prototype matching. The model retrieves content from a global object bank and separately encodes object states, background, and scene attributes, learning these factors through compositional feature reconstruction. In the second learning stage, the representation module and pretrained RAE are frozen, and only a Transformer adapter is trained for image decoding. Reconstruction, editing, and generation share the same latent variables and decoding pathway, without additional raw-image detail at decoding time. We evaluate cross-scene recognition, object discovery, and identity–state interventions, and provide additional analyses of mask supervision and prompt selection. The results examine how shared object content can be reused across scene conditions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Global Object-Centric Learning, Spatial and Semantic Priors, Cross-Scene Object Recognition, Disentangled Representation Learning, Compositional Scene Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iTxMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GOLP combines frozen SAM3 proposals and DINOv2 regional descriptors to learn a shared object-prototype bank, alongside separate state, scene, and background variables. Compositional feature reconstruction and a frozen RAE decoder support recognition, reconstruction, and scene manipulation.

Critical strengths
  • Cross-scene recognition improves substantially over the evaluated baselines on GSO, OCTB, and YCB (Table 1).
  • The shared latent interface supports several useful operations, with qualitative examples of object insertion, removal, and state manipulation.
Critical weaknesses
  1. The contribution beyond foundation-model priors is insufficiently established. A direct SAM3-region + DINOv2-feature clustering baseline is missing. Without this comparison or matched prior access for competing methods, the recognition gains cannot be clearly attributed to the proposed global representation learning.
  2. The central mechanism lacks controlled validation. There are no decisive ablations isolating prototype retrieval, compositional reconstruction, and auxiliary semantic guidance. Appendix A.7 explicitly acknowledges unmatched protocols and training budgets, limiting the conclusions from its mask-supervision comparisons.
  3. The evaluation provides limited evidence of generalization. OCTB and COCO evaluation partitions coincide with validation partitions used for model selection; natural-image results use internal protocols. Moreover, GOLP trails MUFASA on every COCO discovery metric (Table 6), and the collapsed YCB baselines weaken some comparisons.
  4. Content–state separation remains weakly demonstrated. The evidence mainly consists of selected qualitative interventions. Since the scene encoder receives full-image features and conditions object decoding, object information could bypass the prototype bank. Quantitative leakage and intervention tests are needed.
Questions for authors
  1. How does GOLP compare with clustering the same SAM3-region DINOv2 descriptors, and with baselines given equivalent priors?
  2. Can matched ablations establish whether compositional reconstruction improves recognition beyond semantic prototype learning alone?
  3. Can you report independent held-out results and quantify identity preservation, state control, and scene-code leakage?
3 comments

Thank you for your detailed review comments. Below, we clarify our contributions and present the experiments you requested.

1. What do pretrained priors provide?

Thank you for requesting this comparison. We incorporate pretrained priors because they offer useful spatial and category-related information. Our contribution lies in leveraging this information to link object observations to a content bank derived from the entire dataset, and then training on this retrieved content to enable compositional decoding. Instead of using local appearance descriptors directly, we feed the retrieved content into the object decoder; this allows the same representation to generalize across different observations and scene configurations.

Comparison with direct clustering. We have added a K-means-based baseline, using DINOv2 region descriptors, teacher model descriptors cached during the GOLP training process, and a concatenated feature vector (formed by normalizing and combining both). The K-means model is fitted solely on the observations from the training set. In these experiments, we use the same SAM3 object proposals and the same number of categories, with one-to-one matching between proposals and ground truth (GT) objects defined by an IoU threshold of ≥ 0.3.

Table R1. Object matching accuracy (%), with label alignment performed via the Hungarian algorithm during evaluation. “Coverage” refers to the proportion of ground truth (GT) objects that are successfully matched to a shared proposal. For the clustering methods, a random seed of 0 is used; for GOLP, the original reference checkpoint is employed. | Dataset | DINOv2 | Teacher | Joint Features | GOLP | Coverage | | :--- | ---: | ---: | ---: | ---: | ---: | | OCTB | 52.69 | 74.33 | 72.13 | 60.88 | 86.01 | | GSO | 82.16 | 91.08 | 90.38 | 96.24 | 91.61 | | YCB | 48.03 | 60.59 | 65.27 | 54.68 | 83.37 | | COCO | 17.80 | 16.17 | 19.40 | 19.72 | 65.16 | | VOC | 30.53 | 38.38 | 37.02 | 40.53 | 59.17 | |

What can we learn from this comparison? Across all five reported comparisons, our method outperforms direct clustering of DINOv2 features in terms of "object matching accuracy." "Teacher clustering" performs best on the OCTB dataset, while "joint feature clustering" yields the best results on the YCB dataset. GOLP achieves the highest scores on the GSO, COCO, and VOC datasets; however, the 96.40% accuracy observed on the GSO dataset is comparable to the 96.24% achieved by GOLP. Therefore, we do not claim to consistently outperform the direct clustering of the same pre-trained descriptors in terms of recognition performance.

These strong clustering results justify our use of such prior information: our goal is to leverage the category-related knowledge already embedded within these priors, rather than learning everything from scratch. We utilize SAM3 regions to organize object observations, leveraging pre-trained regional semantics to guide the association of these observations with shared content. It should be clarified that the performance gains we achieve—relative to methods employing different pre-trained inputs—reflect the collective advantages of our entire "prior-guided framework," rather than being attributable solely to the "bank learning" component.

Why our objective extends beyond mere clustering. Our goal is to learn reusable object content that facilitates cross-scene recognition, object discovery, reconstruction, and the editing or generation of specific objects. While cluster assignment addresses the question of "which observations belong to the same object," we go further by training the model to decode the associated content based on varying object states. We do not dismiss the utility of clustering as a baseline, nor do we suggest that fixed clustering results cannot be integrated with a decoder. Instead, we distinguish between "identity grouping" and "content learning" geared toward downstream compositional applications. Consequently, when evaluating our global representations, we assess not only recognition accuracy but also their practical utility across a range of scene-level operations.

2. Does joint reconstruction lead to improvements in recognition performance?

Our purpose in introducing the reconstruction task is to learn decodable object content, rather than to use it as an auxiliary objective for classification. While identity learning associates observations with shared representations, it does not inherently dictate how these representations—combined with object states—should be used to reconstruct appearance and shape. Through a joint reconstruction task, we utilize an object feature decoder and a mask decoder to learn this mapping. This is precisely the function of the "reconstruction path" in our design.

Regarding your question about recognition performance, we compared the full GOLP model against a "content-gradient-stop" variant. In this variant, we truncate the gradients of the extracted content before it reaches the object decoder. This prevents the reconstruction process from updating the "shared content" branch, while still allowing reconstruction-based training for other factors and the decoders themselves.

Table R2: Recognition accuracy (%) with and without reconstruction feedback. The ACC metric counts unmatched ground-truth (GT) objects as errors. These results are based on newly trained models (using a random seed of 0) and differ from the original checkpoints and "matched-object" metrics reported in Table R1.

DatasetFull GOLPContent-Gradient-Stop Variant
OCTB48.1649.74
GSO72.9082.80
YCB45.3847.23

In these experiments, we did not observe any improvement in recognition performance resulting from reconstruction feedback. Consequently, we do not characterize "joint reconstruction" as a method that further enhances recognition capabilities beyond semantic identity learning. However, it is important to distinguish between the "feedback mechanism" tested here and the "reconstruction task" itself: even with gradient truncation, the variant still employs the reconstruction task to train its decoders. The relatively high ACC (accuracy) results actually raise a question: should the "shared content" branch also be updated during the reconstruction process? This does not imply that visual decoding relationships can be learned solely through identity assignments.

Why reconstruction remains part of our approach. The global representations we require serve a purpose that extends beyond merely identifying observed objects. In Figure 3, we leverage these representations—in conjunction with an object decoder—to remove or insert specific objects. In Figure 4, by extracting and instantiating these representations, we generate scenes containing objects of specific categories and quantities. To perform such operations, the model must learn how the selected content and its state contribute object features and masks to the scene. The reconstruction process serves as explicit training for this mapping relationship.

Consequently, our core logic lies in learning a representation that simultaneously supports identification, decoding, and reuse within a single framework. We do not interpret a "lack of improvement in ACC" as an "improvement in generation quality" (nor was the latter measured in Reviewer 2’s assessment); rather, we distinguish between the role of reconstruction feedback in "identification" and its role in training "compositional decoding"—the mechanism that enables editing and generation operations.

3. Does identity information originate from a prototype or from other encodings?

We employ a "global prototype" to represent the shared content of an object, while using "state/scene codes" to modulate how that object appears within a specific observation. To investigate this division of labor, we replaced the "soft mixture" with a single predicted prototype, gc^g_{\hat c} (where c^=arg⁡max⁡cyc\hat c=\arg\max_c y_c). Subsequently, while keeping this prototype fixed, we swapped the state and scene codes across different scenes.

Table R3: Identity consistency (%) relative to predictions from the original input. We compare the identity predicted based on the "reconstructed object" against the identity predicted based on the "original observation." The 100% consistency for the input data itself arises from self-comparison and does not reflect actual recognition accuracy.

ConditionOCTBGSOYCB
Original Input Baseline100.00100.00100.00
Soft Reconstruction67.8781.2944.84
Single-Prototype Reconstruction67.5781.6546.10
Single-Prototype + Cross-Scene State/Scene Codes47.1567.9942.07

Substantial identity consistency is maintained even without the use of a soft mixture. Replacing the mixture mechanism with a single prototype results in consistency metric changes of only -0.30, +0.36, and +1.26 percentage points on the OCTB, GSO, and YCB datasets, respectively. In this setup, all observations categorized under the same identity are assigned an identical content vector; in other words, we no longer encode observation-specific variations through distinct mixing weights. Nevertheless, we maintain a largely consistent overall identity. This directly addresses concerns regarding Equation (4): the observed reconstruction consistency does not rely on the specific combination of prototypes used for a given observation. Even if this disentanglement is not absolute, we are still able to reuse the fixed content when other encoding components vary. After swapping "state encodings" and "scene encodings" across different scenarios, we find that identity agreement reaches 47.15% on the OCTB dataset, 67.99% on the GSO dataset, and 42.07% on the YCB dataset. We designed this intervention experiment to test whether the same content representation could be maintained without retaining the original state and scene conditions. The consistency observed in these experiments indicates that cross-scenario content reuse is feasible to a certain extent—meaning that individual prototypes need not be strictly confined to their original encoding combinations.

The decline in consistency following these swaps is also noteworthy; it implies that the degree to which identity information is preserved varies depending on the conditions. Consequently, we argue that these results support the conclusion that "shared object content" and "observation-dependent attributes" have been, to some degree, successfully disentangled.

4. Model Selection, Data Partitioning, and Failures in Benchmarking.

Model selection is performed based on identification accuracy (ACC) on the validation set. In the "Stage-1" control experiments, we select the result that yields the highest ACC on a "fixed-candidate" set within the validation data. Dictionary decoders are selected based on their reconstruction loss during the calibration phase, rather than their performance on intervention tasks.

Regarding the OCTB/COCO benchmark results, the original "test partition" was divided into non-overlapping validation and evaluation subsets—the former used for checkpoint selection, and the latter for reporting final results. In Appendix A.5, these subsets were conflated with the original test set (which contains the same collection of images); we will clarify this distinction in a subsequent revision and provide specific details regarding the partitioning. For the GSO and YCB datasets, the original test partitions were maintained. Furthermore, the validation images used in the OCTB RGB intervention experiments consist of historical data excluded from the evaluator calibration process; consequently, these experiments serve as mechanism diagnostics rather than as entirely independent, novel tests. The internal processing pipeline for COCO/VOC also deviates from the standard procedures prescribed by the official benchmark protocols.

Finally, regarding the "collapsed" runs observed for GOLD/CSGO on the YCB dataset, we do not attribute these to inherent failures of the methods themselves. The aforementioned "same-input control" experiments provide a basis for comparison that does not rely on these specific reproduction outcomes. Additionally, we have retained the results for MUFASA, which demonstrate superior performance on the COCO segmentation task and the VOC overlap metrics. These results constrain the scope of our conclusions concerning segmentation capabilities, while the separate "identity" and "intervention" tests are designed to investigate whether shared content can be recognized and reused across different contexts.

Reviewer_ctwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose GOLP, a global object-centric framework that uses frozen SAM3 masks as spatial priors and frozen DINOv2 regional descriptors as category-related priors. Object observations are assigned to a learnable global object bank, while object states, background, and scene attributes are encoded separately for compositional feature reconstruction. A second stage trains a Transformer adapter to decode the learned representations through a frozen RAE. The method is evaluated on cross-scene recognition, object discovery, editing, and compositional generation.

Critical strengths
  • The motivation of learning reusable cross-scene object content, rather than only within-image object grouping, is relevant to object-centric representation learning.
  • The use of complementary SAM3 spatial priors and DINOv2 semantic priors to guide global identity assignment is reasonable.
  • Cross-scene recognition results on GSO, OCTB, and YCB are strong, and the qualitative editing and generation examples illustrate the intended content-state separation.
Critical weaknesses
  • Technical novelty is limited. The method primarily combines existing components—foundation-model region proposals, DINOv2 features, prototype banks, Gumbel-Softmax assignment, variational attribute encoders, compositional reconstruction, and a pretrained RAE decoder—into a new pipeline.
  • The framework depends heavily on frozen SAM3 and DINOv2 outputs. It remains unclear how much of the improvement comes from the learned global object bank rather than from the strong pretrained priors already providing object regions and category-relevant descriptors.
  • Object discovery results are mixed. Although GOLP achieves the best foreground ARI on several datasets, it underperforms GOLD and CSGO on GSO in ARI-A, mBO, and mIoU. On COCO (Table 6, Appendix A.6), MUFASA outperforms GOLP on all reported metrics, and on VOC MUFASA retains a substantial overlap advantage.
  • Editing and generation are evaluated only qualitatively. Without quantitative measures of editing consistency, identity preservation, background preservation, or generation fidelity, the claimed compositional capabilities are difficult to assess.
  • Ablations are limited. The mask-supervision analysis is descriptive rather than controlled, and there is no matched ablation isolating the contributions of SAM3 masks, DINOv2 descriptors, the global bank, semantic guidance, and the RAE decoder.
  • Scalability to cluttered, open-vocabulary, or many-object scenes is unclear.
Questions for authors
  • How much of the recognition advantage remains if the learned global object bank is removed and DINOv2 descriptors from SAM3 masks are directly clustered?
  • On GSO, why does GOLP have the best ARI-O but lower ARI-A, mBOc, and mIoUc than GOLD and CSGO? Is this due to background leakage, inaccurate object extent, or SAM3 proposal errors?
  • Can the authors provide quantitative evaluation for editing and generation, such as object-count accuracy, identity preservation, background preservation, or image fidelity?
  • How sensitive are the results to SAM3 prompt selection, candidate truncation, and mask quality? The appendix describes frequent candidate-cap saturation.
  • Can the authors provide results of controlled ablations for SAM3 masks, DINOv2 descriptors, the global object bank, semantic guidance, and the RAE decoder under matched training and evaluation settings?
4 comments

Thank you for your detailed review comments. Below, we clarify our contributions and present the experiments you requested.

1. Contributions Beyond Pre-trained Components.

Our contribution is not the proposal of a new segmentation backbone or clustering algorithm. Instead, we leverage regional priors to link "object observations" to a dataset-wide "content bank." We then use the retrieved content—rather than local appearance descriptors—as input for compositional decoding. This approach allows us to associate different observations with the same underlying content, even when an object’s state changes or its presence within a scene varies. Unlike "within-image object grouping," our method enables content sharing across different scenes; unlike "direct clustering of regional descriptors," our approach also involves learning how to decode and manipulate the retrieved content. We do not claim that the concepts of a "shared content bank" or "generative object representation" are, in themselves, entirely novel.

We have added the "direct clustering" baseline experiment you requested. In this setup, K-means is fitted exclusively on the observations obtained during the training phase. For all methods, the candidate objects, the number of clusters, and the logic for one-to-one matching between candidate objects and ground truth (GT)—based on an IoU threshold of ≥ 0.3—remain consistent.

Table R1. Accuracy (%) for matched objects, with label alignment performed using the Hungarian algorithm during evaluation. "Coverage" refers to the proportion of ground truth (GT) instances that were successfully matched.

DatasetDINOv2TeacherJoint featuresGOLPCoverage
OCTB52.6974.3372.1360.8886.01
GSO82.1691.0890.3896.2491.61
YCB48.0360.5965.2754.6883.37
COCO17.8016.1719.4019.7265.16
VOC30.5338.3837.0240.5359.17

On the OCTB and YCB datasets, the results from stronger "teacher" models or joint baselines indicate that these priors contain a significant amount of category (identity) information. On the GSO dataset, the "teacher" model achieves an accuracy of 96.40%, which is comparable to the 96.24% accuracy of GOLP. We incorporate these priors precisely to leverage the category information they contain, thereby enhancing the model's ability to learn global object representations. Compared to previous methods for learning global object representations, GOLP achieves a notable improvement in cross-scene recognition performance specifically by utilizing the prior category information embedded within SAM3 and DINOv2 features. The goal of learning global templates (i.e., global object libraries) is to obtain reusable, invariant object representations—enabling capabilities such as cross-scene object recognition and the editing or generation of scenes involving specific objects—rather than to perform object clustering.

2. GSO Grouping and Overlap; Comparison with MUFASA

We investigated whether the GSO gap stems from "proposal misses." Under the same matching criteria, SAM3 candidate proposals cover 91.61% of the ground truth (GT) objects, whereas our decoded masks cover 68.82%; thus, there is an additional loss in coverage during the decoding process. While the ARI-O metric evaluates assignments specifically at GT foreground pixels, the overlap metric also penalizes the predicted spatial extent of objects. This accounts for potential discrepancies in rankings across different metrics, though it does not justify the lower overlap scores. We maintain the superior performance of GOLD/CSGO on the GSO task, alongside MUFASA’s results on the COCO/VOC tasks. Our central claim is that these representations facilitate cross-scene identity recognition and composition, rather than asserting that we have achieved state-of-the-art segmentation performance. Single-image ARI and matching overlap metrics fail to capture cross-scene identity consistency—as these metrics remain invariant even if object labels are permuted independently within each individual image.

3. Quantitative Editing and Generation.

We have added two evaluations: identity agreement when decoding conditions are altered, and the accuracy of object categories and counts in the final RGB output.

We replaced the "soft mixture" with a single predicted prototype, and swapped state codes and scene codes across different scenes while keeping the prototype fixed.

Table R2: Identity agreement with the original input prediction (%). The "original input" serves as the reference baseline, with its agreement defined as 100% (this does not represent accuracy relative to ground-truth labels).

ConditionOCTBGSOYCB
Original Input (Baseline)100.00100.00100.00
Soft Reconstruction67.8781.2944.84
Single-Prototype Reconstruction67.5781.6546.10
Single-Prototype + Cross-Scene State/Scene Codes47.1567.9942.07

Even after removing the "soft mixture," we maintained a high level of consistency, with a maximum difference of only 1.26 percentage points. While cross-scene swapping reduces agreement, the fact that fixed prototypes can still be reused under certain conditions suggests that GOLP is, to some extent, capable of disentangling invariant object properties, variant object properties, and scene-state attributes.

Quantitative Evaluation of Editing and Generation. Following your suggestion, we have incorporated quantitative tests designed to assess whether the generated images contain the specified objects (and the correct number thereof) and whether the editing operations have inadvertently altered the surrounding content. Our objective extends beyond merely reconstructing a recognizable scene; rather, we aim to leverage learned object representations to modify or combine elements as requested. Consequently, we evaluate the final RGB output directly, rather than inferring task success based on the identity codes or the number of slots provided to the decoder.

We employ a predefined control protocol and utilize an independently calibrated dense-DINO detector for evaluation—a detector that, during operation, has no access to the generated masks, identity codes, or the expected object counts. To qualify as a "perfect category/count match," the output image must contain exactly the requested categories in their specified quantities (including multiple instances of the same category).

Table R3. Quantitative Evaluation Results for Editing and Generation. The "perfect category/count match" metric is assessed based on 64 test cases, which encompass content/state replacement, translation, scaling, addition, and the combination of three objects. The Count Mean Absolute Error (Count MAE) and the Non-target L1 Error (Non-target L1) are evaluated across the full set of 96 test cases, which also include deletion operations and interventions at the scene-code level. | Dataset | Exact Category/Count Matches (out of 64 cases) ↑ | Mean Absolute Error (MAE) in Count ↓ | L1 Error in Non-Target Regions ↓ | | :--- | ---: | ---: | ---: | | OCTB | 4/64 (6.25%) | 1.927 | 0.02474 | | GSO | 49/64 (76.56%) | 0.177 | 0.01598 | | YCB | 0/64 (0.00%) | 1.052 | 0.04871 | |

Can we leverage these representations to achieve specified combinations? On the GSO dataset, we achieved an exact match for both category and count in 49 out of 64 (76.56%) cases. Notably, this includes 9 successful instances out of 11 for content replacement, 9 out of 10 for object addition, and 6 out of 10 for three-object combinations. These quantitative results demonstrate that the learned representations can be used not only to assign identities to observed objects but also to support operations based on category and count constraints, as illustrated in Figures 3–4.

Can we control the number of objects without altering irrelevant content? On the full GSO dataset, we obtained a Mean Absolute Error (MAE) of 0.177 objects per case. This metric evaluates the count independently of category accuracy; our "exact category/count" criterion, by contrast, requires the object identities to be entirely correct. We report both metrics because a result with the correct count but incorrect object categories cannot be considered a success. The results indicate that the proposed GOLP method is effective at controlling the number of objects while leaving irrelevant content unchanged.

Regarding your question about background preservation, we compared the outputs in regions outside the edited area. On the GSO dataset, the L1 error for non-target regions is 0.01598; for the OCTB and YCB datasets, the corresponding values ​​are 0.02474 and 0.04871, respectively. This indicates that the proposed GOLP is effective at preserving the background.

4. The Role of Prompts, Semantic Guidance, and Reconstruction.

Choice of Prompts. We employ fixed "object-level" prompts rather than category lists tailored to specific images. This approach allows us to obtain candidate regions without requiring object labels for each individual image. We use these regions to define our observations, upon which GOLP learns shared "object identities" and "content." In Appendix A.8, we compare the quality of candidate regions generated by different prompts, as the choice of prompt directly determines the scope of regions available for learning. We provide this comparison not to suggest that the final model is insensitive to variations in prompts.

Rationale for Integrating Semantic Guidance and Reconstruction. Thank you for this question; it allows us to clarify the distinct contributions of these components. We incorporate "semantic guidance" to link observations to shared object identities, while "reconstruction" is employed to learn how to generate visual objects from the extracted content and object states. In short, semantic priors are introduced to enhance the clustering performance of GOLP, whereas reconstruction enables the learning of a reusable global object library (acting as cluster centers), which facilitates scene editing and the generation of specific objects.

Can learning the content (i.e., feature representations) yield performance gains when the clustering (i.e., identity assignments) remains fixed? To address this, we compare two scenarios: one using a fixed "teacher dictionary," and another using a dictionary that begins with the same initial state but has its entries continuously updated through a reconstruction process—all while keeping the original identity assignments unchanged. Throughout this process, the decoder is trained consistently, ensuring that the comparison focuses solely on the adaptive optimization of the "shared content," rather than on changes to the clustering strategy or the decoder architecture.

Table R5: Percentage (%) of cases—out of 96 RGB control scenarios—where the model’s output perfectly matches the target object identities and quantities. All experimental conditions utilize the same decoder architecture, initialization method, training schedule, and calibration-loss selection. For the pre-trained, fixed GOLP dictionary, the cost of representation pre-training must also be taken into account.

Dictionary TypeOCTBGSOYCB
Fixed Teacher Dictionary (Fixed Clustering)19.7978.1314.58
Learnable Teacher Dictionary (Fixed Clustering)27.0870.8323.96
Fixed Joint-Feature Clustering17.7175.0017.71
Pre-trained GOLP Dictionary (Fixed)22.9267.7114.58

We have improved the performance of compositional outputs without altering the identity assignments. Following the update to the teacher dictionary, the proportion of instances where both the category and the count are perfectly matched increased from 19.79% to 27.08% on the OCTB dataset, and from 14.58% to 23.96% on the YCB dataset—representing gains of 7.29 and 9.38 percentage points, respectively. Since we kept the category assignments fixed, these improvements do not stem from a reclassification of the observed samples. This highlights the additional benefit of what we term "content learning": by refining the shared representations, the decoder becomes more capable of reliably expressing the target objects, rather than using these representations solely to distinguish between different observed samples.

Clarification regarding RAE. In response to your request for an evaluation of the RAE, we provide the following clarification: Before introducing the image decoding stage, we first learn identity information and object masks during Stage 1. Subsequently, we freeze the Stage 1 model and the pre-trained RAE, training only the connecting adapters. Therefore, the differences in recognition performance shown in Table R4 are not attributable to the RAE.

I thank the authors for the responses and for the additional experimental results. I believe that the work could still be better repositioned to deeply emphasise the novelty beyond powerful pre-trained components, and thus retain my score at this point.

Reviewer_BTaLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GOLP uses SAM3 regions and pretrained visual features to learn a global object bank. It combines the retrieved content with separate state, scene, and background variables for reconstruction and editing. Experiments evaluate cross-scene recognition, object discovery, and scene manipulation.

Critical strengths
  1. Table 1 shows large recognition gains across GSO, OCTB, and YCB. The object-discovery results on OCTB and YCB are also encouraging.

  2. The same object representations support recognition and scene editing. Figures 3–5 show useful examples of object removal, insertion, and state changes.

Critical weaknesses
  1. It is unclear how much of the improvement comes from the proposed learning method versus stronger pretrained inputs. A baseline clustering the same SAM3-region DINOv2/SAM3 features is missing. The mask-supervision study does not resolve this: Appendix A.7 states that the runs differ in training budgets and selection/evaluation procedures.

  2. The evidence for content–state separation is limited. Eq. (4) allows different prototype mixtures for the same predicted identity, and the full-scene code also enters each object decoder. Both could carry observation-specific content. The six swap examples in Appendix A.3 are insufficient to establish where identity information is stored.

Questions for authors

1.How does GOLP compare with clustering the same regional features? How much does joint reconstruction improve recognition under matched settings?

2.Does identity preservation hold when decoding uses a single prototype and state/scene codes are exchanged across scenes?

3.How were checkpoints selected? Since OCTB and COCO are evaluated on validation partitions, are independent test results available?

1 comment

Thank you for your thorough review. We have added the requested analyses: "clustering comparisons based on identical features" and "single-prototype testing."

1. Clustering and Reconstruction Based on Identical Features.

We applied K-means clustering to the training observations using the following features: DINOv2 region descriptors, teacher descriptors (derived from the SAM3 preprocessing used during GOLP training), and a concatenated feature vector formed by normalizing and combining both. All methods utilized the same set of SAM3 candidate objects, employed an identical number of clusters, and followed a "one-to-one matching" protocol between candidate objects and ground truth (GT) instances (IoU ≥ 0.3).

Table R1. Accuracy (%) for matched objects; the Hungarian algorithm was used for label alignment during evaluation. "Coverage" refers to the proportion of ground truth (GT) objects that were successfully matched.

DatasetDINOv2TeacherJoint FeaturesGOLPCoverage
OCTB52.6974.3372.1360.8886.01
GSO82.1691.0890.3896.2491.61
YCB48.0360.5965.2754.6883.37
COCO17.8016.1719.4019.7265.16
VOC30.5338.3837.0240.5359.17

As shown in Table r1, using "Teacher" descriptors yields the best clustering performance on the OCTB and YCB datasets, while DINOv2 features perform the worst. On the GSO, COCO, and VOC datasets, our proposed GOLP method achieves the best clustering results. This indicates that the priors from SAM3 descriptors and DINOv2 features indeed contain useful categorical information—the primary reason we utilize both as inputs for our encoder. Our objective is to leverage these two types of priors—which offer strong categorical information and spatial object awareness (via SAM3 segmentation results)—to enhance clustering and object discovery, thereby facilitating the learning of reusable, globally invariant object representations. Furthermore, it should be noted that the goal of the GOLP method is not merely to achieve superior clustering; rather, by learning global object representations, it enables cross-scene object recognition, targeted scene image editing and generation, and unsupervised object discovery. Can reconstruction improve recognition performance? It should be noted that the scene reconstruction structure in GOLP is not designed to enhance recognition performance; rather, it aims to learn meaningful global object representations that can be reused for scene editing and generation. Without the reconstruction module, there would be no guarantee that the learned global object representations could generate the appearance and shape of specific objects.

2. Single Prototype and Cross-Scene Swapping.

We directly addressed concerns regarding Equation (4) by replacing each mixture component in the equation with garg⁡max⁡cycg_{\arg\max_c y_c}. We then held this single prototype fixed while randomly reassigning state codes and scene codes across different scenes.

Table R3: Consistency in identity prediction between input objects and reconstructed objects (%). Note that the 100% value for "Original Input Reference" represents "self-agreement" rather than accuracy relative to ground-truth labels.

ConditionOCTBGSOYCB
Original Input Reference100.00100.00100.00
Soft Reconstruction67.8781.2944.84
Single Prototype Reconstruction67.5781.6546.10
Single Prototype + Cross-Scene State/Scene Codes47.1567.9942.07

As shown in Table R3, employing a single prototype mechanism yields an overall identity consistency comparable to that of soft reconstruction. This indicates that we can achieve essentially the same level of identity consistency even without encoding "observation-specific variation" into the mixture weights. In the case of cross-scene swapping, while overall consistency decreases, a degree of cross-scene identity consistency is still maintained; this suggests that GOLP effectively decouples invariant object properties, variable object properties, and scene state representations.

3. Checkpoint Selection and Independent Evaluation.

For model selection, we choose the checkpoint that achieves the highest cross-scenario accuracy (ACC) on the validation set. Regarding the OCTB/COCO benchmark results, we have partitioned the original "test" set into two non-overlapping subsets: one for validation and one for final evaluation. Checkpoint selection is based on the former, while the reported benchmark results are based on the latter. Our previous statement that "these two partitions are identical" was incorrect—while they originate from the same parent partition, they contain different sets of images. We will correct this description in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.