acceptodds
Under review as a conference paper at ICLR 2027

ROSE-3D: Render-Free Object-to-Scene Embeddings for Compositional 3D Representations

Abstract

3D scene representations are typically computed from the entire scene at a global level, rather than from reusable representations of the objects it contains. We argue that this constrains their capacity to express composition. A scene consists of objects placed in a shared frame, and the same objects recur across scenes; encoding each scene as a whole precludes reuse, making new arrangements of familiar objects as costly as unfamiliar ones. We treat composition as an operation in representation space: a learned operator maps object representations and their placements (translation, rotation, and scale) to a representation of the scene, without assembly or global encoding. To keep an object's content (shape and appearance) separable from its placement, we encode each 3D asset from its mesh and material maps without rendering, binding local surface orientation and appearance to continuous surface coordinates. We show that predicting a scene encoder's features can be satisfied without reading object content: the operator relies largely on layout and barely changes when objects are substituted. We therefore add two complementary objectives: identification, which requires the representation at each location to identify the object placed there, and language supervision, which aligns the composed representation with a description of the scene. With these objectives, the composed representation follows object content: it responds to substituted objects, including objects unseen during composer training, and after a replacement it shifts toward the new object. On question answering and situated next-step navigation, composed representations are competitive with point-map and point-cloud methods and outperform simpler representations built from the same inputs, while representing each new arrangement in milliseconds.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D scene representation, compositional representation learning, self-supervised learning, spatial semantic pointers, vector symbolic architectures, render-free 3D encoding, 3D question answering, joint-embedding predictive architecture

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SjQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ROSE-3D, a framework that composes reusable object representations into scene-level embeddings without re-encoding the entire scene. Each 3D asset is tokenized directly from its mesh and PBR material maps, using Spatial Semantic Pointers to bind surface normals and DINOv3 material descriptors to continuous surface coordinates. An asset encoder is pretrained on these tokens through JEPA-style masked feature prediction. A composition operator then combines frozen object representations and their placements into a fixed set of 64 latent tokens. The operator is trained to predict the frozen encoder's features of the assembled scene, with additional identification and language supervision to mitigate layout-based shortcuts. Experiments evaluate compositional behavior through controlled object interventions and downstream performance on ScanQA, SQA3D, and MSNN.

Critical strengths
  1. Well-motivated formulation and insightful shortcut analysis. The paper presents a clear and interesting formulation of scene composition using reusable object representations. It also identifies an important layout-based shortcut: when trained with feature prediction alone, the composition operator can rely largely on object layout while ignoring object content.

  2. Efficient scene composition. By separating object encoding from scene composition, the proposed framework can reuse object representations across different arrangements without rendering or re-encoding the entire scene. The reported composition time of 4.6 ms for a new arrangement of 24 previously encoded objects demonstrates its potential efficiency advantage, particularly when objects are repeatedly rearranged.

Critical weaknesses
  1. Potentially unfair benchmark and efficiency comparisons. ROSE-3D is evaluated on InternScenes reconstructions with known object assets and placements, whereas the baselines process original scans. These reconstructions are also included in the unlabeled training corpus of both the asset encoder and the composer, and the language model is additionally adapted using approximately 189k generated QA training pairs. These differences make the downstream comparisons in Table 1 difficult to interpret. Similarly, the reported 4.6 ms measures only the composition of previously encoded objects, while the baseline timings include perception costs. Therefore, the reported speedup does not establish an equivalent end-to-end efficiency advantage.

  2. Limited evidence of compositional correctness and benefits over simpler representations. The main experiments primarily evaluate composition through changes in latent representations under object substitutions, which do not necessarily demonstrate correct object–location correspondence. Furthermore, Table 5 shows that the composer's identification preference changes only marginally after owner permutations or two-object swaps, whereas re-encoding the scene produces clearer shifts. Table 6 also shows that simple pooling of object representations performs within 0.1–1.0 points of ROSE-3D across all reported metrics. These findings raise concerns about whether the learned composition operator associates object content with its placement and provides substantial benefits over simpler aggregation methods.

Questions for authors
  1. Can the authors evaluate the baselines on the same reconstructed scenes, or evaluate ROSE-3D with predicted object instances and retrieved assets from raw scans? Could they also report the cost of obtaining object assets and placements for a more comparable efficiency analysis?

  2. Why does the composer's object-identification preference change so little after owner permutations and two-object swaps? What limits its ability to associate object content with specific spatial placements?

  3. How does SSP-based tokenization compare with conventional positional encoding using the same surface normals and DINOv3 material descriptors under matched training settings?

8 comments

We sincerely thank the reviewer for their thoughtful review. Below, we address each concern.

W1 and Q1: Potentially...

Published methods on the same inputs: Following the reviewer’s suggestion, we evaluated 3D-VisTA on the InternScenes reconstructions used by ROSE-3D. It is fine-tuned from its released pretrained checkpoint with its released ScanQA recipe, using the reconstructed scenes and their known object instances as input. We chose 3D-VisTA among the methods in Table 1 because it provides released code, checkpoints and a fine-tuning recipe that could be run within the rebuttal period. On identical inputs, 3D-VisTA reaches 21.7 EM@1 on ScanQA, against 24.5 for ROSE-3D. Even when given the true category of every object, information ROSE-3D does not receive, it reaches 22.1. The advantage of ROSE-3D therefore does not come from the reconstructed inputs. On the original scans, 3D-VisTA reports 22.4 (Table 1), so ROSE-3D leads under either input, and the reconstructions are not an easier input for the baseline.

Controls on the same inputs: Appendix A.6 also compares representations built from the same reconstructions, using the same adaptation data and language-model pipeline:

  • Scene-level control: It encodes the assembled scene as a whole, as scan-based methods do. It scores 22.6 on ScanQA and 51.6 on SQA3D, against 24.5 and 52.8 for ROSE-3D.
  • Set transformer: A learned baseline over object tokens, matched to ROSE-3D in capacity (103.3M vs. 102.9M parameters), and objectives. It scores 22.6 and 51.4, against 24.5 and 52.8 for ROSE-3D.

Training exposure: To test whether exposure to the benchmark rooms affects the results, we retrained the composition operator with every reconstruction of the rooms in the benchmarks’ evaluation splits excluded (295 scenes, including rescans of the same rooms), and removed the generated QA pairs of those rescans from adaptation. Accuracy is unchanged (SQA3D 52.9 vs 52.8), as is prediction error on held-out scenes (0.347 vs 0.344). The asset encoder is self-supervised on asset geometry and materials, and never sees questions, answers or labels.

Adaptation data: The generated QA pairs are used only to adapt the language model and never enter the composition objective. ROSE-3D and all controls above are adapted with the same pairs, so the differences between them reflect the representation alone. With the same pairs and pipeline, the operator trained with prediction alone reaches 21.5 on ScanQA, against 24.5 with the full objective (Table 2).

Cost: As stated in Section 4.1, the 4.6 ms is the cost of each new arrangement of encoded objects, not an end-to-end cost from raw scans. In this setting, assets and placements are the input, as in simulation, layout generation and asset libraries. When scenes originate from raw scans, objects and placements are obtained once per scene, through instance segmentation (for example, 352 ms for Mask3D; Appendix D.5) followed by asset retrieval and placement estimation, a one-time step outside our measured protocol. After that, the costs are encoding each object once and composing each arrangement. Encoding 24 objects takes 0.54 s, and the cache requires 1.5 to 1.8 MiB per object. Including this encoding cost, composition becomes cheaper than re-encoding the assembled scene after about eight arrangements of the same objects. Under the same protocol (Appendix D.5, Table 10):

ObjectsROSE-3DSet TransformerPooled objectsPooled scene encoding
104.1 ms5.8 ms0.8 ms39 ms
254.5 ms5.9 ms1.0 ms82 ms
506.2 ms5.9 ms1.1 ms203 ms

Methods that reuse cached object features all run in milliseconds, while re-encoding the scene grows to 0.2s at 50 objects. Among the cached methods, ROSE-3D is the most accurate.

Q3: How does SSP...

We trained two asset encoders under identical settings, differing only in the tokenizer (Appendix A.7). One uses our SSP tokens, and the other uses conventional tokens that concatenate Fourier features of the patch coordinate with the same normal and DINOv3 descriptor. A composition operator was then trained on each.

SSP binding yields a clearly better composer. After an object is replaced by another of the same category, the shift toward the new object is about 1.8 times larger (0.211 vs 0.120), and the composed representation depends more strongly on object content. The gain starts in the encoder: with SSP tokens, objects of the same category are separated far better (0.120 vs 0.033).

We attribute this to the fact that concatenated tokens let the encoder pass its inputs through nearly unchanged. SSP binding instead spreads each attribute over a continuous spatial code, so the encoder must learn to separate content from position. This produces the instance-level features that composition relies on.

Continued: W2 and Q2: Limited evidence....

Appendix A.5 tests whether the composed representation identifies the right object at each location, not only whether it changes:

  • Unedited scenes: On held-out scenes, the identification readout prefers the true object over other objects in the scene at 0.74 of points (re-encoded scene: 0.79).
  • After replacing objects: The composed representation follows the edit in 99% of scenes. Its gap to its own unedited reference (0.094) matches that of re-encoding the scene (0.091), so replacement costs re-composition the same as a full re-encoding, at a fraction of the compute.

Figures 3 and 4 (Appendix A.5) visualize this on held-out scenes and after an edit.

Owner permutation and two-object swap: These edits keep the scene’s objects the same and only exchange their placements. The response to these edits reflects the granularity of placement conditioning: ROSE-3D conditions each object on its placement as a whole, whereas the re-encoded scene recomputes features from the assembled geometry at every point and so reflects a new arrangement directly. Edits that change the scene’s content, such as the compatible swap, are registered at the same relative cost as re-encoding (0.094 vs 0.091). An exchange of placements among the same objects leaves the set of composed contents unchanged and shifts the per-location readout less. We also evaluated two finer-grained forms of placement binding (Appendix A.5): explicit scene-frame coordinates, which weakened the reliance on content, and training on swapped scene pairs, which did not improve on the current design. The current conditioning therefore gives the best overall results, and per-location binding that preserves the reliance on content is the extension we discuss in Section 5.

Benefit over simpler representations: Among the controls in Appendix A.6 (W1), pooled objects mean-pools each object’s representation, with an embedding of its placement, into the same grid of cells as the composer’s latent queries. This fixed pooling is a strong control because it is built from our own asset encoder’s object features, so its strength reflects the quality of that encoder. A learned set transformer over the same object tokens, trained with our objectives at matched capacity, does not reach it (ScanQA 22.6 vs 24.0, SQA3D 51.4 vs 51.8), so learning over object tokens is not sufficient by itself. ROSE-3D is the only learned representation that improves over fixed pooling, on ScanQA (24.5 vs 24.0) and SQA3D (52.8 vs 51.8), with MSNN following the same ordering (Table 6). The gain therefore comes from how the operator composes objects.

Dear Reviewer, As the discussion period nears its end, please let us know if our response has sufficiently addressed your concerns.

Reviewer_KybDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method ROSE-3D to represent the 3d scene with multiple decoupled objecs and their correspondng placements rather previous global scene level. Specifically, ROSE-3D uses Spatial Semantic Pointers~(SSP) to represent the geometry and material appearance of objects and a learned operator maps multiple objects and placements to a 3d scene. Additionally, ROSE-3D uses a scene encoder matching, identification and language supervision as objective to improve the learned operator. At last, ROSE-3D is verified from the perspectives of both composition power and downstream tasks. ROSE-3D achieves comparable performance with point-map and point-cloud methods, but significantly speed up the process of new scene representation.

Critical strengths
  1. The motivation to improve the expressive power of various objects is valuable, which can further extend the quantity and quality of limited 3d scene assets. ROSE-3D decomposes the global 3d scene to objects and their placements to make objects resusable.
  2. ROSE-3D proposes a practical and effective method to implement the objects to scene operator, which make ROSE-3D achieve competetive performance with point-map and point-cloud methods and improve the efficience of building scene.
  3. The experiments including both composition and downstream tasks seems quite comprehensive.
Critical weaknesses
  1. My main concern is the generalization ability of ROSE-3D. It seems that both training and testing are done by extracting objects from the original scenes datasets and then combining them for testing. Would it still achieve good results if objects adopted from different scenes were combined?
  2. Besides the quantity of metrics, more visualization of 3d scenes obtained by the learned operator is necessary to be shown.
Questions for authors

Provide more objecte composition results from different scenes and scene visualizations.

2 comments

Dear Reviewer, As the discussion period nears its end, please let us know if our response has sufficiently addressed your concerns.

We sincerely appreciate your positive assessment and constructive feedback. We have provided additional evidence on composing objects from different scenes, showing that re-composition after replacing every object loses the same as re-encoding the scene, and two new figures visualizing the composed representation (Figures 3 and 4). We greatly value your time and look forward to any further feedback before the discussion period ends, so that we can address remaining concerns.

Reviewer_S77YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a compositional 3D representation which natively handles object manipulation, and the presence of multiple instances of the same object type. The method encodes 3D assets without rendering by tessellating a scene into small patches and encoding them into geometry tokens. The method evaluation on tasks like question answering shows competitive results compared to the state of the art.

Critical strengths

The overall idea of learning a compositional scene representation is very interesting for a variety of downstream tasks which likely benefit from this kind of built-in abstraction.

Critical weaknesses
  • Evaluation of scene representation vs. downstream task: A focus of the paper seems to be the learning of the compositional scene representation, the method evaluation, however, is focused on downstream tasks like 3D question answering. While this is a good indicator of overall performance, when only evaluating the downstream task it is hard to judge whether estimation errors are due to the task estimation module or the quality of the compositional scene representation. I think the authors can do a better job on evaluating the quality of the scene representation to judge this part better. For instance, on could compute precision/recall for the objects that where found in the scene and used in the compositional representation.

  • Qualitative Results Visualization: The results of the compositional scene structure are never shown qualitatively. I think it would be very useful to visualize the results of the compositional structure of the scene representation for a few example scenes to see which instances of a scene are actually covered under the same object type or under different object type, or if they are covered at all. From the pure numbers in the tables it is difficult to get an impression.

  • While I generally appreciate the investigation of other scene representations for various downstream tasks, one should also discuss and investigate the limitations of such a representation which is currently missing in the paper. It can be possible that task-relevant information cannot be well encoded into the compositional object-based representation and this information is then missing for the downstream task. A simple example could be the presence of a mirror in the scene which cannot be simply encoded into the compositional scene representation and questions regarding the number of objects in the scene would likely be incorrectly answered. Further limitations are likely regarding very large numbers of objects/instances in a scene, e.g. a large theater with hundreds of seats, as well as the usefulness of the representation in non-human-made environments or scenes with unknown objects.

Overall, I am on the edge regarding the rating of this submission. The compositional approach is very interesting (although also not invented by the authors as stated in the SSP section of the related work) and yields competitive results. On the other hand, the results are also not clearly better than existing methods, and since the "handcrafted" compositional architecture likely limits the generality - as indicated in my previous point - the advantages of the proposed approach are not obvious to me. Moreover, the method evaluation regarding the scene representation vs. downstream task outcome can be better evaluated.

Questions for authors
  • The Cook-Torrance light model (L238ff.) usually requires the position/direction of the light source to compute the shading of a face. How is the light source estimated? or is the light source given by the user, or just set up on some predefined location?

  • It would be helpful if the authors can comment on the limitations of their approach.

2 comments

Dear Reviewer, As the discussion period nears its end, please let us know if our response has sufficiently addressed your concerns.

We sincerely appreciate your constructive feedback. We have provided a direct evaluation of the representation without a language model, two new figures visualizing the composed representation (Figures 3 and 4), a limitations section addressing view-dependent content, large scenes, and unknown objects, and clarifications on novelty, generality, and the shading configuration. We greatly value your time and look forward to any further feedback before the discussion period ends, so that we can address remaining concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank all reviewers for their constructive feedback and for recognizing the motivation of composing reusable object representations (S77Y, KybD, SjQr), the insight of the layout-shortcut analysis (SjQr), the efficiency of composing new arrangements without rendering or re-encoding (KybD, SjQr), and the comprehensiveness of the experiments (KybD). In response to your comments, we have provided the following additional evidence and revisions:

  1. Comparison on the same inputs: 3D-VisTA, fine-tuned on the same InternScenes reconstructions, reaches 21.7 EM@1 on ScanQA (22.1 with true object categories) against 24.5 for ROSE-3D. A learned set transformer baseline at matched capacity and objectives also falls short (Appendix A.6).
  2. Training exposure: A composition operator retrained with every reconstruction of the benchmark evaluation rooms excluded matches the reported model in accuracy and prediction error (Appendix E.2).
  3. Correctness of composition and visualization: Two new figures show, over the surface of held-out scenes, which object the composed representation identifies at each location, and how it follows a scene in which every object is replaced by one from another scene (Figures 3 and 4, Appendix A.5).
  4. SSP tokenization: A matched comparison with conventional positional encoding shows that SSP binding yields a substantially stronger composition operator (Appendix A.7).
  5. Cost: We report cache size, the break-even point against re-encoding, and composition cost across scene sizes for ROSE-3D and alternatives that reuse cached object features (Appendix D.5, Table 10).
  6. Scope and limitations: A new limitations paragraph (Section 5) and a clarification of the shading configuration (Appendix C.1).

Our individual responses, posted on 9 November, provide detailed results and address the remaining methodological and evaluation concerns. We have not yet received replies from the reviewers, and we would welcome any further questions before the discussion period ends. We hope this summary assists the Area Chair in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.