- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ROSE-3D, a framework that composes reusable object representations into scene-level embeddings without re-encoding the entire scene. Each 3D asset is tokenized directly from its mesh and PBR material maps, using Spatial Semantic Pointers to bind surface normals and DINOv3 material descriptors to continuous surface coordinates. An asset encoder is pretrained on these tokens through JEPA-style masked feature prediction. A composition operator then combines frozen object representations and their placements into a fixed set of 64 latent tokens. The operator is trained to predict the frozen encoder's features of the assembled scene, with additional identification and language supervision to mitigate layout-based shortcuts. Experiments evaluate compositional behavior through controlled object interventions and downstream performance on ScanQA, SQA3D, and MSNN.
Critical strengths
-
Well-motivated formulation and insightful shortcut analysis. The paper presents a clear and interesting formulation of scene composition using reusable object representations. It also identifies an important layout-based shortcut: when trained with feature prediction alone, the composition operator can rely largely on object layout while ignoring object content.
-
Efficient scene composition. By separating object encoding from scene composition, the proposed framework can reuse object representations across different arrangements without rendering or re-encoding the entire scene. The reported composition time of 4.6 ms for a new arrangement of 24 previously encoded objects demonstrates its potential efficiency advantage, particularly when objects are repeatedly rearranged.
Critical weaknesses
-
Potentially unfair benchmark and efficiency comparisons. ROSE-3D is evaluated on InternScenes reconstructions with known object assets and placements, whereas the baselines process original scans. These reconstructions are also included in the unlabeled training corpus of both the asset encoder and the composer, and the language model is additionally adapted using approximately 189k generated QA training pairs. These differences make the downstream comparisons in Table 1 difficult to interpret. Similarly, the reported 4.6 ms measures only the composition of previously encoded objects, while the baseline timings include perception costs. Therefore, the reported speedup does not establish an equivalent end-to-end efficiency advantage.
-
Limited evidence of compositional correctness and benefits over simpler representations. The main experiments primarily evaluate composition through changes in latent representations under object substitutions, which do not necessarily demonstrate correct object–location correspondence. Furthermore, Table 5 shows that the composer's identification preference changes only marginally after owner permutations or two-object swaps, whereas re-encoding the scene produces clearer shifts. Table 6 also shows that simple pooling of object representations performs within 0.1–1.0 points of ROSE-3D across all reported metrics. These findings raise concerns about whether the learned composition operator associates object content with its placement and provides substantial benefits over simpler aggregation methods.
Questions for authors
-
Can the authors evaluate the baselines on the same reconstructed scenes, or evaluate ROSE-3D with predicted object instances and retrieved assets from raw scans? Could they also report the cost of obtaining object assets and placements for a more comparable efficiency analysis?
-
Why does the composer's object-identification preference change so little after owner permutations and two-object swaps? What limits its ability to associate object content with specific spatial placements?
-
How does SSP-based tokenization compare with conventional positional encoding using the same surface normals and DINOv3 material descriptors under matched training settings?
8 comments
Continued: W2 and Q2: Limited evidence....
Appendix A.5 tests whether the composed representation identifies the right object at each location, not only whether it changes:
- Unedited scenes: On held-out scenes, the identification readout prefers the true object over other objects in the scene at 0.74 of points (re-encoded scene: 0.79).
- After replacing objects: The composed representation follows the edit in 99% of scenes. Its gap to its own unedited reference (0.094) matches that of re-encoding the scene (0.091), so replacement costs re-composition the same as a full re-encoding, at a fraction of the compute.
Figures 3 and 4 (Appendix A.5) visualize this on held-out scenes and after an edit.
Owner permutation and two-object swap: These edits keep the scene’s objects the same and only exchange their placements. The response to these edits reflects the granularity of placement conditioning: ROSE-3D conditions each object on its placement as a whole, whereas the re-encoded scene recomputes features from the assembled geometry at every point and so reflects a new arrangement directly. Edits that change the scene’s content, such as the compatible swap, are registered at the same relative cost as re-encoding (0.094 vs 0.091). An exchange of placements among the same objects leaves the set of composed contents unchanged and shifts the per-location readout less. We also evaluated two finer-grained forms of placement binding (Appendix A.5): explicit scene-frame coordinates, which weakened the reliance on content, and training on swapped scene pairs, which did not improve on the current design. The current conditioning therefore gives the best overall results, and per-location binding that preserves the reliance on content is the extension we discuss in Section 5.
Benefit over simpler representations: Among the controls in Appendix A.6 (W1), pooled objects mean-pools each object’s representation, with an embedding of its placement, into the same grid of cells as the composer’s latent queries. This fixed pooling is a strong control because it is built from our own asset encoder’s object features, so its strength reflects the quality of that encoder. A learned set transformer over the same object tokens, trained with our objectives at matched capacity, does not reach it (ScanQA 22.6 vs 24.0, SQA3D 51.4 vs 51.8), so learning over object tokens is not sufficient by itself. ROSE-3D is the only learned representation that improves over fixed pooling, on ScanQA (24.5 vs 24.0) and SQA3D (52.8 vs 51.8), with MSNN following the same ordering (Table 6). The gain therefore comes from how the operator composes objects.
Dear Reviewer, As the discussion period nears its end, please let us know if our response has sufficiently addressed your concerns.
We sincerely thank the reviewer for their thoughtful review. Below, we address each concern.
W1 and Q1: Potentially...
Published methods on the same inputs: Following the reviewer’s suggestion, we evaluated 3D-VisTA on the InternScenes reconstructions used by ROSE-3D. It is fine-tuned from its released pretrained checkpoint with its released ScanQA recipe, using the reconstructed scenes and their known object instances as input. We chose 3D-VisTA among the methods in Table 1 because it provides released code, checkpoints and a fine-tuning recipe that could be run within the rebuttal period. On identical inputs, 3D-VisTA reaches 21.7 EM@1 on ScanQA, against 24.5 for ROSE-3D. Even when given the true category of every object, information ROSE-3D does not receive, it reaches 22.1. The advantage of ROSE-3D therefore does not come from the reconstructed inputs. On the original scans, 3D-VisTA reports 22.4 (Table 1), so ROSE-3D leads under either input, and the reconstructions are not an easier input for the baseline.
Controls on the same inputs: Appendix A.6 also compares representations built from the same reconstructions, using the same adaptation data and language-model pipeline:
Training exposure: To test whether exposure to the benchmark rooms affects the results, we retrained the composition operator with every reconstruction of the rooms in the benchmarks’ evaluation splits excluded (295 scenes, including rescans of the same rooms), and removed the generated QA pairs of those rescans from adaptation. Accuracy is unchanged (SQA3D 52.9 vs 52.8), as is prediction error on held-out scenes (0.347 vs 0.344). The asset encoder is self-supervised on asset geometry and materials, and never sees questions, answers or labels.
Adaptation data: The generated QA pairs are used only to adapt the language model and never enter the composition objective. ROSE-3D and all controls above are adapted with the same pairs, so the differences between them reflect the representation alone. With the same pairs and pipeline, the operator trained with prediction alone reaches 21.5 on ScanQA, against 24.5 with the full objective (Table 2).
Cost: As stated in Section 4.1, the 4.6 ms is the cost of each new arrangement of encoded objects, not an end-to-end cost from raw scans. In this setting, assets and placements are the input, as in simulation, layout generation and asset libraries. When scenes originate from raw scans, objects and placements are obtained once per scene, through instance segmentation (for example, 352 ms for Mask3D; Appendix D.5) followed by asset retrieval and placement estimation, a one-time step outside our measured protocol. After that, the costs are encoding each object once and composing each arrangement. Encoding 24 objects takes 0.54 s, and the cache requires 1.5 to 1.8 MiB per object. Including this encoding cost, composition becomes cheaper than re-encoding the assembled scene after about eight arrangements of the same objects. Under the same protocol (Appendix D.5, Table 10):
Methods that reuse cached object features all run in milliseconds, while re-encoding the scene grows to 0.2s at 50 objects. Among the cached methods, ROSE-3D is the most accurate.
Q3: How does SSP...
We trained two asset encoders under identical settings, differing only in the tokenizer (Appendix A.7). One uses our SSP tokens, and the other uses conventional tokens that concatenate Fourier features of the patch coordinate with the same normal and DINOv3 descriptor. A composition operator was then trained on each.
SSP binding yields a clearly better composer. After an object is replaced by another of the same category, the shift toward the new object is about 1.8 times larger (0.211 vs 0.120), and the composed representation depends more strongly on object content. The gain starts in the encoder: with SSP tokens, objects of the same category are separated far better (0.120 vs 0.033).
We attribute this to the fact that concatenated tokens let the encoder pass its inputs through nearly unchanged. SSP binding instead spreads each attribute over a continuous spatial code, so the encoder must learn to separate content from position. This produces the instance-level features that composition relies on.