- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes OctLLM, a unified LLM for 3D generation and understanding that uses Sparse Octree (S-Octree) tokens to explicitly preserve the spatial hierarchy and locality of 3D geometry. It introduces a dual-stream transformer that adds dedicated trainable 3D branches while keeping the pre-trained text–image pathway frozen, preserving general language capabilities. OctLLM achieves state-of-the-art performance among unified multimodal LLMs.
Critical strengths
-
The use of Octree structure provides an explicit and strong 3D geometric prior, which better regularise the representation by preserving spatial structure in an efficient way. The Sparse Octree (S-Octree) further reduces sequence length by removing fine-level details, which are subsequently recovered by a 3D U-Net.
-
A new network architecture design for integrating 3D representation (tokenisation and generation) with pre-trained LLMs. The token-routed dual-stream design introduces dedicated trainable 3D branches while freezing the pre-trained text–image pathway. Shared self-attention still enables cross-modal interaction. This offers a plausible solution to the trade-off between acquiring 3D modelling representation, and retaining LLM's general language ability.
Critical weaknesses
-
Loss of fine geometric details and reliance on completion. The paper uses a 3D-Unet shape completion to recover masked/missing details. This raises questions about whether fine geometric structures, thin parts, and high-frequency details can be faithfully reconstructed, rather than merely plausibly completed. There are a few qualitative visualisations shown in the paper and the appendix, but not sufficiently convincing. The method would benefit from a clearer analysis of the trade-off between sparsification, geometric detail preserving and high-frequency fidelity, and completion errors.
-
Several components build on prior work, including sparse 3D octree tokenisation ( conceptually similar to Trellis's active voxel), 3D RoPE, masked tokens completion using U-net, and flow-based 3D shape generation. The paper combines these components in a potentially effective architecture, but the methodological description does not make the novel contributions of each component sufficiently explicit.
In addition, besides the above two technical critiques, I feel the writing (language , prose) of the paper seems to be overly rhetorically polished but in a rather abstract way, while missing concrete analysis. For example, it repeatedly frames the method through contrasts and memorable formulations—for example, geometry entering as an “explicit 3D language” and adding capacity “without overwriting language ability”, but lacks concrete explanation of the underlying technical contribution. In another example, in the Abstract and introduction section, it use excessively highly symmetrical contrasts, overly polished but abstract phrases, and strong claims with limited qualification, which often obscures the critical technical argument. in short, the clarity of the technical exposition is lacking.
Questions for authors
-
In your section 3.1, the description of S-Octree, you move quickly from explaining random node removal, to preserved shape identity and improved efficiency. Can you please explain the conditions under which these expected benefits will be obtained ?
-
Similarly, the discussion of the dual-stream architecture repeatedly emphasises preserving language ability, such as describing the 3D object , as you show in figure S2. Can you be more direct in specifying which parameters in the LLM model are frozen, which are trainable, and how information flows between the streams? In my view, stronger ablations and comparisons would help establish which design choices are essential to the reported gains.
3 comments
Response to Reviewer 1Zkw (Part 2/2)
Q3: Writing clarity.
A3: We apologize that the original phrasing obscured the technical content, and we thank the reviewer for the candid feedback. In the abstract, introduction, and conclusion, we have replaced figurative phrasing with concrete mechanisms and numbers; all changes are marked in blue in the revised PDF.
Q4: Under what conditions do the S-Octree benefits hold?
A4: Each occupied node above the finest level contributes one byte, which encodes the occupancy of its 8 children. Emptying a penultimate-level node therefore removes exactly one byte, i.e., one 2×2×2 block of finest voxels. If a fraction of the occupied penultimate-level nodes is dropped, about of the tokens disappear, so the saving is about . For large or voluminous shapes, the finest level dominates and the saving approaches (50%). For thin or small shapes, coarse levels make up a larger share of , so the saving is smaller but still substantial. The table below shows six Toys4K objects with :
| Object | Full octree | S-Octree | Reduction | |
|---|---|---|---|---|
| Crate | 2,252 | 2,992 | 1,870 | 37.5% |
| Storage box | 3,652 | 4,612 | 2,790 | 39.5% |
| Box-shaped device | 3,212 | 4,214 | 2,613 | 38.0% |
| Ballpoint pen | 320 | 478 | 320 | 33.0% |
| Pencil | 595 | 867 | 571 | 34.1% |
| Thin pencil | 128 | 288 | 225 | 21.8% |
Shape identity is preserved because all coarser levels are kept, so the overall layout is unchanged and only fine 2×2×2 blocks are removed. Neighbouring blocks on a surface tend to have similar occupancy, which lets the U-Net infer the missing ones.
Q5: Which parameters are frozen or trainable, and how information flows.
A5: All Qwen2.5-VL-7B weights are frozen, including the vision encoder, word embeddings, language head, and the attention projections and FFNs of all 28 blocks. The trainable parameters (≈2.80B) consist of four parts: a copy of and the FFN in each of 12 routed blocks (layers 0, 4, 8, 12, 14, 16, 18, 20, 22, 24, 26, 27), initialized from that block; the embeddings of the 259 new tokens (256 occupancy bytes, <mesh_bos>, <mesh_eos>, and <MASK>); the mesh output head; and the depth embedding.
Routing depends only on token type. In the routed blocks, occupancy-byte and <MASK> tokens use the trainable copies (Eqs. 1–2), while text, image, <mesh_bos>, and <mesh_eos> tokens use only frozen weights; in the other 16 blocks, all tokens use frozen weights. Every block still keeps a single causal self-attention over the whole sequence. The text or image condition therefore reaches the 3D tokens during generation, and the 3D tokens reach the text tokens during captioning.
Each of these choices is isolated by an ablation. Replacing routing with LoRA + UltraChat loses 5.1 MMLU and 18.1 HellaSwag points (Table S5), and removing the mask, the 3D position, or mask suppression raises FID (Table S6). Branch placement and count are studied in Table S7 and Fig. S5, and full-rank branches are compared with routed LoRA in Table S8 and Fig. S6.
We hope these clarifications address your concerns, and we would be grateful if you would consider raising your score.
Dear Reviewer 1Zkw, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.
Response to Reviewer 1Zkw (Part 1/2)
We sincerely thank Reviewer 1Zkw for the careful reading. We also appreciate the recognition that the octree provides an explicit geometric prior and that the dual-stream design is a plausible solution to the capacity–preservation trade-off. All changes in the revised PDF are marked in blue.
Q1: Fine geometric details and reliance on completion.
A1: This is a fair concern, and S-Octree does trade some fine detail for a shorter sequence. However, this trade-off is shared by every LLM-compatible 3D representation; the real question is where to place it. OctLLM predicts structure at 64³, the same resolution as TRELLIS's structure stage and ShapeLLM-Omni's voxels, and in all three the TRELLIS decoder adds fine surface detail. At this resolution, OctLLM's completed voxels are closer to the ground truth than ShapeLLM-Omni's (CD-L2 0.0202 vs. 0.0273, F-score 43.53 vs. 36.58; Table 3).
We chose 64³ because each additional octree level multiplies the sequence length and memory by roughly 2–4×, and depth 6 is the deepest level whose training fits on our 96 GB H20 GPUs:
Within this budget, sparsification and completion do not lower quality. Under the same setup, training on full octrees gives FID 27.54, compared with 26.31 for S-Octrees (App. D), and completion improves every voxel metric of the raw S-Octree (e.g., IoU 5.60→9.75%, F-score 40.24→43.53%; Table 3). Dropping half of the penultimate level thus saves 24% of memory and 37% of epoch time without lowering generation quality. Structures thinner than one 2×2×2 block at 64³ rely most on completion (see Q4 and App. E).
Q2: Novelty of individual components.
A2: We thank the reviewer for this question and agree that the novelty of each component should be stated explicitly. OctLLM's core contributions are a new 3D representation for LLMs and a new way to add it to a pretrained backbone; the ideas borrowed from prior work are redesigned to serve these two designs.
(1) Explicit 3D inside an LLM. To our knowledge, OctLLM is the first multimodal LLM that generates and understands shapes through an explicit hierarchical 3D structure. Each 3D token is the occupancy of a concrete octree node whose coordinates and depth enter the positional encoding, so the LLM operates on the geometry itself. Prior 3D LLMs operate on proxies: VQVAE indices (ShapeLLM-Omni), VAE latents (CG-MLLM, EVA01), or OBJ coordinates written as digit strings (LLaMA-Mesh), all of which hide locality and hierarchy from the model. In the same model, octree tokens outperform VQVAE tokens (FID 33.10 → 26.31, Table S4), and as input they give the best captioning among unified MLLMs.
(2) S-Octree, a new sequence representation. Emptying half of the penultimate-level nodes shortens the sequence by 22–40% (see Q4) while every coarser level is kept, and a 3D U-Net restores the dropped blocks. Peak training memory falls from 58 GB to 44 GB and epoch time by 37%, with no loss in quality (FID 26.31 vs. 27.54 for full octrees). TRELLIS's active voxels are also sparse, but they form an unordered set generated jointly by a flow model, not a coarse-to-fine sequence that an LLM writes token by token and reads back as input.
(3) Token-routed dual-stream backbone. As the reviewer notes, this is a new design for adding 3D to a pretrained LLM. Existing 3D LLMs update the backbone and lose language ability (ShapeLLM-Omni: −34.5 MMLU, −40.7 HellaSwag). OctLLM keeps every pretrained weight frozen and gives 3D tokens full-rank branches with their own embeddings and head, so it matches the backbone exactly on MMLU and HellaSwag. Bringing this principle from 2D models (MoT, X-Fusion) to 3D required choices that we validate: branches in 12 of 28 blocks (Table S7) and full-rank branches instead of LoRA, which is unstable at full scale (Fig. S6).
Other components listed by the reviewer. Our 3D RoPE writes octree coordinates into Qwen2.5-VL's three existing M-RoPE channels, adding no positional parameters (w/o: FID 32.84 vs. 26.31, Table S6). Mask tokens are separate from U-Net completion: adapted from Uni-3DAR, they are position queries for AR generation and are evicted from the KV cache once used (w/o: FID 61.08). Finally, OctLLM itself is not flow-based. TRELLIS's second stage only decodes our 64³ structure into a mesh, as in ShapeLLM-Omni and AR3D-R1; we do not claim it, and Table 3 shows that our gain over ShapeLLM-Omni already exists before this stage.