acceptodds
Under review as a conference paper at ICLR 2027

Octrees as an Explicit 3D Language

Abstract

Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations. Geometry enters as an explicit 3D sequence of octree occupancy tokens. However, full octree sequences grow rapidly with depth; OctLLM therefore randomly empties penultimate-level nodes and omits descendants while preserving shape, yielding a shorter coordinate- and depth-anchored Sparse Octree (S-Octree) for position-aware mask-modeling generation and 3D understanding. On the other front, existing methods introduce a new modality with full fine-tuning or LoRA, but full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. OctLLM instead adds 3D capacity in parameters separate from the pretrained ones: mesh tokens are routed through independent trainable branches in a subset of blocks while text and image tokens retain the frozen vision-language pathway, and the two streams interact through shared self-attention. It trains far fewer parameters than full fine-tuning, yet sets a new state of the art among unified multimodal LLMs, lowering image-to-3D FID by and raising render-grounded captioning by points over ShapeLLM-Omni, while matching the backbone on general language benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D generation, 3D understanding, multimodal large language models, octree representation, autoregressive models, hierarchical 3D representation, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1ZkwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes OctLLM, a unified LLM for 3D generation and understanding that uses Sparse Octree (S-Octree) tokens to explicitly preserve the spatial hierarchy and locality of 3D geometry. It introduces a dual-stream transformer that adds dedicated trainable 3D branches while keeping the pre-trained text–image pathway frozen, preserving general language capabilities. OctLLM achieves state-of-the-art performance among unified multimodal LLMs.

Critical strengths
  1. The use of Octree structure provides an explicit and strong 3D geometric prior, which better regularise the representation by preserving spatial structure in an efficient way. The Sparse Octree (S-Octree) further reduces sequence length by removing fine-level details, which are subsequently recovered by a 3D U-Net.

  2. A new network architecture design for integrating 3D representation (tokenisation and generation) with pre-trained LLMs. The token-routed dual-stream design introduces dedicated trainable 3D branches while freezing the pre-trained text–image pathway. Shared self-attention still enables cross-modal interaction. This offers a plausible solution to the trade-off between acquiring 3D modelling representation, and retaining LLM's general language ability.

Critical weaknesses
  1. Loss of fine geometric details and reliance on completion. The paper uses a 3D-Unet shape completion to recover masked/missing details. This raises questions about whether fine geometric structures, thin parts, and high-frequency details can be faithfully reconstructed, rather than merely plausibly completed. There are a few qualitative visualisations shown in the paper and the appendix, but not sufficiently convincing. The method would benefit from a clearer analysis of the trade-off between sparsification, geometric detail preserving and high-frequency fidelity, and completion errors.

  2. Several components build on prior work, including sparse 3D octree tokenisation ( conceptually similar to Trellis's active voxel), 3D RoPE, masked tokens completion using U-net, and flow-based 3D shape generation. The paper combines these components in a potentially effective architecture, but the methodological description does not make the novel contributions of each component sufficiently explicit.

In addition, besides the above two technical critiques, I feel the writing (language , prose) of the paper seems to be overly rhetorically polished but in a rather abstract way, while missing concrete analysis. For example, it repeatedly frames the method through contrasts and memorable formulations—for example, geometry entering as an “explicit 3D language” and adding capacity “without overwriting language ability”, but lacks concrete explanation of the underlying technical contribution. In another example, in the Abstract and introduction section, it use excessively highly symmetrical contrasts, overly polished but abstract phrases, and strong claims with limited qualification, which often obscures the critical technical argument. in short, the clarity of the technical exposition is lacking.

Questions for authors
  1. In your section 3.1, the description of S-Octree, you move quickly from explaining random node removal, to preserved shape identity and improved efficiency. Can you please explain the conditions under which these expected benefits will be obtained ?

  2. Similarly, the discussion of the dual-stream architecture repeatedly emphasises preserving language ability, such as describing the 3D object , as you show in figure S2. Can you be more direct in specifying which parameters in the LLM model are frozen, which are trainable, and how information flows between the streams? In my view, stronger ablations and comparisons would help establish which design choices are essential to the reported gains.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Zkw (Part 1/2)

We sincerely thank Reviewer 1Zkw for the careful reading. We also appreciate the recognition that the octree provides an explicit geometric prior and that the dual-stream design is a plausible solution to the capacity–preservation trade-off. All changes in the revised PDF are marked in blue.

Q1: Fine geometric details and reliance on completion.

A1: This is a fair concern, and S-Octree does trade some fine detail for a shorter sequence. However, this trade-off is shared by every LLM-compatible 3D representation; the real question is where to place it. OctLLM predicts structure at 64³, the same resolution as TRELLIS's structure stage and ShapeLLM-Omni's voxels, and in all three the TRELLIS decoder adds fine surface detail. At this resolution, OctLLM's completed voxels are closer to the ground truth than ShapeLLM-Omni's (CD-L2 0.0202 vs. 0.0273, F-score 43.53 vs. 36.58; Table 3).

We chose 64³ because each additional octree level multiplies the sequence length and memory by roughly 2–4×, and depth 6 is the deepest level whose training fits on our 96 GB H20 GPUs:

DepthGridAvg. tokensMemory / GPU
6 (ours)64³~1,26044 GB (measured)
7128³~5,000~110 GB (est.)
8256³~20,000~0.4 TB (est.)
9512³~62,000~1.2 TB (est.)
101024³~137,000~2.7 TB (est.)

Within this budget, sparsification and completion do not lower quality. Under the same setup, training on full octrees gives FID 27.54, compared with 26.31 for S-Octrees (App. D), and completion improves every voxel metric of the raw S-Octree (e.g., IoU 5.60→9.75%, F-score 40.24→43.53%; Table 3). Dropping half of the penultimate level thus saves 24% of memory and 37% of epoch time without lowering generation quality. Structures thinner than one 2×2×2 block at 64³ rely most on completion (see Q4 and App. E).

Q2: Novelty of individual components.

A2: We thank the reviewer for this question and agree that the novelty of each component should be stated explicitly. OctLLM's core contributions are a new 3D representation for LLMs and a new way to add it to a pretrained backbone; the ideas borrowed from prior work are redesigned to serve these two designs.

(1) Explicit 3D inside an LLM. To our knowledge, OctLLM is the first multimodal LLM that generates and understands shapes through an explicit hierarchical 3D structure. Each 3D token is the occupancy of a concrete octree node whose coordinates and depth enter the positional encoding, so the LLM operates on the geometry itself. Prior 3D LLMs operate on proxies: VQVAE indices (ShapeLLM-Omni), VAE latents (CG-MLLM, EVA01), or OBJ coordinates written as digit strings (LLaMA-Mesh), all of which hide locality and hierarchy from the model. In the same model, octree tokens outperform VQVAE tokens (FID 33.10 → 26.31, Table S4), and as input they give the best captioning among unified MLLMs.

(2) S-Octree, a new sequence representation. Emptying half of the penultimate-level nodes shortens the sequence by 22–40% (see Q4) while every coarser level is kept, and a 3D U-Net restores the dropped blocks. Peak training memory falls from 58 GB to 44 GB and epoch time by 37%, with no loss in quality (FID 26.31 vs. 27.54 for full octrees). TRELLIS's active voxels are also sparse, but they form an unordered set generated jointly by a flow model, not a coarse-to-fine sequence that an LLM writes token by token and reads back as input.

(3) Token-routed dual-stream backbone. As the reviewer notes, this is a new design for adding 3D to a pretrained LLM. Existing 3D LLMs update the backbone and lose language ability (ShapeLLM-Omni: −34.5 MMLU, −40.7 HellaSwag). OctLLM keeps every pretrained weight frozen and gives 3D tokens full-rank branches with their own embeddings and head, so it matches the backbone exactly on MMLU and HellaSwag. Bringing this principle from 2D models (MoT, X-Fusion) to 3D required choices that we validate: branches in 12 of 28 blocks (Table S7) and full-rank branches instead of LoRA, which is unstable at full scale (Fig. S6).

Other components listed by the reviewer. Our 3D RoPE writes octree coordinates into Qwen2.5-VL's three existing M-RoPE channels, adding no positional parameters (w/o: FID 32.84 vs. 26.31, Table S6). Mask tokens are separate from U-Net completion: adapted from Uni-3DAR, they are position queries for AR generation and are evicted from the KV cache once used (w/o: FID 61.08). Finally, OctLLM itself is not flow-based. TRELLIS's second stage only decodes our 64³ structure into a mesh, as in ShapeLLM-Omni and AR3D-R1; we do not claim it, and Table 3 shows that our gain over ShapeLLM-Omni already exists before this stage.

AuthorsOpenReview ↗

Response to Reviewer 1Zkw (Part 2/2)

Q3: Writing clarity.

A3: We apologize that the original phrasing obscured the technical content, and we thank the reviewer for the candid feedback. In the abstract, introduction, and conclusion, we have replaced figurative phrasing with concrete mechanisms and numbers; all changes are marked in blue in the revised PDF.

Q4: Under what conditions do the S-Octree benefits hold?

A4: Each occupied node above the finest level contributes one byte, which encodes the occupancy of its 8 children. Emptying a penultimate-level node therefore removes exactly one byte, i.e., one 2×2×2 block of finest voxels. If a fraction pp of the N5N_5 occupied penultimate-level nodes is dropped, about pN5pN_5 of the NN tokens disappear, so the saving is about pN5/NpN_5/N. For large or voluminous shapes, the finest level dominates NN and the saving approaches pp (50%). For thin or small shapes, coarse levels make up a larger share of NN, so the saving is smaller but still substantial. The table below shows six Toys4K objects with p=0.5p=0.5:

ObjectN5N_5Full octreeS-OctreeReduction
Crate2,2522,9921,87037.5%
Storage box3,6524,6122,79039.5%
Box-shaped device3,2124,2142,61338.0%
Ballpoint pen32047832033.0%
Pencil59586757134.1%
Thin pencil12828822521.8%

Shape identity is preserved because all coarser levels are kept, so the overall layout is unchanged and only fine 2×2×2 blocks are removed. Neighbouring blocks on a surface tend to have similar occupancy, which lets the U-Net infer the missing ones.

Q5: Which parameters are frozen or trainable, and how information flows.

A5: All Qwen2.5-VL-7B weights are frozen, including the vision encoder, word embeddings, language head, and the attention projections and FFNs of all 28 blocks. The trainable parameters (≈2.80B) consist of four parts: a copy of WQ,WK,WV,WOW_Q, W_K, W_V, W_O and the FFN in each of 12 routed blocks (layers 0, 4, 8, 12, 14, 16, 18, 20, 22, 24, 26, 27), initialized from that block; the embeddings of the 259 new tokens (256 occupancy bytes, <mesh_bos>, <mesh_eos>, and <MASK>); the mesh output head; and the depth embedding.

Routing depends only on token type. In the routed blocks, occupancy-byte and <MASK> tokens use the trainable copies (Eqs. 1–2), while text, image, <mesh_bos>, and <mesh_eos> tokens use only frozen weights; in the other 16 blocks, all tokens use frozen weights. Every block still keeps a single causal self-attention over the whole sequence. The text or image condition therefore reaches the 3D tokens during generation, and the 3D tokens reach the text tokens during captioning.

Each of these choices is isolated by an ablation. Replacing routing with LoRA + UltraChat loses 5.1 MMLU and 18.1 HellaSwag points (Table S5), and removing the mask, the 3D position, or mask suppression raises FID (Table S6). Branch placement and count are studied in Table S7 and Fig. S5, and full-rank branches are compared with routed LoRA in Table S8 and Fig. S6.

We hope these clarifications address your concerns, and we would be grateful if you would consider raising your score.

Dear Reviewer 1Zkw, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.

Reviewer_WbzQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OctLLM, which unifies 3D generation conditioned on images or text with 3D description within Qwen2.5-VL-7B-Instruct. Geometry is encoded as a sparse octree (S-Octree) of depth 6 (64³), where every 8 occupancy bits are packed into a byte token. Before generating each byte, the model receives a <MASK> query containing the target node's coordinates and depth; the query position is determined by the tree structure, requiring the model to predict only the occupancy pattern at that location. The original text and visual pathways remain frozen, while 12 out of 28 layers incorporate trainable full-rank 3D projection and FFN branches. Mesh tokens use a separate vocabulary and output head. The predicted S-Octree is first completed by a separately trained 3D U-Net, then fed together with the original conditioning input into TRELLIS's second stage to generate the final mesh.

Critical strengths
  1. The research problem is valuable. The paper addresses two issues simultaneously: the representation interface between LLMs and 3D geometry, and how to preserve pre-trained capabilities when introducing new modalities.

  2. The representation and prediction mechanisms are well aligned. Octrees provide spatial hierarchy, position queries specify target nodes for prediction, and the model predicts only occupied bytes. The ablation study shows that removing MASK, positional encoding, or MASK suppression leads to degraded performance.

  3. Both generation and understanding results have improved. The FID score for image-to-3D generation shows significant improvement over ShapeLLM-Omni, and the GPT-img score on PointLLM-200 also increases substantially.

Critical weaknesses
  1. The scope of comparison needs to be expanded, and the specific contributions need to be separated from the existing methods.CG-MLLM (ICML 2026) builds on VLM and MOT, using continuous 3D latent representations encoded by Hunyuan3D-VAE for generation and already open-sourced. EVA01 (TOG 2026) adopts a similar architecture and uses sparse tensors encoded by Trellis-VAE. Compared with these prior works, the paper should clearly articulate the added value of its own design. It is recommended to include these unified model under a common protocol for comparison. Additionally, comparisons with state-of-the-art specialized 3D generative models (e.g., Trellis.2, SAM3D) should be provided to evaluate the performance gap between current unified models and dedicated ones.

  2. The source of generation quality remains unclear. The final mesh undergoes completion via a 3D U-Net and refinement in the second stage of TRELLIS, while TRELLIS also re-receives the original conditioning. Therefore, the results in Table 1 reflect contributions from three components: the LLM-predicted structure, the completion network, and the second stage of TRELLIS. OctLLM essentially replaces the first stage of TRELLIS, yet the full version of TRELLIS still performs significantly better (3D FID 18.15 vs. 31.58), indicating that the structural predictions from the LLM remain inferior to the original first-stage output.

  3. Efficiency claims are insufficiently supported. In Table S3, OctLLM's autoregressive time is 37.59 seconds—about 1.9 times longer than ShapeLLM-Omni’s 19.92 seconds—making the term "competitive" inappropriate. Moreover, the statistics only account for the autoregressive phase, excluding the U-Net and the second stage of TRELLIS, thus missing end-to-end timing data.

  4. MASK negates the length reduction benefit of S-Octree. Shortening sequences was the primary motivation for adopting S-Octree. However, during training, a MASK token is inserted before each byte, nearly longer than the full octree without MASK. The memory and time comparisons do not separately account for this overhead.

  5. The evaluation scope for 3D understanding is too narrow. Currently, only captioning on PointLLM-200 is used, lacking tasks such as 3D QA, part localization, or classification. Thus, the term "3D understanding" appears overly broad.

Questions for authors
  1. Can the geometric metrics between the S-Octree and U-Net completion results and the GT (Ground Truth) be reported directly without going through TRELLIS? For example, IoU (Intersection over Union), Chamfer distance, and F-score. Then, compare these results with those of the first stage of Trellis and shapellm-omni. I'm curious whether the final results are due to the difference in input distribution in the second stage or whether the low-level model generated by the unified model is actually worse than the first stage of Trellis.

  2. The conclusion of GPT-img is opposite to that of GPT-ref: OctLLM leads PointLLM on GPT-img, but significantly lags behind on GPT-ref. Does this indicate that this difference is caused by different datasets? Or is it a performance improvement? Meanwhile, the captions were generated by Qwen3.5, and the judge was Qwen3.8-Max. Have there been any tests using non-Qwen judges or manual evaluations for verification?

8 comments
AuthorsOpenReview ↗

Response to Reviewer WbzQ (Part 1/2)

We sincerely thank Reviewer WbzQ for the detailed review, and for recognizing the value of the problem and the alignment between our representation and prediction mechanism. All changes in the revised PDF are marked in blue.

Q1: Comparison with CG-MLLM, EVA01, TRELLIS.2, and SAM3D.

A1: (1) TRELLIS.2 and SAM3D. Thank you for the suggestion. Both are now in Table 1, evaluated under the same Toys4K image-to-3D protocol:

MethodFID (Inc.)↓KID (Inc.)↓FID (DINOv2)↓KID (DINOv2)↓CLIP↑
Task-specific
TRELLIS.216.810.1243.491.4893.91
TRELLIS18.150.1348.472.0692.86
SAM3D23.780.4162.993.1192.72
Multimodal LLMs
ShapeLLM-Omni38.211.02145.149.9886.75
OctLLM31.580.56135.587.9487.31

As expected, dedicated image-to-3D generators remain stronger, and Sec. 4.2 now states this explicitly. The two families, however, differ in scope: dedicated generators devote their architecture and training to this single task, whereas OctLLM is one frozen-backbone model that also performs text-to-3D, captioning, and dialogue. Within unified models, OctLLM substantially narrows the gap. Compared with the previous best unified model, it closes about one third of the FID gap to TRELLIS (20.06 → 13.43) and about half of the KID gap (0.89 → 0.43).

(2) CG-MLLM and EVA01. Both are concurrent works. A fair comparison would require running their released code under our protocol, but CG-MLLM's code was released on Sep 7, 2026, shortly before the ICLR deadline, and EVA01's on Oct 20, 2026, after the deadline. The two methods also take a different route: both generate continuous latents of a pretrained 3D VAE (Hunyuan3D-VAE and TRELLIS sparse latents, respectively), whereas OctLLM generates discrete octree occupancy bytes by next-token prediction. Its 3D tokens therefore come from deterministic octree serialization rather than a learned VAE, each token carries explicit 3D coordinates and depth, and the same sequence is read back as input for captioning.

Q2: Source of generation quality: voxel metrics before TRELLIS.

A2: Table 3 now evaluates image-conditioned 64³ voxels against the ground truth before TRELLIS's second stage:

Structural outputIoU (%)↑CD-L1↓CD-L2↓F-score (%)↑
TRELLIS (stage one)23.840.066180.0083372.87
ShapeLLM-Omni (decoded voxels)7.750.158150.0272536.58
OctLLM (raw S-Octree)5.600.137640.0205240.24
OctLLM (+ 3D U-Net completion)9.750.133250.0201843.53

The gains come from our predicted structure, not from the decoder. After completion, OctLLM outperforms ShapeLLM-Omni on all four metrics. Since ShapeLLM-Omni also decodes with TRELLIS's second stage, OctLLM's advantage is already present in the voxels and is not introduced by the decoder or by a shift in the decoder's input distribution. The raw S-Octree is sparse by design, because half of the penultimate-level nodes are dropped, and completion improves all four metrics.

TRELLIS stage one is indeed more accurate, as expected for a flow model trained solely for this step. However, we respectfully note that OctLLM is not merely a replacement for this stage: the same model predicts structure from text or images, captions 3D inputs, and keeps the backbone's language ability. The TRELLIS decoder is a shared, pluggable component that ShapeLLM-Omni and AR3D-R1 also use.

AuthorsOpenReview ↗

Response to Reviewer WbzQ (Part 2/2)

Q3: Efficiency claims and end-to-end time.

A3: Thank you for this suggestion. App. C.3 now reports end-to-end time to mesh, including AR generation, the 3D U-Net, and TRELLIS decoding (200 Toys4K prompts, one B200, batch size 1):

MethodOutput tokensTime to mesh (s)↓
ShapeLLM-Omni1,02620.87 ± 5.11
AR3D-R11,02621.35 ± 6.01
LLaMA-Mesh4,44472.73 ± 27.93
OctGPT10,03351.20 ± 12.81
OctLLM1,27730.77 ± 13.50

OctLLM is indeed about 10 s slower than the VQ-based ShapeLLM-Omni and AR3D-R1, partly because its sequence length adapts to shape complexity, and we have revised the wording in App. C.3 accordingly. That said, it is 1.7–2.4× faster than OctGPT and LLaMA-Mesh, which also generate explicit geometry, and the extra time over ShapeLLM-Omni buys a clear quality gain on text-to-3D (FID 52.36 → 45.61, Table 1).

Q4: Overhead of mask tokens.

A4: We respectfully disagree that masks negate the benefit of S-Octree. Masks do double the length of generation samples during training, but the two designs are complementary.

Masks are essential, because the 3D position of the next byte cannot be inferred from its index in the sequence. Removing them raises FID from 26.31 to 61.08, by far the largest drop in our ablations (Table S6), so a full octree without masks is not a viable alternative. The relevant comparison is therefore with vs. without S-Octree under the same mask-based model. Under this comparison, S-Octree gives clear savings: it lowers peak training memory from 58 GB to 44 GB (−24%) and epoch time from 2.17 h to 1.37 h (−37%) at comparable FID (App. D, Fig. S8), which brings training within 48 GB GPUs. In other words, S-Octree is what makes mask-based training affordable.

The overhead is also limited to generation samples during training. Understanding samples contain no masks, and at inference each mask is evicted from the KV cache once its byte is sampled, so masks never accumulate in the cached context (App. A.2).

Q5: Scope of 3D understanding.

A5: We believe 3D captioning is a meaningful test of 3D understanding. PointLLM-200 is the standard understanding benchmark for 3D LLMs, and captioning on it is demanding: to name an object and describe its parts and their arrangement, the model must recognize global shape and part structure directly from the S-Octree. On this benchmark, OctLLM leads all unified MLLMs on every metric, with +28.7 GPT-img over ShapeLLM-Omni. It also surpasses the specialized PointLLM-7B under the render-grounded judge, even though it observes geometry only.

Extending OctLLM to 3D QA and part grounding is a natural next step. These tasks mainly require task-specific instruction data rather than a different model. Since OctLLM's representation and architecture are task-agnostic, the same S-Octree input can be paired with such data without architectural changes.

Q6: GPT-ref vs. GPT-img, and judge bias.

A6: PointLLM-200 reference captions focus heavily on color and material, which OctLLM does not describe by design because the S-Octree encodes geometry only (Sec. 4.2). GPT-ref therefore penalizes accurate geometric descriptions, whereas GPT-img scores the caption against renders of the object. For example (render):

  • GT: "White car with gray thick band running all over roof and bonnet having gray side panels and wheels."
  • OctLLM: "a low-slung, two-door sports car … a long hood and short rear deck … a sloping roofline that transitions into a fastback-style rear window … large, multi-spoke wheels …"
  • Scores: GPT-ref 10, GPT-img 82.

To check for judge bias, we re-scored all methods with a non-Qwen judge, DeepSeek-V4.1-Flash (official API):

MethodGPT-ref↑GPT-img↑
PointLLM-13B47.9452.53
PointLLM-7B45.3550.60
ShapeLLM-7B17.3036.73
LLaMA-Mesh9.6020.93
ShapeLLM-Omni21.2120.68
OctLLM35.1651.03

The ranking matches that of Qwen3.8-Max, except that LLaMA-Mesh and ShapeLLM-Omni, which are 0.25 apart, swap places on GPT-img. OctLLM again ranks second on GPT-img and third on GPT-ref, and it leads all multimodal LLMs on both. Our conclusions therefore do not depend on the Qwen judge.

We hope these results address your concerns, and we would be grateful if you would consider raising your score.

Thanks for the detailed response. I sincerely appreciate the effort, and it has helped me better understand the paper. However, I would like to respectfully raise a few remaining concerns.

  1. The octree is introduced to bring sparsity and reduced computation to 3D processing. Simply adding a mask weakens this advantage.
  2. Whether works are concurrent should be judged by their publication or acceptance dates, not by their open-source release dates. CG-MLLM and EVA01 clearly predate the current submission cycle. The authors claim that OctLLM achieves the best image-to-3D generation and 3D captioning results among the compared unified multimodal LLMs, while matching the backbone on MMLU and HellaSwag. Given that claim, it is inappropriate to avoid these two methods. I believe the comparison is necessary.
  3. In the generation results shown by the authors, I observe many holes in the generated shapes. To my knowledge, ShapeLLM-Omni and AR3D-R1 suffer from the same problem. This may be caused by an information bottleneck between the LLM generator and TRELLIS, and it may be a common weakness of 3D unified language models (3DULMs) that rely on an external generator. To my knowledge, CG-MLLM and EVA01 do not have this problem.
  4. I suspect that the current method may not outperform CG-MLLM and EVA01. Results are not the only thing that matters in a paper, but the omission of these directly relevant prior works concerns me. If the performance is not superior, a candid discussion would still be valuable. For example, the authors could analyze the causes, such as the information bottleneck between the 3DULMs and the external generator. They could also discuss potential solutions and any advantages in inference speed or resource consumption. Such an analysis would be meaningful, whereas avoiding the comparison is not.

For these reasons, I regard the paper as borderline and lean toward rejection. In summary, I will maintain my current rating for the present version of the paper.

Reviewer_AZzSOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OctLLM, a multimodal language model that uses sparse octree occupancy sequences as a shared representation for 3D generation and understanding. Shapes are voxelized into octrees, serialized into byte-level occupancy tokens, and shortened by randomly pruning nodes at the penultimate level. Each retained token carries explicit spatial coordinates and depth information. During generation, predictive mask tokens query the next known spatial position, while historical mask tokens are excluded from subsequent attention. OctLLM extends a frozen Qwen2.5-VL-7B backbone with trainable, modality-specific attention projections and feed-forward branches in selected transformer blocks. Text and image tokens retain the pretrained pathway. Generated sparse occupancy is completed by a separate 3D network and passed to TRELLIS, together with the original condition, to produce the final mesh. For understanding, the model directly reads the sparse octree and generates a caption.

Critical strengths
  1. The octree representation provides a useful interface between geometry and language. Each occupancy token carries a spatial position and an octree depth, giving the model explicit information about location and scale. The same representation is used for both shape generation and captioning, without requiring a learned codebook. I find this a sensible design for bringing 3D data into a pretrained language model. The sparse representation also offers a practical way to control sequence length, with the completion network recovering the omitted occupancy before mesh decoding.

  2. The architecture adds 3D capability while keeping the pretrained language pathway intact.Text and image tokens use the frozen backbone, while 3D tokens pass through separate trainable projections and feed-forward layers. Shared attention allows the two streams to exchange information. This gives the model additional capacity for geometry without updating the parameters used by ordinary language tokens. Table 2b supports this design choice, with OctLLM matching the backbone on MMLU and HellaSwag and remaining close on GSM8K. The remaining IFEval difference deserves clarification, but the preservation of performance on the other benchmarks is encouraging.

  3. The generation and captioning results are convincing relative to the compared unified models. On image to 3D generation, OctLLM improves over ShapeLLM-Omni on all five metrics in Table 1. Inception FID decreases from 38.21 to 31.58, and KID decreases from 1.02 to 0.56. The examples in Figure 5 also show better preservation of object structure, including the chair, sofa, and hammer. On PointLLM-200, OctLLM improves over ShapeLLM-Omni on all four metrics in Table 2a, with the render-grounded caption score increasing from 18.14 to 46.88. These results show that the model can use the same geometric representation effectively in both directions.

Critical weaknesses
  1. The claim of exact language preservation needs to distinguish the frozen backbone from the behavior of the complete system. OctLLM’s IFEval score decreases from 75.41 to 69.87. The authors explain that the mesh head sometimes starts a 3D sequence during an ordinary answer and state that masking the mesh vocabulary removes this drop. It would be helpful to report the recovered results explicitly. The paper should also explain how the model chooses between a word token and <mesh_bos>, since Equation (3) describes separate language and mesh prediction losses without fully specifying this decision. This would clarify when preservation holds and how the same assistant switches reliably between conversation and 3D generation.

  2. The evidence for the advantage of full-rank branches is limited to a particular training setup. The token-routed LoRA comparison in Table S8 is a useful control, and the full-rank branches perform better on image-conditioned generation. However, text-conditioned FID is very close, at 30.48 versus 30.93, while KID favors LoRA, at 0.98 versus 1.07. The LoRA parameter count is not reported, making it difficult to separate the effect of full-rank transformations from differences in total capacity. These ablations also use ShapeNet airplanes with eight branches, while the main model uses a broader dataset and twelve branches. Reporting the parameter budgets and the training evidence behind the stated LoRA instability would help establish how broadly this conclusion applies.

Questions for authors
  1. Could you report the language benchmark results with and without mesh-vocabulary masking? This would help verify the explanation for the IFEval drop. How does the system determine when to suppress mesh outputs while still allowing users to request 3D generation in the same conversation?

  2. How does the model decide whether to produce an ordinary word or <mesh_bos>? Equation (3) defines separate language and mesh prediction losses, but the supervision for this transition is less clear. Please explain how the boundary token is trained and how the two output heads are combined during inference.

  3. What are the trainable parameter counts and computational costs of the full-rank and token-routed LoRA variants in Table S8? Given their similar text-conditioned results, how much of the image-generation improvement can be attributed to the adaptation design rather than the parameter budget? Could you also provide the training curves and optimization settings underlying the reported LoRA instability on the full dataset?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AZzS

We sincerely thank Reviewer AZzS for the positive assessment and the insightful questions, which helped us sharpen the language-preservation claim. All changes in the revised PDF are marked in blue.

Q1: Language scores with mesh-vocabulary masking, and when mesh output should be suppressed.

A1: Table 2b now reports OctLLM with <mesh_bos> suppressed at inference ("no mesh"):

MethodMMLUHellaSwagGSM8KIFEval
Qwen2.5-VL-7B (backbone)64.6567.4483.5475.41
OctLLM64.6567.4483.0169.87
OctLLM (no mesh)64.6567.4483.8575.97

With <mesh_bos> suppressed, OctLLM is on par with the backbone on all four benchmarks. The IFEval drop therefore comes from unintended mesh starts, not from any change to the language pathway. We have revised Sec. 4.2 to separate the two levels: text and image tokens are processed exactly as in the frozen backbone, whereas the complete system may occasionally emit <mesh_bos> in the middle of an answer.

"No mesh" is a diagnostic setting rather than a deployment requirement. In mixed conversations, the model itself decides when to start a mesh (see Q2), and an application can disable <mesh_bos> for text-only sessions if desired. The spurious starts are a data issue rather than an architectural one: every training conversation involves 3D, so the model never sees examples of when not to emit <mesh_bos>. Adding a small set of text-only conversations is expected to address this without touching the frozen pathway.

Q2: How the model chooses between a word and <mesh_bos>.

A2: <mesh_bos> is predicted like an ordinary next token. During training, the loss at every text position, including the one whose target is <mesh_bos>, is cross-entropy over the full vocabulary. The logits of the new tokens come from the mesh head and are concatenated with the frozen language-head logits, so <mesh_bos> competes directly with words and is supervised by the same loss as text (Ttext\mathcal{T}_{\text{text}} in Eq. 3). Mesh-byte targets instead use a cross-entropy restricted to the 256 occupancy bytes (Tmesh\mathcal{T}_{\text{mesh}}).

Inference follows the same split. Before <mesh_bos>, occupancy bytes cannot be sampled, and <mesh_bos> competes with words. Once it is sampled, decoding switches to the octree loop, where only the 256 occupancy bytes are allowed. <mesh_eos> is not predicted; it is appended once the octree is complete (App. A.2). In addition, the response preamble ends with ":" (system prompt, App. A.2), which gives the model a consistent pattern that leads into <mesh_bos>.

Q3: Parameter budgets of full-rank vs. token-routed LoRA, and training stability.

A3: Thank you for this suggestion. The rank-256 LoRA in the original submission had only 0.648B parameters, so we added a budget-matched rank-768 variant (Table S8):

MethodParamsFID (img)KID (img)FID (text)KID (text)
Full-rank branches (8 layers)1.866B26.310.8030.481.07
Routed LoRA, r=256 (28 layers)0.648B32.441.0630.930.98
Routed LoRA, r=768 (28 layers)1.938B22.600.5123.770.52

As the reviewer suspected, the parameter budget matters. At a matched budget, routed LoRA generates better on this single-category subset, likely because it adapts all 28 layers rather than eight, and we have revised the wording in the paper accordingly. Both variants preserve language, because this property comes from token routing, which they share.

At full scale, however, the decisive factor is optimization stability. On the full dataset, we compared routed LoRA (r=256, α=512) with eight full-rank branches whose FFN width is 4,736, so that both have about 0.648B parameters, under identical optimizer settings (AdamW, peak LR 1e-4, cosine decay, 10% warmup). LoRA shows repeated loss spikes and gradient-norm spikes above 5,000, whereas full-rank training converges steadily (Fig. S6). This is why OctLLM uses full-rank branches.

Thanks reviewers to provide the proper response to my concerns. I will keep my score.

I have one more question not regarding the paper review but would like to discuss with authors.

As you mentioned in the response to Review 1Zkw, "OctLLM predicts structure at 64³", which is a pretty low resolution but I can also understand that you followed the similar setting as Trellis and ShapeLLM-Omni, I wonder do you see any potential chance to apply to the scene level generation?

Thank you for the positive feedback and the interesting question. We see two possible directions for extending OctLLM to scene-level generation.

1. Direct scene generation. A single 64³ grid would provide limited detail for an entire scene. OctGPT [1] demonstrates the potential of octree-based models for scene generation, but increasing the resolution in OctLLM would substantially increase token count and memory usage. Adaptive octree refinement, inspired by [2], could reduce this cost by allocating finer cells to complex regions and coarser cells elsewhere. This would require adapting our tokenization and decoding pipeline and training on scene data.

2. Object generation and scene assembly. An LLM could plan the scene layout, generate individual objects with OctLLM, and assemble them by specifying their positions, orientations, and scales. Separate generation would ease the per-generation token bottleneck and preserve more detail per object. A backbone with stronger planning and tool-use capabilities, potentially supported by scene-specific SFT or RL, could help maintain spatial relationships and resolve collisions.

Both are promising directions, and we hope to continue exploring them in future work. We also hope that our response has addressed your concerns.

[1] Wei et al. OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation. SIGGRAPH, 2025.

[2] Deng et al. Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenization. ICCV, 2025.

Thanks for the discussion. I have no more question and concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. Reviewers found the octree interface between geometry and language sensible and well aligned with the prediction mechanism (AZzS, WbzQ, 1Zkw). They also noted that the dual-stream design adds 3D capacity while keeping the pretrained language ability (AZzS, 1Zkw), and found the gains over prior unified models convincing (AZzS, WbzQ).

We have uploaded a revised PDF with all changes marked in blue. The main revisions are:

  1. Dedicated 3D generators. TRELLIS.2 and SAM3D are added to Table 1. (WbzQ)
  2. Voxel quality before mesh decoding. Table 3 reports IoU, Chamfer distance, and F-score against TRELLIS stage one and ShapeLLM-Omni. (WbzQ)
  3. Language benchmarks with <mesh_bos> suppressed. See Table 2b and Sec. 4.2. (AZzS)
  4. Full-rank branches vs. token-routed LoRA at matched budgets, plus full-dataset training curves. See Table S8 and Fig. S6. (AZzS)
  5. End-to-end time to mesh, including the 3D U-Net and TRELLIS decoding. See Table S3. (WbzQ)
  6. Writing. The abstract, introduction, and conclusion now describe the mechanisms more concretely. (1Zkw)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.