acceptodds
Withdrawn from ICLR 2027

HiSuper-VLM: Multi-Scale Scene Understanding via Hierarchical Superpoints

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Vision-Language Models (3D-VLMs) have shown strong potential for 3D scene understanding. However, scaling to larger, more complex scenes challenges current 3D-VLMs in both efficiency and accuracy. This is largely due to their use of variable-length scene tokens that grow with scene complexity and the decoupling of scene representation and understanding, which limits VLM's adaptive reasoning over scenes with different scales. To address this, we propose HiSuper-VLM, a 3D-VLM framework leveraging hierarchical superpoints to construct compact yet expressive representations, providing a foundation for VLM understanding across multiple scales and hierarchical levels while tightly coupling representation with reasoning. Specifically, for representation, we construct hierarchical superpoints, using fixed-length large-scale superpoints as initial visual tokens for VLMs while retaining fine-scale superpoints for information completeness. For understanding, we perform multi-level queries across VLM layers, from coarse to fine, enhancing reasoning by refining 3D representations during the process. Experimental results demonstrate that HiSuper-VLM significantly outperforms state-of-the-art methods on multiple indoor and outdoor 3D scene understanding benchmarks. Code will be publicly released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M1NZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HiSuper-VLM, a framework designed to improve the accuracy–efficiency trade-off in large-scale 3D scene understanding. It organizes point clouds into hierarchical superpoints and introduces an Instruction-aware Superpoint Aggregation Module (ISAM) that combines instruction-conditioned aggregation with intra-level interaction to produce a fixed number of high-level visual tokens. An Embedded Superpoint Query Module (ESQM) then uses the LLM’s text-to-visual attention to select relevant regions, retrieve lower-level details, and inject them through a fixed set of learnable tokens. Experiments with OPT-1.3B and LLaMA2-7B cover single-room, multi-room, street-level, and city-level tasks.

Critical strengths
  • Balancing visual token count against detail preservation is an important challenge for large scenes. The proposed combination of compact high-level superpoints and task-relevant lower-level features addresses this challenge coherently.
  • The experiments cover question answering, scene captioning, and embodied planning across four scene scales. These comparisons support practical benefits beyond simply using a larger language model.
  • Figure 4 reports reductions relative to LSceneLLM in visual token count from 640 to 160, inference memory from 75.6 GB to 29.3 GB, and first-token latency from 210 ms to 105 ms, alongside improved results on major tasks. Although the measurement protocol requires further clarification, these results indicate promising potential for efficient 3D VLMs.
  • The supplementary material includes incremental component ablations and analyses of hierarchy depth, text cross-attention, learnable token count, top-k selection, and ESQM insertion depth.
Critical weaknesses
  • Scalability is a central claim, yet §3.2 still extracts features for all NN input points and constructs an input-dependent set of initial superpoints. Fixing the high-level visual token count therefore bounds only part of the LLM-side computation; it does not establish how point encoding, superpoint generation, hierarchy construction, or total memory scale with scene size. The paper does not report end-to-end latency, peak memory, and accuracy as functions of point count or initial superpoint count. Although Figure 4 states that settings are matched, the measurement dataset, input size, inference batch size, numerical precision, output length, and inclusion of feature extraction and preprocessing are insufficiently specified. Section A.2.2 does report 57.5 MB of superpoint memory and a 4.33% generation-time overhead, so the issue is not a complete absence of overhead analysis. Rather, these averages lack a clearly specified workload and scaling analysis. The 8611.8 MB total inference memory in §A.2.2 and 29.3 GB in Figure 4 may reflect different configurations, but their relationship is unclear.
  • Table 5 baseline uses an SA module without the proposed hierarchical superpoints. Adding ISAM-Agg introduces both the hierarchical representation and instruction-aware aggregation. Consequently, the CIDEr increase from 109.59 to 116.25 cannot be attributed specifically to instruction conditioning. A more informative comparison would preserve the hierarchy and token budget while replacing instruction-aware aggregation with mean pooling, max pooling, or instruction-independent learned aggregation. The ESQM ablations primarily test module inclusion rather than whether LLM-attention-based selection outperforms random or simpler selection strategies under the same retrieval budget. This distinction matters because LSceneLLM already uses LLM attention to identify relevant regions and retrieve fine-grained information. The paper should isolate the additional benefits of hierarchical superpoints and injection through fixed learnable tokens.
  • Eqn. (1) contains multiple learnable tokens, with their number specified as 32 in the supplement. However, Eqns. (2)–(3) do not explain how the similarities between a superpoint and these 32 tokens are converted into aggregation weights yielding a single F-dimensional superpoint feature. Even if Eq. (2) is interpreted as omitting a transpose, a reduction or fusion operation over the token dimension remains unspecified. This concerns the core ISAM computation.
  • The abstract describes coarse-to-fine, multi-level queries across LLM layers, whereas §3.4 explicitly describes only retrieval from sLs_L to sL−1s_{L-1}, and §A.2.7 specifies insertion at layer 8. With the default L=3L=3, it is unclear whether and how ESQM subsequently accesses s1s_1, or whether the implementation performs only one s3s_3 → s24s_{24} refinement. although hierarchy depth, learnable token count, top-k, and insertion depth are provided, the per-level superpoint counts, aggregation group size M, intra-level neighborhood size, and attention-score aggregation across heads and text positions remain unclear.
  • The indoor QA, captioning, and planning tasks in Tables 1–2 are primarily evaluated using CIDEr, ROUGE, and METEOR. These metrics measure agreement with reference text but do not directly verify target localization, spatial-relation correctness, or whether generated plans satisfy scene constraints. Figures 5, 7, and 8 provide useful qualitative examples and attention visualizations, but not corresponding quantitative grounding or planning-correctness evaluations.
  • Section 4.2.2 claims consistent superiority over all baselines on single-room tasks, but Table 2 contains clear exceptions: HiSuper-VLM obtains a Scene Captioning CIDEr of 3.43 versus ChatScene’s 3.67, and ROUGE of 23.62 versus LL3DA’s 24.62. In Table 3, its overall accuracy of 57.1% exceeds the other listed general-purpose models but remains below the specialist NuScenesQA model’s 58.1%. The claims should distinguish leadership among the compared general-purpose models from superiority over every baseline. Some improvements are also small, including gains over LSceneLLM of 0.16 in XR-QA METEOR, 0.10 in XR-Caption ROUGE, and 0.7 percentage points in NuScenes-QA accuracy. Neither the main paper nor the supplement reports run-to-run variance or confidence intervals, despite random centroid sampling being part of the method.
Questions for authors

See the Weaknesses.

Reviewer_ufNeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiSuper-VLM represents 3D scenes with hierarchical superpoints, combining compact coarse tokens with selective retrieval of finer details. Its main contributions are instruction-aware aggregation and an attention-guided retrieval module. Experiments cover indoor, street, and city scenes, with ablations and comparisons of accuracy, memory, and latency.

Critical strengths

(1) The accuracy and effiency tradeoff is compelling to me. Hi Super improves over LScenesLLM over XR-QA (table 1) and City-3DQZ (table 4) with alot less visual tokens and better throughput (figure 3). This potentially suggest affectiveness of hierarchy.

(2)The ablations support design choices, particularly instruction-aware aggregation, which improves XR-QA CIDEr from 109.59 to 116.25 (Table 5). This suggests that the proposed representation plays an important role in improving performance.

Critical weaknesses

ESQM (Embedded Superpoint Query Module ) from what i understand first selects the top-k superpoints by text-to-visual attention and then injects their finer children into the LLM (Sec. 3.4). Table 5 removes the whole module and Table 7 removes only its text cross-attention, so the 1.55 CIDEr improvement in Table 5 could come from the extra fine-grained features alone rather than from choosing them by attention. Since ESQM is one of the two main contributions, this limits how much of the improvement can be credited to attention-guided selection vs fine-grained features alone.

Questions for authors

Evidence Request: Does ESQM still improve results when the top-k superpoints are chosen at random, with the same k and number of learnable tokens i.e. does attention-guided selection really works? This would show whether attention selects more useful details than chance.

Reviewer_Pmg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In my view, this paper proposes HiSuper-VLM for efficient 3D scene understanding using hierarchical superpoints. I find the method reasonably designed, and it achieves solid results on several benchmarks. However, I am not fully convinced by the novelty of the proposed modules, nor by the justification of their individual contributions—this is the main point I would like the authors to address more convincingly.

Critical strengths

The paper addresses a relevant problem in large-scale 3D scene understanding. The hierarchical superpoint design is reasonable and helps reduce visual tokens while retaining scene information. The experiments cover different scene scales and show improvements in both accuracy and efficiency.

Critical weaknesses

The technical novelty is somewhat limited, as hierarchical superpoint aggregation and attention-based querying have been explored in previous studies. The improvement brought by ESQM is relatively small compared with ISAM, raising questions about its actual contribution. The paper lacks a detailed analysis of how the fixed-length representation performs as scene complexity increases. The Top-k querying strategy may miss small objects or important spatial details, especially in complex scenes.

Questions for authors

I have three questions: Could the authors further clarify the main differences between HiSuper-VLM and existing hierarchical 3D-VLM methods? Why is ESQM necessary given its relatively small improvement in the ablation study? How does the method handle scenes with many objects or complex spatial relationships when using a fixed number of visual tokens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.