- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HiSuper-VLM, a framework designed to improve the accuracy–efficiency trade-off in large-scale 3D scene understanding. It organizes point clouds into hierarchical superpoints and introduces an Instruction-aware Superpoint Aggregation Module (ISAM) that combines instruction-conditioned aggregation with intra-level interaction to produce a fixed number of high-level visual tokens. An Embedded Superpoint Query Module (ESQM) then uses the LLM’s text-to-visual attention to select relevant regions, retrieve lower-level details, and inject them through a fixed set of learnable tokens. Experiments with OPT-1.3B and LLaMA2-7B cover single-room, multi-room, street-level, and city-level tasks.
Critical strengths
- Balancing visual token count against detail preservation is an important challenge for large scenes. The proposed combination of compact high-level superpoints and task-relevant lower-level features addresses this challenge coherently.
- The experiments cover question answering, scene captioning, and embodied planning across four scene scales. These comparisons support practical benefits beyond simply using a larger language model.
- Figure 4 reports reductions relative to LSceneLLM in visual token count from 640 to 160, inference memory from 75.6 GB to 29.3 GB, and first-token latency from 210 ms to 105 ms, alongside improved results on major tasks. Although the measurement protocol requires further clarification, these results indicate promising potential for efficient 3D VLMs.
- The supplementary material includes incremental component ablations and analyses of hierarchy depth, text cross-attention, learnable token count, top-k selection, and ESQM insertion depth.
Critical weaknesses
- Scalability is a central claim, yet §3.2 still extracts features for all input points and constructs an input-dependent set of initial superpoints. Fixing the high-level visual token count therefore bounds only part of the LLM-side computation; it does not establish how point encoding, superpoint generation, hierarchy construction, or total memory scale with scene size. The paper does not report end-to-end latency, peak memory, and accuracy as functions of point count or initial superpoint count. Although Figure 4 states that settings are matched, the measurement dataset, input size, inference batch size, numerical precision, output length, and inclusion of feature extraction and preprocessing are insufficiently specified. Section A.2.2 does report 57.5 MB of superpoint memory and a 4.33% generation-time overhead, so the issue is not a complete absence of overhead analysis. Rather, these averages lack a clearly specified workload and scaling analysis. The 8611.8 MB total inference memory in §A.2.2 and 29.3 GB in Figure 4 may reflect different configurations, but their relationship is unclear.
- Table 5 baseline uses an SA module without the proposed hierarchical superpoints. Adding ISAM-Agg introduces both the hierarchical representation and instruction-aware aggregation. Consequently, the CIDEr increase from 109.59 to 116.25 cannot be attributed specifically to instruction conditioning. A more informative comparison would preserve the hierarchy and token budget while replacing instruction-aware aggregation with mean pooling, max pooling, or instruction-independent learned aggregation. The ESQM ablations primarily test module inclusion rather than whether LLM-attention-based selection outperforms random or simpler selection strategies under the same retrieval budget. This distinction matters because LSceneLLM already uses LLM attention to identify relevant regions and retrieve fine-grained information. The paper should isolate the additional benefits of hierarchical superpoints and injection through fixed learnable tokens.
- Eqn. (1) contains multiple learnable tokens, with their number specified as 32 in the supplement. However, Eqns. (2)–(3) do not explain how the similarities between a superpoint and these 32 tokens are converted into aggregation weights yielding a single F-dimensional superpoint feature. Even if Eq. (2) is interpreted as omitting a transpose, a reduction or fusion operation over the token dimension remains unspecified. This concerns the core ISAM computation.
- The abstract describes coarse-to-fine, multi-level queries across LLM layers, whereas §3.4 explicitly describes only retrieval from to , and §A.2.7 specifies insertion at layer 8. With the default , it is unclear whether and how ESQM subsequently accesses , or whether the implementation performs only one → refinement. although hierarchy depth, learnable token count, top-k, and insertion depth are provided, the per-level superpoint counts, aggregation group size M, intra-level neighborhood size, and attention-score aggregation across heads and text positions remain unclear.
- The indoor QA, captioning, and planning tasks in Tables 1–2 are primarily evaluated using CIDEr, ROUGE, and METEOR. These metrics measure agreement with reference text but do not directly verify target localization, spatial-relation correctness, or whether generated plans satisfy scene constraints. Figures 5, 7, and 8 provide useful qualitative examples and attention visualizations, but not corresponding quantitative grounding or planning-correctness evaluations.
- Section 4.2.2 claims consistent superiority over all baselines on single-room tasks, but Table 2 contains clear exceptions: HiSuper-VLM obtains a Scene Captioning CIDEr of 3.43 versus ChatScene’s 3.67, and ROUGE of 23.62 versus LL3DA’s 24.62. In Table 3, its overall accuracy of 57.1% exceeds the other listed general-purpose models but remains below the specialist NuScenesQA model’s 58.1%. The claims should distinguish leadership among the compared general-purpose models from superiority over every baseline. Some improvements are also small, including gains over LSceneLLM of 0.16 in XR-QA METEOR, 0.10 in XR-Caption ROUGE, and 0.7 percentage points in NuScenes-QA accuracy. Neither the main paper nor the supplement reports run-to-run variance or confidence intervals, despite random centroid sampling being part of the method.
Questions for authors
See the Weaknesses.