- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present POINTTOOL, an encoder-free and tool-augmented framework for 3D point cloud reasoning. Built around three primary components—Alignment-Free Patch Tokenization (AFPT), Decoupled Point-Text LoRA (DPL), and Program-Supervised Tool Reasoning (PSTR)—the model is evaluated alongside PointCloud-TR, a newly introduced benchmark featuring:
- 13K 3D objects
- 39.8K question-answer pairs
- 35.6K executable tool trajectories Experiments indicate SOTA performance on standard 3D perception tasks and improved reasoning accuracy on PointCloud-TR. However, while the tool-augmented workflow and benchmark are promising, there are significant reservations regarding the paper's core technical novelty and the true source of its performance gains.
Critical strengths
-
Effective Tool Integration: The interactive action-observation framework (PSTR) is well-designed, effectively handling precise geometric measurements, spatial reasoning, and external knowledge retrieval.
-
Valuable Benchmark Dataset: The PointCloud-TR benchmark, featuring 35.6K executable trajectories, provides a solid resource for evaluating multi-step, tool-augmented 3D reasoning.
-
Informative Controlled Experiments: Component ablations and controlled baselines—such as evaluating an "answer-only" model and equipping PointLLM with PSTR—effectively isolate the impact of tool execution from program supervision.
Critical weaknesses
1. Limited Novelty of Decoupled Point-Text LoRA (DPL)
DPL adapts established 2D encoder-free vision-language paradigms to 3D point clouds:
- Modality Adaptation & Aggregation: Decoupled adapter branches and bidirectional token attention mirror EVEv2 and VoRA.
- Feature Distillation: Distilling pre-trained Point-BERT features into intermediate LLM states replicates VoRA’s teacher-guided framework. The authors should explicitly cite EVEv2 and VoRA, clarifying technical innovations beyond changing the input modality.
2. Incremental Design of AFPT and PSTR
-
AFPT: Patch construction relies on standard PointNet++ primitives (FPS, kNN, max-pooling).
-
PSTR: Tool reasoning relies on standard action-observation loops and supervised trajectory imitation rather than new reasoning or reinforcement-learning algorithms.
3. Gains Driven Primarily by Data and Tool Supervision
Improvements stem mainly from tool interaction and dataset scaling rather than the proposed architecture:
-
Minor Architectural Impact: Table 3 shows DPL adds only ~0.5–0.6 points, and feature distillation adds ~0.4–0.6 points.
-
Dominant Tool/Data Impact: PSTR tool reasoning adds +12.3 percentage points to Overall Perception–Web accuracy (Table 2), while PointCloud-TR fine-tuning adds +1.5 points to VQA (Table 1).
4. Template Bias and Lack of Statistical Rigor
-
Generalization Limits: PointCloud-TR perception tasks rely heavily on fixed question templates and annotated ShapeNet parts, leaving transferability to open-ended, real-world 3D scenes unverified.
-
Statistical Significance: Because several reported architectural gains are small (~0.3–0.8 points), repeated-run statistics or confidence intervals are needed to rule out random seed variance.
Questions for authors
-
Methodological Novelty & Prior Art: How does DPL fundamentally differ from 2D encoder-free paradigms like EVEv2 and VoRA, specifically regarding modality-specific adaptation, bidirectional visual attention, and feature distillation? Could the authors provide an empirical comparison against a VoRA-style adaptation using identical point tokens and training data?
-
Isolated Component Ablations: Could the authors separately ablate the point/text LoRA branches, bidirectional point attention, and feature distillation while strictly controlling for trainable parameters? The current component ablation (Table 3) does not fully isolate the individual contributions of these mechanisms.
-
Controlled Performance Attribution: Given that PointCloud-TR and PSTR yield substantial performance gains (Table 2), could the authors provide controlled comparisons by equipping existing encoder-free 3D LMMs (e.g., ENEL, SAGE) with the exact same dataset and tool supervision?
-
Tool Reasoning Generalization: How well does PSTR generalize to unseen question templates, non-templated spatial queries, and multi-step tool sequences beyond those present in the training trajectories?
-
Evaluation on 3D Scene-Level Datasets: How does POINTTOOL perform on complex 3D scene-level grounding and visual reasoning benchmarks, such as ScanRefer or Nr3D, beyond single-object point cloud datasets?
3 comments
Thank you for your response. I remain unconvinced by the paper’s methodological novelty. For example, using FPS and kNN to construct local patches is standard practice in point cloud processing and has also been used in encoder-free 3D LLMs such as ENEL. I find the dataset interesting and valuable, and on that basis I will maintain my Weak Accept rating.
Thank you for your thoughtful assessment and constructive feedback. This work represents an initial exploration of leveraging tool interaction to enhance 3D point-cloud understanding, with contributions in modality-specific adaptation, executable point-cloud operations, and program-supervised reasoning. We will further investigate more advanced and effective approaches in future work. We appreciate your recognition of the value of PointCloud-TR and your consideration of our work.
Q1: Further Clarification on the Contributions of DPL, AFPT, and PSTR.
Distinction between DPL and EVEv2/VoRA. Unlike EVEv2’s full-parameter decoupling and VoRA’s vision-only LoRA/ViT distillation, DPL keeps the LLM frozen and introduces dual-stream lora branches for both point and text tokens. It also adds bidirectional attention within point-token blocks and uses a Point-BERT for distillation. As shown in Q2's table, DPL outperforms a VoRA-inspired baseline. These choices are driven by the unique properties of 3D point clouds.
Design of AFPT. Unlike 2D regular grids, point clouds are unordered and non-uniformly sampled. AFPT naturally addresses this by using FPS and kNN to build local patches with explicit center positions, leaving object-level aggregation to LLM.
Design of PSTR. While PSTR does not propose a new general reasoning algorithm, its contribution lies in integrating program-supervised tool interaction with a point-cloud LLM. The LLM directly issues executable operations on raw point clouds and conditions decisions on structured observations.
Overall, the integration above and PointCloud-TR—rather than individual primitives—constitute the core contribution of PointTool.
Q2: More detailed ablation results.
We ablate modality-specific LoRA, bidirectional point attention, and feature distillation under identical settings(mean ± std over 3 seeds).
Decoupling outperforms shared and single-branch variants. Furthermore, bidirectional attention and feature distillation each deliver consistent additional gains under the same budget, validating their complementary roles.
Q3: Response to the Controlled Performance Attribution.
We integrate PSTR into two encoder-free 3D LMMs (ENEL, SAGE) and train all models on identical Stage-II data, tools, and protocols.
PSTR generalizes effectively across 3D LMM backbones. PointTool's top performance shows the gains come not just from tool supervision, but also from AFPT and DPL. This also explains the modest gains of DPL and AFPT on conventional benchmarks, which primarily assess object-level semantic understanding—an aspect our method does not specifically target—and offer a narrow range for improvement.
Q4: Clarification on the generalization of PSTR.
Generalization to unseen tool combinations.
We construct a 1,300-question compositional generalization test set by entirely excluding the four complete tool combinations listed in the table from training.
PointTool achieves the highest accuracy, outperforming all other PSTR baselines by a large margin. It even reaches 47.8% on the challenging Locate→Crop→Color composition. This proves the gains are not merely from memorizing individual tools or seen sequences.
Generalization to unseen question formulations.
We further assess linguistic generalization by paraphrasing only the question texts under strict constraints. Compared with other methods, PointTool exhibits strong robustness.
Q5: Evaluation on 3D Scene-Level Datasets.
Scene-level benchmarks are valuable. However, our paper focuses on object-centric point-cloud understanding, which differs from multi-instance scene grounding in input organization and reasoning granularity. Extending our method to the scene level would require new scene-level grounding mechanisms, making it a promising direction for future work.