acceptodds
Under review as a conference paper at ICLR 2027

Unleashing the Value of Raw Point Clouds for 3D Understanding with LLMs

Abstract

Standard 3D point cloud language models typically rely on pretrained point encoders to extract 3D features and project them into representations processable by large language models (LLMs). However, these encoders are often originally optimized for pre-defined tasks, resulting in a representation bottleneck that may lead to a substantial mismatch with downstream 3D understanding. Moreover, existing models largely follow a single pass closed-book generation, limiting their ability to actively inspect task-relevant 3D regions or retrieve necessary external knowledge. In this work, we present PointTool, an encoder-free and tool-augmented framework for 3D reasoning. PointTool introduces Alignment-Free Patch Tokenization (AFPT) to locally aggregate raw point clouds while preserving fine-grained geometric structure, and Decoupled Point-Text LoRA (DPL) to facilitate global aggregation and 3D knowledge internalization within a frozen LLM. Building upon these designs, Program-Supervised Tool Reasoning (PSTR) enables the model to explicitly operate on raw 3D data and external resources through executable tools, supporting interactive 3D reasoning grounded in spatial evidence. We further introduce PointCloud-TR, a benchmark comprising 13K 3D objects, 39.8K questions, and 35.6K tool trajectories to facilitate training and evaluation of complex 3D reasoning. Experiments demonstrate that PointTool achieves strong performance on both standard and complex 3D understanding tasks, validating the effectiveness of the proposed framework.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point Clouds, 3D Understanding, Cross-modal Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_sXq9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present POINTTOOL, an encoder-free and tool-augmented framework for 3D point cloud reasoning. Built around three primary components—Alignment-Free Patch Tokenization (AFPT), Decoupled Point-Text LoRA (DPL), and Program-Supervised Tool Reasoning (PSTR)—the model is evaluated alongside PointCloud-TR, a newly introduced benchmark featuring:

  • 13K 3D objects
  • 39.8K question-answer pairs
  • 35.6K executable tool trajectories Experiments indicate SOTA performance on standard 3D perception tasks and improved reasoning accuracy on PointCloud-TR. However, while the tool-augmented workflow and benchmark are promising, there are significant reservations regarding the paper's core technical novelty and the true source of its performance gains.
Critical strengths
  • Effective Tool Integration: The interactive action-observation framework (PSTR) is well-designed, effectively handling precise geometric measurements, spatial reasoning, and external knowledge retrieval.

  • Valuable Benchmark Dataset: The PointCloud-TR benchmark, featuring 35.6K executable trajectories, provides a solid resource for evaluating multi-step, tool-augmented 3D reasoning.

  • Informative Controlled Experiments: Component ablations and controlled baselines—such as evaluating an "answer-only" model and equipping PointLLM with PSTR—effectively isolate the impact of tool execution from program supervision.

Critical weaknesses

1. Limited Novelty of Decoupled Point-Text LoRA (DPL)

DPL adapts established 2D encoder-free vision-language paradigms to 3D point clouds:

  • Modality Adaptation & Aggregation: Decoupled adapter branches and bidirectional token attention mirror EVEv2 and VoRA.
  • Feature Distillation: Distilling pre-trained Point-BERT features into intermediate LLM states replicates VoRA’s teacher-guided framework. The authors should explicitly cite EVEv2 and VoRA, clarifying technical innovations beyond changing the input modality.

2. Incremental Design of AFPT and PSTR

  • AFPT: Patch construction relies on standard PointNet++ primitives (FPS, kNN, max-pooling).

  • PSTR: Tool reasoning relies on standard action-observation loops and supervised trajectory imitation rather than new reasoning or reinforcement-learning algorithms.


3. Gains Driven Primarily by Data and Tool Supervision

Improvements stem mainly from tool interaction and dataset scaling rather than the proposed architecture:

  • Minor Architectural Impact: Table 3 shows DPL adds only ~0.5–0.6 points, and feature distillation adds ~0.4–0.6 points.

  • Dominant Tool/Data Impact: PSTR tool reasoning adds +12.3 percentage points to Overall Perception–Web accuracy (Table 2), while PointCloud-TR fine-tuning adds +1.5 points to VQA (Table 1).


4. Template Bias and Lack of Statistical Rigor

  • Generalization Limits: PointCloud-TR perception tasks rely heavily on fixed question templates and annotated ShapeNet parts, leaving transferability to open-ended, real-world 3D scenes unverified.

  • Statistical Significance: Because several reported architectural gains are small (~0.3–0.8 points), repeated-run statistics or confidence intervals are needed to rule out random seed variance.

Questions for authors
  1. Methodological Novelty & Prior Art: How does DPL fundamentally differ from 2D encoder-free paradigms like EVEv2 and VoRA, specifically regarding modality-specific adaptation, bidirectional visual attention, and feature distillation? Could the authors provide an empirical comparison against a VoRA-style adaptation using identical point tokens and training data?

  2. Isolated Component Ablations: Could the authors separately ablate the point/text LoRA branches, bidirectional point attention, and feature distillation while strictly controlling for trainable parameters? The current component ablation (Table 3) does not fully isolate the individual contributions of these mechanisms.

  3. Controlled Performance Attribution: Given that PointCloud-TR and PSTR yield substantial performance gains (Table 2), could the authors provide controlled comparisons by equipping existing encoder-free 3D LMMs (e.g., ENEL, SAGE) with the exact same dataset and tool supervision?

  4. Tool Reasoning Generalization: How well does PSTR generalize to unseen question templates, non-templated spatial queries, and multi-step tool sequences beyond those present in the training trajectories?

  5. Evaluation on 3D Scene-Level Datasets: How does POINTTOOL perform on complex 3D scene-level grounding and visual reasoning benchmarks, such as ScanRefer or Nr3D, beyond single-object point cloud datasets?

3 comments

Q1: Further Clarification on the Contributions of DPL, AFPT, and PSTR.

Distinction between DPL and EVEv2/VoRA. Unlike EVEv2’s full-parameter decoupling and VoRA’s vision-only LoRA/ViT distillation, DPL keeps the LLM frozen and introduces dual-stream lora branches for both point and text tokens. It also adds bidirectional attention within point-token blocks and uses a Point-BERT for distillation. As shown in Q2's table, DPL outperforms a VoRA-inspired baseline. These choices are driven by the unique properties of 3D point clouds.

Design of AFPT. Unlike 2D regular grids, point clouds are unordered and non-uniformly sampled. AFPT naturally addresses this by using FPS and kNN to build local patches with explicit center positions, leaving object-level aggregation to LLM.

Design of PSTR. While PSTR does not propose a new general reasoning algorithm, its contribution lies in integrating program-supervised tool interaction with a point-cloud LLM. The LLM directly issues executable operations on raw point clouds and conditions decisions on structured observations.

Overall, the integration above and PointCloud-TR—rather than individual primitives—constitute the core contribution of PointTool.

Q2: More detailed ablation results.

We ablate modality-specific LoRA, bidirectional point attention, and feature distillation under identical settings(mean ± std over 3 seeds).

DPL ConfigTrainable LoRA ParamsCap.GPT-4Cap.S-BERTCls.GPT-4VQA GPT-4
Shared LoRA,r=64r=64159.9M51.6±0.349.4±0.156.1±0.350.6±0.4
Shared LoRA,r=128r=128319.8M51.8±0.249.5±0.256.3±0.350.9±0.3
VoRA-style,r=128r=128319.8M52.2±0.349.9±0.256.7±0.251.4±0.3
Point-only LoRA(text branch frozen)159.9M51.9±0.349.4±0.256.4±0.350.8±0.4
Text-only LoRA(point branch frozen)159.9M51.3±0.349.1±0.255.8±0.450.2±0.4
Decoupled LoRA,r=64r=64 per modality319.8M52.1±0.349.7±0.256.5±0.251.1±0.3
+ bidirectional point-block attention319.8M52.3±0.250.0±0.256.8±0.251.4±0.3
+ feature distillation (full DPL)319.8M52.7±0.250.3±0.157.2±0.251.8±0.3

Decoupling outperforms shared and single-branch variants. Furthermore, bidirectional attention and feature distillation each deliver consistent additional gains under the same budget, validating their complementary roles.

Q3: Response to the Controlled Performance Attribution.

We integrate PSTR into two encoder-free 3D LMMs (ENEL, SAGE) and train all models on identical Stage-II data, tools, and protocols.

ModelPerceptionWebOverall$_{\mathrm{P+W}}$RoutingExecution
PointLLM+PSTR57.652.957.380.189.3
ENEL+PSTR59.554.259.181.089.5
SAGE+PSTR60.855.160.481.989.8
PointTool+PSTR63.756.863.283.290.1

PSTR generalizes effectively across 3D LMM backbones. PointTool's top performance shows the gains come not just from tool supervision, but also from AFPT and DPL. This also explains the modest gains of DPL and AFPT on conventional benchmarks, which primarily assess object-level semantic understanding—an aspect our method does not specifically target—and offer a narrow range for improvement.

Q4: Clarification on the generalization of PSTR.

Generalization to unseen tool combinations.

We construct a 1,300-question compositional generalization test set by entirely excluding the four complete tool combinations listed in the table from training.

Held-out tool combinationQuestionsAnswer-onlyPointLLM+PSTRENEL+PSTRSAGE+PSTRPointTool+PSTR
(Locate→Count)×2→Calculate38450.854.956.758.162.4
Locate→Crop→Count31242.750.352.854.058.7
Locate→Crop→Color27822.638.740.942.147.8
(Locate→Color)×232649.954.256.457.761.3
Weighted average1,30042.650.252.353.658.1

PointTool achieves the highest accuracy, outperforming all other PSTR baselines by a large margin. It even reaches 47.8% on the challenging Locate→Crop→Color composition. This proves the gains are not merely from memorizing individual tools or seen sequences.

Generalization to unseen question formulations.

We further assess linguistic generalization by paraphrasing only the question texts under strict constraints. Compared with other methods, PointTool exhibits strong robustness.

TemplatePointLLM+PSTR(Original)PointLLM+PSTR(Rewritten)PointTool+PSTR(Original)PointTool+PSTR(Rewritten)
Measurement,open56.852.165.462.0
Count,open53.248.961.858.6
Relation58.552.767.663.9
Localisation51.647.460.757.9

Q5: Evaluation on 3D Scene-Level Datasets.

Scene-level benchmarks are valuable. However, our paper focuses on object-centric point-cloud understanding, which differs from multi-instance scene grounding in input organization and reasoning granularity. Extending our method to the scene level would require new scene-level grounding mechanisms, making it a promising direction for future work.

Thank you for your response. I remain unconvinced by the paper’s methodological novelty. For example, using FPS and kNN to construct local patches is standard practice in point cloud processing and has also been used in encoder-free 3D LLMs such as ENEL. I find the dataset interesting and valuable, and on that basis I will maintain my Weak Accept rating.

Thank you for your thoughtful assessment and constructive feedback. This work represents an initial exploration of leveraging tool interaction to enhance 3D point-cloud understanding, with contributions in modality-specific adaptation, executable point-cloud operations, and program-supervised reasoning. We will further investigate more advanced and effective approaches in future work. We appreciate your recognition of the value of PointCloud-TR and your consideration of our work.

Reviewer_FSiHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PointTool, which enables a language model to select relevant parts of 3D objects based on questions, invoke measurement, counting, or information-retrieval tools, and answer using the returned results. The method includes: AFPT, which partitions raw point clouds into local patches and converts them to model input; DPL, which separately adjusts the point cloud and text processing paths; and PSTR, which learns tool usage from reference programs and execution results. The language model backbone remains frozen; the first stage uses an existing point cloud model for training guidance, which is not called during online inference. The authors construct PointCloud-TR, covering perception, web knowledge, and no-tool questions, and evaluate model performance on standard 3D tasks and the new benchmark.

Critical strengths

(1) The executable task and data construction have value. PointCloud-TR links questions, reference programs, and actual execution results, specifying templates, tool parameters, return content, and failure handling. Part annotations are used for question generation and reference calls, not provided to the model during autonomous inference. Perception questions are split by object, supporting evaluation of part localization, tool-parameter filling, and measurement completion on unseen objects. The authors commit to releasing generated data and checkpoints.

(2) Tool-task gains have targeted comparisons and are not all small. In Table 2, answer-only training scores 54.4 while PointTool+PSTR scores 63.2 (+8.8); PointLLM with the same tools and program supervision scores 57.3 (5.9 below PointTool). The former supports the joint benefit of program training and online execution; the latter supports the overall scheme’s added value under tested tool conditions. Appendix diagnostics show 70% of autonomous Locate calls meet region-selection criteria, with answer accuracy at 76% vs. 36% for successful vs. failed localization.

Critical weaknesses

(1) Gains on standard benchmarks are limited and do not yet support broad point cloud understanding improvement. In Table 1, PointTool’s GPT-4-scored improvements over the best prior per task are +1.11 (captioning), +0.79 (classification), +0.30 (QA). The † version using the same training data as baselines shows only +0.31 and +0.19 on captioning and classification. Compared to same-scale 7B priors the margins are larger (+3.87 over SAGE-7B), so not all gains are sub-1-point. Still, overall standard-task results have not demonstrated convincing broad advantage.

(2) Larger tool-task gains are concentrated on a highly templated custom benchmark, with insufficient generalization evidence. Perception questions use 10 templates across 14 ShapeNet-Part categories; training and testing share the same templates and program skeletons, with tests split by object. Perception accounts for 93.2% of the overall score, and 84.5% of perception test programs involve only two calls. The 63.2 overall thus mainly reflects short-flow tasks under known templates. Shared templates do not equal leakage, but combined with limited standard-task gains, the acceptance case is weakened.

Questions for authors
  1. Do standard-task and same-tool-comparison advantages hold under unified evaluation conditions? Please specify the GPT-4 evaluation version and baseline evaluation criteria for Table 1, the full training configuration of Table 3’s “Original,” and the starting point and updatable parameters of PointLLM+PSTR; if repeated evaluations or paired statistics are available, please report them. Most standard-task increments are small, so these details will affect my assessment of effect magnitude and stability.

  2. Can the larger tool-task gains extend to call combinations not seen during training? The existing object-holdout and call-length analyses still use the same set of templates and program skeletons, and cannot distinguish execution of familiar workflows from composition of new ones. At minimum, one holdout setting could be added: each individual tool appears in training, but the test programs’ combinations do not, compared against a baseline with the same tool access.Please also provide numerical tolerances and per-template scores to help assess coverage of existing gains.

  3. Have No-tool test questions been excluded from the background instruction training corpus? Section A.3 (p. 14) states No-tool questions are drawn from Xu et al.'s instruction data, while Section C.2 (p. 16) describes the second stage retaining 70,274 complex instructions in a mixed composition before adding the PointCloud-TR training set. Please use object IDs and QA records to confirm whether No-tool validation/test questions have been removed from this background corpus and other training sources. If these test QA pairs remain in the training corpus, PointCloud-TR’s internal category-based splits cannot prevent them from re-entering training.

  4. Does Search require a correct query to return evidence for a given question? Please explain what is returned for empty queries, wrong-category queries, or irrelevant queries, and whether retrieval is performed against a shared corpus or always returns the pre-saved article for each question.

8 comments

Q1: Response to the implementation of the evaluation.

GPT-4 evaluation settings. Following official protocols, we use task-specific GPT-4 versions and prompts. Full prompts are summarized in Appendix D.

ConfigurationCaptioning&Classification3D MM-Vet VQA
Modelgpt-4-0613gpt-4-0125-preview
Temp / Top-p1 / 11 / 1 (default)
Max tokens2048Default
PromptsNo system prompt; Official PointLLM promptSystem: "You are a helpful AI assistant."; User: Official 3D MM-Vet prompt
ParsingExtract score (0–100) or T/F; drop invalid.Accept [0, 100]; retry invalid; average 5 valid scores.
RetryExponential backoff with jitter (max 40).1-s delay, unlimited.

Full configurations of the baselines in Table 3. Both baselines use the same frozen Point-BERT and Vicuna-7B, training only the projection layer and a shared LoRA to control variables. They differ in data and initialization: the “Original” baseline is trained solely on PointLLM data (excluding PointCloud-TR) to isolate representation effects, while PointLLM-7B + PSTR initializes from PointLLM v1.2 and uses PointTool's Stage-2 data (including PointCloud-TR) to isolate architectural contributions under identical tool-reasoning settings.

Repeated evaluations and paired statistical analysis. We independently repeat the evaluation three times on the same model outputs and report the mean and standard deviation, which range from 0.2 to 0.5 points and indicate low variability across GPT-4 judgments.

ModelCap.GPT-4Cls.GPT-4VQA GPT-4
PointLLM(Reproduced)44.9±0.353.1±0.341.4±0.5
ENEL(Reproduced)51.1±0.255.6±0.343.9±0.4
PointTool†52.8±0.257.3±0.351.9±0.4
PointTool53.6±0.257.9±0.253.4±0.4

We further perform paired bootstrap analysis (10,000 iterations) and McNemar's test on the same test set to demonstrate the robustness of the observed improvements in tool reasoning.

ComparisonMetricImprovement95% Confidence IntervalMcNemar's p-value
PointLLM+PSTRPerception+6.1[4.5,7.7]< 0.001
PointLLM+PSTROverall$_{\rm P+W}$+5.9[4.4,7.4]< 0.001
PointLLM+PSTRRouting+3.1[2.0,4.2]< 0.001
Answer-onlyOverall$_{\rm P+W}$+8.8[7.1,10.5]< 0.001

Limited improvement in standard benchmarks. PSTR primarily provides supervision for fine-grained spatial operations, while conventional benchmarks mainly assess object-level semantic understanding—an aspect for which our method does not incorporate dedicated design. Additionally, these benchmarks offer a narrow dynamic range for improvement.

Q2: Performance on unseen call-combination tasks.

Since only 16.2% (4,016/24,861) of the training trajectories contain multiple tool calls, we construct a 1,300-question compositional generalization test set by entirely excluding the four complete tool combinations listed in the table from training.

Held-out tool combinationQuestionsAnswer-onlyPointLLM-7B + PSTRENEL-7B + PSTRSAGE-7B + PSTRPointTool-7B + PSTR
(Locate→Count)×2→Calculate38450.854.956.758.162.4
Locate→Crop→Count31242.750.352.854.058.7
Locate→Crop→Color27822.638.740.942.147.8
(Locate→Color)×232649.954.256.457.761.3
Weighted average1,30042.650.252.353.658.1

PointTool achieves the highest accuracy on four unseen tool combinations and maintains strong performance on complex multi-step operations, demonstrating compositional generalization rather than mere tool memorization.

Q3: Clarification about the No-tool test.

We audited the revised evaluation subsets: the No-tool questions are newly constructed on 3,000 held-out objects. As shown in Table R6, both object-ID and record-level checks against all training sources confirm zero overlap, proving no data leakage.

Evaluation subset#QuestionsShared objects with PointCloud-TR training setMatching object–question records in any training source
No-tool Validation27300
No-tool Test34400
Perception Validation2,27300
Perception Test4,45100
Web Validation22900
Web Test32400

We further clarify that No-tool questions are used exclusively for evaluating Routing and are excluded from Answer and Overall which are computed using only Perception and Web.

Q4: The dependency between Search and the correct Query.

Search uses query-conditioned retrieval over a fixed 999-article corpus, not question identity: -Mechanism: A correct query returns the supporting evidence; wrong or empty queries return irrelevant or empty results (empty results count as executed tool calls but yield no evidence). -Data: Questions and queries exclude category names, forcing the model to infer the category from the point cloud. Train and test sets have zero overlap. -Verification: On 324 test questions, the reference query succeeds 95.7% of the time, while substituted queries succeed in only 6 cases, confirming retrieval strictly depends on the generated query.

AuthorsOpenReview ↗

Kind Reminder: Our Responses Await Your Feedback

Dear Reviewer FSiH,

Please accept our sincere apologies for this gentle reminder. We are grateful for your careful review and have provided thorough responses to all your questions.

With the discussion deadline approaching, we would deeply appreciate the opportunity to hear your feedback on our replies. Should you have any remaining doubts, we are glad to clarify or revise further at your convenience.

Thank you again for your valuable time and guidance.

Thank you for the detailed response. Before going through the rest, I would appreciate a brief clarification on the GPT-4 evaluations. My understanding is that gpt-4-0613 and gpt-4-0125-preview were retired from the OpenAI API on Oct 23 and Mar 26, 2026. Could you share which judge model and access route were used for the repeated evaluations in your response to me and the multi-seed ablations in your responses to Reviewers F437 and sXq9? This would help me interpret the new results.

Reviewer_BnVqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PointTool, a framework for 3D point cloud understanding with LLMs. It introduces AFPT to preserve local geometric information, DPL to adapt point and text representations within the LLM, and PSTR to enable tool-based 3D reasoning. The authors also construct PointCloud-TR, a dataset with executable tool trajectories. Experiments on Objaverse, 3D MM-Vet, and PointCloud-TR show improvements over existing methods.

Critical strengths

1、The proposed framework is well-motivated and provides a reasonable approach to preserving fine-grained geometric information while enabling global feature aggregation within the LLM. Integrating point cloud representations with explicit geometric tools also offers a useful direction for 3D reasoning.

2、The paper is generally well-organized, and the proposed PointCloud-TR dataset is a useful contribution. The results on tool-based reasoning are encouraging and demonstrate the potential of the proposed framework.

Critical weaknesses

1、The experimental comparisons are not sufficiently comprehensive. Although several 3D LLMs are included in Table 1, the tool-reasoning comparisons in Table 2 are rather limited. Moreover, the improvements from adding PointCloud-TR are relatively small on standard benchmarks (e.g., +0.8 for Captioning and +0.6 for Classification). More comparisons and analysis would help establish the effectiveness of the proposed components.

2、The evaluation could be more convincing. Most tool-reasoning experiments are conducted on the authors' own dataset, which relies on predefined question templates and tool sequences. It is unclear how well the approach generalizes to unseen reasoning tasks. Also, while Appendix H reports inference latency, the end-to-end cost of multi-step tool reasoning is not clearly evaluated. Finally, I could not find the exact GPT-4 evaluation prompts and judge settings, which are important for reproducing the reported results.

Questions for authors

1、Could the authors include more recent 3D LLM baselines under the same tool-reasoning setting? It would also be helpful to better explain the relatively small improvements obtained from adding PointCloud-TR.

2、How well does PointTool generalize to unseen question templates or more challenging 3D reasoning benchmarks?

3、Could the authors report the full inference time for multi-step tool reasoning, including tool execution? Also, could the exact GPT-4 evaluation prompts and model settings be provided?

4、The related work could be expanded to include several relevant studies on 3D LLMs and reasoning, such as PGLLM [1], PerLA [2], and NS3D [3]. Discussing their connections and differences with PointTool would help clarify the contributions of this work.

[1] Tiankai Chen et al. "Test-Time Optimization of 3D Point Cloud LLM via Manifold-Aware In-Context Guidance and Refinement." ICLR, 2026.

[2] Guofeng Mei et al. "PerLA: Perceptive 3D Language Assistant." CVPR, 2025.

[3] Joy Hsu et al. "NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations." CVPR, 2023.

I would consider raising my score if the authors can address these concerns convincingly.

3 comments

Q1: Broader Baselines and Limited Gains in Table 1.

(1) Additional 3D LMM baselines. We additionally equip two encoder-free 3D LMMs, ENEL and SAGE, with the same PSTR tool space and program-supervised training data under identical tool definitions, interaction protocols, training splits, and held-out test sets.

ModelPerceptionWebOverall$_{\mathrm{P+W}}$RoutingExecution
PointLLM+PSTR57.652.957.380.189.3
ENEL+PSTR59.554.259.181.089.5
SAGE+PSTR60.855.160.481.989.8
PointTool+PSTR63.756.863.283.290.1

The table shows that our encoder-free design provides a representation better suited to tool-mediated reasoning. Furthermore, PSTR’s cross-backbone gains demonstrate that program-supervised tool interaction is a general approach for point-cloud reasoning.

(2) Response to the limited improvement in Table 1. PSTR shows relatively modest gains on conventional benchmarks, partly because it emphasizes different capabilities. Specifically, it targets fine-grained spatial operations rather than object-level semantic understanding, an aspect our method does not specifically design for; the tool supervision aims at interactive reasoning instead. Additionally, these benchmarks offer a narrow dynamic range for improvement.

Q2: Response to the generality of PointTool.

(1) Generalization to unseen tool combinations. Since only 16.2% (4,016/24,861) of the training trajectories contain multiple tool calls, we construct a 1,300-question compositional generalization test set by entirely excluding the four complete tool combinations listed in the table from training.

Held-out tool combinationQuestionsAnswer-onlyPointLLM-7B + PSTRENEL-7B + PSTRSAGE-7B + PSTRPointTool-7B + PSTR
(Locate→Count)×2→Calculate38450.854.956.758.162.4
Locate→Crop→Count31242.750.352.854.058.7
Locate→Crop→Color27822.638.740.942.147.8
(Locate→Color)×232649.954.256.457.761.3
Weighted average1,30042.650.252.353.658.1

On the four unseen tool combinations, PointTool achieves the highest weighted accuracy, outperforming other PSTR-equipped baselines. It also maintains strong performance on challenging combinations requiring multiple spatial operations. These results demonstrate that the gains reflect compositional generalization, rather than merely memorizing individual tools or reproducing seen sequences.

(2) Generalization to unseen question formulations. We further assess linguistic generalization by paraphrasing only the question texts under strict constraints (limited word overlap with training templates and verified semantic equivalence), while keeping other inputs unchanged. Compared with other methods, PointTool exhibits strong robustness, with only a marginal 3.3-point drop.

TemplatePointLLM+PSTR (Original)PointLLM+PSTR (Rewritten)PointTool+PSTR (Original)PointTool+PSTR (Rewritten)
Measurement,open56.852.165.462.0
Count,open53.248.961.858.6
Relation58.552.767.663.9
Localisation51.647.460.757.9

Q3: More details on reasoning efficiency and evaluation.

Task#CallsPreprocessing(ms)LLM Generation(ms)Tool Execution(ms)Total Latency(ms)
One-shot answering00.5±0.0419±0.30.0±0.0420±0.3
Count20.4±0.04,768±2864±0.44,834±28
Measurement20.4±0.15,091±171.1±0.05,094±17
Colour20.4±0.04,838±250.8±0.04,840±24
Localisation20.7±0.05,050±270.8±0.05,052±27
Relation30.4±0.07,751±481.5±0.07,754±48
Comparison50.4±0.010,657±702.2±0.010,661±70
Ratio50.4±0.110,748±612.2±0.010,752±61
Web10.8±0.13,234±121.1±0.03,236±12
No-tool00.8±0.13,068±6.70.0±0.03,070±6.7
Average-0.5±0.05,426±2216±0.15,443±22

Across all questions, LLM generation accounts for 99.7% and tool execution for only 0.3%. The additional cost stems from multi-step reasoning rather than lightweight tool execution, whose performance gains remain substantial.

Q4: Further clarification of the evaluation details.

GPT-4 evaluation settings. Following official protocols, we use task-specific GPT-4 versions and prompts. Full prompts are in Appendix D, with complete configurations, parsing, and retry strategies summarized below.

ConfigurationCaptioning & Classification3D MM-Vet VQA
Modelgpt-4-0613gpt-4-0125-preview
Temp / Top-p1 / 11 / 1 (default)
Max tokens2048Default
PromptsNo system prompt; Official PointLLM promptSystem: "You are a helpful AI assistant."; User: Official 3D MM-Vet prompt
ParsingExtract score (0–100) or T/F; drop invalid.Accept [0, 100]; retry invalid; average 5 valid scores.
RetryExponential backoff with jitter (max 40).1-s delay, unlimited.

Q5: Suggestions for expanded Related Work.

We thank for the constructive suggestions. We have added the connections and differences between PointTool and PerLA, NS3D, and PGLLM to Sec.2.

I think authors have addressed my concerns, so I have decided to increase my rating.

We are grateful to reviewer for the insightful suggestions and are glad that our response has effectively addressed your concerns.

Reviewer_F437OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

POINTTOOL combines raw-point patch tokenization, point/text adaptation within an LLM, and program-supervised tool use for 3D understanding. The paper also introduces PointCloud-TR to evaluate perception, web-assisted reasoning, and decisions about whether to invoke tools.

Critical strengths
  1. Operating directly on raw point-cloud regions is a useful direction for evidence-grounded 3D reasoning.
  2. The answer-only and PointLLM+PSTR controls in Table 2 help distinguish the benefits of additional data and tool supervision.
Critical weaknesses
  1. The supplied default implementation does not reproduce the described DPL configuration. Section 4.2 specifies separate Point/Text LoRA and bidirectional attention within the point block. The initialization script does not enable vora_lora_decoupled, and the training code defaults to shared LoRA when this flag is absent. The supplied forward path also does not show the described point-block attention-mask modification. This leaves the relationship between the provided code and the reported DPL experiments unresolved.
  2. The evaluation implementation conflicts with the stated protocol. Appendices B/D specify fixed Wikipedia snapshots, but eval_toolqa.py uses live retrieval. It also counts direct answers without tool calls as successful executions, whereas Appendix D requires at least one emitted call for a successful Execution trajectory. These discrepancies affect reproducibility and interpretation of the tool metrics.
Questions for authors
  1. Can the authors supply the actual DPL configuration, point-block attention implementation, and logs corresponding to the reported ablations?
  2. Which evaluator produced the tables? Please reconcile snapshot retrieval and the Execution numerator/denominator with the released implementation, and indicate whether corrected evaluation changes the results.
2 comments

Q1: Clarification of the description and implementation details of DPL.

We thank the reviewer for carefully checking our paper and code. We clarify that the initial release’s DPL configuration differed from the paper because ablation-related modifications were not fully reverted before release; we apologize for the confusion.

Importantly, the main results were produced with the intended DPL design. We have now aligned the released code with the paper. We further make the following clarifications:

(1) Correspondence between the paper description and code implementation. The table below summarizes the correspondence between the DPL described in Sec 4.2 and their implementations in the aligned codebase.

ComponentImplementation in the aligned codebaseConfiguration
Modality-specific Adaptationpointllm/model/dpl.py: DecoupledLoRALinear、inject_decoupled_loradpl_lora_decoupled=true,r=64,α=128,dropout 0.05
Global Aggregationdpl.py: build_point_bidirectional_mask; pointllm/model/pointllm.py: Forward pass during the full-prompt passEnabled whenever AFPT is used
Feature distillationpointllm/model/afpt.py: PointBertTeacher、DistillHeads; pointllm/train/train.py: drop_distillation-Remove in the Stage IILayers {8, 16}; loss weight 1; Stage I only

(2) Clarification of the initialization configuration. Although the initial release code did not explicitly set vora_lora_decoupled, both initialization checkpoints used in the original experiments record vora_lora_decoupled=true in their config.json, confirming that the Point/Text LoRA branches were decoupled in the main experiments. For bidirectional point attention, the released forward path did not reflect the main-experiment configuration because ablation-related modifications were inadvertently retained. Thus, the discrepancy lies in the initial code snapshot and its default configuration, not in the DPL design used for the reported results.

(3) Additional multi-seed component ablation experiments. We further strengthened the empirical evidence for DPL with additional component ablations using three independent random seeds.

DPL ConfigurationTrainable LoRA ParamsCap. GPT-4Cap. S-BERTCls. GPT-4VQA GPT-4
Shared LoRA, r=64r=64159.9M51.6 ± 0.349.4 ± 0.156.1 ± 0.350.6 ± 0.4
Shared LoRA, r=128r=128319.8M51.8 ± 0.249.5 ± 0.256.3 ± 0.350.9 ± 0.3
VoRA-style, r=128r=128319.8M52.2 ± 0.349.9 ± 0.256.7 ± 0.251.4 ± 0.3
Point-only LoRA (text branch frozen)159.9M51.9 ± 0.349.4 ± 0.256.4 ± 0.350.8 ± 0.4
Text-only LoRA (point branch frozen)159.9M51.3 ± 0.349.1 ± 0.255.8 ± 0.450.2 ± 0.4
Decoupled LoRA, r=64r=64 per modality319.8M52.1 ± 0.349.7 ± 0.256.5 ± 0.251.1 ± 0.3
+ bidirectional point-block attention319.8M52.3 ± 0.250.0 ± 0.256.8 ± 0.251.4 ± 0.3
+ feature distillation (full DPL)319.8M52.7 ± 0.250.3 ± 0.157.2 ± 0.251.8 ± 0.3

(4) Additional implementation and training records. To facilitate verification and reproducibility, we provide the following additional materials in the supplementary package:

-config.json of original checkpoints documenting the main-experiment Point/Text LoRA setup. -Aligned implementation and its code commit. -Full training logs of LoRA branch injection (layers and target projection modules). -trainer_state.json for three random seeds, covering losses and optimization across both stages. -Attention-mask implementation and verification tests for bidirectional point / causal text attention.

Q2: Response to Reviewer Comment on Evaluation Implementation.

We thank the reviewer for carefully checking the evaluation implementation. All reported results were computed using the released eval_toolqa.py; the discrepancies arise from imprecise descriptions in the appendix.

(1) Wikipedia retrieval and versioned evidence. The “fixed snapshots” refer to article versions used during benchmark construction. For each Web question, we record the source article title, revision date, and supporting evidence, retrieving the article via the Wikipedia API during evaluation. As these articles describe stable object categories, retrieved content changes little. We audited all 81 articles (e.g., Knife: revision ID 1372749145, 2026-09-02): of 324 supporting quotations, 321 appear verbatim in recorded revisions, and 312 remain in the latest versions (as of 2026-11-11). We will clarify this in the appendix.

(2) Execution metric. Execution measures whether all tool calls in a trajectory can be successfully parsed and executed. A direct-answer trajectory with no tool calls contains no failed calls and thus counts as successful; whether a tool should be called is assessed separately by Routing. The Appendix D phrase “at least one emitted call” is inaccurate; we will revise it to match the implementation and specify the numerator and denominator.

AuthorsOpenReview ↗

Kind Reminder: Our Responses Await Your Feedback

Dear Reviewer F437,

Please accept our sincere apologies for this gentle reminder. We are grateful for your careful review and have provided thorough responses to all your questions.

With the discussion deadline approaching, we would deeply appreciate the opportunity to hear your feedback on our replies. Should you have any remaining doubts, we are glad to clarify or revise further at your convenience.

Thank you again for your valuable time and guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.