- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether the pretrained language model backbone needs to be updated when a 3D point cloud modality is introduced. Using a frozen Point-BERT encoder and a three-layer MLP projector, the authors compare projector-only training against joint training of the projector and LoRA on Llama-3.1-8B and Qwen3.5-4B/9B. Evaluation covers 3D object classification, captioning, and retention of existing language, vision, and spatial abilities. Projector-only training achieves roughly twice the sample throughput of joint training, remains competitive on several 3D benchmarks, and preserves the original backbone capabilities through freezing.
Critical strengths
The paper controls the encoder, the corresponding backbone for each group, data order, and batch settings, and reports performance as a function of both GPU time and number of samples. Table 8 documents the throughput advantage across all three backbones. Evaluation examines both new modality learning and retention of existing abilities. Table 4 reveals substantial instruction-following degradation in several jointly trained models, whereas projector-only training leaves the backbone unchanged and therefore provides a clear guarantee that capabilities on the original input path do not drift through parameter updates. In addition, re-evaluating existing 3D models with the same judge model and using three captioning judge models improves comparability.
Critical weaknesses
The exploration of joint training baselines is insufficient to support the claim that an unavoidable trade-off exists between forgetting and performance. Appendix A uses a constant learning rate without warmup, and the LoRA learning rate for Llama is ten times that of Qwen; Llama then exhibits the most severe capability collapse. The curriculum learning ablation addresses only projector training and therefore does not test the common alternative discussed in the introduction, namely warming up the projector before joint training. Without a controlled comparison of learning rate choices or a staged joint training baseline, the observed degradation may stem partly from the chosen training scheme rather than from an inevitable cost of updating the backbone.
Questions for authors
-
Under the same total GPU budget, how does projector-only training compare with projector warmup followed by joint training, and with a reduced LoRA learning rate for Llama? Can these alternatives reduce the loss of existing capabilities while maintaining competitive 3D performance?
-
Does this conclusion hold for part referring and question answering tasks included in the training corpus, rather than only object-level classification and captioning?
5 comments
Thank you for the response. When you complete them, please present them together with the existing results in a single comparison table. I believe these will strengthen the paper's central claim.
As requested, here are the two new configurations together with the existing results in a single table, which we will add to the manuscript.
Table R7. Joint-training variants under the same 16 GPU-hour budget. P/J rows show mean ± std over 3 seeds, while staged/reduced-LR rows are single-seed.
| Model | M40. | Obj. | Omni. | 3D MM-Vet | Held-out QA | Part-referring | IFEval | IFBench | GSM8K | HumanEval | MMLU-Pro | RefSpatial | LingoQA |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P-Qwen3.5-4B (= base LM) | 59.8 ± 0.5 | 67.6 ± 5.9 | 44.4 ± 3.2 | 47.1 ± 1.8 | 66.1 ± 1.2 | 39.0 ± 1.2 | 80.4 | 29.3 | 90.9 | 82.9 | 45.0 | 20.9 | 70.4 |
| J-Qwen3.5-4B (paper recipe) | 55.6 ± 3.4 | 72.7 ± 2.5 | 46.8 ± 0.8 | 50.3 ± 1.0 | 66.4 ± 0.3 | 41.7 ± 0.1 | 38.8 | 17.7 | 82.0 | 72.0 | 45.5 | 1.8 | 58.8 |
| Staged Qwen3.5-4B (4 h P → 12 h J) | 60.5 | 70.0 | 44.1 | 50.9 | 67.4 | 41.3 | 75.8 | 26.7 | 91.2 | 80.5 | 45.3 | 23.1 | 65.8 |
| P-Llama-3.1-8B (= base LM) | 60.1 ± 2.3 | 65.7 ± 1.5 | 43.1 ± 1.6 | 39.8 ± 1.6 | 62.9 ± 0.8 | 34.0 ± 1.4 | 72.5 | 26.3 | 87.0 | 64.0 | 37.3 | – | – |
| J-Llama-3.1-8B (LoRA LR 2e-4, paper) | 52.1 ± 5.4 | 68.5 ± 4.7 | 41.1 ± 1.4 | 46.6 ± 2.3 | 63.5 ± 1.4 | 37.3 ± 1.9 | 10.4 | 16.3 | 0.6 | 0.0 | 11.2 | – | – |
| J-Llama-3.1-8B (LoRA LR 2e-5) | 55.1 | 69.1 | 39.2 | 42.0 | 65.3 | 41.4 | 62.8 | 19.3 | 84.5 | 62.8 | 36.1 | – | – |
The new configurations demonstrate the following:
- Reduced LoRA learning rate (Llama-3.1-8B, 2e-5; projector rate unchanged at 1e-3): The lower rate avoids the collapse seen in Table 4, but the backbone still drifts. Specifically, instruction following drops substantially (IFEval 72.5 → 62.8, IFBench 26.3 → 19.3), and 3D accuracy still trails projector-only training on M40. and Omni. (55.1 vs 60.1 and 39.2 vs 43.1).
- Staged training (Qwen3.5-4B: 4 h projector-only, then 12 h joint on the same data stream): Staged training reaches essentially projector-only 3D accuracy (M40. 60.5 vs 59.8, Omni. 44.1 vs 44.4), so its joint phase adds little for 3D understanding, while still altering the backbone (IFEval −4.6, LingoQA −4.6, IFBench −2.7, HumanEval −2.4 relative to the base model). Its main gain over projector-only is on QA-style tasks (3D MM-Vet 50.9 vs 47.1).
- Across all joint configurations: instruction-following ability degrades in every case (IFEval −4.6 to −62.1, IFBench −2.7 to −11.7 points), whereas projector-only training leaves the backbone unchanged by construction and scores at least as high as standard joint training on M40. at every checkpoint.
These results still support our central claim. The alternatives reduce, but do not remove, the cost of updating the backbone, while projector-only training avoids it entirely at competitive 3D performance and roughly twice the throughput. We will add both configurations to the manuscript.
Part-referring and QA: On the held-out questions (Tables R4–R5 in the general comment), projector-only and joint training are within 0.6 points on QA for every backbone. On part-referring, the difference is small and narrows as the backbone gets stronger (3.3, 2.7 and 0.9 points for Llama-3.1-8B, Qwen3.5-4B and Qwen3.5-9B). We hypothesize that this residual gap reflects the capability of the frozen backbone rather than an inherent limitation of projector-only training, since a more capable backbone needs less adaptation to make use of the projected 3D tokens, so we expect the gap to close further with stronger backbones.
Thank you for the detailed review. We agree the joint-training baselines should be explored further, and are running two additional joint configurations under the same 16-hour budget:
For context on our settings, the Qwen learning rates follow PointLLM’s two training stages (2e-3 for the projector, 2e-5 for the language model). For Llama, we initially used these same rates but observed unstable training, which led to the values in Appendix A.3. Our launched run now isolates the effect of the LoRA rate.
On part-referring and QA: we are evaluating on held-out QA and part-referring examples in the format of our training data, as well as on 3D MM-Vet, and will report both.