acceptodds
Under review as a conference paper at ICLR 2027

Projector Is All You Train

Abstract

The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.

Primary area:
generative models
Keywords:
multimodal large language model, large language model, 3D generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2K3UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether the pretrained language model backbone needs to be updated when a 3D point cloud modality is introduced. Using a frozen Point-BERT encoder and a three-layer MLP projector, the authors compare projector-only training against joint training of the projector and LoRA on Llama-3.1-8B and Qwen3.5-4B/9B. Evaluation covers 3D object classification, captioning, and retention of existing language, vision, and spatial abilities. Projector-only training achieves roughly twice the sample throughput of joint training, remains competitive on several 3D benchmarks, and preserves the original backbone capabilities through freezing.

Critical strengths

The paper controls the encoder, the corresponding backbone for each group, data order, and batch settings, and reports performance as a function of both GPU time and number of samples. Table 8 documents the throughput advantage across all three backbones. Evaluation examines both new modality learning and retention of existing abilities. Table 4 reveals substantial instruction-following degradation in several jointly trained models, whereas projector-only training leaves the backbone unchanged and therefore provides a clear guarantee that capabilities on the original input path do not drift through parameter updates. In addition, re-evaluating existing 3D models with the same judge model and using three captioning judge models improves comparability.

Critical weaknesses

The exploration of joint training baselines is insufficient to support the claim that an unavoidable trade-off exists between forgetting and performance. Appendix A uses a constant learning rate without warmup, and the LoRA learning rate for Llama is ten times that of Qwen; Llama then exhibits the most severe capability collapse. The curriculum learning ablation addresses only projector training and therefore does not test the common alternative discussed in the introduction, namely warming up the projector before joint training. Without a controlled comparison of learning rate choices or a staged joint training baseline, the observed degradation may stem partly from the chosen training scheme rather than from an inevitable cost of updating the backbone.

Questions for authors
  1. Under the same total GPU budget, how does projector-only training compare with projector warmup followed by joint training, and with a reduced LoRA learning rate for Llama? Can these alternatives reduce the loss of existing capabilities while maintaining competitive 3D performance?

  2. Does this conclusion hold for part referring and question answering tasks included in the training corpus, rather than only object-level classification and captioning?

5 comments

Thank you for the detailed review. We agree the joint-training baselines should be explored further, and are running two additional joint configurations under the same 16-hour budget:

  1. A staged schedule (first projector-only training, then joint training for the remaining time), and
  2. Llama-3.1-8B with the same LoRA learning rate used for the Qwen backbones (2e-5).

For context on our settings, the Qwen learning rates follow PointLLM’s two training stages (2e-3 for the projector, 2e-5 for the language model). For Llama, we initially used these same rates but observed unstable training, which led to the values in Appendix A.3. Our launched run now isolates the effect of the LoRA rate.

On part-referring and QA: we are evaluating on held-out QA and part-referring examples in the format of our training data, as well as on 3D MM-Vet, and will report both.

Thank you for the response. When you complete them, please present them together with the existing results in a single comparison table. I believe these will strengthen the paper's central claim.

As requested, here are the two new configurations together with the existing results in a single table, which we will add to the manuscript.

Table R7. Joint-training variants under the same 16 GPU-hour budget. P/J rows show mean ± std over 3 seeds, while staged/reduced-LR rows are single-seed.

ModelM40.Obj.Omni.3D MM-VetHeld-out QAPart-referringIFEvalIFBenchGSM8KHumanEvalMMLU-ProRefSpatialLingoQA
P-Qwen3.5-4B (= base LM)59.8 ± 0.567.6 ± 5.944.4 ± 3.247.1 ± 1.866.1 ± 1.239.0 ± 1.280.429.390.982.945.020.970.4
J-Qwen3.5-4B (paper recipe)55.6 ± 3.472.7 ± 2.546.8 ± 0.850.3 ± 1.066.4 ± 0.341.7 ± 0.138.817.782.072.045.51.858.8
Staged Qwen3.5-4B (4 h P → 12 h J)60.570.044.150.967.441.375.826.791.280.545.323.165.8
P-Llama-3.1-8B (= base LM)60.1 ± 2.365.7 ± 1.543.1 ± 1.639.8 ± 1.662.9 ± 0.834.0 ± 1.472.526.387.064.037.3––
J-Llama-3.1-8B (LoRA LR 2e-4, paper)52.1 ± 5.468.5 ± 4.741.1 ± 1.446.6 ± 2.363.5 ± 1.437.3 ± 1.910.416.30.60.011.2––
J-Llama-3.1-8B (LoRA LR 2e-5)55.169.139.242.065.341.462.819.384.562.836.1––

The new configurations demonstrate the following:

  • Reduced LoRA learning rate (Llama-3.1-8B, 2e-5; projector rate unchanged at 1e-3): The lower rate avoids the collapse seen in Table 4, but the backbone still drifts. Specifically, instruction following drops substantially (IFEval 72.5 → 62.8, IFBench 26.3 → 19.3), and 3D accuracy still trails projector-only training on M40. and Omni. (55.1 vs 60.1 and 39.2 vs 43.1).
  • Staged training (Qwen3.5-4B: 4 h projector-only, then 12 h joint on the same data stream): Staged training reaches essentially projector-only 3D accuracy (M40. 60.5 vs 59.8, Omni. 44.1 vs 44.4), so its joint phase adds little for 3D understanding, while still altering the backbone (IFEval −4.6, LingoQA −4.6, IFBench −2.7, HumanEval −2.4 relative to the base model). Its main gain over projector-only is on QA-style tasks (3D MM-Vet 50.9 vs 47.1).
  • Across all joint configurations: instruction-following ability degrades in every case (IFEval −4.6 to −62.1, IFBench −2.7 to −11.7 points), whereas projector-only training leaves the backbone unchanged by construction and scores at least as high as standard joint training on M40. at every checkpoint.

These results still support our central claim. The alternatives reduce, but do not remove, the cost of updating the backbone, while projector-only training avoids it entirely at competitive 3D performance and roughly twice the throughput. We will add both configurations to the manuscript.

Part-referring and QA: On the held-out questions (Tables R4–R5 in the general comment), projector-only and joint training are within 0.6 points on QA for every backbone. On part-referring, the difference is small and narrows as the backbone gets stronger (3.3, 2.7 and 0.9 points for Llama-3.1-8B, Qwen3.5-4B and Qwen3.5-9B). We hypothesize that this residual gap reflects the capability of the frozen backbone rather than an inherent limitation of projector-only training, since a more capable backbone needs less adaptation to make use of the projected 3D tokens, so we expect the gap to close further with stronger backbones.

Reviewer_jyPjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that fine-tuning the LM backbone is unnecessary for training multimodal large language models (MLLMs) and that projector-only training offers the additional benefit of preserving the backbone’s existing capabilities. The authors report that projector-only training achieves competitive performance on 3D object classification and captioning benchmarks compared with PointLLM. Those models use a fixed point cloud encoder and three LM backbones(Qwen3.5-4B, Qwen3.5-9B, and Llama-3.1-8B-Instruct). In a separate comparison against joint training of the projector and backbone LoRA adapters, they report competitive 3D classification performance under a 16-hour training time. They further emphasize that freezing the backbone preserves its existing capabilities by construction, whereas joint training produces performance degradation on several language, vision, and spatial reasoning benchmarks.

Critical strengths
  1. The paper addresses the useful question of whether training the LM backbone is necessary for MLLM training, evaluating performance on a new modality alongside the preservation of the LM backbone's existing capabilities.
Critical weaknesses
  1. The paper argues that adapting the language model is unnecessary for building MLLMs, but the experimental scope provides limited support for the broader conclusions. The experimental setup includes one encoder, one Llama model, and two model sizes from the same Qwen3.5 family, and cover only 3D tasks. Changes in existing vision and spatial reasoning capabilities are tested only on Qwen3.5. The P/J comparison is also limited to 16-hour training time with a single seed, leaving convergence and variability across seeds unclear. In addition, the external baselines in Tables 2 and 3 are limited to PointLLM models, whose LM backbones and training data differ from those used by the authors’ models, which are trained on the PointLLM-v2 dataset.
Questions for authors
  1. Does the comparison between projector-only and joint training remain consistent with longer training time and different random seeds?
  2. The paper concludes that adapting the language model is unnecessary for building MLLMs. Could the authors clarify whether this is meant to hold in general MLLM tasks, or only for the 3D tasks and backbones tested here?
3 comments

Thank you for the review.

On seeds: we are training two additional seeds for both regimes on Qwen3.5-4B and Llama-3.1-8B with configurations identical to the paper’s runs (projector-only and joint runs share each seed’s data order), and will report mean ± std at each GPU-hour threshold.

On training length: our comparison is intentionally at a fixed wall-clock budget, which reflects the constraint a practitioner faces, and we report intermediate checkpoints (2–16 h) to show the conclusion does not depend on a single stopping point. The 16-hour budget is a fixed comparison budget rather than a convergence point, since our aim is not to train the best possible 3D MLLM, but rather to compare the two training regimes fairly under the same budget, and at 16 hours projector-only training is already consistently competitive with existing 3D MLLM baselines, so we did not train further.

On generality: our claims are about 3D point-cloud MLLMs, and we will make this explicit in the title, abstract and conclusion. Extending the study to other modalities involves a different set of encoders, datasets and benchmarks, and is beyond what a single paper can control for, but we do highlight it as future work.

On baselines: Appendix C (Tables 9–10) also reports ShapeLLM, PointLLM-R and MiniGPT-3D, re-evaluated under our judge.

Following up on our earlier reply, the seed runs are complete (general comment, Tables R1–R2, R6).

We ran the two extra seeds for Qwen3.5-9B, in addition to Qwen3.5-4B and Llama-3.1-8B, and re-evaluated every checkpoint. The central comparison holds across seeds. On M40., projector-only is at least as accurate as joint training at all 15 backbone × checkpoint combinations and in all 9 paired seeds at 16 h. On Obj. and Omni., the two regimes are within about one standard deviation, except for Qwen3.5-4B on Obj., where joint training is ahead (Table R1). Captioning precision is likewise stable across seeds and essentially identical between regimes (Table R6). The seeds also show that the paper's Qwen3.5-4B seed was a low draw for projector-only on Obj., which explains the outlier discussed in Section 4.1.2. Tables 2, 3, 9–11 and Figure 3 will report mean ± std.

Thank you for the responses. The additional seed runs address my concern about different random seeds.

Two concerns remain. First, it is not clear what evidence supports the claim that a fixed wall-clock budget reflects a constraint practitioners actually face. Second, my concern that the external baselines in Tables 2 and 3 are PointLLM models, which differ in LM backbone and training data from the authors’ models trained on PointLLM-V2, does not appear to be addressed.

I will maintain my score.

Reviewer_SghMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on 3D Multimodal LLMs, i.e. an LLM that accepts a 3D point cloud as input to solve a 3D understanding task. The fundamental question they investigate is: do we need to fine-tune the language model backbone, or is it sufficient to train only the projector? To investigate this, they consider a PointLLM-v2 style architecture with these two different training strategies, and performed evaluations on 3D classification and captioning tasks.

Critical strengths

The paper is very clear. Experimental setup looks sound with careful controls.

Critical weaknesses

Fundamentally, this paper remains incremental. Freezing the language model and only training the projector is an established approach, and many VLM works already use it. So, the paper verifies this for 3D domain, but on it’s own this is actually a narrow scope, which is not reflected in the title as well.

Even for 3D domain, the experiment focus is narrow. Captioning and classification tasks are well-solvable with a frozen language model indeed. What about tasks where we’d actually need to adapt the language model such as VQA tasks involving reasoning?

Overall, given these weaknesses, I am leaning towards rejection.

Questions for authors

As I mentioned in weaknesses, even for 3D domain, the experiment focus is narrow. Captioning and classification tasks are well-solvable with a frozen language model indeed. What about tasks where we’d actually need to adapt the language model such as 3D VQA tasks involving reasoning, referring, etc.?

2 comments

Thank you for the review. We agree the title over-states the scope and will revise it to make the 3D setting explicit.

On novelty: Training only the projector has precedents in 2D, e.g. LLaVA’s first stage and frozen-LLM models such as BLIP-2. Our contribution is a controlled comparison for a much more computationally expensive and untested modality. This matters for 3D in particular, where existing 3D MLLMs (PointLLM, ShapeLLM, MiniGPT-3D, PointLLM-R) all adapt the language model.

On task scope: We followed the standard evaluation protocol of the PointLLM line of work for comparability. We focus on object-level tasks because scene-level 3D QA typically relies on different encoders and often on additional image inputs, which would confound a comparison that holds the encoder fixed. We agree, however, that QA-style evaluation is valuable and plan on adding 3D MM-Vet, whose questions cover knowledge, spatial and embodied reasoning, with all baselines re-evaluated. We will report these results in our full response.

Following up on our earlier reply, the QA evaluations we committed to are complete (general comment, Tables R3–R5).

Reasoning-oriented QA: On 3D MM-Vet (Table R3), projector-only Qwen3.5-4B and Qwen3.5-9B score above every 3D MLLM baseline, and joint training scores somewhat higher than projector-only. The per-category breakdown shows where:

Table R8. 3D MM-Vet by question category (mean over 3 seeds)

ModelRecognitionKnowledgeLang. gen.SpatialEmbodiedOverall
P-Qwen3.5-4B51.655.239.239.844.447.1
J-Qwen3.5-4B46.157.642.749.354.550.3
P-Qwen3.5-9B47.555.543.544.854.249.5
J-Qwen3.5-9B49.660.244.145.255.651.6
P-Llama-3.1-8B43.752.227.540.229.339.8
J-Llama-3.1-8B45.654.938.443.047.746.6

These results show that different capabilities come from different parts of training. Grounding the 3D input (recognizing the object, knowing its properties, and answering questions in the training format) emerges from the projector alone. Projector-only training is within a few points of joint training on recognition and knowledge questions, and within 0.6 points on held-out QA (Table R4). What adapting the backbone adds is instruction following on unfamiliar question types: on embodied questions, projector-only models more often describe the correctly identified object instead of answering (analysis in the revised appendix). This gap narrows as the backbone gets stronger (+6.8, +3.2 and +2.1 on 3D MM-Vet for Llama-3.1-8B, Qwen3.5-4B and Qwen3.5-9B), which suggests it reflects the capability of the frozen backbone rather than an inherent limitation of projector-only training. A more capable backbone already follows such instructions without adaptation, so we expect the gap to close further with stronger backbones. Adapting the backbone, by contrast, comes with measurable drift (Table R7 in reviewer 2K3U’s thread). Our goal is not to build the highest-scoring 3D MLLM, but to measure what training only the projector achieves compared with also adapting the backbone. Even so, projector-only Qwen models exceed every published baseline on 3D MM-Vet.

 

Title: Our proposed revised title is "Projector Is All You Train: Frozen-Backbone Training for 3D Multimodal LLMs", and the object-level focus will be stated as a limitation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.