- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PaLoRA combines direct training of selected Transformer projection matrices with LoRA elsewhere. Short GRPO probes estimate adapter-gate sensitivity, and matrices with low variability-normalized scores receive direct updates after reinitialization. On five mathematics benchmarks, the reported mean gains over LoRA are 2.15, 3.18, and 4.26 percentage points for Qwen2.5-1.5B, 3B, and 7B, respectively.
Critical strengths
Table 2 reports improvements over LoRA across all five benchmarks at all three model sizes. At 3B and 7B, mean accuracy is within approximately one percentage point of Full-FT while training only a small fraction of the model parameters. The probe–select–reinitialize procedure is clearly defined, and the paper appropriately acknowledges that adapter sensitivity does not diagnose insufficient rank.
Critical weaknesses
W1. The current evidence does not isolate the benefit of the allocation rule. Table 2 compares methods with unequal parameter budgets, while Table 3 omits the parameter costs of alternative placements. Matching the number of selected matrices does not control for differences in trainable parameter counts. Moreover, all final training runs use a single seed (42); results from three probe seeds do not establish the reproducibility of the reported gains. Budget-matched comparisons against LoRA and random/high-score placements, together with repeated final training runs for the key comparisons, are needed to establish a reliable benefit from the allocation rule. W2. Important experimental details remain insufficiently documented. Table 1 specifies only the 1.5B PaLoRA configuration, while the 3B/7B settings, baseline ranks and learning rates, and selection procedures for the hybrid baselines are insufficiently specified in Sections 4.2–4.3. Appendix A provides little additional experimental detail beyond Figure 1, and the promised separate final-checkpoint results are absent. Reproducibility requires complete configurations and lists of selected matrices. The paper should also report final-checkpoint results and runtime and peak-memory measurements that include probing costs. Validation curves and an accuracy–cost plot would make the performance and efficiency comparisons easier to assess.
Questions for authors
- Can you provide matched-budget allocation comparisons and verify whether the main gains persist across independent final-training seeds (W1)?
- Can you supply the missing configurations, final-checkpoint results, and complete-pipeline cost measurements, with validation curves and an accuracy–cost visualization (W2)? These would materially affect my assessment.