- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper extends a formal-language comparison of fine-tuning and ICL to LoRA and QLoRA. It finds that method rankings depend on checkpoint selection and learning rate. With validation-selected checkpoints, only full FT retains a statistically significant advantage over single-pass ICL after multiple-comparison correction; training to epoch 50 reverses the average ranking.
Critical strengths
- The validation-selected rerun strengthens the evidence beyond test-selected peaks and confirms that late-training degradation persists on fresh test strings.
- Implementation checks address consequential issues, including padding errors, silent batch-size changes, and computing-environment differences.
Critical weaknesses
-
Limited novelty. This paper primarily extends an existing experimental framework using standard LoRA and QLoRA methods. It provides limited additional insight into when or why parameter-efficient adaptation changes the FT–ICL comparison.
-
Limited robustness and scope. Validation-selected results cover only the 3B, 32-training-string setting. The 12 paired observations share grammar structures, and adapter gains are concentrated on Gβ Latin. Neither adapter’s advantage survives multiple-comparison correction, leaving limited evidence for a stable advantage across language conditions.
Questions for authors
-
Which finding provides the strongest substantive advance beyond extending the previous study to adapters?
-
How sensitive are the adapter gains to excluding individual languages, and how should shared grammar structures affect the statistical interpretation?