acceptodds
Under review as a conference paper at ICLR 2027

Does Fine-Tuning Still Beat In-Context Learning Under LoRA?

Abstract

Does the advantage of fine-tuning (FT) over in-context learning (ICL) on synthetic formal languages persist under parameter-efficient adaptation? We compare full FT, LoRA and 4-bit QLoRA with single-pass ICL on Qwen2.5 base models from 0.5B to 3B parameters, using matched sets of 32 training strings and a published discriminative ROC-AUC evaluation. Under the published protocol, which selects each run's best epoch using test scores, all tested fine-tuning methods achieve higher mean AUC than ICL at every evaluated model size. However, at 3B, this ranking reverses when the same runs are evaluated after 50 epochs. To test whether the advantage survives validation-based checkpoint selection, we repeat the 3B comparison across four languages, choosing epochs on fresh validation strings and evaluating on separate held-out test strings. Full FT, LoRA and QLoRA achieve mean test AUCs of 0.942, 0.921 and 0.919, respectively, versus 0.873 for ICL; only the full-FT advantage remains statistically significant after correction for multiple comparisons. Selecting epochs on the test strings instead raises mean AUC only slightly, while all three fine-tuning methods again fall below ICL on average at epoch 50. Under the published protocol, adapter-versus-full-FT comparisons also depend on the adapter learning rate. Thus, in this 32-example setting at 3B, validation-based stopping preserves full FT's advantage over single-pass ICL, with weaker evidence for the adapters, whereas training for 50 epochs reverses the comparison.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, LoRA, QLoRA, in-context learning, formal languages, evaluation methodology, checkpoint selection, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EqtVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends a formal-language comparison of fine-tuning and ICL to LoRA and QLoRA. It finds that method rankings depend on checkpoint selection and learning rate. With validation-selected checkpoints, only full FT retains a statistically significant advantage over single-pass ICL after multiple-comparison correction; training to epoch 50 reverses the average ranking.

Critical strengths
  1. The validation-selected rerun strengthens the evidence beyond test-selected peaks and confirms that late-training degradation persists on fresh test strings.
  2. Implementation checks address consequential issues, including padding errors, silent batch-size changes, and computing-environment differences.
Critical weaknesses
  1. Limited novelty. This paper primarily extends an existing experimental framework using standard LoRA and QLoRA methods. It provides limited additional insight into when or why parameter-efficient adaptation changes the FT–ICL comparison.

  2. Limited robustness and scope. Validation-selected results cover only the 3B, 32-training-string setting. The 12 paired observations share grammar structures, and adapter gains are concentrated on Gβ Latin. Neither adapter’s advantage survives multiple-comparison correction, leaving limited evidence for a stable advantage across language conditions.

Questions for authors
  1. Which finding provides the strongest substantive advance beyond extending the previous study to adapters?

  2. How sensitive are the adapter gains to excluding individual languages, and how should shared grammar structures affect the statistical interpretation?

Reviewer_iooyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits Ghosh et al. (2026), who report that full fine-tuning (FT) beats ICL on synthetic formal languages. It adds LoRA (r=8/64/256) and 4-bit QLoRA on Qwen2.5 0.5B/1.5B/3B with 32 training strings.

Critical strengths

The paper tests whether the full-FT > ICL result of Ghosh et al. extends to LoRA/QLoRA, using matched 32-string training sets The work is well reproducible.

Critical weaknesses

The fine-tuning methods can use validation scores to pick the best of 50 epochs, but ICL uses a fixed single-pass prompt without similar tuning of repetition count or example order. Section 3 shows that repeating the examples twice hurts performance in one experiment (AUC drops from 0.937 to 0.744). This explains the choice for that experiment, but does not show that one pass is the best choice in every setting.

Figure 1 shows that test AUC drops from around 0.92–0.94 at the validation-selected checkpoints to about 0.76 at epoch 50, below ICL's 0.873. This shows that training longer can hurt performance. what is the explanation of why performance drops here and when LoRA behaves differently from Full FT.

Questions for authors

1.Does the fine-tuning advantage persist when ICL receives a small, predefined configuration search on the same validation set?

Reviewer_KLgcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the formal-language comparison of Ghosh et al. (2026) to LoRA and 4-bit QLoRA. Using Qwen-2.5 models from 0.5B to 3B and matched sets of 32 training strings, the authors study how adaptation method, checkpoint selection, learning rate, and training duration affect discriminative ROC-AUC. The findings include that a validation-selected rerun at 3B preserves full fine-tuning's advantage over single-pass ICL, with weaker evidence for the adapters; continuing training to epoch 50 reverses the ranking, etc.

Critical strengths

The paper asks a useful follow-up question about whether a full-fine-tuning result extends to parameter-efficient adaptation. The validation-selected rerun is an important control, and the authors carefully distinguish performance decline during training from test-based checkpoint selection.

Critical weaknesses

(1) The ICL baseline receives substantially less model-selection effort. Fine-tuning selects checkpoints using validation data, whereas ICL uses a fixed single-pass prompt, motivated by one repetition check in a reproduction run. Matching the 32 training strings therefore does not match the model-selection budget. Tuning ICL repetition count, example ordering, and prompt format on the same validation data would provide a stronger comparison. The total training and validation information available to each method should also be made explicit.

(2) The statistical evidence covers few independent language structures. The twelve language-and-seed pairs at 3B derive from only two grammar structures with different alphabets and training samples. Evaluating more independently sampled grammars and reporting uncertainty at the grammar level would better support conclusions about the relative learning capabilities of these methods.

(3) The contribution beyond the replication needs a stronger explanation. The epoch-50 reversal is consistent with small-sample overfitting, while validation-based stopping preserves the full-FT advantage. Would be stronger for the paper if it explained a method-specific phenomenon, like why LoRA at rank 64 shows little late decline with 1024 strings while full FT still degrades. Controlled data-size and rank sweeps together with analyses of memorization and grammatical generalization, could turn this observation into a more substantial finding.

Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.