- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the training–inference mismatch in fine-tuning large language models, an adversarial fine-tuning method is proposed. During fine-tuning, Gumbel-Softmax sampling is used to obtain a differentiable next-token representation and to enable the computation of a biased but low-variance pathwise gradient, while a CNN-based discriminator and several stabilization techniques are used to limit potential drift. An exploratory experiment is conducted to demonstrate the effectiveness of the proposed method.
Critical strengths
- To explicitly train a task-specific generator, a joint objective is used, comprising an adversarial loss and an SFT loss.
- To alleviate training instability, several stabilization methods, such as progressively unfreezing layers in the discriminator and learning-rate annealing, are jointly used.
Critical weaknesses
- Both adversarial training and Gumbel-Softmax sampling have been investigated in previous work. This paper merely combines them for fine-tuning large language models, so the novelty is limited.
- The experimental evaluation is limited in scope: the method is not tested across a variety of SFT-trained models, and the assessment relies on an LLM-based evaluator.
Questions for authors
More experiments are needed to demonstrate the effectiveness of the proposed method, such as a comparison between Gumbel-Softmax sampling and standard sampling methods, and an ablation study of the proposed method.
We thank the reviewer for their careful reading and deep technical analysis and their valuable feedback as such.
W1 Review: "Severely limited empirical scope undermines the central claim. The entire empirical study uses a 360M-parameter SmolLM2 model, a single epoch of adversarial training on Dolly-15k, maximum generation length of 64 tokens, and 50 LLM-generated prompts per category for evaluation. This is fine as a proof of concept but it is not enough to support the implicit claim that adversarial fine-tuning with Gumbel-Softmax is a viable complement to SFT at the scale where the training–inference mismatch actually matters (i.e., long-form generation). At 64 tokens, exposure bias is a much smaller problem than at typical assistant-response lengths (hundreds to thousands of tokens). Worse, the SFT baseline is trained for 5 epochs (and was selected as the best of 10 epochs, see Appendix B) while the adversarial model is trained for a single epoch, so the comparison is not compute-matched. The headline +9.9% and +6.5% numbers would shrink substantially—or vanish—against a 1-epoch SFT baseline or against an SFT model trained with the same total FLOPs. Given how small the absolute score differences are (3.04 → 3.34 on a 1–10 scale), this is a major threat to validity."
W1 Response: We partially accept this fault as-is with experiments already completed. We simply did not have access to the level of compute required to run an experiment with a larger model on higher generation lengths. Further, we should note that the 5 epoch SFT baseline was chosen against all other epoch amounts due to its higher performance. We do choose to report a 1-epoch SFT baseline as a data-matched comparison while the main baseline remains the 5-epoch one. When doing this experiment, we recognise that the 5-epoch model strictly outperforms the 1-epoch SFT model. Second, we choose not to include a compute-matched baseline as since the SFT model performance has an approximate optimum at 5 epochs and is decreasing from thereafter, added training would harm behavior. If we use compute time as an approximate indicator for total FLOPS, it would require about 30 epochs of SFT to approximately equalize the number of FLOPS used. This would likely hurt the SFT model more than a less fine-tuned version as research suggests LLMs are optimal after just a few epochs of finetuning. This would then serve as waste of compute.
W2 Review: "Evaluation is LLM-judge-only, on LLM-generated prompts, with no human study and no automatic metric. The authors acknowledge that LLMs can be biased evaluators (citing Wang et al., 2024) and mitigate by averaging 20 scoring evaluations per prompt, but the deeper issue is that (a) the 50 prompts per category are themselves synthetic (Section 7.3, item 1), so the study is evaluating model output on AI-generated prompts scored by an AI judge, which risks a closed-loop artifact; (b) no human evaluation is performed even on a small subsample; (c) no standard automatic metric (BLEU/ROUGE against reference responses, perplexity on held-out Dolly data, MAUVE, etc.) is reported, which would have provided a non-LLM signal at trivial cost; and (d) the statistical analysis is a paired t-test without multiple-comparison correction across three categories and six configurations, so the two significant results (p=0.04 and p=0.03, uncorrected) are not robust to standard corrections (Bonferroni across 3 categories would push Summarization to p≈0.09). The paper's own conclusion correctly calls this preliminary evidence, but the framing elsewhere in the abstract and introduction does not match that caution."
W2 Response: First we would like to thank the reviewer in particular for this very useful feedback. To address (a) and (b) we perform a human evaluation on a subset of 25 of the 150 total generated prompts for only the 0.15 | 0.85 vs SFT-5 comparison, observing a +10.1% relative improvement from the 5 epoch sft baseline. To address (c) we calculate mean per-token perplexity on a held out subset of Dolly 15k. Though this may not be a perfect measure of response quality, per-token perplexity serves as an approximate reflection of the models confidence in its generative ability. We observe that our 0.15|0.85 model has the lowest mean per-token perplexity, indicative of clearer and somewhat confident generation. We hope these added empirical evaluations can sufficiently strengthen the, relatively thin, empirical section we had previously. Also, during the rebuttals, we have made concerted efforts to implement a more cautious, exploratory framing consistent with our scale of experiment and our statistical significance.
Once again, thanks for your review, we will respond shortly to the questions in a seperate comment.