acceptodds
Under review as a conference paper at ICLR 2027

Anchored Adversarial Fine-Tuning of Language Models with Gumbel-Softmax

Abstract

Language models trained with supervised fine-tuning generate the next token based on ground-truth tokens during training but on their own predictions during inference, introducing a training–inference mismatch that can cause errors to compound over a generated sequence. We investigate generative adversarial fine-tuning as a means of introducing autoregressively generated response segments into language-model training, exposing the model to its own predictions during training. To permit gradient propagation to the generator, we leverage Gumbel-Softmax sampling rather than standard discrete token sampling to obtain a differentiable next-token representation, allowing generative adversarial network-based training. Using a convolutional discriminator, which distinguishes generated responses from reference ones, and a transformer language model as the generator, we attempt to improve training stability and response quality through the use of regularization techniques. In particular, during training we progressively unfreeze the discriminator's convolutional layers and optimize the generator using a weighted combination of adversarial and standard supervised fine-tuning objectives. Using an LLM-based evaluator, our best anchored configuration improves over the supervised fine-tuning baseline in all three evaluated categories, including relative gains of 9.9$ percent in Summarization and Science, respectively. These results provide preliminary evidence that the adversarial generation objective can complement supervised fine-tuning in the studied setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Fine Tuning, Adversarial Learning, Generative Adversarial Networks

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_c92WOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

To address the training–inference mismatch in fine-tuning large language models, an adversarial fine-tuning method is proposed. During fine-tuning, Gumbel-Softmax sampling is used to obtain a differentiable next-token representation and to enable the computation of a biased but low-variance pathwise gradient, while a CNN-based discriminator and several stabilization techniques are used to limit potential drift. An exploratory experiment is conducted to demonstrate the effectiveness of the proposed method.

Critical strengths
  1. To explicitly train a task-specific generator, a joint objective is used, comprising an adversarial loss and an SFT loss.
  2. To alleviate training instability, several stabilization methods, such as progressively unfreezing layers in the discriminator and learning-rate annealing, are jointly used.
Critical weaknesses
  1. Both adversarial training and Gumbel-Softmax sampling have been investigated in previous work. This paper merely combines them for fine-tuning large language models, so the novelty is limited.
  2. The experimental evaluation is limited in scope: the method is not tested across a variety of SFT-trained models, and the assessment relies on an LLM-based evaluator.
Questions for authors

More experiments are needed to demonstrate the effectiveness of the proposed method, such as a comparison between Gumbel-Softmax sampling and standard sampling methods, and an ablation study of the proposed method.

Reviewer_xrHuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates adversarial fine tuning of language models using Gumbel-Softmax to propagate surrogate gradients through autoregressive generation. A CNN discriminator distinguishes generated responses from reference responses, while the generator combines adversarial loss with an SFT anchoring objective. Progressive discriminator unfreezing and adaptive learning rates are used to stabilize training. Experiments on SmolLM2-360M show modest improvements in LLM judged response quality for the best anchored configuration over an SFT baseline.

Critical strengths

The framework introduces self-generated response segments into training while retaining token level supervision. SFT anchoring appropriately addresses the limitation that distinguishing reference from generated text does not directly ensure factual or task specific correctness.

Critical weaknesses
  1. Evaluation uses only a 360M model, responses capped at 64 tokens, and 50 self-generated prompts per category, scored by a single LLM judge. This provides limited evidence for addressing error accumulation over longer generations or improving larger language models
  2. Gumbel-Softmax text generation, adversarial language model training, and combining supervised and adversarial objectives have precedents acknowledged by the paper. The contribution primarily combines these techniques with stabilization heuristics, but lacks direct comparisons with closely related methods and controlled ablations establishing the benefits of progressive unfreezing and the other components.
Questions for authors

All experiments use a 360M parameter model and cap responses at 64 tokens. What evidence supports the method’s effectiveness and computational feasibility for larger models and substantially longer generations? Since the motivation concerns error accumulation during autoregressive generation, can the current short response experiments adequately assess this benefit? Please clarify the runtime and memory bottlenecks that imposed these limits.

Comments on the paper

1 comment

We thank the reviewer for their careful reading and deep technical analysis and their valuable feedback as such.

W1 Review:\textbf{W1 Review:} "Severely limited empirical scope undermines the central claim. The entire empirical study uses a 360M-parameter SmolLM2 model, a single epoch of adversarial training on Dolly-15k, maximum generation length of 64 tokens, and 50 LLM-generated prompts per category for evaluation. This is fine as a proof of concept but it is not enough to support the implicit claim that adversarial fine-tuning with Gumbel-Softmax is a viable complement to SFT at the scale where the training–inference mismatch actually matters (i.e., long-form generation). At 64 tokens, exposure bias is a much smaller problem than at typical assistant-response lengths (hundreds to thousands of tokens). Worse, the SFT baseline is trained for 5 epochs (and was selected as the best of 10 epochs, see Appendix B) while the adversarial model is trained for a single epoch, so the comparison is not compute-matched. The headline +9.9% and +6.5% numbers would shrink substantially—or vanish—against a 1-epoch SFT baseline or against an SFT model trained with the same total FLOPs. Given how small the absolute score differences are (3.04 → 3.34 on a 1–10 scale), this is a major threat to validity."

W1 Response:\textbf{W1 Response:} We partially accept this fault as-is with experiments already completed. We simply did not have access to the level of compute required to run an experiment with a larger model on higher generation lengths. Further, we should note that the 5 epoch SFT baseline was chosen against all other epoch amounts due to its higher performance. We do choose to report a 1-epoch SFT baseline as a data-matched comparison while the main baseline remains the 5-epoch one. When doing this experiment, we recognise that the 5-epoch model strictly outperforms the 1-epoch SFT model. Second, we choose not to include a compute-matched baseline as since the SFT model performance has an approximate optimum at 5 epochs and is decreasing from thereafter, added training would harm behavior. If we use compute time as an approximate indicator for total FLOPS, it would require about 30 epochs of SFT to approximately equalize the number of FLOPS used. This would likely hurt the SFT model more than a less fine-tuned version as research suggests LLMs are optimal after just a few epochs of finetuning. This would then serve as waste of compute.

W2 Review:\textbf{W2 Review:} "Evaluation is LLM-judge-only, on LLM-generated prompts, with no human study and no automatic metric. The authors acknowledge that LLMs can be biased evaluators (citing Wang et al., 2024) and mitigate by averaging 20 scoring evaluations per prompt, but the deeper issue is that (a) the 50 prompts per category are themselves synthetic (Section 7.3, item 1), so the study is evaluating model output on AI-generated prompts scored by an AI judge, which risks a closed-loop artifact; (b) no human evaluation is performed even on a small subsample; (c) no standard automatic metric (BLEU/ROUGE against reference responses, perplexity on held-out Dolly data, MAUVE, etc.) is reported, which would have provided a non-LLM signal at trivial cost; and (d) the statistical analysis is a paired t-test without multiple-comparison correction across three categories and six configurations, so the two significant results (p=0.04 and p=0.03, uncorrected) are not robust to standard corrections (Bonferroni across 3 categories would push Summarization to p≈0.09). The paper's own conclusion correctly calls this preliminary evidence, but the framing elsewhere in the abstract and introduction does not match that caution."

W2 Response:\textbf{W2 Response:} First we would like to thank the reviewer in particular for this very useful feedback. To address (a) and (b) we perform a human evaluation on a subset of 25 of the 150 total generated prompts for only the 0.15 | 0.85 vs SFT-5 comparison, observing a +10.1% relative improvement from the 5 epoch sft baseline. To address (c) we calculate mean per-token perplexity on a held out subset of Dolly 15k. Though this may not be a perfect measure of response quality, per-token perplexity serves as an approximate reflection of the models confidence in its generative ability. We observe that our 0.15|0.85 model has the lowest mean per-token perplexity, indicative of clearer and somewhat confident generation. We hope these added empirical evaluations can sufficiently strengthen the, relatively thin, empirical section we had previously. Also, during the rebuttals, we have made concerted efforts to implement a more cautious, exploratory framing consistent with our scale of experiment and our statistical significance.

Once again, thanks for your review, we will respond shortly to the questions in a seperate comment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.