acceptodds
Under review as a conference paper at ICLR 2027

Self-Generative Adversarial Fine-Tuning for Large Language Models

Abstract

Post-training is central to making large language models (LLMs) useful, but it relies on high-quality examples and feedback that are scarce and costly to collect. Iterative self-improvement makes better use of the available data by generating new training examples, provided the model can assess what it generates. Existing methods accept model-generated data without a grounded correction signal or rely on self-evaluation scores that may bias toward the model's own style. We introduce Self-Generative Adversarial fine-tuning for LLM (SGALM), which learns that assessment from real examples. At each round, one LLM generates complete examples from few-shot prompts and learns to distinguish its generations from the real data. The generation is rewarded to be realistic while the discrimination is trained by the truth, instantiating a generative adversarial game without a external feedback or new human labels. Across tasks from different domains, SGALM significantly improves over SFT and existing self-improvement method on different backbones. SGALM also serves as a synthetic-data engine and a realness-based filter, whose generated data continues to improve downstream training. We also diagnose discriminator score and generation diversity which provide evidence against self-evaluation bias and mode collapse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, Self-Evolving, Generative Adversarial Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2Tn8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors propose SGALM, an adversarial framework that fine-tunes LLMs by using them as both generators that produce data following a distribution similar to real data and discriminators that distinguish real data from generated data, with each role trained using its corresponding objective.

The discriminator score D(z)D(z) is computed from the probabilities of the Real/Fake tokens, and the discriminator is trained using a cross-entropy loss based on D(z)D(z).

The generator produces data conditioned on in-context examples and is trained through policy-gradient reinforcement learning using a reward derived from D(z)D(z).

The authors provide a theoretical analysis of few-shot generation and zero-shot target alignment, report that SGALM achieves strong downstream task performance compared to other baseline methods, and demonstrate that SGALM can serve as a synthetic-data engine.

Critical strengths

1. Applicability of SGALM.

SGALM demonstrates strong performance compared to other baseline methods across tasks with varying amounts of training data.

In particular, its superior performance over SFT and other self-play baselines on MBPP with only 0.1k training examples suggests that SGALM is well suited to data-scarce settings where self-play can be particularly useful.

Furthermore, the fact that using SGALM as a synthetic-data engine leads to better performance than Self-Instruct and RFT suggests that the proposed framework can be effective in settings where obtaining large amounts of data is difficult.

2. Training in an in-context learning setting.

It is interesting that the authors theoretically analyze the relationship between the in-context few-shot generation objective and zero-shot target alignment, and empirically demonstrate improvements in SGALM's zero-shot downstream performance.

Critical weaknesses

1. Concerns about the experimental setup.

In Section 5.7 (SGALM as a synthetic-data engine), SGALM is compared against Self-Instruct, RFT, and Self-Instruct + RFT, but the experimental configurations of these methods are not clearly specified.

In particular, if RFT refers to Rejection-sampling Fine-Tuning, it should involve a data filtering process, but it is unclear how this filtering was performed during data synthesis.

Since the use of SGALM as a synthetic-data engine is one of the main contributions of the paper, I believe these experimental details should be described more thoroughly.

Additionally, the substantial discrepancy between the GSM8K performance of Qwen2.5-3B/1.5B-Instruct reported in Tables 1 and 3 and the performance reported in the Qwen2.5 Technical Report [1] (this paper: 67.95/50.72; Qwen2.5 Technical Report: 86.7/73.2) raises further concerns about the experimental setup.

2. The models used in the experiments are relatively outdated.

The experiments are conducted using relatively old models, such as Qwen2.5, Llama-3.2, and Llama-3.1, raising questions about whether the proposed method can generalize to more capable, recent models.

[1] Qwen et al. “Qwen2.5 Technical Report.” arXiv preprint arXiv:2412.15115 (2025).

Questions for authors

1. Could you provide more details about the experimental setup in Section 5.7?

Additionally, if time and computational resources permit, I would appreciate evaluations on other models or tasks in Section 5.7. It is difficult to determine whether SGALM truly outperforms the other methods based on only a single model and task.

2. Could you clarify the exact evaluation and scoring protocols?

In particular, could you explain why the GSM8K performance of the Qwen2.5-Instruct models differs substantially from the results reported in the Qwen2.5 Technical Report?

3. In Table 1, the GSM8K performance of SFT improves by only 0.7 percentage points over the base model, which seems somewhat unusual.

Could you provide more details about the training configurations of SFT and the other baselines (e.g., the number of training steps and the total amount of training data used) to ensure a fair comparison with SGALM?

4. Could you provide results from applying the proposed framework to more recent models, such as Qwen3 or Qwen3.5? If the tasks used in the paper are already saturated by these models, results on different tasks would also be acceptable.

I would be willing to reconsider my score based on the authors' responses to these questions.

3 comments

Response to Reviewer 2Tn8

We sincerely thank the reviewer for recognizing SGALM's usefulness in data-scarce settings and the connection between in-context generation and zero-shot evaluation. We appreciate the specific questions about reproducibility; the evaluation discrepancy and baseline descriptions need to be resolved explicitly.

W1 and Q1. Experimental settings and broader evaluations

Response.

1. Experimental settings in Section 5.7. The source is SGALM-trained Qwen2.5-3B-Instruct, which generates complete GSM8K examples using four-shot ICL. A separate Qwen2.5-1.5B-Instruct recipient is fine-tuned on real data plus the retained synthetic examples. Retained scales of 1×1\times, 2×2\times, and 4×4\times use candidate pools of 8×8\times, 16×16\times, and 32×32\times the real training-set size. SGALM retains the top 1/81/8 by its learned Real/Fake score; RFT matches its total candidate count at each scale.

RFT groups sampled responses by their extracted final answers and retains responses from groups in descending frequency until the budget is filled, sampling uniformly within the final group if necessary. To make fair comparison, it uses neither ground-truth answers nor an external correctness verifier. Self-Instruct generates new complete examples; Self-Instruct+RFT first proposes new questions and then applies the same answer-frequency selection to their sampled responses. Implementation settings are in Appendix C.8, and generation/discrimination prompts and examples are in Appendix C.11.2. We will make these selection rules and budgets explicit in Section 5.7.

2. Existing appendix results and newly added experiments. Appendix C.7 (Tables 10–11) already reports synthetic-data scaling for the Qwen2.5-1.5B-Instruct recipient on GSM8K and compares score-based and random filtering from the same SGALM candidate pool:

Existing filtering comparisonRetained 1×1\timesRetained 2×2\timesRetained 4×4\times
Top 1/81/8 of SGALM candidates58.3259.4559.89
Random 1/81/8 of the same candidate pool56.8256.1755.03

The appendix also evaluates SGALM fine-tuning on additional models and tasks. Appendix C.3 (Table 7) covers Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct on GSM8K, ARC-C, and MBPP, including D-only/G-only ablations. Appendix C.1 (Table 5) evaluates UltraChat200k training on six benchmarks, with a 62.7 average for SGALM versus 61.6 for SPIN. Appendix C.4 (Table 8) reports Qwen2.5-3B-Instruct on IFEval, where strict prompt-level accuracy is 49.4 versus 47.6 for SPIN.

To extend the synthetic-data-engine experiment to another task and recipient family, we added ARC-C with a Llama-3.2-3B-Instruct recipient, using the existing ARC-C-trained Qwen2.5-3B source. SGALM's top-score filter reaches 60.27 versus 58.45 for random selection from the same pool, a 1.82-point improvement.

Each pipeline generates 8N8N candidate responses and retains NN examples for NN real training examples. RFT samples eight responses per existing question; Self-Instruct+RFT proposes NN new questions and samples eight responses to each. Recipient training-token budgets, validation rules, format checks, and deduplication are matched across methods; real-only SFT repeats its data to match the training-token budget.

ARC-C data for Llama-3.2-3B-Instruct recipientRetained synthetic examplesRecipient accuracyGenerated tokensSynthesis/scoring GPU-hours
Real only, matched training-token budget057.96 ± 0.470.00M0.00
Real + Self-InstructNN58.62 ± 0.453.39M3.22
Real + RFTNN58.79 ± 0.391.71M1.65
Real + Self-Instruct+RFTNN59.16 ± 0.421.91M2.03
Real + SGALM, random 1/81/8NN58.45 ± 0.523.62M3.41
Real + SGALM, top 1/81/8NN60.27 ± 0.473.62M4.04

Accuracy is mean ± standard deviation across three independent synthesis/training seeds. Generated tokens include question proposals and repeated responses. GPU-hours cover synthesis and scoring, excluding source-model adaptation and recipient SFT.

Q2. Evaluation/scoring protocols and the GSM8K discrepancy

Response. We evaluate GSM8K using zero-shot chain-of-thought prompting. While the technical report uses four-shot chain-of-thought prompting. Our code has been released at SGALM_Code, with the GSM8K evaluation command in the README.

Re-evaluating the released checkpoints under a common configuration yields 72.43 for Qwen2.5-1.5B-Instruct, close to the 73.2 cited from the Qwen2.5 Technical Report, Tables 9–10. The table separates the submitted scores, reproduction under the original configuration, and evaluation under the common configuration from the technical report. Within the common configuration, SGALM exceeds the corresponding SFT controls by 1.84 points for 3B and 1.79 points for 1.5B.

Checkpoint/training dataSubmitted GSM8K (paper)Reproduced 0-shot configurationReproduced 4-shot configuration
Qwen2.5-1.5B-Instruct, unchanged50.7250.7672.43
Qwen2.5-1.5B-Instruct, real-data SFT57.1257.1174.87
Qwen2.5-1.5B-Instruct, real + SGALM 4×4\times59.8959.9776.66

The released evaluation command runs lm_eval with --tasks gsm8k_cot_zeroshot, --num_fewshot 0, and greedy decoding (temperature=0, do_sample=False, repetition_penalty=1.0). Four-shot contexts are used only for training-time synthetic generation. The score changes across configurations do not isolate which protocol factors explain the discrepancy; we will release both configurations with per-example outputs and extracted answers.

Q3. SFT training budget and comparability

Response. The code to reproduce has been released at SGALM_Code.

The submission reports 68.61 for SFT versus 67.95 for the unchanged model on GSM8K. All fine-tuned baselines, including SFT, have matched real-data exposure; the self-play methods also have matched total real-plus-synthetic exposure. The existing D-only/G-only comparisons in Tables 1 and 7 test the training mechanisms, as discussed under Reviewer SmC9 (W1(ii)).

The original SFT run uses 6,973 real examples for four epochs and 436 optimizer steps, yielding the reported 0.66-point GSM8K gain. The table specifies each method's data, updates, and checkpoint rule. All use batch size 64, with AdamW for SFT/Self-Instruct and RMSProp for policy/preference updates.

Original GSM8K settingUnique real examplesSynthetic examples per roundTotal epochs/roundsOptimizer stepsLearning rate and checkpoint rule
SFT6,97304 epochs4365e-7; best validation epoch among 4
Self-Instruct6,9736,9734 rounds × 1 epoch8725e-7; best validation round among 4
SPIN6,9736,9734 rounds × 1 epoch4361e-6; best validation round among 4
SGALM6,9736,9734 rounds × 1 epoch4365e-7; best validation round among 4

All methods use the same 6,973/500 training/validation split. SGALM uses 55,784 discriminator labels across four rounds.

W2 and Q4. Results on a newer model

Response. The added Qwen3-8B comparison covers seven benchmarks, including MATH-500, where SGALM scores 86.53 versus 84.40 for SFT and 85.73 for IRIS. All methods use non-thinking mode and shared inference limits. The complete results are reproduced below; the training and evaluation settings are described under Reviewer ocuV (Q3–Q4).

Qwen3-8B methodARC-CTruthfulQAWinoGrandeGSM8KHellaSwagMMLUMATH-500GPU-hours
Base68.4359.0874.1992.4978.3578.8884.200.0
SFT, matched training-token budget69.08 ± 0.2759.65 ± 0.3574.40 ± 0.2892.87 ± 0.2378.58 ± 0.1879.06 ± 0.1284.40 ± 0.35125.3
SPIN69.68 ± 0.2659.96 ± 0.3574.56 ± 0.2893.18 ± 0.2078.74 ± 0.1879.13 ± 0.1384.80 ± 0.35192.5
SPACE70.11 ± 0.2660.51 ± 0.3574.93 ± 0.2593.35 ± 0.1678.91 ± 0.1779.28 ± 0.1685.20 ± 0.40195.1
T-SPIN70.34 ± 0.2560.84 ± 0.3375.14 ± 0.2893.56 ± 0.2079.13 ± 0.1779.46 ± 0.1585.53 ± 0.31237.0
IRIS70.62 ± 0.2160.99 ± 0.3574.80 ± 0.2893.76 ± 0.1979.29 ± 0.1879.65 ± 0.1485.73 ± 0.42206.8
SGALM71.33 ± 0.2661.83 ± 0.3474.85 ± 0.2894.06 ± 0.1979.73 ± 0.1779.42 ± 0.1586.53 ± 0.31214.2

Fine-tuned scores are mean ± standard deviation across three independent training seeds; Base is a single checkpoint evaluation. GPU-hours are per seed on 4 × A100-80GB and include generation, scoring, and training.

Thank you for the detailed response and additional experiments, which have addressed most of my concerns, so I will raise my score accordingly.

Reviewer_ocuVOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-generative adversarial fine-tuning framework for large language models, namely SGALM. The key idea is to employ a single LLM as both the generator and the discriminator. The generator uses few-shot prompts to create complete new examples, and the discriminator is trained to distinguish real examples from generated ones. Its output probabilities for “Real” and “Fake” provide a continuous reward for improving generation through policy gradient updates. The authors also evaluate SGALM on mathematical reasoning, science question answering, code generation, and instruction following. The theoretical analysis revisits the classical GAN optimality results and, under additional assumptions, connects few-shot distribution matching to zero-shot alignment with the target distribution.

Critical strengths
  1. Improving models with self-generated data when real data is limited is an important problem in LLM post-training. This paper resolves this problem by contrasting real and generated examples, offering an effective way to data-scarce scenarios.

  2. This paper proposes using the same model for both generation and discrimination, without a separate classification head. This makes the approach easy to understand and implement.

Critical weaknesses
  1. The paper lacks a detailed discussion of prior work [1], which proposes a GAN-based self-play fine-tuning method and provides theoretical optimality guarantees. The authors should provide a more careful comparison of the methods and experimental settings.

[1] SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models. 2025.

Questions for authors
  1. Why does SGALM use the probabilities of “Real” and “Fake” as a quality signal? This design seems trivial, and the paper does not sufficiently explain why these probabilities reliably reflect sample quality.

  2. SGALM is largely based on the standard GAN framework, and its technical novelty appears limited.

  3. The experiments lack comparisons with recent self-play fine-tuning methods [1,2]. The main experiments focus on Qwen2.5-3B. Although additional models are evaluated in the appendix, experiments with newer and larger models would provide stronger evidence of the method’s effectiveness.

  4. The main evaluation focuses on GSM8K, ARC-Challenge, and MBPP. A broader and more systematic evaluation, covering multi-domain knowledge, commonsense reasoning, and more challenging mathematical reasoning tasks, would strengthen the paper.

[1] Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs. 2025.

[2] IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning. 2026.

5 comments

Response to Reviewer ocuV

We sincerely thank the reviewer for pointing out important related works missing from our discussion, including SPACE and T-SPIN. We have already added both works to the references and cited them. We will provide a detailed comparison of them with SGALM in later manuscript.

[1] SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models. NeurIPS 2025.

[2] Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs. NeurIPS 2025.

W1 and Q2. Relationship to SPACE and technical novelty

Response. SGALM's ICL-based generation operates on complete samples zz, which may be documents, instruction–response pairs, or other text sequences (Section 3). It supports data beyond (x,y)(x,y) pairs and can generate entirely new samples, including new questions together with their answers, making the trained model a synthetic-data engine.

Both SGALM and SPACE build on adversarial distribution matching. In SPACE, the model generates responses to existing prompts, and its NCE classifier uses the likelihood ratio between the current policy and the previous-iteration opponent. With synthetic/real ratio μ\mu,

=\sigma\\!\left(\log p_\theta(y\mid x)-\log p_{\theta_t}(y\mid x)-\log\mu\right).$$ For feedback, SGALM trains an explicit source classifier over complete samples through the LLM's Real/Fake output probabilities: $$D_{\mathrm{SGALM}}(z) =\frac{p_\theta(\mathrm{Real}\mid D_{\mathrm{Prompt}},z)} {p_\theta(\mathrm{Real}\mid D_{\mathrm{Prompt}},z)+p_\theta(\mathrm{Fake}\mid D_{\mathrm{Prompt}},z)}.$$ Binary cross-entropy trains this score, which supplies a detached reward for the policy-gradient generation objective within shared parameters. Section 5.7 evaluates the trained model as a synthetic-data generator and filter. Theorem 4.3 connects context-marginal few-shot distribution matching to zero-shot alignment; the classical GAN equilibrium results are attributed in Section 4.1 and Appendix B. The comparison with SPACE, T-SPIN, and IRIS is summarized below. We will expand this comparison in the revised paper. | Method | Generated training example | Feedback parameterization | Main optimization distinction | |---|---|---|---| | [SPACE](https://arxiv.org/html/2512.07175v1) | New response to an existing prompt | NCE classifier from current/opponent response likelihoods | Separate real/synthetic classification terms; opponent copied between rounds | | [T-SPIN](https://arxiv.org/html/2601.08198v1) | New response to an existing prompt | Relative scores of real, current synthetic, and initial-policy responses | Historical-response anchor and entropy-based reference-free formulation | | [IRIS](https://arxiv.org/html/2604.20933v1) | New response to an existing prompt | Current/opponent log-likelihood ratio with tilted-risk weighting | Rényi-order-controlled real/synthetic terms and adaptive order schedule | | SGALM | New complete sample via ICL: a document, an $(x,y)$ pair, or other text | Explicit Real/Fake output probabilities | Source-classification update plus detached-reward policy-gradient generation update |

Q1. Why Real/Fake probabilities provide useful feedback

Response. Our setting assumes that real examples are high-quality and correct, but costly to obtain. If the generated samples can not be discriminated from real ones by a capable discriminator, then the generated samples are likely to be high-quality and correct. Through repeated comparisons of real and generated examples, the discriminator learns to distinguish generated data from this trusted reference. The generator then learns to produce samples that the trained discriminator cannot distinguish from real data. The underlying idea is that matching the distribution of high-quality, correct examples lets us generate more examples of that quality; Real/Fake probabilities provide the learned feedback for this process.

The held-out diagnostics support this relationship: the scores distinguish correct from incorrect generated responses with AUROC 0.86 on GSM8K and 0.79 on MBPP. Each dataset uses four responses for each of 25 held-out prompts; task-specific correctness verifiers are used only for diagnosis.

Existing diagnosticGSM8KMBPP
Mean score for correct generated responses0.410.56
Mean score for incorrect generated responses0.020.15
Correctness AUROC0.860.79
Score: original / semantic corruption / surface change0.93 / 0.01 / 0.790.70 / 0.11 / 0.64

The population analysis formalizes this intuition: at the ideal adversarial equilibrium, pG=pTp_G=p_T. With balanced source sampling and a fixed generator, the optimal discriminator is

D∗(z)=pT(z)pT(z)+pG(z).D^*(z)=\frac{p_T(z)}{p_T(z)+p_G(z)}.

This score measures target-to-generator relative density. Under the clean-target assumption, an incorrect sample outside the target support has D∗(z)=0D^*(z)=0. It therefore provides a distribution-based learning signal, although a learned discriminator's score is not a calibrated probability of correctness.

Q3–Q4. Recent baselines, a newer backbone, and broader evaluation

Response. We added a seven-benchmark Qwen3-8B comparison with SPACE, T-SPIN, and IRIS. SGALM achieves the highest mean score on five of seven tasks, including 86.53 on MATH-500 versus 85.73 for IRIS.

All methods start from the same checkpoint and use a fixed 20k-example UltraChat training subset with a disjoint 1k-example validation subset. Iterative methods share the per-round candidate budget, up to four rounds with one training epoch per round; SFT receives a matched training-token budget. Method-specific losses follow the authors' implementations, with the same validation-search allowance and checkpoint rule.

All methods use Qwen3-8B's non-thinking mode, identical task configurations, and shared per-task inference limits. Metrics are answer exact match for GSM8K and MATH-500, MC2 for TruthfulQA, and the frozen task configurations' accuracy/normalized accuracy for the other benchmarks. Benchmark test labels are excluded from training and checkpoint selection.

Qwen3-8B methodARC-CTruthfulQAWinoGrandeGSM8KHellaSwagMMLUMATH-500GPU-hours
Base68.4359.0874.1992.4978.3578.8884.200.0
SFT, matched training-token budget69.08 ± 0.2759.65 ± 0.3574.40 ± 0.2892.87 ± 0.2378.58 ± 0.1879.06 ± 0.1284.40 ± 0.35125.3
SPIN69.68 ± 0.2659.96 ± 0.3574.56 ± 0.2893.18 ± 0.2078.74 ± 0.1879.13 ± 0.1384.80 ± 0.35192.5
SPACE70.11 ± 0.2660.51 ± 0.3574.93 ± 0.2593.35 ± 0.1678.91 ± 0.1779.28 ± 0.1685.20 ± 0.40195.1
T-SPIN70.34 ± 0.2560.84 ± 0.3375.14 ± 0.2893.56 ± 0.2079.13 ± 0.1779.46 ± 0.1585.53 ± 0.31237.0
IRIS70.62 ± 0.2160.99 ± 0.3574.80 ± 0.2893.76 ± 0.1979.29 ± 0.1879.65 ± 0.1485.73 ± 0.42206.8
SGALM71.33 ± 0.2661.83 ± 0.3474.85 ± 0.2894.06 ± 0.1979.73 ± 0.1779.42 ± 0.1586.53 ± 0.31214.2

Fine-tuned scores are mean ± standard deviation across three independent training seeds; Base is a single checkpoint evaluation. GPU-hours are per seed on 4 × A100-80GB and include generation, scoring, and training, including T-SPIN's initial-policy anchor generation.

Thanks for the detailed response. I still have the following concerns:

  • To W1: My understanding is that DSGALMD_{\text{SGALM}} is essentially a special case of DSPACED_{\text{SPACE}} with μ=1\mu=1, i.e., when positive and negative examples are balanced. This limits the novelty of the proposed method.

  • To Q1: The response mainly explains the basic idea of self-play fine-tuning, but does not directly address why the probabilities of “Real” and “Fake” should be used as a quality signal. For example, could the probabilities of alternative labels, such as “Positive” and “Negative,” serve the same purpose? The rationale for this choice remains unclear.

Reviewer_SmC9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SGALM, a self-improvement framework that adapts generative adversarial training to large language models. A single LLM generates complete examples from few-shot prompts and learns to distinguish its generations from real data. Discrimination uses “Real”/“Fake” token probabilities, while generation receives discriminator-derived rewards through policy-gradient updates. This grounds self-improvement in existing examples without requiring external reward models or additional human annotations. The authors establish sufficient conditions connecting idealized few-shot distribution matching to zero-shot target alignment. Experiments show improvements over supervised fine-tuning and self-improvement baselines in mathematics, scientific reasoning, and coding. Beyond improving the model itself, SGALM serves as a synthetic-data generator and filter, producing training examples that improve a separate model’s downstream performance.

Critical strengths
  1. An interesting discovery about an auxiliary training mechanism. The author discovers that a GAN like training, where an LLM serving as both the generator and the discriminator, could improve the performance of downstream tasks like QA.
  2. A strong theoretical analysis Sufficient theoretical analysis is provided in both the main text and the supplementary materials, including but not limited to the convergence analysis, and the illustration of not overfitting on the dataset themselves.
  3. A synthetic data generator by the side. The GAN-like training mechanism also brought forth a synthetic data generator, which is likelly to serve as a data augmentation mechanism for subsequent training tasks.
Critical weaknesses
  1. The training mechanism still require the existence of training data. Although the authors do not use the data in directly training these tasks, the data is still seen in the training process of the LLM backbone itself. This yields two deficits. (i) This partially breaks the claim of training data free assumption, as the current GAN-like training process cannot go without the existence of real data. (ii) There is an ambiguity in the attribution of the performance gains. We are not sure whether the performance gain is from the GAN like training, or it is from the extra acquaintance of the dataset in the autoregressive manner, as autoregressive pretraining still gives performance gain on downstream tasks.
  2. A typo issue. The author had better wrap the citations in a parenthesis via \citep{}
Questions for authors
  1. Can the author provide more theoretical analysis to prove the cross-task transfer to illustrate the capability of the proposed mechnism having an emerging capability of cross task transfer?
  2. Can the author swap the presentation of the experiement results, letting cross task trasfer at Appendix C.1 to be the main result, while letting the current in-distribution validaton result to be supplementary?

Trying to answer these two questions can showcase the claim of not requiring extra human annotations, and will make this paper logically comprehensive.

6 comments

Response to Reviewer SmC9

We sincerely thank the reviewer for recognizing the usefulness of the auxiliary adversarial training mechanism, the few-shot/zero-shot analysis, and the synthetic-data application. Your comments help us clarify both the supervision setting and what the evidence establishes.

W1(i). Supervision from existing real data

Response. The paper does not assume training-data-free learning. Our claim is self-improvement without additional human annotations or an external reward model. The abstract describes learning without new human labels or external feedback, and Section 3 states: “We are given a dataset D\mathcal{D} of real samples drawn from a target distribution pTp_T.” This dataset supplies the generation contexts and real examples for source discrimination; Real/Fake labels follow automatically from sample provenance. We will make this more explicit.

W1(ii). Attribution of improvements beyond additional data exposure

Response. The comparisons in Tables 1, 5, 7, and 8 already control data exposure. All fine-tuned baselines have the same real-data exposure as SGALM, including SFT in Tables 1, 7, and 8. All self-play methods also have the same total data exposure to real and synthetic examples, with matched synthetic-sample budgets and the same four-shot setting for ICL-based generation (Section 5.1 and Appendix C.8). These existing controls show that SGALM's gains cannot be attributed simply to more data exposure.

The D-only and G-only ablations in Tables 1 and 7 further test the training mechanisms. On GSM8K in Table 1, D-only and G-only reach 69.83 and 69.98, respectively, versus 68.61 for SFT; combining the objectives gives 72.71. These results support the effectiveness of both source-discrimination learning and the generator objective, and the benefit of combining them.

W2. Typo

Thanks for your careful reading. We will correct the citation formatting and typographical errors.

Q1. Few-shot generation and zero-shot alignment

Response. We do not assume cross-task transfer. Theorem 4.3 concerns few-shot generation and zero-shot behavior for the same target distribution under the shared latent-domain and identifiability assumptions.

Appendix C.1 follows the general-data self-play setting used by SPIN: one model is fine-tuned on UltraChat200k and evaluated on six benchmarks. UltraChat is treated as a broad instruction distribution covering the domains represented by these benchmarks. The six-benchmark results therefore assess multi-domain improvement within this general target distribution. We will make this relationship explicit.

Q2. Promoting the six-benchmark evaluation to the main text

Response. Thanks for your kind advice. We will consider moving the six-benchmark results from Appendix C.1 into the main experimental section to compare with SPIN under the matched UltraChat200k setting.

Reviewer_SmC9OpenReview ↗

Further question about your reply in W1.

Throughout my reading of your paper, I believe the real use of this work is to deploy existing LLMs to domains where nearly no existing datasets is avaiable, as the current version of the abstract claims. However, in your response to w1, you said the dataset is still required, so does section 3. I think there is a huge misalignment inside, and if the author choose to weaken the claim in the abstract, the theoretical contribution will be shrinked by a large extent, and the reviewer is likely to re-assess this paper.

Reviewer_SmC9OpenReview ↗

About the possibility of a demo or theoretical induction for cross-task transfer.

Your reply to Q1 also makes me curious about the possibility of cross-task transfer. Is it likely that the aothor produce a demo or theoretical illustration to showcase the possibility of cross-task transfer? This is also of much practical importance.

Note: This will by no means hurt my overall assessment, no matter what the result is.

Comments on the paper

2 comments

Dear Reviewers, please take a look at the authors' rebuttal and follow up in the discussion to settle any remaining concerns by Nov 18 AoE. Thank you!

Dear reviewers, just a brief reminder that November 18 (AoE) is the last day of author–reviewer discussion.

SmC9: Thank you for following up. Please feel free to share any additional comments before the deadline.

ocuV: Thank you for following up. Please feel free to share any additional comments before the deadline.

2Tn8: Thank you for following up. Please feel free to share any additional comments before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.