- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The authors propose SGALM, an adversarial framework that fine-tunes LLMs by using them as both generators that produce data following a distribution similar to real data and discriminators that distinguish real data from generated data, with each role trained using its corresponding objective.
The discriminator score is computed from the probabilities of the Real/Fake tokens, and the discriminator is trained using a cross-entropy loss based on .
The generator produces data conditioned on in-context examples and is trained through policy-gradient reinforcement learning using a reward derived from .
The authors provide a theoretical analysis of few-shot generation and zero-shot target alignment, report that SGALM achieves strong downstream task performance compared to other baseline methods, and demonstrate that SGALM can serve as a synthetic-data engine.
Critical strengths
1. Applicability of SGALM.
SGALM demonstrates strong performance compared to other baseline methods across tasks with varying amounts of training data.
In particular, its superior performance over SFT and other self-play baselines on MBPP with only 0.1k training examples suggests that SGALM is well suited to data-scarce settings where self-play can be particularly useful.
Furthermore, the fact that using SGALM as a synthetic-data engine leads to better performance than Self-Instruct and RFT suggests that the proposed framework can be effective in settings where obtaining large amounts of data is difficult.
2. Training in an in-context learning setting.
It is interesting that the authors theoretically analyze the relationship between the in-context few-shot generation objective and zero-shot target alignment, and empirically demonstrate improvements in SGALM's zero-shot downstream performance.
Critical weaknesses
1. Concerns about the experimental setup.
In Section 5.7 (SGALM as a synthetic-data engine), SGALM is compared against Self-Instruct, RFT, and Self-Instruct + RFT, but the experimental configurations of these methods are not clearly specified.
In particular, if RFT refers to Rejection-sampling Fine-Tuning, it should involve a data filtering process, but it is unclear how this filtering was performed during data synthesis.
Since the use of SGALM as a synthetic-data engine is one of the main contributions of the paper, I believe these experimental details should be described more thoroughly.
Additionally, the substantial discrepancy between the GSM8K performance of Qwen2.5-3B/1.5B-Instruct reported in Tables 1 and 3 and the performance reported in the Qwen2.5 Technical Report [1] (this paper: 67.95/50.72; Qwen2.5 Technical Report: 86.7/73.2) raises further concerns about the experimental setup.
2. The models used in the experiments are relatively outdated.
The experiments are conducted using relatively old models, such as Qwen2.5, Llama-3.2, and Llama-3.1, raising questions about whether the proposed method can generalize to more capable, recent models.
[1] Qwen et al. “Qwen2.5 Technical Report.” arXiv preprint arXiv:2412.15115 (2025).
Questions for authors
1. Could you provide more details about the experimental setup in Section 5.7?
Additionally, if time and computational resources permit, I would appreciate evaluations on other models or tasks in Section 5.7. It is difficult to determine whether SGALM truly outperforms the other methods based on only a single model and task.
2. Could you clarify the exact evaluation and scoring protocols?
In particular, could you explain why the GSM8K performance of the Qwen2.5-Instruct models differs substantially from the results reported in the Qwen2.5 Technical Report?
3. In Table 1, the GSM8K performance of SFT improves by only 0.7 percentage points over the base model, which seems somewhat unusual.
Could you provide more details about the training configurations of SFT and the other baselines (e.g., the number of training steps and the total amount of training data used) to ensure a fair comparison with SGALM?
4. Could you provide results from applying the proposed framework to more recent models, such as Qwen3 or Qwen3.5? If the tasks used in the paper are already saturated by these models, results on different tasks would also be acceptable.
I would be willing to reconsider my score based on the authors' responses to these questions.
3 comments
Part II
Q2. Evaluation/scoring protocols and the GSM8K discrepancy
Response. We evaluate GSM8K using zero-shot chain-of-thought prompting. While the technical report uses four-shot chain-of-thought prompting. Our code has been released at SGALM_Code, with the GSM8K evaluation command in the README.
Re-evaluating the released checkpoints under a common configuration yields 72.43 for Qwen2.5-1.5B-Instruct, close to the 73.2 cited from the Qwen2.5 Technical Report, Tables 9–10. The table separates the submitted scores, reproduction under the original configuration, and evaluation under the common configuration from the technical report. Within the common configuration, SGALM exceeds the corresponding SFT controls by 1.84 points for 3B and 1.79 points for 1.5B.
| Checkpoint/training data | Submitted GSM8K (paper) | Reproduced 0-shot configuration | Reproduced 4-shot configuration |
|---|---|---|---|
| Qwen2.5-1.5B-Instruct, unchanged | 50.72 | 50.76 | 72.43 |
| Qwen2.5-1.5B-Instruct, real-data SFT | 57.12 | 57.11 | 74.87 |
| Qwen2.5-1.5B-Instruct, real + SGALM | 59.89 | 59.97 | 76.66 |
The released evaluation command runs lm_eval with --tasks gsm8k_cot_zeroshot, --num_fewshot 0, and greedy decoding (temperature=0, do_sample=False, repetition_penalty=1.0). Four-shot contexts are used only for training-time synthetic generation. The score changes across configurations do not isolate which protocol factors explain the discrepancy; we will release both configurations with per-example outputs and extracted answers.
Q3. SFT training budget and comparability
Response. The code to reproduce has been released at SGALM_Code.
The submission reports 68.61 for SFT versus 67.95 for the unchanged model on GSM8K. All fine-tuned baselines, including SFT, have matched real-data exposure; the self-play methods also have matched total real-plus-synthetic exposure. The existing D-only/G-only comparisons in Tables 1 and 7 test the training mechanisms, as discussed under Reviewer SmC9 (W1(ii)).
The original SFT run uses 6,973 real examples for four epochs and 436 optimizer steps, yielding the reported 0.66-point GSM8K gain. The table specifies each method's data, updates, and checkpoint rule. All use batch size 64, with AdamW for SFT/Self-Instruct and RMSProp for policy/preference updates.
| Original GSM8K setting | Unique real examples | Synthetic examples per round | Total epochs/rounds | Optimizer steps | Learning rate and checkpoint rule |
|---|---|---|---|---|---|
| SFT | 6,973 | 0 | 4 epochs | 436 | 5e-7; best validation epoch among 4 |
| Self-Instruct | 6,973 | 6,973 | 4 rounds × 1 epoch | 872 | 5e-7; best validation round among 4 |
| SPIN | 6,973 | 6,973 | 4 rounds × 1 epoch | 436 | 1e-6; best validation round among 4 |
| SGALM | 6,973 | 6,973 | 4 rounds × 1 epoch | 436 | 5e-7; best validation round among 4 |
All methods use the same 6,973/500 training/validation split. SGALM uses 55,784 discriminator labels across four rounds.
W2 and Q4. Results on a newer model
Response. The added Qwen3-8B comparison covers seven benchmarks, including MATH-500, where SGALM scores 86.53 versus 84.40 for SFT and 85.73 for IRIS. All methods use non-thinking mode and shared inference limits. The complete results are reproduced below; the training and evaluation settings are described under Reviewer ocuV (Q3–Q4).
| Qwen3-8B method | ARC-C | TruthfulQA | WinoGrande | GSM8K | HellaSwag | MMLU | MATH-500 | GPU-hours |
|---|---|---|---|---|---|---|---|---|
| Base | 68.43 | 59.08 | 74.19 | 92.49 | 78.35 | 78.88 | 84.20 | 0.0 |
| SFT, matched training-token budget | 69.08 ± 0.27 | 59.65 ± 0.35 | 74.40 ± 0.28 | 92.87 ± 0.23 | 78.58 ± 0.18 | 79.06 ± 0.12 | 84.40 ± 0.35 | 125.3 |
| SPIN | 69.68 ± 0.26 | 59.96 ± 0.35 | 74.56 ± 0.28 | 93.18 ± 0.20 | 78.74 ± 0.18 | 79.13 ± 0.13 | 84.80 ± 0.35 | 192.5 |
| SPACE | 70.11 ± 0.26 | 60.51 ± 0.35 | 74.93 ± 0.25 | 93.35 ± 0.16 | 78.91 ± 0.17 | 79.28 ± 0.16 | 85.20 ± 0.40 | 195.1 |
| T-SPIN | 70.34 ± 0.25 | 60.84 ± 0.33 | 75.14 ± 0.28 | 93.56 ± 0.20 | 79.13 ± 0.17 | 79.46 ± 0.15 | 85.53 ± 0.31 | 237.0 |
| IRIS | 70.62 ± 0.21 | 60.99 ± 0.35 | 74.80 ± 0.28 | 93.76 ± 0.19 | 79.29 ± 0.18 | 79.65 ± 0.14 | 85.73 ± 0.42 | 206.8 |
| SGALM | 71.33 ± 0.26 | 61.83 ± 0.34 | 74.85 ± 0.28 | 94.06 ± 0.19 | 79.73 ± 0.17 | 79.42 ± 0.15 | 86.53 ± 0.31 | 214.2 |
Fine-tuned scores are mean ± standard deviation across three independent training seeds; Base is a single checkpoint evaluation. GPU-hours are per seed on 4 × A100-80GB and include generation, scoring, and training.
Thank you for the detailed response and additional experiments, which have addressed most of my concerns, so I will raise my score accordingly.
Part I
Response to Reviewer 2Tn8
We sincerely thank the reviewer for recognizing SGALM's usefulness in data-scarce settings and the connection between in-context generation and zero-shot evaluation. We appreciate the specific questions about reproducibility; the evaluation discrepancy and baseline descriptions need to be resolved explicitly.
W1 and Q1. Experimental settings and broader evaluations
Response.
1. Experimental settings in Section 5.7. The source is SGALM-trained Qwen2.5-3B-Instruct, which generates complete GSM8K examples using four-shot ICL. A separate Qwen2.5-1.5B-Instruct recipient is fine-tuned on real data plus the retained synthetic examples. Retained scales of 1×, 2×, and 4× use candidate pools of 8×, 16×, and 32× the real training-set size. SGALM retains the top 1/8 by its learned Real/Fake score; RFT matches its total candidate count at each scale.
RFT groups sampled responses by their extracted final answers and retains responses from groups in descending frequency until the budget is filled, sampling uniformly within the final group if necessary. To make fair comparison, it uses neither ground-truth answers nor an external correctness verifier. Self-Instruct generates new complete examples; Self-Instruct+RFT first proposes new questions and then applies the same answer-frequency selection to their sampled responses. Implementation settings are in Appendix C.8, and generation/discrimination prompts and examples are in Appendix C.11.2. We will make these selection rules and budgets explicit in Section 5.7.
2. Existing appendix results and newly added experiments. Appendix C.7 (Tables 10–11) already reports synthetic-data scaling for the Qwen2.5-1.5B-Instruct recipient on GSM8K and compares score-based and random filtering from the same SGALM candidate pool:
The appendix also evaluates SGALM fine-tuning on additional models and tasks. Appendix C.3 (Table 7) covers Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct on GSM8K, ARC-C, and MBPP, including D-only/G-only ablations. Appendix C.1 (Table 5) evaluates UltraChat200k training on six benchmarks, with a 62.7 average for SGALM versus 61.6 for SPIN. Appendix C.4 (Table 8) reports Qwen2.5-3B-Instruct on IFEval, where strict prompt-level accuracy is 49.4 versus 47.6 for SPIN.
To extend the synthetic-data-engine experiment to another task and recipient family, we added ARC-C with a Llama-3.2-3B-Instruct recipient, using the existing ARC-C-trained Qwen2.5-3B source. SGALM's top-score filter reaches 60.27 versus 58.45 for random selection from the same pool, a 1.82-point improvement.
Each pipeline generates 8N candidate responses and retains N examples for N real training examples. RFT samples eight responses per existing question; Self-Instruct+RFT proposes N new questions and samples eight responses to each. Recipient training-token budgets, validation rules, format checks, and deduplication are matched across methods; real-only SFT repeats its data to match the training-token budget.
Accuracy is mean ± standard deviation across three independent synthesis/training seeds. Generated tokens include question proposals and repeated responses. GPU-hours cover synthesis and scoring, excluding source-model adaptation and recipient SFT.