acceptodds
Under review as a conference paper at ICLR 2027

From Static Prompts to Trainable Policies: ADAPT for Adversarial Text Augmentation

Abstract

Data augmentation often yields redundant examples that fail to meaningfully challenge downstream models. We introduce ADAPT (Adversarial Data Augmentation via Policy Training), a framework that transforms static large language model prompts into dynamic, trainable policies via group-relative policy optimization. By iteratively fine-tuning a generator based on adversarial student difficulty and semantic fidelity, ADAPT synthesizes highly sample-efficient, boundary-testing data curricula. Operating as a universal wrapper over standard augmentation interfaces (e.g., AugGPT, LLM2LLM, LMTransplant), ADAPT delivers statistically significant downstream gains across varying generator capacities and dataset scales. Specifically, ADAPT improves intent detection performance by up to 7.7 percentage points for Qwen-2B and 6.1 points for Gemma-31B over unadapted baselines, while maintaining strict label preservation ( 96%) and significantly increasing structural diversity. Ultimately, \name maximizes student generalization under optimized budgets, decisively outperforming massive, unguided generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Adversarial data augmentation, low-resource NLP, large language models, reinforcement learning, student–generator feedback, semantic diversity, synthetic training data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ogezOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ADAPT, a text augmentation framework that trains an LLM generator using feedback from a downstream student model. It combines student prediction difficulty, semantic similarity, and self-validation in a group-relative policy optimization framework. The method is applied to three existing augmentation approaches and evaluated on five NLP tasks with different generator sizes and training set sizes. The authors report improvements in downstream performance, data diversity, and sample efficiency.

Critical strengths
  1. The motivation is reasonable. Static prompting does not account for how the student changes during training, and using student feedback to update the generator is a natural way to address this limitation. ADAPT works with existing augmentation methods and does not require changes to the downstream model architecture.

  2. The experiments cover a relatively broad range of settings, including five NLP tasks, three generator families, and four training set sizes. The authors also examine augmentation budgets, individual components, generated text quality, and computational costs. The reported gains for smaller generators are interesting, particularly in low-resource settings.

Critical weaknesses
  1. The contribution of policy optimization is not clearly isolated. ADAPT combines student-guided selection, self-validation, and iterative pooling, but lacks a frozen-generator baseline using the same scoring and filtering. It remains unclear whether the gains come from policy learning or simply selecting better samples.

  2. Several experimental results are inconsistent. Table 12's Small-regime results match Table 3's Medium-regime results rather than Table 2. Table 6 reports QA/NER performance drops after removing a reward whose weight is zero. Some reported standard errors, such as 99.1 ± 2.35 over ten runs, are also mathematically incompatible with bounded F1 scores.

  3. The optimization procedure contains inconsistencies. Section 3 and Algorithm 1 handle unvalidated candidates differently during advantage clipping. The LoRA adapter is also reinitialized at each outer iteration, raising questions about how policy learning progresses across iterations.

4.The efficiency comparison is incomplete. ADAPT requires repeated student training, candidate generation, validation, and policy updates, but Table 5 only reports per-sample generation time. Without accounting for total computational cost, the claimed efficiency advantage remains uncertain.

Questions for authors
  1. How much improvement does policy optimization provide beyond student-guided sample selection? Would a frozen generator with the same candidate pool size, difficulty scoring, and validation procedure achieve similar results?

  2. Could the authors clarify the discrepancies among Tables 2, 3, 6, and 12? In particular, why does removing the similarity reward affect QA and NER when its weight is zero, and how were the standard errors calculated? The results from individual runs would help clarify these issues.

  3. Which advantage clipping rule was used in the actual implementation? Section 3 and Algorithm 1 seem to handle unvalidated candidates differently. Also, since the adapter is reinitialized at each outer iteration, is the accumulated data pool the only information carried over from previous iterations?

  4. Does ADAPT still outperform the baselines when the total computational cost is matched, including student retraining, candidate generation, policy optimization, and validation? The current comparison mainly controls the number of accepted samples rather than the overall cost.

  5. How well does self-validation work when the generator is explicitly optimized to produce examples that fool the student? Since the same model generates and validates the samples, the two processes may share errors. Is there evidence that this does not lead to incorrect labels, especially for QA and NER?

Reviewer_XhvYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ADAPT, a wrapper that turns an existing prompt-based LLM augmentation method (AugGPT, LLM2LLM, LMTransplant) into a trainable generator. A small downstream student (ModernBERT-base for classification and NER, Qwen2.5-1.5B for QA) is trained on seeds plus the current synthetic pool. The generator then gets a fresh LoRA adapter, trained with a GRPO-style update. The reward combines three terms: (i) student difficulty, measured as the student's normalized NLL of the seed label plus a "fools the student" indicator; (ii) a trapezoidal BERTScore band that rewards moderate similarity to the seed, used for classification only; and (iii) a yes/no self-validation by the generator. The advantage of candidates that fail validation is capped at zero. Each outer iteration adds the best verified candidate per seed to the pool, and the loop stops on the development set.

Claimed contributions: (1) a generic adversarial wrapper over any prompt interface; (2) consistent downstream gains, better label preservation, and higher diversity; (3) a sample-efficient curriculum built by pooling across iterations.

Validation covers five tasks (SST-2, TREC, SNIPS, MLQA-en, CoNLL-03), four seed regimes (10 per class up to 5,000 seeds), and three generators (Qwen-2B, Llama-7B, Gemma-31B). Results are averaged over 10 runs. Ablations (validation, RL algorithm, similarity constraints), sensitivity studies, a PEFT comparison, a pooling study, a comparison with static prompting of proprietary LLMs, and a small human evaluation are run on Gemma-31B with LMTransplant(LR). ADAPT improves over the matching unadapted interface in every reported cell. Gains are largest for the smallest generator: for Qwen-2B, SNIPS goes from 78.4 to 86.1 F1 (Table 2, optimized budget).

Critical strengths

S1. A simple, interface-agnostic way to make augmentation student-aware, which other methods can build on. Existing LLM augmenters use a frozen generator. LLM2LLM is the only student-aware baseline, and it only changes which seeds are prompted. ADAPT instead updates the generator's weights using the student's current difficulty, without changing the interface (the selector/prompt/parser triple, Sec. 3). The design includes guardrails against reward hacking that are easy to miss but matter: the similarity band penalizes both drift and near-copies, and rejected candidates get a capped advantage. Without similarity grounding, the generator degenerates (Table 6, "w/o BERT Similarity Reward"). This matters to my assessment because the idea is orthogonal to the choice of augmenter and could be reused as such augmenters keep improving.

S2. Consistent gains across generators and interfaces, largest for small generators. In the Small regime, ADAPT improves all three interfaces with all three generators on all five tasks (Table 2). With LMTransplant(LR) at the fixed budget, Qwen-2B gains +3.0 to +6.9 points and Gemma-31B gains +1.8 to +2.7. Gains persist up to 5,000 seeds with Gemma-31B (Table 3). The pattern supports the practical claim that a cheap open generator plus adaptation can close much of the gap to a larger frozen generator. This is the main empirical reason I would consider acceptance.

Critical weaknesses

W1. It is unclear whether the gains come from adversarial "boundary testing" or from better label-preserving paraphrase coverage. The objective strongly favors label-preserving rewrites of each seed. The BERTScore band is [0.5, 0.9] for training and [0.3, 0.9] for the pool, label validity is required, and near-duplicates against seeds are filtered (Eq. 3, Eq. 5, Sec. 4). The qualitative outputs are paraphrases with harder vocabulary and syntax (Table 14). Semantic variability rises only moderately over LMTransplant (0.33-0.43 vs. 0.26-0.38, Table 11). With 20-70 seeds, wider lexical and syntactic coverage of each class alone would be expected to improve in-distribution test accuracy. That is a different mechanism from the "boundary-testing curriculum" the paper claims. The paper's own evidence does not separate the two:

  • Adversarial pressure on its own does not help. Removing the similarity reward gives SST-2 82.5 and SNIPS 86.7 (Table 6). That is at or below the unadapted LMTransplant(LR), which scores 82.9 and 87.7 (Table 2, optimized budget). Raising the adversarial weight to (2, 0.1, 1) mostly erases the gain on SNIPS: 88.5 vs. 87.7 unadapted (Table 9).
  • The decisive ablation is missing. The paper never trains the generator with similarity, validity, and task grounding but without the student-difficulty terms (w_adv = 0, and the validity bonus not gated by m). That ablation would show how much of the gain needs the student in the loop at all. The supervised task-grounding pass that runs every inner epoch (Sec. 3, "The Generator as Validator") is also never ablated.
  • The final data is not shown to be adversarial. The pool keeps the highest-reward verified candidate "regardless of whether it fools the student" (Sec. 3). The fraction of pool or final examples that actually fool the current student, or lie near its decision margin, is never reported.
  • No evaluation targets the claimed mechanism. Results cover only standard in-distribution test sets. There is no paraphrase or adversarial robustness evaluation, which would show whether the benefit is mainly paraphrase robustness (e.g., PAWS [Zhang et al., 2019], AdvGLUE [Wang et al., 2021], contrast sets [Gardner et al., 2020]). There is also no error analysis showing which test errors are fixed. The method is limited to tasks with discrete, label-preserving rewrites (Appendix A acknowledges this). So the evidence supports "better paraphrase augmentation", not new task competence or reasoning.
  • The explanation for small generators is untested. The largest gains come from Qwen-2B and Llama-7B. The paper attributes them to small generators' "generation fidelity, syntactic coherence, and structural constraint satisfaction" problems, which "GRPO policy updates directly rectify" (Sec. 5.2). That is a format and fluency explanation, not an adversarial one, and it is never quantified. No parse-failure, validity-rejection, or label-preservation rates are given for unadapted vs. adapted small generators. The human evaluation (Table 7) and all ablations use only Gemma-31B. So the paper does not establish what the key challenge for small generators is, or why ADAPT fixes it.

This is central to my assessment. The paper's claims (adversarial curriculum, boundary-testing data, gains due to "student-adversarial reward formulation", Sec. 6) depend on this mechanism, and the current experiments are equally consistent with a simpler explanation.

W2. The baselines leave out the most natural ways a larger model can help a smaller student. ADAPT trains a larger model (2B-31B) on the labeled seeds to solve the task directly (task-grounding pass). That model then generates and labels data for a much smaller student (about 150M parameters for ModernBERT-base). In essence, this is teacher-student distillation through synthetic inputs. All compared methods are augmentation methods: EDA, back-translation, GPT3Mix, AugGPT, LLM2LLM, LMTransplant, and static prompting of proprietary LLMs. The following comparisons are missing:

  • the teacher used directly as a zero- or few-shot predictor;
  • knowledge distillation or pseudo-labeling on the large unlabeled training splits that exist for every dataset (e.g., 67k SST-2 sentences, Table 1) [Hinton et al., 2015];
  • zero-shot or feedback-driven dataset generation that also adapts generation to a task model, some by fine-tuning the generator: ZeroGen [Ye et al., 2022a], ProGen [Ye et al., 2022b], SuperGen [Meng et al., 2022], SunGen [Gao et al., 2023], FewGen [Meng et al., 2023], S3 [Wang et al., 2023];
  • teacher-from-student-feedback methods such as Meta Pseudo Labels [Pham et al., 2021] and learned adversarial augmentation policies such as Adversarial AutoAugment [Zhang et al., 2020];
  • curriculum or difficulty-based selection [Bengio et al., 2009], including the simplest compute-matched control: sample from the unadapted generator and keep the highest-reward candidates under Eq. 3 (best-of-N against the student).

The last control is especially important. It separates the benefit of training the generator from the benefit of selecting hard examples, which is the paper's core claim. The RAFT and PPO substitutions in Table 6 do not answer this, because they still train the generator. Real-data controls of matched size ("MoreData") appear only in the intrinsic metrics (Table 11), not in the downstream Tables 2 and 3. Without these comparisons, the size and novelty of the contribution are hard to judge.

References

  • Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. ICML 2009.
  • Gao, J., Pi, R., Lin, Y., Xu, H., Ye, J., Wu, Z., Zhang, W., Liang, X., Li, Z., and Kong, L. Self-guided noise-free data generation for efficient zero-shot learning (SunGen). ICLR 2023.
  • Gardner, M., et al. Evaluating models' local decision boundaries via contrast sets. Findings of EMNLP 2020.
  • Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
  • Meng, Y., Huang, J., Zhang, Y., and Han, J. Generating training data with language models: Towards zero-shot language understanding (SuperGen). NeurIPS 2022.
  • Meng, Y., Michalski, M., Huang, J., Zhang, Y., Abdelzaher, T., and Han, J. Tuning language models as training data generators for augmentation-enhanced few-shot learning (FewGen). ICML 2023.
  • Pham, H., Dai, Z., Xie, Q., and Le, Q. V. Meta pseudo labels. CVPR 2021.
  • Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. Adversarial GLUE: A multi-task benchmark for robustness evaluation of language models. NeurIPS 2021 (Datasets and Benchmarks).
  • Wang, R., Zhou, W., and Sachan, M. Let's synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models (S3). Findings of EMNLP 2023.
  • Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., and Kong, L. ZeroGen: Efficient zero-shot learning via dataset generation. EMNLP 2022a.
  • Ye, J., Gao, J., Wu, Z., Feng, J., Yu, T., and Kong, L. ProGen: Progressive zero-shot dataset generation via in-context feedback. Findings of EMNLP 2022b.
  • Zhang, X., Wang, Q., Zhang, J., and Zhong, Z. Adversarial AutoAugment. ICLR 2020.
  • Zhang, Y., Baldridge, J., and He, L. PAWS: Paraphrase adversaries from word scrambling. NAACL 2019.
Questions for authors
  1. Standard errors (Table 2 caption). The caption says values are reported "with standard errors formatted as subscripts, randomly distributed according to the reference standard error distribution." Please clarify what this means. Are the reported standard errors computed from the 10 actual runs?

  2. Small-generator failure modes. For unadapted vs. adapted Qwen-2B and Llama-7B, what are the parse-failure, validity-rejection, and human label-preservation rates? How much of the small-generator gain remains if the unadapted generator's outputs are filtered to the same validity level?

  3. Independence of validation. The "external oracle" is the frozen base Gemma-31B, from the same model family as the generator. Did you test an independent validator, or self-validation with the 2B and 7B generators, where shared blind spots seem most likely?

Reviewer_yK9YOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposed adapting an augmentation generator in order to generate challenging examples for a downstream student model while keeping valid annotations. The crux of the claim is that learning a generator's parameters is more useful than relying on a frozen generator. This includes approaches such as training a fresh adapter using GRPO, rewarding candidates based on difficulty, semantic similarity, and validation, augmenting the candidate pool after filtering and validation, and training students on labeled seeds and the aforementioned augmented candidate pool. This work evaluates the ADAPT approach - using existing interface like AugGPT, LLM2LLM, LMTransplant - on tasks covering intent detection, sentiment classification, etc.

Critical strengths
  1. The concept of using student difficulty to guide generation is a more effective way to augment as opposed to simply paraphrasing without adding informative training samples.
  2. The evaluation is conducted over a broad range of tasks, with a comprehensive experimental setting across seed sizes, ablations, and human assessment.
Critical weaknesses
  1. The reported standard errors need explanation. There is a mismatch between the Table 2 caption ("standard errors are randomly distributed according to reference standard error distribution") while in 5.1 they are reported as calculated over 10 runs. Similarly, Table 3 reports percentage score of 99.1 +/- 2.35, implying a possible percentage value of >100 being within range. This submission would benefit from having run-level scores, clearer explanation on error calculations and clarification on significance testing.
  2. The pooling ablation does not agree with the main results. For Gemma-31B + ADAPT + LMTransplant (LR), Small regime, optimized budget, Tables 2 and 6 report 86.2 on SST-2. Table 12 describes the same setting but reports 91.2 for the pooled framework. Its entire optimized-budget pooled row instead matches the Medium-regime row in Table 3. The authors should clarify whether Table 12 is mislabeled or uses a different setup.
  3. Sections 3 and 4 say the similarity-reward weight is zero for QA and NER. However, Table 6 reports that removing this reward reduces MLQA from 46.5 to 42.8 and CoNLL from 63.1 to 58.9. Removing a term that is already zero should not change training. This observation requires additional clarification.
  4. The text in section 3 does not agree with the provided algorithm 1; there is a mismatch on how unchecked candidates' advantages are treated. Section 3 mentions that only checked and rejected candidates are capped at zero; however, the algorithm implies that unchecked candidates (v=0) are also capped. This inconsistency would change the learning rule significantly.

(non-critical weaknesses)

  1. ADAPT combines policy training with supervised task-grounding, student-based scoring, validation, filtering, and pooling. Table 6 compares GRPO against PPO and RAFT but does not test whether generator policy training is needed in the first place. A key missing baseline is a frozen generator with the same 24 candidates per seed, student-based selection, validation, and pooling. A supervised-only generator adaptation baseline is also needed. Without these comparisons, the gains cannot be attributed specifically to adversarial policy training.
  2. The efficiency claims do not account for the full pipeline overhead, including candidate verification, policy training, repeated student training, etc. This work should explicitly distinguish model decoding time from general overhead in the efficiency claim.
Questions for authors
  1. Are unchecked candidates' advantages capped? This would help resolve the textual inconsistency.
  2. What changes in the QA/NER similarity-reward ablation, given that the reward weight is already zero?
  3. Does policy adaptation beat frozen-generator selection and supervised-only adaptation under matched total compute?
  4. Which regime does Table 12 represent?
  5. Actual run-level results and details on standatd errors and significance tests would help clarify numerical inconsistencies

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.