acceptodds
Under review as a conference paper at ICLR 2027

Meta-Weighted Self-Training for Language Models under Extreme Data Scarcity

Abstract

Adapting language models from only a small set of prompt–completion pairs is challenging: direct fine-tuning overfits, while self-training on model-generated data can reinforce errors. We propose , a meta-weighted self-training framework for extreme data scarcity that assigns each synthetic example a weight reflecting its : examples are upweighted if imitating them leads, after several adaptation steps, to the largest likelihood improvement on the real demonstrations. The weights therefore assign credit over future training dynamics rather than ranking examples by current sample quality. Starting from a frozen pretrained backbone with LoRA adapters, MWeiST generates synthetic completions and optimises these weights via a meta-objective on the held-out real demonstrations. To make this tractable at scale, we introduce , a first-order approximation that replaces backpropagation through the inner optimisation with gradient alignment between per-example updates and the meta-loss; a double-gradient implementation recovers all alignments in two backward passes per step, independent of batch size. Working from as few as 80 real demonstrations, we adapt five backbones spanning 270M to 4B parameters on three summarisation benchmarks. MWeiST attains the lowest held-out perplexity in every model–dataset cell and improves completion cross-entropy over all five baselines in the large majority of paired comparisons. GradAlign tracks the exact BPTT meta-gradient closely while admitting substantially larger inner batches at the same memory budget. Learning which synthetic examples to train on is therefore practical at scales where exact meta-gradients are not.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
self-training, meta-learning, data reweighting, meta-gradients, low-resource adaptation, synthetic data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JpjiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MWeiST, a framework for closed-loop self-training with self-generated data, and GradAlign, a first-order method for estimating synthetic example value via gradient alignment. MWeiST builds on a frozen pretrained backbone with LoRA adapters. It generates synthetic completions and optimizes their weights using a meta-objective on real demonstrations. GradAlign approximates training value by aligning each example's parameter-update gradient with the meta-loss gradient, using only two backward passes per step via a double-gradient implementation. Experiments cover three summarization datasets (SAMSum, DialogSum, XSum), each with 80 real demonstrations and 500 held-out examples, and five pretrained models (270M–4B). MWeiST is compared against LoRA, DoRA, confidence filtering, ReST, and L2RW on perplexity, target-completion cross-entropy, and character-level F-score. Results show consistent gains in likelihood-based metrics, but less consistent gains in reference-overlap metrics.

Critical strengths
  • Effective meta-weighted self-training under extreme data scarcity. MWeiST consistently improves held-out likelihood metrics in the main evaluation, achieving the lowest perplexity across all 15 model–dataset combinations and the lowest completion cross-entropy in 13 of 15 combinations. Paired bootstrap tests further support the reported improvements over the evaluated baselines.
  • A computationally practical approximation to multi-step meta-gradients. GradAlign avoids the substantial memory cost of exact truncated backpropagation through time. The paper evaluates its agreement with exact BPTT at multiple tested model scales and unroll lengths, and reports end-to-end runtime and memory measurements. These results provide empirical support for the proposed approximation within the evaluated settings.
Critical weaknesses
  • Improvements in likelihood metrics do not always translate into better generation quality. MWeiST achieves strong performance in perplexity and completion cross-entropy, but its improvements on text-overlap metrics such as ChrF, ROUGE-L, and BLEU are inconsistent. This suggests that lower predictive loss does not necessarily imply higher actual quality of generated summaries.
  • The reliability of GradAlign has not been established across a sufficiently broad range of settings. GradAlign is a biased approximation that omits the curvature terms captured by exact truncated backpropagation through time (BPTT). Although the paper demonstrates high agreement with the exact method under specific model scales and unroll lengths, it provides no theoretical error bound, and the validation remains limited in scope.
Questions for authors
  • MWeiST consistently improves perplexity and completion cross-entropy, but its improvements on reference-overlap metrics such as ChrF, ROUGE-L, and BLEU are less consistent. Could the authors clarify why these likelihood improvements do not consistently translate into better generation quality? Have human evaluations or other task-specific quality metrics been considered to further validate the practical benefits of MWeiST?
3 comments
AuthorsOpenReview ↗

Generation Quality, LLM Judge and GradAlign Error Bounds

Thank you for the clear summary and questions. Your points led to these new analyses.

W1 (likelihood vs. overlap). We compared MWeiST with uniform self-training (same pipeline, all weights 1) on ChrF and ROUGE-L across 45 cells (five backbones, three datasets, three splits; paired bootstrap, n = 500; Table 7). MWeiST wins/ties/loses 28/17/0 on ChrF and 29/16/0 on ROUGE-L. A blind LLM judge also prefers MWeiST to every self-training baseline (57.8-59.3%; Tables 19-20).

Cross-entropy and overlap metrics are complementary: cross-entropy scores the model's probability of the reference independent of decoding, while ChrF and ROUGE-L score one decoded output, which also depends on decoding settings and stopping. Our primary claim remains reduced held-out loss (Section 4.3), and the checks above show generation quality moving in the same direction. Your comment led us to recheck why ChrF barely separated the methods. We found a bug in batched decoding at evaluation time of Table 1 for left-padding tokenizers (Gemma-3, GraniteMoE), whose outputs mostly repeated the prompt, and regenerated all outputs after fixing it. Training was unaffected: synthetic data is generated by a separate routine that pads on the right. ChrF and ROUGE-L now separate the methods more often (Tables 1 and 6) and never favor uniform self-training.

Q1 (human or task-specific evaluation). We ran a blind LLM judge (Gemini 3.8 Flash) on Gemma-3 1B outputs for 200 held-out examples per dataset; it sees the source and two unlabeled summaries in both orders (Tables 19-20, Appendix H).

  • Overall preference. Counting ties half, MWeiST wins 57.8% [54.9, 60.5] against Confidence Filtering and ReST and 59.3% [56.4, 62.1] against L2RW. Preference is significant on SAMSum and XSum, on par on DialogSum; length-controlled rates remain 58.0-59.5%.
  • Faithfulness alone. Preference is 55.6-56.6%; every 95% CI excludes 50%.
  • Against the untrained base model. Preference is 59.7% [56.8, 62.5] overall and 56.8% [54.3, 59.2] on faithfulness.

W2 (GradAlign reliability; no error bound). GradAlign's bias comes solely from replacing exact BPTT's ∏s>t(I−αHs)\prod_{s>t}(I-\alpha H_s) alignment factor with the identity for an example used at step tt. We derived a formal bound under smoothness assumptions (Appendix G, Theorems G.4-G.5):

  • (i) To first order, the dropped part is α∑s>tHs\alpha\sum_{s>t}H_s: per-example error grows linearly with α\alpha times accumulated curvature over remaining inner steps and vanishes for the last steps.
  • (ii) GradAlign cannot flip an example's meta-gradient sign unless its gradient is nearly orthogonal to gmetag_{\text{meta}}.
  • (iii) Its direction remains a descent direction of the exact truncated objective while relative error is below one.

We measured the error against exact BPTT at T = 10, 75 and 150 (270M, first cycle; Table 18, Appendix G.5): relative L2 error grows roughly linearly in T (0.0035, 0.023 and 0.046), while direction holds. Across the original five Table 5 settings, mean cosine is ≥ 0.9992, with no sign flips among the 20% largest exact meta-gradients over 162,000 paired values.

We also trained three further cells (Gemma-3 270M and 1B DialogSum, 1B XSum) with both GradAlign and exact BPTT for three cycles (Table 5). Cosine stays ≥ 0.993 in every meta-iteration, and the GradAlign- and BPTT-trained runs of each cell finish within 5e-5 nats of each other. At 4B, where exact BPTT fits only inner batch 2, cosine stays ≥ 0.989 in every meta-iteration (Table 5).

Inner learning rate. We swept 0.5x to 4x the default against exact BPTT (one cycle, T = 75; Table 5). Error grows with α\alpha: at 1B, mean cosine stays ≥ 0.9999 and mean relative L2 error roughly doubles with each doubling of α\alpha (0.0032 to 0.0200). At 270M, cosine stays ≥ 0.989 in every meta-iteration (mean relative error 0.0120 to 0.1317). Sign agreement is ≥ 97.7%, with no top-20% flips.

We believe this work opens a useful direction for meta-learned weighting of self-generated data, and the settings where gains are smaller point to clear next steps.

I appreciate the authors' efforts in addressing my concerns. Nevertheless, I remain concerned about the novelty and overall significance of the contributions. As a result, I am still inclined to maintain my Weak Reject recommendation.

Thank you for the follow-up. We respectfully disagree with the assessment of novelty and significance, for the following reasons.

Novelty. MWeiST learns which self-generated completions help the model after several adaptation steps, using real-demonstration loss to assign their weights. It relearns these weights each time the model generates a new pool. L2RW and Meta-Weight-Net use one-step look-ahead on fixed datasets; Confidence Filtering and ReST select completions by confidence or reward.

GradAlign makes this multi-step weighting practical at LLM scale. At 4B, it supports an inner batch size of 16, whereas exact BPTT fit only a batch of 2 on a 96 GB GPU. Appendix A presents our integrated recipe for bringing multi-step meta-weighting into the LLM self-training loop. Our implementation contribution is this integrated design: double-gradient computation provides all per-example estimates in two backward passes per inner step, while mixed precision and rematerialisation make repeated meta-updates fit in memory.

Significance. The multi-step extension delivers gains with only 80 real examples: MWeiST beats one-step L2RW on perplexity in all 15 model–dataset combinations and on completion cross-entropy in 14. It also beats GRPO baselines using ROUGE-L or ChrF rewards on perplexity in 10/10 comparisons and completion cross-entropy in 9/10. A blind LLM judge also significantly favors MWeiST over all the competitor methods, with preference rates of 57.8–59.3% pooled across the three datasets.

Appendix G formally bounds the approximation error, and GradAlign closely tracks exact BPTT across the tested scales.

Reviewer_WGRdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MWeiST, a meta-weighted self-training framework designed for language model adaptation under extreme data scarcity. MWeiST learns weights for synthetic examples based on its long-term training utility. It evaluates the future utility of each synthetic example by looking ahead at how training on it impacts the training performance on real demonstrations. GradAlign further approximates multi-step meta-gradients using gradient alignment. Experiments across five models and three summarization datasets demonstrate the consistent performance improvements of MWeiST.

Critical strengths
  1. The meta-learned weights add a clear margin over uniform self-training at a matched budget, making meta-weighted self-training feasible at billion-parameter scale.
  2. MWeiST evaluates synthetic examples through their induced improvement on real demonstrations, providing a training-oriented criterion beyond generation confidence or reference overlap.
  3. The evaluation covers different model architectures and various relevant metrics.
Critical weaknesses
  1. The significant improvements are almost entirely confined to likelihood-based metrics like PPL, while other metrics (e.g., ChrF and BLEU) show marginal differences compared to baselines. This raises questions about whether the method truly improves downstream generation quality.
  2. While empirically effective, GradAlign discards second-order curvature terms in BPTT. The paper reports strong agreement with exact truncated BPTT at 270M and 1B, but corresponding validation at 3B–4B and sensitivity analysis with respect to the inner-loop learning rate are not provided.
  3. Table 10 reports quite larger wall-clock runtime of MWeiST than those of LoRA, while some CCE improvements are modest. Comparisons with well-tuned baselines under matched computational budgets are needed to clarify whether the additional meta-optimization cost is justified.
  4. The contributions of across-prompt and within-prompt reweighting are not separately proved. A comparison with per-prompt normalization that preserves equal total weight across prompts would help assess the benefit of reallocating training weight between prompts.
Questions for authors

Please refer to the weakness above

2 comments
AuthorsOpenReview ↗

(1/2): Generation Quality and GradAlign at Larger Learning Rates and Scale

Thank you for the detailed, concrete review. Your four points led to these new analyses.

W1 (gains confined to likelihood metrics). We added two generation checks; both show gains:

  • Blind LLM judge. Gemini 3.8 Flash compared Gemma-3 1B output pairs in both orders on 200 held-out examples per dataset (Tables 19-20, Appendix H). It prefers MWeiST over Confidence Filtering and ReST (57.8% [54.9, 60.5]), L2RW (59.3% [56.4, 62.1]), LoRA and DoRA (58.0% [54.9, 61.0], 57.9% [54.8, 60.8]) and the untrained base model (59.7% [56.8, 62.5]). Against the trained baselines, preference is significant on SAMSum and XSum, on par on DialogSum. It persists under length control (58.0-59.8%) and on faithfulness alone.
  • Controlled overlap comparison. We compared MWeiST with uniform self-training, which fixes weights at 1 in the same pipeline, across 45 cells (five backbones, three datasets, three splits; paired bootstrap, n = 500; Table 7). MWeiST wins/ties/loses 28/17/0 on ChrF, 29/16/0 on ROUGE-L, 43/2/0 on perplexity and 41/4/0 on cross-entropy.

Likelihood remains the primary claim (Section 4.3): gains over self-training baselines hold in 45/45 perplexity and 39/45 cross-entropy comparisons (Table 6). Your comment led us to recheck why ChrF barely separated the methods. We found a bug in batched decoding at evaluation time of Table 1 for left-padding tokenizers (Gemma-3, GraniteMoE), whose outputs mostly repeated the prompt, and regenerated all outputs above after fixing it. Training was unaffected: synthetic data is generated by a separate routine that pads on the right. ChrF and ROUGE-L now separate the methods more often (Tables 1 and 6) and never favor uniform self-training.

W2 (inner learning rate; GradAlign at 3B-4B). We compared GradAlign with exact BPTT at 0.5x, 1x, 2x and 4x the default inner learning rate (Gemma-3 270M and 1B, one cycle, T = 75; new rows in Table 5):

Inner LR2e-44e-48e-41.6e-3
270M cosine1.00000.99990.99950.9953
270M rel. L20.01200.02330.05170.1317
270M sign agreement0.99880.99920.99570.9770
1B cosine1.00001.00001.00000.9999
1B rel. L20.00320.00550.01020.0200
1B sign agreement0.99850.99920.99880.9967

Entries average five meta-iterations. Error grows roughly linearly with α\alpha over the default range, roughly doubling with each doubling of α\alpha at 1B. At 270M it grows faster at 4x (0.0517 to 0.1317), accumulating over meta-iterations, but cosine stays ≥ 0.989 in every meta-iteration. Sign agreement is ≥ 97.7% in every setting, with no flips among the 20% largest meta-gradients.

This matches theory: GradAlign drops ∏s>t(I−αHs)\prod_{s>t}(I-\alpha H_s), whose first-order part is α∑s>tHs\alpha\sum_{s>t}H_s, so error should scale with αT\alpha T. Varying T at fixed α\alpha on 270M, first-cycle relative L2 error is 0.0035, 0.023 and 0.046 at T = 10, 75 and 150 (Table 18, Appendix G.5). Appendix G adds a formal bound (Theorems G.4-G.5): GradAlign cannot flip an example's meta-gradient sign unless its gradient is nearly orthogonal to gmetag_{\text{meta}}, and its direction remains a descent direction of the exact truncated objective while relative error is below one.

We also compared GradAlign with exact BPTT on Gemma-3 4B (SAMSum, one cycle, T = 75; Table 5). Exact BPTT fits a 96 GB GPU only at inner batch 2, so each meta-iteration updates 150 of the 1,200 examples. GradAlign keeps cosine ≥ 0.989 with exact BPTT in every meta-iteration (mean 0.9945), with no sign flips among the 20% largest meta-gradients of the updated examples. The 3B-4B runs keep the default α\alpha with T ≤ 150, so αT≤0.06\alpha T \le 0.06: tested directly at 270M (T = 150, Table 5) and half the largest value in the sweep (αT=0.12\alpha T = 0.12 at 4x). Learned weights beat matched uniform self-training on held-out cross-entropy in all 9 blocks at 3B-4B (Table 7).

AuthorsOpenReview ↗

(2/2): Cost vs. Tuned Baselines and Across-Prompt Reweighting

W3 (cost vs. well-tuned baselines at matched compute). MWeiST improves held-out likelihood at higher training cost than direct LoRA/DoRA. Uniform self-training isolates meta-weighting cost with identical prompts, generation budget, retraining optimizer and schedule. We timed each method on Gemma-3 1B (one GPU, including n = 500 evaluation; three-split means per dataset): wall-clock is 0.9-1.4 h for LoRA/DoRA, 3.8-5.0 h for uniform self-training and 7.1-8.3 h for MWeiST.

The self-training loop (1,200 candidates per cycle, seven cycles) accounts for about half the gap from direct fine-tuning. Meta-weighting adds 65-88%, lowering completion cross-entropy against uniform self-training in 42/45 same-split comparisons (Table 7). On 1B SAMSum it is 1.792 vs. 1.861, in every split. Confidence Filtering, ReST and L2RW share the loop and budget. Against GRPO-style reward weights (ROUGE-L or ChrF reward, three seeds each) used instead of meta-gradient weights in the same loop, MWeiST leads on perplexity in 10/10 model-dataset cells and on cross-entropy in 9/10 (Table 15). The rank-8 LoRA output adapter leaves inference cost unchanged.

We also ran uniform self-training for 17 cycles on one 1B SAMSum split; at its per-cycle cost, that is slightly more training time than MWeiST's 7 cycles (Appendix C.4). On that split's 480 non-monitoring held-out examples, its completion cross-entropy worsened from 1.853 after 7 cycles to 2.032, and ChrF fell from 19.00 to 16.99; MWeiST achieved 1.785 and 19.86 after 7 cycles (both differences significant under the paired bootstrap). This supports the value of meta-weighting.

We also tuned LoRA/DoRA under a search budget capped at one MWeiST training run (9 optimizer and learning-rate settings). Results depend on the setting: MWeiST keeps lower held-out cross-entropy at Gemma-3 4B XSum (1.533 vs. 1.640-1.775, all three splits), while tuned direct fine-tuning is stronger at 1B SAMSum and 4B DialogSum. MWeiST was not re-tuned for these runs; giving it the same search budget is a natural next step.

Direct fine-tuning also has a structural advantage, most visible on overlap metrics such as ChrF. It uses the 80 gold completions as token targets, so it can fit the surface form of the references (length, phrasing, style), and the held-out examples follow the same distribution. MWeiST never trains on gold completions: it trains only on self-generated data, and the gold completions enter only through the meta-loss that sets the sample weights. We attribute much of the overlap gap to this difference in access to reference text.

W4 (across- vs. within-prompt reweighting). Eq. 5 constrains all 1,200 examples globally (80 prompts × 15 completions); learned weights perform both between-prompt reallocation and within-prompt ranking. We decomposed the final-cycle weights of the 15 main runs (Appendix D). Restoring equal 1/80 prompt shares would move a median 33% of total weight, and between-prompt variance accounts for a median 58% of weight variance, versus about 7% after shuffling weights across prompts.

On Gemma-3 270M, we tested your proposed control and its converse (one split, 500 held-out examples, paired bootstrap; Table 14, Appendix F.5):

  • Equal shares rescales each prompt's 15 weights to 1/80 total at every meta-iteration and before retraining, retaining only within-prompt ranking.
  • Prompt-level assigns each completion its prompt's mean weight, retaining only between-prompt reallocation.

Completion cross-entropy changes relative to MWeiST (parentheses: fraction of its gain over uniform self-training lost):

VariantSAMSumDialogSumXSum
Equal shares (within-prompt only)+0.055 (73%)+0.015 (70%)+0.003 (81%)
Prompt-level (between-prompt only)+0.016 (21%)+0.005 (24%)+0.0003 (9%)

All differences are significant (95% CIs exclude 0); perplexity agrees. Between-prompt reallocation supplies most of the gain, with a smaller significant contribution from within-prompt ranking. ChrF agrees on SAMSum (equal shares: -0.5 points, significant) and ties on DialogSum and XSum.

We believe this work opens a useful direction for meta-learned weighting of self-generated data, and the settings where gains are smaller point to clear next steps.

Reviewer_vN9nOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MWeiST a framework for adapting LLMs under extreme data scarcity. This method frames synthetic data adaptation as a bilevel optimization problem: it assigns learnable importance weights to synthetic examples, optimized via look-ahead training trajectories, to improve likelihood on real demonstrations. The authors also introduce GradAlign, an extension of FOMAML to LLMs. Extensive experiments and analysis demonstrate that MWeiST achieves low held-out perplexity and cross entropy loss compared to standard PEFT methods.

Critical strengths
  • The paper is well written and very clear. The analysis in the appendix are comprehensive and they give some nice insights about the method functionalities.
  • GradAlign bypass the memory intensive high-order backpropagation due to the bilevel optimization, while maintaining high correlation with exact meta-gradients.
  • Generating synthetic data and directly weight them based on their quality is a good idea to overcome the problem of data scarcity.
Critical weaknesses
  • The evaluation is done on 80 examples and it primary focuses on summarization tasks. An extension to different tasks and even less examples could further strengthening the paper.
  • The paper idea is very similar to existing work (e.g., [1]) although it is implemented in the context of LLMs. Furthermore, even GradAlign is simply an extension of FOMAML. A clarification on the paper contribution is necessary.

[1] Vettoruzzo, Anna, Mohamed-Rafik Bouguelia, and Thorsteinn Rögnvaldsson. "Efficient few-shot human activity recognition via meta-learning and data augmentation." Neural Computing and Applications 37.23 (2025): 19461-19477.

Questions for authors
  • Are the weights learned by the model transferable across data from same or different domains?
  • How does MWeiST prevent the synthetic pool from collapsing, given that it only reweights but not “correct” poor generations?
  • What is the method sensitivity to the number of meta-iterations and inner SGD steps? Is the method robust at scale when the number of trainable parameters increases? It is well known that this is a bottleneck of MAML.
2 comments
AuthorsOpenReview ↗

(1/2): Data Scale, Task Coverage and Relation to [1]

Thank you for the review and for highlighting the clarity and appendix analyses.

W1 (80 examples, mostly summarization). Table 1 uses 80 real training examples and 500 held-out examples per dataset. Figure 10 varies the training pool from 40 to 200: at ntr=40n_{tr}=40, completion cross-entropy decreases every cycle (-0.008 nats after three cycles, as at ntr=80n_{tr}=80).

We trained the self-training baselines on the same 40 examples (Gemma-3 270M, SAMSum, 480 held-out examples; Table 21, Appendix I.2). MWeiST has lower perplexity than uniform self-training (all weights 1), Confidence Filtering, ReST and L2RW, and lower completion cross-entropy than the first three (paired 95% CIs exclude zero).

Appendix C.3 also evaluates four non-summarization tasks, GSM8K and Alpaca (Table 8), JFLEG grammar correction and E2E data-to-text (Table 9), as well as CNN/DailyMail. Against self-training baselines, MWeiST significantly lowers completion cross-entropy in 13/15 GSM8K comparisons (no losses), 9/9 on E2E and 15/15 on CNN/DailyMail. Alpaca is mixed. Broader task coverage remains our main future direction.

W2 (relation to [1]; GradAlign vs. FOMAML). We will cite and discuss [1], which meta-learns a MAML initialization and one weight per hand-designed augmentation type, shared across examples. MWeiST learns no initialization. Each cycle, it generates 1,200 completions and learns a weight for each from its effect on real-demonstration loss after a T-step look-ahead.

GradAlign is first-order in the FOMAML sense (Section 3.3.2). Dropping inner Jacobians leaves one gradient for an initialization, but one alignment per example and inner step for per-example weights, naively costing one backward pass per example. GradAlign computes all alignments with two backward passes per inner step, independent of batch size, and reduces to L2RW at T=1T=1. Appendix G derives GradAlign from the exact truncated meta-gradient and bounds the error of dropping the inner Jacobians, including conditions under which it preserves each example's meta-gradient sign and gives a descent direction.

We will state the contributions as:

  • (i) closed-loop, per-example bilevel reweighting of self-generated data under extreme scarcity;
  • (ii) GradAlign, extending L2RW's one-step estimator to multi-step look-ahead at LLM scale;
  • (iii) agreement with exact BPTT (Figure 2, Table 5) and gains over one-step L2RW: lower perplexity in 15/15 cells and cross-entropy in 14/15 (Table 1).
AuthorsOpenReview ↗

(2/2): Weight Transfer, Pool Collapse and K/T Sensitivity

Q1 (transferability of the weights). Weights belong to individual completions and reset to 1 with each new pool (Algorithm 1, lines 2-3); they do not carry over to new data. We correlated per-prompt mean weights across cycles and across backbones (Appendix I.1). Weighting patterns persist within a run: consecutive cycles have median Spearman correlation 0.88 across the 15 cells. Across backbones, 27 of 30 model pairs are at chance, consistent with weights measuring a completion's effect on a particular model.

We also evaluated each final MWeiST adapter zero-shot on the other two datasets (Figure 12, Appendix I.1). Adapters transfer within-domain: the DialogSum adapter lowers SAMSum cross-entropy for 4 of 5 backbones (0.017-0.058 nats), matching the in-domain adapter for two. Across formats (dialogue vs. news), 14 of 20 pairs stay within 0.005 nats of the base model.

Q2 (pool collapse). Reweighting controls completions' effects on the model without correcting them. Three safeguards counter collapse:

  • (1) Regenerate the pool from the same 80 real prompts and re-learn weights each cycle (Algorithm 1).
  • (2) Use real-demonstration meta-loss to push completions whose look-ahead training hurts it to wmin⁡=0.01w_{\min}=0.01.
  • (3) Clip weights to [0.01, 10] and renormalize to mean 1.

We computed the effective sample size of the weights in every cycle and cell: it stays at 29-85% of the pool (Appendix D). Figure 3's template preferences reallocate weight between prompts while reducing real-data loss.

We added base-model cross-entropy to Table 13, which compares MWeiST with uniform self-training on identical regenerated pools. Uniform self-training ends above the base model in 6 of 14 cells; MWeiST beats it in 13/14, and in 42/45 same-split comparisons on 500 held-out examples (Table 7). Weighting thus prevents drift.

We also ran uniform self-training for 17 cycles on one 1B SAMSum split, slightly more training time than MWeiST's 7 cycles (6.75 h vs. 6.46 h; Appendix C.4). On the 480 held-out examples its drift worsens: completion cross-entropy rises from 1.853 to 2.032 and ChrF falls from 19.00 to 16.99, while MWeiST reaches 1.785 and 19.86 after 7 cycles (both differences significant under the paired bootstrap).

Q3 (sensitivity to K and T; scale). Figure 9 sweeps K∈{1,3,5,10}K\in\{1,3,5,10\} and T∈{1,5,25,75}T\in\{1,5,25,75\} on Gemma-3 1B/SAMSum. Gains are monotone in both, depending mainly on K⋅TK\cdot T (Spearman -0.98 with final cross-entropy). Equal-K⋅TK\cdot T cells differ by at most 0.002 nats. K=T=1K=T=1 behaves like uniform self-training. Gains saturate at our default: K=10K=10 vs. 55 gives 1.834 vs. 1.836 for about 60% more wall-clock.

We repeated the grid on Gemma-3 270M for DialogSum and XSum (3×33\times3 grids, three cycles each; Table 22, Appendix I.3). Final cross-entropy again decreases monotonically in both KK and TT, correlates with K⋅TK\cdot T at -0.98 (Spearman) on both datasets, and differs by at most 0.0003 nats between equal-K⋅T=25K\cdot T=25 cells.

Scale: MAML differentiates through the inner loop with respect to an initialization of size ∣ϕ∣|\phi|. Our 1,200 weight meta-parameters are independent of model size. GradAlign avoids trajectory backpropagation, using two backward passes per inner step and storing only adapter states (Table 4).

  • At 270M, MWeiST's peak memory matches uniform self-training's (27.3 vs. 27.2 GiB, Table 10).
  • At 4B, GradAlign fits inner batch 16, versus only 2 for exact BPTT (Table 2).
  • Wall-clock still grows with model size: 27 min per cycle at 270M vs. 66 min for GraniteMoE-3B (Table 3).

We also raised the number of trainable parameters to LoRA rank 32 and 128 instead of 8 (α = 2r, other settings unchanged; Gemma-3 1B, SAMSum, 480 held-out examples; Appendix I.4, Table 23). MWeiST's advantage over uniform self-training grows with rank. On the same split, its cross-entropy gain rises from 0.066 nats at rank 8 to 0.150 and 0.194, and its ChrF gain from +1.05 to +2.99 and +4.65 (paired 95% CIs exclude zero). MWeiST improves with rank (cross-entropy 1.789, 1.744, 1.707), while uniform self-training does not on cross-entropy or ChrF. Training time is the same at rank 32 and 128 (4.89 vs. 4.75 h).

We believe this work opens a useful direction for meta-learned weighting of self-generated data, and the settings where gains are smaller point to clear next steps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.