- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MWeiST, a framework for closed-loop self-training with self-generated data, and GradAlign, a first-order method for estimating synthetic example value via gradient alignment. MWeiST builds on a frozen pretrained backbone with LoRA adapters. It generates synthetic completions and optimizes their weights using a meta-objective on real demonstrations. GradAlign approximates training value by aligning each example's parameter-update gradient with the meta-loss gradient, using only two backward passes per step via a double-gradient implementation. Experiments cover three summarization datasets (SAMSum, DialogSum, XSum), each with 80 real demonstrations and 500 held-out examples, and five pretrained models (270M–4B). MWeiST is compared against LoRA, DoRA, confidence filtering, ReST, and L2RW on perplexity, target-completion cross-entropy, and character-level F-score. Results show consistent gains in likelihood-based metrics, but less consistent gains in reference-overlap metrics.
Critical strengths
- Effective meta-weighted self-training under extreme data scarcity. MWeiST consistently improves held-out likelihood metrics in the main evaluation, achieving the lowest perplexity across all 15 model–dataset combinations and the lowest completion cross-entropy in 13 of 15 combinations. Paired bootstrap tests further support the reported improvements over the evaluated baselines.
- A computationally practical approximation to multi-step meta-gradients. GradAlign avoids the substantial memory cost of exact truncated backpropagation through time. The paper evaluates its agreement with exact BPTT at multiple tested model scales and unroll lengths, and reports end-to-end runtime and memory measurements. These results provide empirical support for the proposed approximation within the evaluated settings.
Critical weaknesses
- Improvements in likelihood metrics do not always translate into better generation quality. MWeiST achieves strong performance in perplexity and completion cross-entropy, but its improvements on text-overlap metrics such as ChrF, ROUGE-L, and BLEU are inconsistent. This suggests that lower predictive loss does not necessarily imply higher actual quality of generated summaries.
- The reliability of GradAlign has not been established across a sufficiently broad range of settings. GradAlign is a biased approximation that omits the curvature terms captured by exact truncated backpropagation through time (BPTT). Although the paper demonstrates high agreement with the exact method under specific model scales and unroll lengths, it provides no theoretical error bound, and the validation remains limited in scope.
Questions for authors
- MWeiST consistently improves perplexity and completion cross-entropy, but its improvements on reference-overlap metrics such as ChrF, ROUGE-L, and BLEU are less consistent. Could the authors clarify why these likelihood improvements do not consistently translate into better generation quality? Have human evaluations or other task-specific quality metrics been considered to further validate the practical benefits of MWeiST?
3 comments
I appreciate the authors' efforts in addressing my concerns. Nevertheless, I remain concerned about the novelty and overall significance of the contributions. As a result, I am still inclined to maintain my Weak Reject recommendation.
Thank you for the follow-up. We respectfully disagree with the assessment of novelty and significance, for the following reasons.
Novelty. MWeiST learns which self-generated completions help the model after several adaptation steps, using real-demonstration loss to assign their weights. It relearns these weights each time the model generates a new pool. L2RW and Meta-Weight-Net use one-step look-ahead on fixed datasets; Confidence Filtering and ReST select completions by confidence or reward.
GradAlign makes this multi-step weighting practical at LLM scale. At 4B, it supports an inner batch size of 16, whereas exact BPTT fit only a batch of 2 on a 96 GB GPU. Appendix A presents our integrated recipe for bringing multi-step meta-weighting into the LLM self-training loop. Our implementation contribution is this integrated design: double-gradient computation provides all per-example estimates in two backward passes per inner step, while mixed precision and rematerialisation make repeated meta-updates fit in memory.
Significance. The multi-step extension delivers gains with only 80 real examples: MWeiST beats one-step L2RW on perplexity in all 15 model–dataset combinations and on completion cross-entropy in 14. It also beats GRPO baselines using ROUGE-L or ChrF rewards on perplexity in 10/10 comparisons and completion cross-entropy in 9/10. A blind LLM judge also significantly favors MWeiST over all the competitor methods, with preference rates of 57.8–59.3% pooled across the three datasets.
Appendix G formally bounds the approximation error, and GradAlign closely tracks exact BPTT across the tested scales.
Generation Quality, LLM Judge and GradAlign Error Bounds
Thank you for the clear summary and questions. Your points led to these new analyses.
W1 (likelihood vs. overlap). We compared MWeiST with uniform self-training (same pipeline, all weights 1) on ChrF and ROUGE-L across 45 cells (five backbones, three datasets, three splits; paired bootstrap, n = 500; Table 7). MWeiST wins/ties/loses 28/17/0 on ChrF and 29/16/0 on ROUGE-L. A blind LLM judge also prefers MWeiST to every self-training baseline (57.8-59.3%; Tables 19-20).
Cross-entropy and overlap metrics are complementary: cross-entropy scores the model's probability of the reference independent of decoding, while ChrF and ROUGE-L score one decoded output, which also depends on decoding settings and stopping. Our primary claim remains reduced held-out loss (Section 4.3), and the checks above show generation quality moving in the same direction. Your comment led us to recheck why ChrF barely separated the methods. We found a bug in batched decoding at evaluation time of Table 1 for left-padding tokenizers (Gemma-3, GraniteMoE), whose outputs mostly repeated the prompt, and regenerated all outputs after fixing it. Training was unaffected: synthetic data is generated by a separate routine that pads on the right. ChrF and ROUGE-L now separate the methods more often (Tables 1 and 6) and never favor uniform self-training.
Q1 (human or task-specific evaluation). We ran a blind LLM judge (Gemini 3.8 Flash) on Gemma-3 1B outputs for 200 held-out examples per dataset; it sees the source and two unlabeled summaries in both orders (Tables 19-20, Appendix H).
W2 (GradAlign reliability; no error bound). GradAlign's bias comes solely from replacing exact BPTT's ∏s>t(I−αHs) alignment factor with the identity for an example used at step t. We derived a formal bound under smoothness assumptions (Appendix G, Theorems G.4-G.5):
We measured the error against exact BPTT at T = 10, 75 and 150 (270M, first cycle; Table 18, Appendix G.5): relative L2 error grows roughly linearly in T (0.0035, 0.023 and 0.046), while direction holds. Across the original five Table 5 settings, mean cosine is ≥ 0.9992, with no sign flips among the 20% largest exact meta-gradients over 162,000 paired values.
We also trained three further cells (Gemma-3 270M and 1B DialogSum, 1B XSum) with both GradAlign and exact BPTT for three cycles (Table 5). Cosine stays ≥ 0.993 in every meta-iteration, and the GradAlign- and BPTT-trained runs of each cell finish within 5e-5 nats of each other. At 4B, where exact BPTT fits only inner batch 2, cosine stays ≥ 0.989 in every meta-iteration (Table 5).
Inner learning rate. We swept 0.5x to 4x the default against exact BPTT (one cycle, T = 75; Table 5). Error grows with α: at 1B, mean cosine stays ≥ 0.9999 and mean relative L2 error roughly doubles with each doubling of α (0.0032 to 0.0200). At 270M, cosine stays ≥ 0.989 in every meta-iteration (mean relative error 0.0120 to 0.1317). Sign agreement is ≥ 97.7%, with no top-20% flips.
We believe this work opens a useful direction for meta-learned weighting of self-generated data, and the settings where gains are smaller point to clear next steps.