- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Meta-TA learns two controls for a CPPO-style update: the position-weight floor and the prefix-divergence budget. It updates these controls using query reward after a temporary policy update. At exact behavior matching the constraints are inactive, so the method first takes a detached warm-up step, then differentiates through a support update with a soft mask. A development set selects the allocator. The chosen allocator is frozen while a fresh full model is trained with a hard mask. The paper also tests structured allocators, query overfitting, and online refresh during continual mathematical training. On Qwen3-8B-Base, AIME improves from 29.01 to 31.95 with the same deployment token budget. Including selection raises the total cost of one deployment from 1.00 to 4.33 CPPO units.
Critical strengths
- The paper identifies an obstacle to applying meta-gradients in this setting: a behavior-matched update has no allocator signal through an inactive constraint. It gives a detached warm-up and differentiable surrogate that can make the allocator affect the support update. This is a concrete adaptation of meta-learning to CPPO-style controls. Appendix A limits the formal result to admissible schedules, without claiming global policy feasibility or monotonic improvement.
- The selected allocator transfers to fresh policy training and yields a 2.94 ± 0.23 AIME-point gain over CPPO across five paired training seeds. Separate support, query, development, and final evaluation sets help make this result credible. The paper also distinguishes repeated generations from training seeds and includes the full selection cost. Its query-reuse experiments reveal substantial overfitting on small pools (Tables 7, 13-15), a useful warning for others learning update-rule parameters.
Critical weaknesses
- The experiments do not show that meta-gradient selection is a worthwhile advance over simpler tuning. The main allocator has just two parameters, but Table 6 includes only the meta-gradient selector. Comparing against published CPPO calibration and tuning baseline learning rates does not replace a grid, random, or derivative-free search over the same allocator with matched query access and compute. Selection costs 3.291 CPPO units, bringing one deployment to 4.328 units (Table 13). Reuse could amortize that cost, but its practical value depends on the allocator transferring without reselection. Position shuffling retains a 2.44-point gain out of 2.94 for the full method (Table 7), also weakening the attribution to detailed positional allocation. The results establish that the selected settings help; they leave the additional value of the learning procedure unresolved.
- The numerical gradient results need to be reconciled with the deterministic check described in Appendix A.5. That check requires fixed stopped quantities and recommends sweeping perturbation size. Table 5 and Figure 10 report correlation 0.85 and sign agreement 80.6%, with correlation falling to 0.67 early in training. Those figures alone do not validate differentiation of the same cached surrogate. Finite precision, branch crossings, or resampling could explain the differences, but the paper needs to separate them. This matters because the differentiable update is central to the claimed contribution, and its temporary LoRA/SGD computation must transfer to full-model AdamW deployment (Appendix B.2).
Questions for authors
- How does meta-gradient selection compare with search over the same admissible two-parameter allocator, using identical query/development access and matched total compute? For the proposed cost amortization, across which changes in training conditions has the same allocator been reused without reselection? Also, how does the no-warm-up result exactly match CPPO when default CPPO uses a sequence-adaptive prefix budget and the main frozen allocator emits a fixed pair?
- Do Table 5 and Figure 10 compare derivatives of exactly the cached function in Appendix A.5, or do they resample rewards or recompute stopped statistics? Please report finite-difference step sizes and relative errors, and repeat the analysis excluding branch crossings. This would distinguish numerical error, stochastic variation, and comparisons of different functions.
Reply to Reviewer Gujj
Thank you for the precise questions. Several of the new experiments follow them directly.
Q1a / W1: Search over the same allocator with the same access and compute. BO now searches the same admissible two-parameter allocator (new §4.3, Table 2). It uses Meta-TA's evaluator, checkpoint bank, support, query, and development pools, finalist rule (5 finalists × 20 development episodes), and fixed-pair deployment. At equal total compute (4.33 units), BO uses 552 query episodes and Meta-TA uses 200.
Q1b: Reuse conditions. We reused the 8B selection without reselection under three changes (Table 3, five paired seeds):
Where reuse applies, the cost per deployment is 1.037 + 3.291/N CPPO runs, under 1.5× at 8 deployments.
Q1c: How does no-warm-up match CPPO exactly? With zero warm-up the allocator gradient is exactly zero (Prop. 1; all 24 zero-warm-up audit checks agree), so the allocator stays at its initialization. In the submitted ablation, this arm then deployed the unchanged default CPPO rule, with its sequence-adaptive budget, rather than the fixed initial pair. Its realized KL (0.01514) is the same as CPPO's, and its zero gain holds by construction. Table 12 and the text now say this. The fixed-pair counterfactual you describe is now its own arm in Table 2: it reaches 30.07 AIME, and Meta-TA beats it by 1.50 in all 15 paired runs. App. C.4 also shows that the two rules accept identical tokens at behavior matching and diverge once the policy moves.
W1: Shuffling keeps 2.44 of the 2.94 points. A shuffle keeps the learned weights and changes only their positions, so it tests order, not allocation. To test the allocation itself, we added flat rules that replace the learned weights with one weight matched to Meta-TA's KL or active-token rate. These lose 0.89 and 1.01 points, in every seed, and no flat rule can match both quantities. The gain therefore comes mainly from the non-uniform allocation; positional order adds about half a point on top, as two independent estimates agree (0.56 from a new shuffle arm that matches both quantities, 0.50 from the original ablation). We revised the text to attribute it this way.
Q2 / W2: Did Table 5 and Figure 10 differentiate the cached function? No, and your suspicion was right. Their finite differences regenerated query responses under common random numbers (h = 1e-3, with sensitivity steps 5e-4 and 2e-3), so they compared different quantities. We relabeled them as an estimator-agreement diagnostic and added the cached check (App. C.4, Table 11, Fig. 11):
This separates the three sources you named: numerical error follows a V-shape in h with a floor set by precision, branch crossings leave a plateau that does not depend on h and disappears when branches are frozen, and the original disagreement came from comparing different quantities. On transfer from LoRA/SGD to AdamW, the full-training audit in Table 4 places the Meta-TA pair second of six, 0.16 points from the best (details in our reply to Reviewer 81ec).
We hope these answers settle your questions, and we would be glad to clarify anything else during the discussion.