acceptodds
Under review as a conference paper at ICLR 2027

Learning Where to Constrain: Meta-Learned Trust Allocation for Language-Model Reasoning

Abstract

Policy updates in auto-regressive reasoning change both token probabilities and the prefixes on which later decisions depend. Recent trust-region methods for LLM reinforcement learning account for token position and accumulated prefix divergence to improve training stability. However, a fixed or prespecified allocation may not suit different response lengths or changing task distributions. To study whether these constraints should instead be adaptive, we introduce Meta-TA (Meta-learned Trust Allocation), which learns an admissible position-weight floor and prefix-divergence budget by differentiating through the policy update. We show why a one-step update starting from the rollout policy provides no allocator gradient through an inactive constraint. A detached warm-up followed by a differentiable soft-constrained update enables allocator learning when the constraints become active. We then freeze the learned allocation and evaluate its transfer to fresh policy training. Our main evaluation uses the Qwen model family on mathematical reasoning, and we extend the study to abrupt task changes, gradual distribution shifts, and recurring tasks to examine adaptation and retention. To mitigate allocator overfitting, we select among saved allocator checkpoints using a separate subset and examine whether repeated updates on the query subset lead to overfitting. Decoupling allocator-regularization from policy-entropy helps distinguish restrictions on the learned update rule from changes in policy exploration. Together, these comparisons examine when learned constraints improve LLM reinforcement learning reasoning and if those gains justify additional cost.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, mathematical reasoning, large language models, meta-learning, policy optimization, trust-region methods, long-horizon reasoning, training stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GujjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Meta-TA learns two controls for a CPPO-style update: the position-weight floor and the prefix-divergence budget. It updates these controls using query reward after a temporary policy update. At exact behavior matching the constraints are inactive, so the method first takes a detached warm-up step, then differentiates through a support update with a soft mask. A development set selects the allocator. The chosen allocator is frozen while a fresh full model is trained with a hard mask. The paper also tests structured allocators, query overfitting, and online refresh during continual mathematical training. On Qwen3-8B-Base, AIME improves from 29.01 to 31.95 with the same deployment token budget. Including selection raises the total cost of one deployment from 1.00 to 4.33 CPPO units.

Critical strengths
  1. The paper identifies an obstacle to applying meta-gradients in this setting: a behavior-matched update has no allocator signal through an inactive constraint. It gives a detached warm-up and differentiable surrogate that can make the allocator affect the support update. This is a concrete adaptation of meta-learning to CPPO-style controls. Appendix A limits the formal result to admissible schedules, without claiming global policy feasibility or monotonic improvement.
  2. The selected allocator transfers to fresh policy training and yields a 2.94 ± 0.23 AIME-point gain over CPPO across five paired training seeds. Separate support, query, development, and final evaluation sets help make this result credible. The paper also distinguishes repeated generations from training seeds and includes the full selection cost. Its query-reuse experiments reveal substantial overfitting on small pools (Tables 7, 13-15), a useful warning for others learning update-rule parameters.
Critical weaknesses
  1. The experiments do not show that meta-gradient selection is a worthwhile advance over simpler tuning. The main allocator has just two parameters, but Table 6 includes only the meta-gradient selector. Comparing against published CPPO calibration and tuning baseline learning rates does not replace a grid, random, or derivative-free search over the same allocator with matched query access and compute. Selection costs 3.291 CPPO units, bringing one deployment to 4.328 units (Table 13). Reuse could amortize that cost, but its practical value depends on the allocator transferring without reselection. Position shuffling retains a 2.44-point gain out of 2.94 for the full method (Table 7), also weakening the attribution to detailed positional allocation. The results establish that the selected settings help; they leave the additional value of the learning procedure unresolved.
  2. The numerical gradient results need to be reconciled with the deterministic check described in Appendix A.5. That check requires fixed stopped quantities and recommends sweeping perturbation size. Table 5 and Figure 10 report correlation 0.85 and sign agreement 80.6%, with correlation falling to 0.67 early in training. Those figures alone do not validate differentiation of the same cached surrogate. Finite precision, branch crossings, or resampling could explain the differences, but the paper needs to separate them. This matters because the differentiable update is central to the claimed contribution, and its temporary LoRA/SGD computation must transfer to full-model AdamW deployment (Appendix B.2).
Questions for authors
  1. How does meta-gradient selection compare with search over the same admissible two-parameter allocator, using identical query/development access and matched total compute? For the proposed cost amortization, across which changes in training conditions has the same allocator been reused without reselection? Also, how does the no-warm-up result exactly match CPPO when default CPPO uses a sequence-adaptive prefix budget and the main frozen allocator emits a fixed pair?
  2. Do Table 5 and Figure 10 compare derivatives of exactly the cached function in Appendix A.5, or do they resample rewards or recompute stopped statistics? Please report finite-difference step sizes and relative errors, and repeat the analysis excluding branch crossings. This would distinguish numerical error, stochastic variation, and comparisons of different functions.
1 comment
AuthorsOpenReview ↗

Reply to Reviewer Gujj

Thank you for the precise questions. Several of the new experiments follow them directly.

Q1a / W1: Search over the same allocator with the same access and compute. BO now searches the same admissible two-parameter allocator (new §4.3, Table 2). It uses Meta-TA's evaluator, checkpoint bank, support, query, and development pools, finalist rule (5 finalists × 20 development episodes), and fixed-pair deployment. At equal total compute (4.33 units), BO uses 552 query episodes and Meta-TA uses 200.

  • On AIME, BO reaches 31.06 and Meta-TA 31.57, so Meta-TA is ahead by 0.51 in every selection block (3 blocks × 5 deployments).
  • In a separate equal-feedback cohort (200 query episodes each), Meta-TA leads by 1.06, again in every block.
  • Meta-TA also beats an adaptive schedule tuned with the same development access by 1.01, and the fixed pair at CPPO's defaults by 1.50, in all 15 paired runs.

Q1b: Reuse conditions. We reused the 8B selection without reselection under three changes (Table 3, five paired seeds):

  • On a held-out DAPO-Math subject mixture, it beats CPPO by 2.07 and the reused BO pair by 0.53, ahead of both in every seed.
  • On a smaller model (Qwen3-1.7B-Base), it beats CPPO by 0.40 in every seed.
  • The gains are largest where responses are long: on GSM8K training, whose solutions are short, accuracy is unchanged (−0.20), consistent with a late-position mechanism.

Where reuse applies, the cost per deployment is 1.037 + 3.291/N CPPO runs, under 1.5× at 8 deployments.

Q1c: How does no-warm-up match CPPO exactly? With zero warm-up the allocator gradient is exactly zero (Prop. 1; all 24 zero-warm-up audit checks agree), so the allocator stays at its initialization. In the submitted ablation, this arm then deployed the unchanged default CPPO rule, with its sequence-adaptive budget, rather than the fixed initial pair. Its realized KL (0.01514) is the same as CPPO's, and its zero gain holds by construction. Table 12 and the text now say this. The fixed-pair counterfactual you describe is now its own arm in Table 2: it reaches 30.07 AIME, and Meta-TA beats it by 1.50 in all 15 paired runs. App. C.4 also shows that the two rules accept identical tokens at behavior matching and diverge once the policy moves.

W1: Shuffling keeps 2.44 of the 2.94 points. A shuffle keeps the learned weights and changes only their positions, so it tests order, not allocation. To test the allocation itself, we added flat rules that replace the learned weights with one weight matched to Meta-TA's KL or active-token rate. These lose 0.89 and 1.01 points, in every seed, and no flat rule can match both quantities. The gain therefore comes mainly from the non-uniform allocation; positional order adds about half a point on top, as two independent estimates agree (0.56 from a new shuffle arm that matches both quantities, 0.50 from the original ablation). We revised the text to attribute it this way.

Q2 / W2: Did Table 5 and Figure 10 differentiate the cached function? No, and your suspicion was right. Their finite differences regenerated query responses under common random numbers (h = 1e-3, with sensitivity steps 5e-4 and 2e-3), so they compared different quantities. We relabeled them as an estimator-agreement diagnostic and added the cached check (App. C.4, Table 11, Fig. 11):

  • Step sizes: 15 values from 1e-8 to 1e-1 in float64 and float32. We report all 2,160 checks.
  • Relative error away from branch crossings: the median is 8.0e-10 (max 1.6e-8) in float64 at h = 1e-4, and 8.4e-4 (max 2.1e-2) in float32 at h = 1e-2.
  • Branch crossings: about 8% at every h under the piecewise function, and 6.1e-10 with branches frozen.
  • Signs: they agree on every non-degenerate check in both precisions.

This separates the three sources you named: numerical error follows a V-shape in h with a floor set by precision, branch crossings leave a plateau that does not depend on h and disappears when branches are frozen, and the original disagreement came from comparing different quantities. On transfer from LoRA/SGD to AdamW, the full-training audit in Table 4 places the Meta-TA pair second of six, 0.16 points from the best (details in our reply to Reviewer 81ec).

We hope these answers settle your questions, and we would be glad to clarify anything else during the discussion.

Reviewer_81ecOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Meta-TA, which uses meta gradients to select the position and prefix constraints of a CPPO update rule. A detached warmup activates the constraints, and a subsequent differentiable support update allows query rewards to train the allocator. In the main setting, the selected allocation is frozen and transferred to a fresh policy training run. The paper also studies conditioned allocation and periodic adaptation during continual learning. Experiments on mathematical reasoning report improvements over the compared baselines, with substantial additional cost for allocator selection.

Critical strengths
  1. The reset protocol provides meaningful evidence of allocation transfer. Algorithm 1 resets both the policy and optimizer before full training. The improvements in Table 1 therefore support the usefulness of the selected allocation beyond the temporary policies used during selection. This is a consequential distinction when assessing whether allocator learning produces a reusable result.

  2. The paper identifies a concrete obstacle to learning through constraints. Proposition 1 explains why inactive masks remove the allocator gradient at the rollout policy. The detached warmup followed by a differentiable constrained update provides a coherent response to that obstacle. This observation is useful for designing optimization procedures that learn through masking rules.

  3. Several controls improve the interpretation of the empirical results. Sections 4.3 to 4.5 distinguish response horizon, replay, and allocator regularization from other sources of improvement. The cost accounting also includes allocator selection. These choices make the reported reasoning and retention benefits more informative.

Critical weaknesses
  1. The central contribution is not distinguished from ordinary parameter tuning. The position and prefix constraints are inherited from CPPO, and the main deployed allocator consists of two fixed parameters. The methodological question is therefore whether meta gradients provide a useful advantage over simpler ways of selecting those parameters. Appendix B.3 and Table 6 do not establish this comparison. They show that the proposed selector improves over the published CPPO configuration, but leave its advantage over a selector using the same parameter family, data access, and total budget unresolved. This distinction matters because allocator selection itself costs several baseline training runs. Reusability alone cannot justify the procedure, since parameters found by a simpler search could also be reused. The current evidence supports the usefulness of a selected configuration more strongly than it supports the necessity or computational value of the proposed learning method.

  2. The meta objective is insufficiently connected to the training outcome it is meant to improve. Appendix B.2 learns the allocation through brief adaptation in a LoRA parameter subspace with SGD and a soft mask. Deployment uses full model training with AdamW and hard masks. Stationary selection also evaluates temporary updates along a checkpoint trajectory produced by default CPPO. These differences change the update directions and policy states through which the allocation is evaluated. Table 8 illustrates the resulting concern by showing that an intervention can improve query performance while weakening frozen transfer. Successful retraining with the final selected allocation establishes an empirical benefit, but does not establish how reliably this surrogate selects allocations for the actual deployment process. Evidence connecting candidate quality under the probe to candidate quality after fresh full training is needed to justify this central design choice.

  3. The evidence does not establish that the core meta gradient is computed reliably. Appendix C.2 and Table 5 compare automatic differentiation with finite differences on a cached computation, yet report substantial disagreement in gradient signs. Appendix A.5 explains that sampling noise cannot account for disagreement when both calculations use the same cached inputs. The paper does not resolve whether the discrepancy comes from gate boundaries, numerical precision, or a mismatch between the computations being compared. This concern directly affects the claimed learning mechanism. Observing a nonzero gradient and an improved final configuration does not verify that the implemented update follows the stated meta objective. A reproducible numerical check that resolves the disagreement is needed to support the algorithmic claim.

Questions for authors
  1. How does Meta-TA compare with a simpler selector over the same constraint parameters when data access and total selection and training budgets are matched? This comparison would determine whether the contribution extends beyond finding a better CPPO configuration.

  2. What evidence shows that the temporary LoRA and SGD probe reliably selects allocations for full model training with AdamW and hard masks? In particular, do candidate rankings under the selection objective agree with their rankings after fresh full training?

  3. What explains the gradient sign disagreements in Table 5 when both calculations use cached inputs? Can the discrepancy be resolved by controlling gate boundaries and precision and showing numerical error as the finite difference step varies?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 81ec

Thank you for the careful review. Each of your three concerns led to a new study.

W1 / Q1: A simpler selector over the same family, data, and budget. BO now searches the same admissible (w_min, r_b) coordinates (new §4.3, Table 2). It shares everything else with Meta-TA: the evaluator (support generation, warm-up, soft-masked update, query scoring), the checkpoint bank, the support, query, and development pools, the finalist rule, and the fixed-pair deployment. Each arm has 3 selection blocks with 5 fresh deployments each. On AIME:

  • at equal total compute (4.33 units), BO reaches 31.06 with 552 query episodes and Meta-TA 31.57 with 200, so Meta-TA is ahead by 0.51 in every block
  • at equal feedback (200 query episodes each), BO reaches 30.73 and Meta-TA 31.79, so Meta-TA is ahead by 1.06 in every block

Meta-TA also beats a tuned adaptive schedule by 1.01 and the fixed pair at CPPO's defaults by 1.50, in all 15 paired runs. With the same parameter family, data, and budget, the meta-gradient finds a better configuration than search, with far fewer queries. App. B.7 now describes the BO baseline. The advantage also survives reuse: on a held-out subject mixture, the reused Meta-TA pair beats the reused BO pair by 0.53, in every seed.

W2 / Q2: Does the probe pick allocations that hold up under full training? For new Table 4, we fixed six candidates in advance: the initialization, the Meta-TA and BO selections, a local perturbation, and a broad and a weak point. We deployed each in five fresh training runs with full-model AdamW and the hard mask, then scored them on an independent 1,000-problem panel that no selector sees.

  • The pair Meta-TA's hypergradient reached ranks second of six, 0.16 points behind the best. The BO pair is 0.79 behind.
  • A probe that updates the full model with AdamW, as deployment does, ranks the six exactly as full training does (Spearman 1.00) and picks the winner.
  • The original LoRA/SGD probe also singles out the weak candidate, and the full-model probe raises the rank agreement over all six from Spearman 0.49 to 1.00 by also ordering the competitive candidates, which lie within about a point of each other.

Selection therefore lands close to the best deployment allocation, and the revision recommends the full-model probe for score-based steps such as choosing finalists.

W3 / Q3: Gradient sign disagreement. The disagreement came from comparing two different computations; the gradient itself was not at fault. The former Table 5 reported a diagnostic whose finite differences regenerated query responses under common random numbers, so it set a score-function hypergradient against finite differences of a different, stochastic quantity. We are sorry this was confused with the cached check described in App. A.5. We have now run that check as specified (App. C.4, Fig. 11, Table 11). It uses 12 categorical fixtures that instantiate Eqs. 2–7 with every stopped quantity cached, 15 step sizes from 1e-8 to 1e-1, and both float64 and float32. Each check is labeled by whether its perturbation crosses a gate branch. This separates the three explanations you raised:

  • Mismatched computations (the actual cause): once the computations match, the median relative error away from branch crossings is 8.0e-10 in float64 (h = 1e-4) and 8.4e-4 in float32 (h = 1e-2). The error traces the usual truncation/roundoff V-shape in h.
  • Gate boundaries: checks that cross a branch level off near 8% at every h. Freezing the branches at their unperturbed decisions brings this to 6.1e-10, so the residual is the kink, not the derivative.
  • Precision: float32 raises the error floor to between 1e-4 and 1e-3, as expected. Signs agree on every non-degenerate check in both precisions.

The chain rule from raw to physical coordinates agrees to 5e-18, and every zero-warm-up check gives a gradient of exactly zero, as Prop. 1 predicts. We release all 2,160 checks with the scripts.

We hope this resolves your concerns and would be grateful if you would reconsider your score in light of the new evidence.

Dear Reviewer 81ec,

We wanted to follow up on our response to your three concerns.

The revision adds the matched-resource Bayesian-optimization comparison (§4.3, Table 2), the six-candidate full-training audit (§4.5, Table 4), and a corrected deterministic gradient check on cached categorical fixtures (Appendix C.4). Our response also explains why the earlier finite-difference diagnostic compared different computations.

For the probe question, Table 4 distinguishes the original LoRA/SGD probe’s imperfect ranking from the stronger agreement of the full-model AdamW probe. The allocation reached by Meta-TA ranks second among the six audited candidates, 0.16 points behind the best.

Could you let us know whether these results address your concerns, or which aspect would benefit from further clarification? We would welcome the opportunity to respond during the remaining discussion period.

Thank you again for the careful review.

Dear Reviewer 81ec,

Thank you again for your time and thoughtful feedback. We would like to follow up on our response and ask whether you have any remaining questions or concerns. We welcome further discussion and would be glad to provide any additional clarification.

The additional studies directly address your three questions:

  • Meta-TA exceeds Bayesian optimization by 0.51 AIME points at matched total compute; its allocation finishes within 0.16 points of the best of six audited candidates; and the corrected cached categorical audit shows numerical gradient agreement.
  • The full-model AdamW probe also reproduces the deployment ranking of those six candidates (§§4.3–4.5; Appendix C.4). These results strengthen the evidence for effective selection, successful transfer to full training, and gradient reliability.

Would you kindly consider updating your recommendation to 3 (weak acceptance) on this basis? Thank you for your consideration.

Best regards,

The authors

Thanks for the author's responses. I have increased my score.

Reviewer_g3YCOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Meta-TA, a meta-learning framework that learns position-dependent trust-region constraints for LLM reinforcement learning. It uses bilevel optimization to adjust constraint parameters based on rewards from a temporarily updated policy on held-out prompts. Experiments on mathematical reasoning benchmarks show improvements over existing methods, particularly for longer responses. Additional experiments investigate continual learning, adaptation to distribution shifts, and computational costs.

Critical strengths
  1. Technically coherent formulation. The paper provides a principled application of meta-gradient optimization to trust-region hyperparameter adaptation. The warm-up strategy addresses the inactive-constraint gradient issue, making the proposed optimization procedure feasible.

  2. Relatively comprehensive empirical evaluation. The experiments demonstrate improvements on mathematical reasoning benchmarks and investigate response length, continual learning, overfitting, and computational overhead. These analyses provide useful evidence of the method's effectiveness.

Critical weaknesses
  1. Limited methodological novelty. The proposed method largely combines CPPO's existing position-dependent trust-region constraints with established meta-gradient hyperparameter optimization. Since the main formulation only learns two constraint parameters, the methodological contribution appears incremental. The paper does not sufficiently establish why this combination constitutes a substantial advance over existing adaptive optimization methods.

  2. High computational cost and insufficient comparison with simpler alternatives. Meta-TA requires repeated policy updates and additional query rollouts to optimize the constraint parameters, resulting in approximately 4.33× the total compute of CPPO for a single training run. Despite this substantial overhead, the paper lacks comparisons with simpler alternatives, such as grid search, Bayesian optimization, or adaptive hyperparameter schedules under matched computational budgets. Without these comparisons, it is unclear whether the proposed bilevel optimization is necessary or offers a meaningful efficiency advantage.

Questions for authors
  1. Necessity of meta-gradient optimization. Since the proposed method primarily learns two trust-region hyperparameters, how does it compare with simpler alternatives such as grid search, Bayesian optimization, or adaptive hyperparameter schedules under matched computational budgets? Is bilevel optimization essential to achieving the reported improvements?

  2. Computational efficiency and practical applicability. Meta-TA incurs approximately 4.33× the total computational cost of CPPO for a single training run. How many subsequent training runs are required to amortize the allocator-selection cost? How well does a learned allocator transfer across different model sizes, training datasets, or task distributions without requiring additional meta-training?

  3. Source of performance improvements. To what extent do the improvements come from learning a meaningful position-dependent allocation rather than simply finding better overall trust-region hyperparameters? Could the authors provide comparisons with fixed constraints matched for overall divergence or masking rates to isolate the contribution of the learned allocation?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer g3YC

Thank you for the review and for your positive comments on the formulation and the scope of the experiments. We take your questions in order; the general comment has the full table.

W1: Novelty. Our contribution is not the tuning of two CPPO parameters. Meta-TA belongs to the family of meta-gradient methods that learn parts of the update rule from their effect on later performance, such as meta-gradient RL (Xu et al., 2018), STAC, and Adaptive Interest (Klissarov et al., 2022), which learns the state weights of emphatic TD by differentiating through the update. Meta-TA does the same for the trust region, and the two-parameter allocation is its simplest case. Our claims are about making this work and testing it. First, we show that any update-rule parameter acting only through masks that are inactive at behavior matching gets an exactly zero hypergradient from a one-step update at the rollout policy. This includes PPO clipping ranges (Prop. 1, Remark 1). A detached warm-up restores the signal, and a cached audit confirms the implemented derivative to a median relative error of 8e-10. Second, with resources matched, learning the rule through the update beats derivative-free search and an adaptive schedule. Third, a full-training audit shows that the learned rule is close to the best of six candidates. We kept the allocator at two parameters on purpose, because that makes a fair comparison with search affordable. We expect search to get harder with more parameters and the gradient to matter more, but we have not tested this yet.

W2 / Q1: Simpler alternatives at matched compute. New Table 2 has 3 selection blocks per arm and 5 fresh deployments per block. At 4.33 CPPO units of total compute:

  • BO over the same two parameters, with 552 query episodes: 31.06 AIME
  • adaptive schedule with tuned KL feedback: 30.56
  • Meta-TA, with 200 query episodes: 31.57

Meta-TA beats BO by 0.51 in every selection block, and it beats the adaptive schedule by 1.01 and the fixed pair by 1.50 in all 15 paired runs. In a separate cohort where BO gets Meta-TA's 200 query episodes, Meta-TA leads by 1.06, again in every block. All selectors share the evaluator, data pools, finalist rule, and deployment, so only the way candidates are proposed differs. Meta-TA also needs 2.8× fewer query episodes at the same total compute, and with equal episodes BO trails by 1.06; on MATH-500 and OlympiadBench the two are on par. Its pair also beats BO's under fresh full training (Table 4) and when reused (below). We used BO as the derivative-free baseline because it is the standard strong choice for low-dimensional continuous search and is usually at least as sample-efficient as grid or random search.

Q2: Amortization and transfer. If N deployments share one selection, each costs 1.037 + 3.291/N CPPO runs on average: under 2× at 4, 1.5× at 8, and 1.25× at 16 (Fig. 1b). Without any reselection (Table 3, five paired seeds):

  • On a held-out DAPO-Math subject mixture (8B), the Meta-TA pair beats CPPO by 2.07 and the reused BO pair by 0.53, ahead of both in every seed.
  • On Qwen3-1.7B-Base, it beats CPPO by 0.40 in every seed.
  • The gains are largest where responses are long: on GSM8K training, whose solutions are short, accuracy is unchanged (−0.20), consistent with the late-position mechanism of §4.4.

Q3: Learned allocation vs. overall strength. We added exactly these controls (§4.4, Fig. 12). A flat, position-independent rule calibrated to Meta-TA's behavior KL loses 0.89 AIME points, and one calibrated to its active-outward-token rate loses 1.01. Meta-TA is ahead of both in every seed, and each match holds within tolerance across seeds and training phases. The two cannot be matched together: at matched KL the flat rule accepts up to 2.4 points fewer outward tokens, and at matched rate its KL goes above the tolerance. The learned allocation combines a divergence level and an acceptance rate that no single flat weight achieves.

We hope this answers your questions. If it does, we would appreciate it if you would take the new results into account in your assessment.

Dear Reviewer g3YC,

We wanted to follow up on our response and revised manuscript as the discussion period approaches its end.

Your questions motivated the matched-resource comparisons in §4.3, Table 2. At equal total compute, Meta-TA improves AIME over Bayesian optimization by 0.51 points on average, with a positive difference in each of the three selection blocks, using 200 versus 552 query episodes. The revision also includes the adaptive-schedule comparison, flat constraints matched for divergence or masking rate (§4.4), and unchanged allocator reuse with explicit amortization accounting (§4.6).

We would appreciate hearing whether these additions address your concerns about the value of meta-gradient selection and the source of the gains. If any point remains unresolved, we would be glad to clarify it before discussion closes.

Thank you again for your time and feedback.

Dear Reviewer g3YC,

Thank you again for your time and thoughtful feedback. We would like to follow up on our response and ask whether you have any remaining questions or concerns. We welcome further discussion and would be glad to provide any additional clarification.

  • We added matched-resource comparisons in §4.3, Table 2, motivated by your questions. At equal total compute, Meta-TA improves AIME over Bayesian optimization by 0.51 points on average, with a positive difference in each of the three selection blocks, using 200 versus 552 query episodes.
  • Our revision also includes the adaptive-schedule comparison, flat constraints matched for divergence or masking rate (§4.4), and unchanged allocator reuse with explicit amortization accounting (§4.6).

If our response and additional results have addressed your concerns, we would be grateful if you could consider a more favorable score in light of this evidence.

Thank you for your consideration.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

Summary of the revision and new experiments

Thank you all for the careful reviews. All three of you found the vanishing-hypergradient analysis and the warm-up construction useful; Reviewers 81ec and Gujj found that resetting the policy and optimizer before retraining gives credible evidence of transfer; and Reviewer g3YC found the evaluation broad. The question you share is whether learning the allocation with a meta-gradient beats simpler ways of choosing it, given the same data and compute. We ran each comparison you asked for, and the results are below. Changes in the revised PDF are in blue, with a summary on the first appendix page. All new numbers are recomputed from the released per-repetition counts.

1. Meta-TA against simpler selectors at matched resources (new §4.3, Table 2, Fig. 1). On Qwen3-8B-Base, each arm runs in 3 independent selection blocks, and each block's choice is deployed in 5 fresh training runs. Three-year AIME Avg@16:

ArmAIMEQuery episodesSelection workTotal work
CPPO28.65––1.00
Fixed pair at CPPO defaults30.07––1.04
Adaptive schedule (KL feedback, tuned)30.56–3.274.33
BO, equal compute31.065523.294.33
BO, equal feedback (separate cohort)30.732001.812.85
Meta-TA31.572003.294.33

Work is in units of one default CPPO training run and includes one deployment.

At equal compute, Meta-TA scores 0.51 AIME points above BO, ahead in each of the three blocks, and on par on MATH-500 and OlympiadBench. Both spend the same total compute, but Meta-TA puts 1.48 units into the hypergradient and needs 2.8× fewer query episodes, while BO spends the same compute on extra evaluations. When BO gets the same 200 query episodes, it trails by 1.06 points, again in every block. BO sees only the query scores of the pairs it tries, and everything else (evaluator, pools, finalist rule, deployment) is shared; App. B.7 describes it. Meta-TA also beats the adaptive schedule by 1.01, the fixed pair by 1.50, and CPPO by 2.92, in all 15 paired runs. Against CPPO it gains 1.08 on MATH-500 and 1.07 on OlympiadBench.

2. The shape of the allocation matters, not just its strength (§4.4). Flat constraints calibrated to Meta-TA's behavior KL or to its active-outward-token rate lose 0.89 and 1.01 AIME points, in every seed. No flat strength can match both quantities at once.

3. Selection picks allocations that hold up in deployment (§4.5, Table 4). We fixed six candidates in advance and deployed each in five fresh training runs, scoring them on an independent 1,000-problem panel. Meta-TA's pair ranks second, 0.16 points behind the best; BO's pair is 0.79 behind. A probe that updates the full model with AdamW, as deployment does, ranks the six exactly as full training does.

4. We verified the hypergradient (§4.5, App. C.4). We implemented the cached finite-difference check from App. A.5: 12 fixtures, 2,160 checks, 15 step sizes from 1e-8 to 1e-1, float64 and float32, split by whether a perturbation crosses a gate branch. Away from branch crossings the median relative error is 8.0e-10 in float64. Signs agree on every non-degenerate check, and every zero-warm-up check gives a gradient of exactly zero, as Prop. 1 predicts. The former Table 5 was a different diagnostic: its finite differences regenerated query responses under common random numbers. It is now labeled as such.

5. Cost and reuse (§4.6, Table 3). Selection is a one-time cost. If N deployments share one selection, each costs 1.037 + 3.291/N CPPO runs on average:

Deployments sharing one selection14816
Work per deployment (CPPO runs)4.331.861.451.24

Reused without reselection on a held-out subject mixture, the Meta-TA pair beats CPPO by 2.07 points and the reused BO pair by 0.53, ahead of both in every seed. On Qwen3-1.7B it beats CPPO by 0.40.

6. Clarifications. The no-warm-up ablation deployed the unchanged CPPO rule (its realized KL equals CPPO's), so its zero gain holds by construction. Proposition 1 also covers PPO clipping ranges (Remark 1). The main text still fits in 10 pages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.