acceptodds
Under review as a conference paper at ICLR 2027

Learning to Ask for Help: Budget-Constrained Decision Routing via Dual-Advantage GRPO

Abstract

Small language models (SLMs) on devices answer queries locally and request cloud assistance when needed. Each cloud request consumes resources, motivating a prescribed allowance for collaboration. Existing collaborative post-training methods use a fixed assistance reward, but the resulting cloud usage can drift as local competence changes. DA-GRPO treats expected cloud usage as a constraint: task reward and cloud cost yield two group-relative advantages, combined by a multiplier updated from measured usage relative to the target. The method retains the clipped GRPO surrogate and adds no measurable training overhead in our configuration. On Qwen2.5-1.5B-Instruct and Llama-3.2-3B-Instruct across mathematical reasoning, code generation, and knowledge benchmarks, DA-GRPO achieves the highest mean joint accuracy in all eight settings under common task-specific training budget targets, improving on the strongest fixed-reward baseline by 1.8 to 7.0 percentage points. In a separate budget-control experiment, mean usage is 0.313 at the end of training and 0.314 at in-distribution test time against a target of 0.30. Cloud usage increases from 0.10 on the easiest MATH level to 0.49 on the hardest. The same framework supports changing targets during continued training and cloud-token costs, while test-time usage under distribution shift can exceed the training target.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, edge-cloud collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vEuKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DA-GRPO, a budget-constrained reinforcement learning framework for local-cloud language model collaboration. The method combines group-relative task-reward and cloud-cost advantages through a dynamically updated Lagrangian multiplier, allowing a local SLM to jointly learn reasoning and help-seeking decisions under a prescribed cloud-usage budget. Experiments on two SLMs across mathematical reasoning, code generation, and knowledge benchmarks demonstrate improved joint accuracy over fixed-reward collaborative training baselines. Additional analyses investigate budget tracking, difficulty-dependent cloud usage, changing budget targets, and token-aware costs.

Critical strengths

S1. Practically relevant problem formulation. The paper addresses resource-aware local-cloud collaboration, where balancing response quality and cloud usage is practically important. Incorporating explicit resource constraints into collaborative post-training provides a meaningful alternative to manually tuning fixed assistance rewards.

S2. Simple and empirically effective framework. DA-GRPO integrates cost-aware optimization into GRPO without introducing an additional critic or routing model. The method achieves consistent mean joint-accuracy improvements across the evaluated benchmarks, supported by additional analyses of budget adaptation, difficulty-dependent routing, and training efficiency.

Critical weaknesses

W1. Insufficient problem justification and limited methodological novelty.

The paper treats collaboration-ratio drift under fixed assistance rewards as a fundamental limitation. However, decreasing cloud usage may naturally reflect improving local model competence and a more cost-effective collaboration strategy, rather than an undesirable behavior. Moreover, the prescribed budgets lack concrete deployment-level justification, leaving unclear when enforcing a predefined budget is preferable to optimizing the evolving accuracy–cost tradeoff. Methodologically, the proposed dual-advantage construction is algebraically equivalent to applying standard GRPO to the Lagrangian-shaped reward r−λcr-\lambda c, while the multiplier update follows conventional primal-dual optimization. The methodological contribution beyond existing constrained RL techniques therefore appears limited.

W2. Insufficient controlled experiments and deployment-oriented validation.

Although extensive experiments are presented, several limitations weaken the empirical conclusions. First, comparisons with fixed-reward baselines do not isolate the contribution of adaptive dual optimization; a controlled comparison between fixed and adaptive λ\lambda under identical reward formulations and training settings is missing. Second, the paper does not sufficiently justify coupling budget control with local SLM training, compared with independent, inference-time adjustable routers such as RouteLLM and Hybrid LLM. Third, accuracy improvements are not consistently evaluated under matched actual cloud usage, particularly for TACO-Verified and MMLU. Moreover, budget satisfaction exhibits across-seed variability and substantial violations under distribution shifts, while changing budgets requires continued training. Finally, the respective contributions of local reasoning and routing improvements remain insufficiently disentangled, several gains lack statistical significance analysis, and end-to-end inference costs are not comprehensively evaluated.

Questions for authors

Q1. Problem motivation and methodological contribution. Why should collaboration-ratio drift be considered undesirable when it may naturally reflect improving local competence? What practical constraints motivate the selected budget targets? Furthermore, what is the essential methodological contribution beyond applying standard primal-dual optimization to GRPO?

Q2. Necessity of dynamic dual optimization. Can the authors compare fixed and adaptive λ\lambda using identical Lagrangian-shaped rewards, GRPO optimizers, and training budgets? Does adaptive optimization consistently improve accuracy or budget stability under matched actual cloud usage?

Q3. Necessity of coupled routing and local training. What advantages does joint optimization offer over independent, budget-adjustable routers such as RouteLLM or Hybrid LLM? Can the authors provide accuracy–cost comparisons across multiple budgets, accounting for retraining requirements and inference overhead?

Q4. Attribution and robustness of empirical gains. Can the authors report actual test-time cloud usage for all benchmarks and quantitatively separate local reasoning improvements from routing improvements? How robust are the reported gains across additional seeds and under distribution shifts?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vEuK (1/2)

We thank the reviewer for the detailed review. Part 1 covers the motivation and the contribution. Part 2 covers the experimental requests.

[W1, Q1] Why is drift undesirable, and why a prescribed budget?

We agree that falling usage can be a good outcome. Our concern is that the operator does not choose the level. Under a fixed assistance term, usage follows the reward coefficient and the current competence of the model, and it can move in either direction. The new runs in the general response show both directions in one run. Usage first peaks near 0.8 against an allowance of 0.30. It then falls to 0.03 to 0.12 and stays near 0.18. The first phase overspends the budget. The second phase leaves budget unused that would buy accuracy.

The budget in problem (1) is a ceiling. When usage is below τ, the projected update (5) moves λ toward 0, so the method never forces the model to spend. A deployment that prefers to save cost sets a smaller τ. Section 1 names the cases we have in mind: a metered API contract, a link budget, and a policy on what may leave the device. In each case, the allowance is fixed before accuracy is considered, and it is stated as a share of queries or as tokens. A fixed assistance term cannot state such an allowance. It sets a price, and the usage that a price buys changes as the model learns. We will add these examples to Section 3:

A budget is the largest share of queries, or of tokens, that a deployment may send to the cloud, for example under a metered API contract, a per-device rate limit, or a policy on what may leave the device. Problem (1) treats it as a ceiling. When usage is below τ, the projected update moves λ toward 0. A fixed assistance term instead fixes a price, and the usage it buys changes with the competence of the model.

[W1, Q1] Contribution beyond primal-dual GRPO

We agree that the dual-weighted advantage gives the same update as standard GRPO on the reward r − λc. Section 4 states this equivalence (lines 198 to 199). We do not claim a new advantage estimator, and we will remove wording that suggests one. We claim the following:

  1. We treat cloud usage in collaborative post-training as a constraint on a help action that the local model generates itself. Prior collaborative post-training (Fang et al., 2026) uses a fixed assistance term instead.
  2. We show that a price set in advance does not reliably hold a budget. Eq. (4) explains why. Help is worth its price when the reward gain from the cloud exceeds λ. As the local model improves, this gain shrinks for more prompts, so a fixed price buys less help over time. The new runs show this for the fixed term and for fixed λ = 0.3 and 0.5. A fixed λ holds τ = 0.30 only near 0.71, the price that adaptive λ finds in one run (general response, Section 2).
  3. One scalar update from measured usage sets the price inside an unmodified clipped GRPO surrogate. It needs no critic, cost model or router, and it adds no forward or backward pass. Section 4 also shows where the pressure acts. Only groups that disagree about asking carry a direct cost gradient, and our reply to Reviewer Fb7b reports how often this happens.

We revise the contribution statement as follows:

We do not propose a new advantage estimator. For fixed λ, the dual-weighted advantage equals the group-centered advantage of r − λc (Section 4). Our contributions are 1) a constrained formulation of collaborative post-training, in which a budget sets usage instead of a hand-tuned assistance reward, 2) evidence that a fixed assistance term does not hold a budget during post-training, and 3) a controller that sets the price of help from measured usage inside an unmodified GRPO update.
AuthorsOpenReview ↗

Response to Reviewer vEuK (2/2)

[W2, Q2] Fixed versus adaptive λ under identical rewards

We agree that fixed λ under the identical reward is the right control, and Section 2 of the general response reports it. We trained GRPO on the identical reward r − λc with λ fixed at 0.3, 0.5, 0.71 and 1.0. Over steps 101 to 300, usage averages 0.83, 0.44 and 0.001 at λ = 0.3, 0.5 and 1.0. At λ = 0.71, where adaptive λ settles, usage averages 0.287. Adaptive λ averages 0.296 and 0.298 over the same steps. A fixed λ therefore holds the budget only when it equals the price that adaptation finds. With adaptive λ, the price moves from 0.50 to about 0.70 in the first 50 steps, while usage is above the target. By Eq. (4), the right price depends on the gap between cloud and local reward, so it is not known before training.

[W2, Q3] Why couple the decision with local training?

An inference-time router such as RouteLLM or Hybrid LLM decides over a frozen model. The local model is then trained as if it had to answer every query (Section 2). With a help action, the local model learns which queries it can solve while it learns to solve them, and no second model runs at inference. We agree that a router has one clear advantage: it can change the budget without retraining. Our conclusion states that DA-GRPO sets the budget during training. We will add RouteLLM-style and Hybrid-LLM-style routers on the locally tuned model, swept over thresholds, and compare accuracy against measured usage.

[W2, Q4] Measured usage, attribution and robustness

  • Measured usage. We will report the measured ratio of every Table 1 cell from the same evaluation pass as its accuracies, including TACO-Verified and MMLU (general response, Section 3).
  • Seeds. In the new runs, usage over steps 101 to 300 is 0.296 ± 0.042 and 0.298 ± 0.050 for the two DA-GRPO seeds.
  • Distribution shift. The constraint holds the expected usage on the training distribution. Under shift, usage can exceed τ, and Section 5.4 and our conclusion state this. A strict cap at test time needs an extra mechanism, and we leave it to future work.
  • Local reasoning versus routing. At each checkpoint's own ratio, we will compare its help requests with random and oracle routing on the same local model. This separates allocation from local competence.
  • Significance. We will report paired bootstrap intervals over test items and seeds for every comparison in the revised tables.

We thank the reviewer again. Please let us know if this response addresses your concerns, or if further clarification would help.

Reviewer_s8LiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DA-GRPO for budget-constrained local-cloud collaboration, combining group-relative reward and cost advantages with an adaptive multiplier. Specifically, DA-GRPO uses a budget-constrained reinforcement learning framework that enables small language models (SLMs) to decide when to answer locally and when to request assistance from a more capable cloud LLM. Experiments evaluate joint accuracy, budget tracking, and token costs across two small language models.

Critical strengths
  • The paper addresses a practical weakness of fixed assistance rewards: cloud usage can drift as the local model improves. The proposed update is simple and fits within the GRPO training procedure.
  • The proposed DA-GRPO is simple and practical. It adds only a group-relative cost advantage and a scalar dual update on top of GRPO, introducing no extra critic, router, or forward/backward pass, and no measurable training overhead.
Critical weaknesses
  • DA-GRPO changes both the reward formulation and the way cloud usage is controlled. This makes it hard to tell whether the gains come from adaptive budget control or simply from using a different reward. A fixed-λ ablation with the same reward and training setup would help clarify this.
  • It is unclear whether DA-GRPO really learns better routing decisions. The reported local accuracy is measured on different subsets of questions selected by each method, so higher accuracy does not necessarily mean better routing. The fixed-set experiment helps, but it still does not separate improvements in local model ability from better routing decisions.
  • Small suggestion none blocking. The authors are encouraged to insert figures in PDF or vector format to improve their clarity and readability.
Questions for authors
  • How much of the improvement actually comes from updating λ? Could the authors compare adaptive λ with fixed λ under the same reward and training setup?
  • Could the authors evaluate routing quality under matched cloud budgets while controlling for differences in local model ability?
2 comments
AuthorsOpenReview ↗

Response to Reviewer s8Li

We thank the reviewer for the positive assessment of the method's simplicity, and for two focused questions.

[W1, Q1] Is the gain from adapting λ or from a different reward?

The reviewer is right that our comparisons change two things at once. The fixed-reward baselines pay a fixed assistance term for a correct assisted answer (0.2, 0.05 or 0.4 by task). DA-GRPO pays the task reward minus the price λ. We ran the suggested ablation (general response, Section 2). It trains GRPO on the identical reward r − λc with λ fixed at 0.3, 0.5, 0.71 and 1.0, with the same data, optimizer and group size. Only the dual step differs.

Over steps 101 to 300, usage averages 0.83 at λ = 0.3, 0.44 at λ = 0.5 and 0.001 at λ = 1.0. At λ = 0.71, usage averages 0.287, against 0.296 and 0.298 for adaptive λ. At λ = 0.3 and 0.5, usage keeps falling during training. At λ = 0.71, each 50-step average from step 101 stays between 0.27 and 0.30. The budget is therefore held by the price level, and adaptation is how we find this level without a search. A guess of 0.5 or 1.0 misses the target widely.

Adaptive λ starts at 0.50 and rises to about 0.70 while usage is above the target. Over steps 101 to 300, it settles at 0.71 to 0.72 and usage averages 0.30. By Eq. (4), the right price depends on the gap between cloud and local reward. This gap depends on the local model, so the price is not known before training.

[W2, Q2] Does DA-GRPO learn better routing, controlling for local ability?

We agree that accuracy on the retained subset mixes local ability with routing. The fixed-set control in Figure 3 isolates only the local model. We will add a decomposition that holds the local model fixed:

  1. We evaluate each checkpoint with the cloud disabled on the full test set. This gives the local correctness of every item.
  2. At the checkpoint's own measured ratio ρ, we compute joint accuracy under three routers on the same local model. They are its own help requests, random routing at ρ, and oracle routing at ρ. The oracle sends exactly the items that the local model gets wrong.
  3. Routing quality is the share of the gap between random and oracle routing that the model's own requests close. The local model is the same in all three, so this share does not depend on local ability.

Comparing methods at matched ρ then separates local competence (accuracy with the cloud disabled) from allocation (routing quality). We will report both for all methods and seeds, with paired bootstrap intervals.

[W3] Vector figures

Thank you. We will replace all figures with vector PDFs.

Please let us know if this response addresses your concerns, or if further clarification would help.

Thank you for your response. I decide to keep my score.

Reviewer_Fb7bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DA-GRPO for jointly training a small language model to answer queries locally and selectively request assistance from a fixed cloud LLM under an expected usage budget. The method computes group-centered task-reward and cloud-cost advantages, combines them using an adaptive Lagrange multiplier, and updates the multiplier according to the observed budget violation. Experiments on two SLMs across mathematics, code generation, and knowledge benchmarks show improvements over fixed-reward collaborative training and routing baselines. Additional experiments investigate budget tracking, difficulty-dependent assistance, changing training targets, token-based costs, and optimization overhead.

Critical strengths
  1. A practical formulation of budget-aware collaboration during post-training. Replacing a fixed assistance reward with an adaptive Lagrangian penalty addresses a concrete limitation of collaborative post-training: the same fixed incentive may produce different cloud usage as local competence changes. The method is simple to integrate into GRPO, requires no additional critic or inference-time router, and introduces negligible measured training overhead. The experiments also demonstrate adaptation to changing budgets during continued training, providing a useful implementation strategy for resource-constrained LLM systems.

  2. Encouraging empirical results and informative budget-control analyses. The paper evaluates two SLMs across four benchmarks and compares against routing-based and collaborative-training baselines. DA-GRPO achieves the highest mean joint accuracy in all eight reported settings, with particularly meaningful improvements on mathematics benchmarks where realized usage is approximately matched. The additional comparisons of alternative dual formulations, difficulty-dependent cloud usage, token-based budgets, and fixed-set local performance provide useful evidence about the behavior and limitations of learned assistance policies.

Critical weaknesses
  1. Limited methodological novelty beyond standard primal–dual optimization. The proposed dual-weighted advantage in Eq. (3) is algebraically identical to group-centering the Lagrangian-shaped reward r−λcr-\lambda c. Thus, the main algorithmic mechanism is a standard adaptive Lagrange multiplier combined with an existing group-relative policy optimizer, rather than a fundamentally new advantage estimator. The theoretical results establish unbiasedness only for an idealized, unclipped estimator and characterize fixed points of the mean dual update; they provide no convergence or feasibility guarantees for the implemented GRPO procedure. Moreover, Table 2 does not establish the advantage of the proposed formulation over the objective-level dual at matched realized cost: the latter obtains higher joint accuracy (82.4 versus 80.4) while consuming more cloud assistance (0.346 versus 0.314). These observations make it difficult to distinguish the contribution of the specific advantage formulation from the benefits of standard adaptive constraint pricing.

  2. Insufficient matched-cost evidence for the central accuracy claims. Table 1 reports the highest mean joint accuracy across eight settings, but realized test-time cloud usage is unavailable for TACO-Verified and MMLU, which account for four settings. Matching training budget targets does not guarantee matched inference-time resource consumption, especially since the paper itself demonstrates that cloud usage can change substantially under distribution shift. Even in the dedicated budget-control experiment, the test-time collaboration ratio is 0.314 ± 0.070 against a target of 0.30, indicating approximate average tracking rather than reliable budget satisfaction across runs. Without realized cost measurements and accuracy–cost comparisons on all evaluated tasks, the reported accuracy advantages cannot be consistently interpreted as improvements under equivalent resource constraints.

Questions for authors
  1. Matched-cost evaluation. Could the authors report the realized test-time collaboration ratios for TACO-Verified and MMLU for all compared methods, including variability across seeds? Do DA-GRPO's accuracy improvements persist when methods are compared at matched realized cloud usage rather than common training budget targets?

  2. Contribution of the advantage formulation. Since Air−λAicA_i^r-\lambda A_i^c equals the group-centered advantage of ri−λcir_i-\lambda c_i, does DA-GRPO differ from standard GRPO with an adaptively Lagrangian-shaped reward when normalization and clipping conventions are held fixed? Which empirical benefits specifically depend on the proposed advantage-level construction rather than the primal–dual controller itself?

  3. Effectiveness of cost feedback. As noted in Section 4, the cost advantage vanishes when all rollouts in a group make the same assistance decision. How frequently does this occur during training, and how does it affect the responsiveness of the dual controller, particularly during the initial periods of near-universal cloud offloading shown in Figure 4?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fb7b (1/2)

We thank the reviewer for the careful review. Q3 led us to log the share of mixed groups at every step, and the new runs below report it.

[W1, Q2] Contribution of the advantage formulation

DA-GRPO does not differ from GRPO on the shaped reward r − λc when normalization and clipping are held fixed. Section 4 states this (lines 198 to 199 and 238 to 243). The benefits we claim come from the primal-dual controller. They do not come from the advantage-level form. We keep the advantage-level form for practical reasons. It retains one clipped GRPO surrogate, and a further resource adds only one cost advantage and one scalar update. We will state this plainly and revise the contribution statement (general response, Section 5).

[W1] Scope of the theory

We agree with the reviewer's reading of the theory. Proposition A.2 covers the unclipped, unnormalized estimator. Proposition B.1 characterizes the stationary points of the mean dual dynamics. The paper does not claim convergence or feasibility for the implemented procedure (lines 200 to 203 and 234 to 235). The new runs behave as Proposition B.1 describes. Over steps 101 to 300, λ settles at 0.71 to 0.72 and the batch usage averages 0.30.

[W1] Objective-level dual versus advantage-level dual

We agree that the paper does not show either form to be more accurate at matched cost. Section 5.3 states this (lines 396 to 397), and we make no such claim. Before clipping, the two forms share the same surrogate, and the benefit we claim comes from the controller that both use. In the revision, we compare the two forms at matched measured usage.

[Q3] How often is the cost advantage zero?

We now log, at every step, the share of groups whose responses disagree about asking. These mixed groups are the only ones with a nonzero cost advantage. In the two new DA-GRPO runs, the share is as follows.

PhaseMixed groups (seed 1 / seed 2)Batch usage
Steps 1–100.64 / 0.62rises from 0.50 to about 0.90
Steps 11–300.48 / 0.47peaks at 0.94, then falls to about 0.65
Steps 201–3000.36 / 0.340.296 / 0.286

During the early phase of near-universal offloading, about half of the groups still carry a cost signal. λ rises from 0.50 to about 0.70 within 50 steps. The 10-step moving average of usage reaches the band 0.27 to 0.33 at step 56 and step 79. Late in training, about two thirds of the groups are unanimous, and usage still averages 0.30 in this phase. We will add this measurement to Section 4, as a figure with λ over the same steps.

AuthorsOpenReview ↗

Response to Reviewer Fb7b (2/2)

[W2, Q1] Matched-cost evidence

We agree that common training targets do not establish matched test-time cost, and Section 5.2 says so for TACO-Verified and MMLU. In the revision, we recompute every cell of Table 1 from one evaluation pass. Each cell reports the measured ratio, local accuracy and joint accuracy, with mean and standard deviation over seeds, including TACO-Verified and MMLU. We also place every method on a plot of joint accuracy against measured usage. This plot answers the reviewer's second question directly, because it compares methods at equal measured cost.

[W2] Budget satisfaction across runs

We agree that the objective gives approximate tracking of expected usage, not a hard cap, and the paper uses this wording. In the new runs, usage over steps 101 to 300 is 0.296 ± 0.042 and 0.298 ± 0.050 for the two DA-GRPO seeds. With the fixed assistance term, it is 0.182 ± 0.039 and 0.180 ± 0.048 against the same target. We will also report test-time usage for these checkpoints.

We thank the reviewer again for these questions. Please let us know if this response addresses your concerns, or if further clarification would help.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and constructive questions. The three reviews ask for the same three controls: 1) fixed versus adaptive λ under the same reward, 2) comparisons at matched measured usage, and 3) a separation of local competence from routing. We report new runs for the first, and we describe how the revision answers the other two. Per-reviewer replies follow.

1. New budget-control runs with step-level logging

To answer Reviewer Fb7b's question about unanimous groups, we trained new runs that log the share of mixed groups at every step. The setting is Qwen2.5-1.5B-Instruct on MATH-lighteval with τ = 0.30, 300 updates and two seeds per method. The table reports the collaboration ratio of the training batches.

MethodSeedSteps 1–50Steps 51–100Steps 101–300 (mean ± sd)Mean λ, steps 201–300
GRPO, fixed assistance term 0.210.3620.1190.182 ± 0.039n/a
GRPO, fixed assistance term 0.220.3450.0310.180 ± 0.048n/a
DA-GRPO10.6910.3330.296 ± 0.0420.709
DA-GRPO20.6920.3390.298 ± 0.0500.721

With the fixed assistance term, usage drifts as Section 3 describes. It peaks near 0.8 within ten steps, falls to 0.03 to 0.12 in steps 51 to 100, and then stays near 0.18, well below the target. With DA-GRPO, usage also overshoots early and peaks at 0.94. During this phase, λ rises from 0.50 to about 0.70. The 10-step moving average of usage enters the band 0.27 to 0.33 at step 56 (seed 1) and step 79 (seed 2). Over steps 101 to 300, usage averages 0.296 and 0.298, and λ settles at 0.71 to 0.72. This agrees with the stationary-point interpretation of Proposition B.1.

2. Fixed versus adaptive λ under the identical reward (all reviewers)

We trained GRPO on the identical reward r − λc with λ held fixed at 0.3, 0.5, 0.71 and 1.0, in the setting of Section 1 with one seed each. The value 0.71 is where adaptive λ settles. Only the dual step differs from DA-GRPO.

MethodSeedSteps 1–50Steps 51–100Steps 101–300 (mean ± sd)Mean λ, steps 201–300
Fixed λ = 0.310.9570.9860.831 ± 0.1400.30
Fixed λ = 0.510.8100.5140.436 ± 0.0570.50
Fixed λ = 0.7110.5100.3040.287 ± 0.0430.71
Fixed λ = 1.010.1750.0010.001 ± 0.0021.00
DA-GRPO, adaptive λ10.6910.3330.296 ± 0.0420.709
DA-GRPO, adaptive λ20.6920.3390.298 ± 0.0500.721

A fixed λ holds the target only near the price that adaptive λ finds. At λ = 0.3, usage stays above 0.95 from step 11 to step 150, and it is still 0.64 in steps 251 to 300. At λ = 1.0, usage is at most 0.01 after step 50, so the model almost never asks. At λ = 0.5, usage averages 0.44 and falls slowly, from 0.47 in steps 101 to 150 to 0.40 in steps 251 to 300. At λ = 0.71, usage averages 0.287, close to the target. From step 101 on, each 50-step average stays between 0.27 and 0.30. Early on, it also overshoots less, because adaptive λ starts at 0.50. Budget tracking therefore depends on the price level. Adaptation finds this price in one run, without a search over λ, and a guess of 0.5 or 1.0 misses the target widely.

3. Matched measured usage (Fb7b, vEuK)

We agree that common training targets do not establish matched test-time cost. In the revision, every cell of Table 1 reports the measured ratio, local accuracy and joint accuracy from one evaluation pass, with mean and standard deviation over seeds. This includes TACO-Verified and MMLU. We also add a plot of joint accuracy against measured usage for every method, so that methods are compared at equal cost.

4. Local competence versus routing (s8Li, vEuK)

We hold the local model fixed and evaluate it with the cloud disabled on the full test set. At the checkpoint's own measured ratio, we compare its own help requests with random and oracle routing on the same local model. The share of the gap between random and oracle routing that its own requests close measures routing quality, independent of local ability.

5. Positioning

Reviewers vEuK and Fb7b note that the dual-weighted advantage equals the group-centered advantage of r − λc. We agree, and Section 4 states this. We claim three contributions: 1) a constrained formulation of collaborative post-training, 2) evidence that a fixed assistance term does not hold a budget, and 3) a controller that sets the price of help from measured usage inside an unmodified GRPO update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.