- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DA-GRPO, a budget-constrained reinforcement learning framework for local-cloud language model collaboration. The method combines group-relative task-reward and cloud-cost advantages through a dynamically updated Lagrangian multiplier, allowing a local SLM to jointly learn reasoning and help-seeking decisions under a prescribed cloud-usage budget. Experiments on two SLMs across mathematical reasoning, code generation, and knowledge benchmarks demonstrate improved joint accuracy over fixed-reward collaborative training baselines. Additional analyses investigate budget tracking, difficulty-dependent cloud usage, changing budget targets, and token-aware costs.
Critical strengths
S1. Practically relevant problem formulation. The paper addresses resource-aware local-cloud collaboration, where balancing response quality and cloud usage is practically important. Incorporating explicit resource constraints into collaborative post-training provides a meaningful alternative to manually tuning fixed assistance rewards.
S2. Simple and empirically effective framework. DA-GRPO integrates cost-aware optimization into GRPO without introducing an additional critic or routing model. The method achieves consistent mean joint-accuracy improvements across the evaluated benchmarks, supported by additional analyses of budget adaptation, difficulty-dependent routing, and training efficiency.
Critical weaknesses
W1. Insufficient problem justification and limited methodological novelty.
The paper treats collaboration-ratio drift under fixed assistance rewards as a fundamental limitation. However, decreasing cloud usage may naturally reflect improving local model competence and a more cost-effective collaboration strategy, rather than an undesirable behavior. Moreover, the prescribed budgets lack concrete deployment-level justification, leaving unclear when enforcing a predefined budget is preferable to optimizing the evolving accuracy–cost tradeoff. Methodologically, the proposed dual-advantage construction is algebraically equivalent to applying standard GRPO to the Lagrangian-shaped reward , while the multiplier update follows conventional primal-dual optimization. The methodological contribution beyond existing constrained RL techniques therefore appears limited.
W2. Insufficient controlled experiments and deployment-oriented validation.
Although extensive experiments are presented, several limitations weaken the empirical conclusions. First, comparisons with fixed-reward baselines do not isolate the contribution of adaptive dual optimization; a controlled comparison between fixed and adaptive under identical reward formulations and training settings is missing. Second, the paper does not sufficiently justify coupling budget control with local SLM training, compared with independent, inference-time adjustable routers such as RouteLLM and Hybrid LLM. Third, accuracy improvements are not consistently evaluated under matched actual cloud usage, particularly for TACO-Verified and MMLU. Moreover, budget satisfaction exhibits across-seed variability and substantial violations under distribution shifts, while changing budgets requires continued training. Finally, the respective contributions of local reasoning and routing improvements remain insufficiently disentangled, several gains lack statistical significance analysis, and end-to-end inference costs are not comprehensively evaluated.
Questions for authors
Q1. Problem motivation and methodological contribution. Why should collaboration-ratio drift be considered undesirable when it may naturally reflect improving local competence? What practical constraints motivate the selected budget targets? Furthermore, what is the essential methodological contribution beyond applying standard primal-dual optimization to GRPO?
Q2. Necessity of dynamic dual optimization. Can the authors compare fixed and adaptive using identical Lagrangian-shaped rewards, GRPO optimizers, and training budgets? Does adaptive optimization consistently improve accuracy or budget stability under matched actual cloud usage?
Q3. Necessity of coupled routing and local training. What advantages does joint optimization offer over independent, budget-adjustable routers such as RouteLLM or Hybrid LLM? Can the authors provide accuracy–cost comparisons across multiple budgets, accounting for retraining requirements and inference overhead?
Q4. Attribution and robustness of empirical gains. Can the authors report actual test-time cloud usage for all benchmarks and quantitatively separate local reasoning improvements from routing improvements? How robust are the reported gains across additional seeds and under distribution shifts?
2 comments
Response to Reviewer vEuK (2/2)
[W2, Q2] Fixed versus adaptive λ under identical rewards
We agree that fixed λ under the identical reward is the right control, and Section 2 of the general response reports it. We trained GRPO on the identical reward r − λc with λ fixed at 0.3, 0.5, 0.71 and 1.0. Over steps 101 to 300, usage averages 0.83, 0.44 and 0.001 at λ = 0.3, 0.5 and 1.0. At λ = 0.71, where adaptive λ settles, usage averages 0.287. Adaptive λ averages 0.296 and 0.298 over the same steps. A fixed λ therefore holds the budget only when it equals the price that adaptation finds. With adaptive λ, the price moves from 0.50 to about 0.70 in the first 50 steps, while usage is above the target. By Eq. (4), the right price depends on the gap between cloud and local reward, so it is not known before training.
[W2, Q3] Why couple the decision with local training?
An inference-time router such as RouteLLM or Hybrid LLM decides over a frozen model. The local model is then trained as if it had to answer every query (Section 2). With a help action, the local model learns which queries it can solve while it learns to solve them, and no second model runs at inference. We agree that a router has one clear advantage: it can change the budget without retraining. Our conclusion states that DA-GRPO sets the budget during training. We will add RouteLLM-style and Hybrid-LLM-style routers on the locally tuned model, swept over thresholds, and compare accuracy against measured usage.
[W2, Q4] Measured usage, attribution and robustness
- Measured usage. We will report the measured ratio of every Table 1 cell from the same evaluation pass as its accuracies, including TACO-Verified and MMLU (general response, Section 3).
- Seeds. In the new runs, usage over steps 101 to 300 is 0.296 ± 0.042 and 0.298 ± 0.050 for the two DA-GRPO seeds.
- Distribution shift. The constraint holds the expected usage on the training distribution. Under shift, usage can exceed τ, and Section 5.4 and our conclusion state this. A strict cap at test time needs an extra mechanism, and we leave it to future work.
- Local reasoning versus routing. At each checkpoint's own ratio, we will compare its help requests with random and oracle routing on the same local model. This separates allocation from local competence.
- Significance. We will report paired bootstrap intervals over test items and seeds for every comparison in the revised tables.
We thank the reviewer again. Please let us know if this response addresses your concerns, or if further clarification would help.
Response to Reviewer vEuK (1/2)
We thank the reviewer for the detailed review. Part 1 covers the motivation and the contribution. Part 2 covers the experimental requests.
[W1, Q1] Why is drift undesirable, and why a prescribed budget?
We agree that falling usage can be a good outcome. Our concern is that the operator does not choose the level. Under a fixed assistance term, usage follows the reward coefficient and the current competence of the model, and it can move in either direction. The new runs in the general response show both directions in one run. Usage first peaks near 0.8 against an allowance of 0.30. It then falls to 0.03 to 0.12 and stays near 0.18. The first phase overspends the budget. The second phase leaves budget unused that would buy accuracy.
The budget in problem (1) is a ceiling. When usage is below τ, the projected update (5) moves λ toward 0, so the method never forces the model to spend. A deployment that prefers to save cost sets a smaller τ. Section 1 names the cases we have in mind: a metered API contract, a link budget, and a policy on what may leave the device. In each case, the allowance is fixed before accuracy is considered, and it is stated as a share of queries or as tokens. A fixed assistance term cannot state such an allowance. It sets a price, and the usage that a price buys changes as the model learns. We will add these examples to Section 3:
[W1, Q1] Contribution beyond primal-dual GRPO
We agree that the dual-weighted advantage gives the same update as standard GRPO on the reward r − λc. Section 4 states this equivalence (lines 198 to 199). We do not claim a new advantage estimator, and we will remove wording that suggests one. We claim the following:
We revise the contribution statement as follows: