acceptodds
Withdrawn from ICLR 2027

Learning When to Spawn Agents via Quality-Guided Cost Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A language-model recruitment policy must decide when additional agents justify their execution cost. With Group Relative Policy Optimization (GRPO), adding a cost penalty to the reward can make cost differences determine task-level advantages whenever all sampled trajectories have the same quality. We show why reducing the penalty coefficient may have little effect: normalization approximately cancels its scale when weighted cost dispersion is large relative to the numerical stability term. Quality-Guided Cost Optimization (QGC) uses the group's quality state to choose which cost comparisons to retain. It assigns zero task scores when quality is zero, compares team size alone when quality is identical and positive, and combines separately normalized quality and full estimated cost when quality varies. Positive quality includes partial success. The positive-quality branch therefore favors smaller teams at the same observed progress without penalizing output length. Across three Qwen3-8B training seeds on controlled compositional programming tasks, QGC raises mean success from 44.56% to 46.03% (+1.46 percentage points before rounding) and reduces mean estimated cost by 28.78% relative to Quality-Only. Against Guided Direct Cost, it gains 1.95 percentage points with 1.51% lower cost. The gains vary across seeds and models. Branch comparisons show higher success at slightly higher cost relative to full-cost tie feedback, and lower cost with a small success loss relative to tie suppression. These results support choosing cost comparisons according to observed task progress, while leaving matched-resource branch benefits to be established.

Primary area:
reinforcement learning
Keywords:
large language model agents, multi-agent systems, reinforcement learning, group relative policy optimization, agent recruitment, cost-aware learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eDrcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Quality-Guided Cost Optimization (QGC), a modification to GRPO reward construction for learning agent recruitment policies. It observes that within-group normalization can effectively cancel the cost coefficient when trajectories have identical quality. QGC introduces different reward rules depending on whether all trajectories fail, achieve equal positive quality, or have varying quality. On compositional programming tasks, QGC reports a 1.46% success improvement over Quality-Only on Qwen3-8B, with lower estimated cost.

Critical strengths

The observation about cost coefficient cancellation under within-group normalization is mathematically reasonable and potentially useful. The paper also includes ablation comparisons to investigate different reward constructions.

Critical weaknesses
  1. The recruitment problem and cost model are insufficiently motivated.

The benchmark consists of programming tasks with relatively explicit subtask decompositions. In many cases, assigning workers according to the number of independent subtasks appears to be a natural heuristic. Yet no simple structure-aware recruitment baseline is evaluated. It is therefore unclear whether learning is necessary or offers meaningful advantages.

The cost model is also inadequately justified. It uses the maximum estimated text length across execution slots plus a fixed penalty of 50 per recruited worker. The scheduling assumptions, the choice of this penalty, and its relationship to actual execution costs are not established. Since this proxy determines both the training rewards and reported efficiency gains, these limitations are consequential.

  1. The experimental evidence does not convincingly establish the proposed method's advantage.

The reported gain is very modest, only 1.46 percentage points, with considerable variation across three training seeds. One seed performs worse than both Quality-Only and GDC. No paired statistical tests or confidence intervals are provided.

More importantly, the tie-suppressed control (TSC) achieves comparable success to QGC (45.76% versus 45.48%). Thus, the benefit of the proposed positive-quality tie-handling mechanism remains unclear. The reported cost reduction is also largely achieved by GDC, suggesting that much of the improvement comes from introducing a cost penalty rather than the proposed method.

The main comparisons additionally use different cost coefficients (QGC at 0.03 versus GDC/NC at 0.1), making it difficult to attribute the observed differences to reward construction.

Questions for authors
  • How does QGC compare against simple structure-aware recruitment heuristics?

  • Can the authors provide statistically supported comparisons under matched resource budgets and appropriately tuned cost coefficients?

  • How sensitive are the results to the worker penalty, and how does the cost proxy relate to actual token consumption or latency?

Reviewer_iS49OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies quality-cost rewards for training an agent recruitment policy with GRPO. It shows that when trajectories within a group have identical quality, normalization can approximately cancel the scale of the cost coefficient, giving cheap failures positive advantages. QGC assigns zero task scores to all-zero-quality groups, compares team size alone in equal positive-quality groups, and separately normalizes quality and proxy cost when quality varies. Experiments on compositional programming tasks report success, proxy cost, cross-model results, and branch-rule diagnostics.

Critical strengths
  1. The derivation of coefficient cancellation in equal-quality groups is relevant to the training objective. It explains why merely reducing the cost weight may not remove a preference for cheap failed trajectories.
Critical weaknesses
  1. QGC's mean success gain over QO is only 1.46 percentage points, and one Qwen3-8B run reverses it. More importantly, Appendix C.1 says that the task counts, decoding repetitions, and integer success counts for Tables 1–3 are unspecified; Appendix C.4 says common-checkpoint, evaluator, and unchanged-weight checks are undocumented for the aggregate tables. These strange contents make the reliability of the experimental evaluation questionable.
  2. Table 1 uses λ=0.03 for QGC but λ=0.1 for GDC and NC, and Figure 3's development budgets do not match realized cost. In Table 3, QGC gains success over FCT with slightly higher proxy cost and more total output tokens, while it trades some success for lower proxy cost against TSC. The current comparisons do not establish a better quality-cost frontier from the proposed rule itself, and proxy-cost reductions do not demonstrate actual token or latency savings.
Questions for authors
  1. What are the independent task counts, decoding repetitions, and integer success counts for Tables 1–3? Were all methods evaluated on the same held-out tasks with the same evaluator and strictly frozen checkpoints, and can you provide paired task-level outcomes?
  2. With equal tuning budgets and matched realized cost, what are the success-cost curves for QGC, GDC, NC, FCT, and TSC? Does the Table 3 difference remain attributable to the positive-quality-tie rule at a common operating point?
  3. Do the Pareto relationships in Tables 1 and 3 persist when Equation 2's proxy cost is replaced by tokenizer-counted total tokens and measured latency?
Reviewer_5tnmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce Quality-Guided Cost Optimization (QGC), a reward design approach for training language-model agents to decide when to recruit additional workers during task execution. The authors identify an issue in cost-aware GRPO training, where reward normalization can cause cost differences to dominate the advantage signal when sampled trajectories have identical task quality. To address this, QGC uses three quality-dependent reward rules, suppressing cost feedback when all trajectories fail, comparing team sizes when trajectories achieve the same positive quality, and combining separately normalized quality and full estimated cost when quality varies. The method is evaluated on compositional programming tasks using several language models, with comparisons against alternative reward formulations and analyses of task success and agent recruitment behavior.

Critical strengths
  1. In this paper, I especially like that the paper identifies a meaningful issue in cost-aware GRPO training, where reward normalization can cause cost differences to dominate the advantage signal even when all trajectories fail. The mathematical analysis provides a clear explanation of this behavior and helps motivate the need to reconsider how quality and cost rewards are combined.

  2. The paper includes several useful ablation studies and comparisons across different model backbones, which help illustrate the trade-offs between task success and execution cost. And I like that the authors report results across multiple training seeds and openly discuss cases where the proposed method does not consistently outperform the baselines.

Critical weaknesses
  1. In this paper, I found the methodological novelty and technical depth of this work rather limited. While I think the motivation is reasonable, the proposed approach largely relies on three manually designed reward rules, without a convincing explanation of why these particular choices are necessary or better than other alternatives. The so-called quality-guided optimization is essentially a heuristic switch between predefined reward functions, and I do not see a sufficiently novel or principled mechanism beyond existing multi-objective reward optimization methods.

  2. The experimental results are not particularly convincing to me, especially considering the modest and inconsistent performance improvements across models and training seeds. The evaluation is restricted to controlled compositional programming tasks, and the reported cost reductions rely on a manually defined proxy rather than actual computational expenses. And I found it difficult to tell whether the method genuinely learns better agent recruitment decisions or simply encourages smaller teams through its reward design.

Questions for authors
  1. I want to better understand the reasoning behind the three reward branches, particularly why zero-quality groups should completely ignore cost while equal positive-quality groups should only consider team size.

  2. Could you please provide a more comprehensive comparison with properly tuned baselines under matched training budgets to clarify how much improvement actually comes from the proposed reward design?

  3. How can we tell whether the learned policy is making better recruitment decisions rather than simply reducing the number of workers across tasks?

  4. Besides, I concerned that the current evaluation is too narrow to support the broader claims about efficient multi-agent systems. Would you consider testing QGC on more realistic and diverse agentic environments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.