acceptodds
Under review as a conference paper at ICLR 2027

Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning

Abstract

Training tool-using language models requires costly interaction, yet identical rewards within a rollout group provide no group-relative learning signal. We connect harness evaluation and rollout allocation through reward disagreement in Model Context Protocol (MCP) environments. We first introduce Discord, which tests harness edits using exact paired inference and decision-preserving early stopping: only disagreeing paired outcomes favor one harness over another. This distinction between evaluation volume and useful evidence inspires BRACE(Bank-calibrated Rollout Allocation under Correlated Episodes). BRACE combines historical task evidence, uncertainty in task success rates, and a correlated reward model to forecast mixed-reward groups before allocating generation budget. Controlled harness experiments favor Discord, while live MCP experiments expose a retention–cost trade-off. At three Qwen3-VL model scales, BRACE exceeds the strongest plotted baseline peaks with fewer new training rollouts; historical-bank collection is counted separately. It also leads the compared methods on BFCL and NESTFUL without further training. Ablations probe the bank, estimator, and allocation score; prospective evaluation checks forecast calibration. These results support forecasting reward contrast before generation.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, LLM agents, tool use, Model Context Protocol

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oeYJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses two problems in tool-using language agents: efficient evaluation of agent harness modifications and effective rollout allocation for reinforcement learning. For the first problem, the authors propose DISCORD, which evaluates candidate harness edits by comparing paired task outcomes between the incumbent and candidate configurations. It uses exact statistical testing and early stopping to reduce evaluation costs while preserving the retention decision. For the second problem, the authors introduce BRACE, which leverages historical rollout outcomes to predict the likelihood of mixed-reward groups. Based on these predictions, it assigns sampling weights to eligible tasks before generating fixed-size rollout groups, aiming to reduce uninformative training rollouts caused by identical within-group rewards.

Critical strengths
  1. The paper investigates two practically important issues in tool-using agents: the high cost of evaluating harness modifications and the inefficient use of rollout budgets during RL training. Both are relevant to improving agent performance under limited computational resources.
  2. The experiments include comparisons against multiple baselines and evaluations across three Qwen3-VL model sizes.
Critical weaknesses
  1. Although the paper identifies two relevant problems, the motivation for addressing them together is not sufficiently developed. DISCORD focuses on evaluating and retaining harness modifications without changing model parameters, whereas BRACE focuses on allocating training tasks to optimize model parameters. Their connection appears primarily conceptual, through the use of outcome disagreement, rather than methodological or operational.
  2. The proposed approaches build on relatively intuitive principles: using paired outcome disagreements to evaluate competing configurations and prioritizing tasks expected to produce informative reward variation. However, the paper does not sufficiently clarify which methodological components constitute the principal novelty relative to existing paired statistical testing, early-stopping procedures, and adaptive task-sampling strategies. The authors should more explicitly explain why the proposed mechanisms are preferable to simpler alternatives and which components are essential to the reported improvements. A stronger discussion of the limitations of existing approaches and the specific mechanisms by which DISCORD and BRACE overcome them would improve the technical contribution.
  3. Although BRACE achieves the best solve rates across the three model scales, its improvements are less consistent on other evaluation metrics. In Table 2, BRACE obtains the best result on only three of the nine reported model-scale/metric combinations. These observations indicate potential trade-offs between task-solving performance, tool-use reliability, and interaction efficiency that are not sufficiently explained.
Questions for authors

What is the substantive connection between the two proposed methods beyond their shared reliance on reward disagreement? Are they intended to form a unified optimization framework, or are they independent approaches addressing different problems? If the latter, what motivates presenting them as a single contribution?

1 comment

Thank you for identifying the need for a clearer connection, a more precise novelty claim, and an explicit discussion of metric trade-offs. We have revised the framing and added controls to address these points.

Why present DISCORD and BRACE together? Both address expensive interactions during development of the same agent, but at different decision points: harness retention and policy training. The conceptual progression is from measuring useful contrast to predicting where it will occur. In DISCORD, paired agreement cannot distinguish the candidate from the incumbent; in BRACE, unanimous rewards cannot produce a group-centered reward signal. DISCORD motivates BRACE through this principle, rather than supplying a required intermediate output. The Introduction, Figure 1, and Sections 2–4 now distinguish the common motivation from the different statistical objects. We make no claim of a jointly optimized pipeline or an experimentally established gain from composing the two procedures.

Novelty relative to simpler alternatives. We now explicitly credit paired exact testing and curtailment as established tools. DISCORD's contribution is their operational use and evaluation for harness retention, including stopping without changing the prescribed full-test decision. BRACE addresses prospective allocation through the combination of capped historical task evidence, posterior uncertainty, correlated-group forecasts, and eligibility checks. Uniform sampling ignores task evidence; post-generation filtering cannot recover the cost of groups already generated. PLR, TSCL, TRACE, and VIP provide comparisons to other adaptive rules. The revised baseline description also makes clear that TSCL uses historical evidence, while its bank snapshot and eligible pool differ from BRACE's.

Which components matter? Table 3 reports that removing the bank lowers solve rate by up to 4.3 percentage points. The prospective correlation-aware forecast reduces degeneracy-prediction error from 0.591 to 0.056 over 83 evaluation arm-cycles, excluding the cycles used to fit correlation. Proposition 1 states the important boundary: at fixed group size and known task success probability, correlation correction preserves ranking, so calibration alone does not establish improved task ordering or policy performance.

The original VIP-style row changed score and eligibility together. Section 5.5 now adds a 4B matched-eligibility control: BRACE achieves 14.9% versus 13.2% for VIP-style allocation. Without the mask, the rates remain 14.9%/13.2%. Thus the 1.7-point gap persists with eligibility held fixed; the interaction is 0.0 points at reported precision. This supports the score comparison at 4B without attributing the full gap specifically to correlation correction. The main text also reports early-window mixed-group yields of 55.8/50.2/43.3 per 1,000 attempts for BRACE versus 26.2/21.9/21.9 for uniform GRPO. These measure realized allocation yield, not full-training compute savings.

Solve rate, call reliability, and turns. We agree that these metrics capture different outcomes. Section 5.3 now states explicitly that BRACE leads the three solve-rate columns, but not the six invalid-call/turn columns. Relative to the base, invalid calls fall from 6.5% to 4.6% at 4B but rise from 16.6% to 19.3% at 2B and from 6.8% to 9.0% at 8B. BRACE targets the availability of a training reward signal, not an explicit call-validity or trajectory-length objective. Short trajectories may also end in failure, so mean turns alone do not establish successful-task efficiency. We retain the full metrics and scope the claim to task completion, rather than claiming uniform improvement in tool-use reliability.

Presentation and additional validation. The main 10 pages now contain the organizing principle, component evidence, cost trade-offs, and a new text-only Qwen3-4B-Instruct-2507 control (Table 4). BRACE achieves 13.6% mean AWM solve rate versus 12.4% for PLR, 12.1% for uniform GRPO, and 11.4% for TRACE. The revised text distinguishes descriptive gains from corrected paired significance. The appendix supplies detailed protocols and limitations rather than carrying the main answers to the reviews.

We hope these changes make both the contribution and the strength of its supporting evidence substantially easier to assess.

Reviewer_giprOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DISCORD and BRACE to allocate evaluation and training budgets for tool-using agents based on reward disagreement. DISCORD uses exact paired testing to decide whether to retain harness modifications, while BRACE combines historical task outcomes, success-rate uncertainty, and rollout correlation to select tasks likely to produce mixed-reward groups for GRPO training.

Critical strengths

The paper provides detailed experimental settings and evaluates transfer to additional benchmarks.

Critical weaknesses
  1. The connection between DISCORD and BRACE is weak. Beyond the shared motivation of reward disagreement, the two methods address separate problems and are evaluated independently. As presented, the paper feels like two loosely connected studies.
  2. The practical motivation for DISCORD is unclear. It relies on repeated evaluations to decide whether to retain harness modifications, potentially requiring substantial interaction costs. I am not convinced that the paper establishes a realistic use case where this form of statistically driven iterative optimization is necessary and worth the cost.
  3. The experimental validation on BRACE is limited, particularly in its settings, as detailed below.
Questions for authors
  1. Why is BRACE evaluated exclusively on Qwen3-VL models rather than commonly used text-only LLMs? Since the adopted evaluation tasks do not rely on visual inputs, please provide evidences that the gains also hold for more well-trained LLMs.
  2. The proposed methods do not appear to rely on properties specific to tool-use agents or MCP. In particular, BRACE seems broadly applicable to task selection in group-relative RL with binary rewards. Could the authors clarify which aspects of the design are specific to this setting? The current framing, especially the title, suggests domain-specific contributions that are not clearly established. I suggest revising it to distinguish the methods’ general scope from their experimental setting.
1 comment

Thank you for highlighting the connection between the methods, DISCORD's practical cost, and the scope of the BRACE experiments. We address each concern bellow:

Connection between DISCORD and BRACE. The Introduction and Figure 1 now explain the shared decision principle and its two applications. An agent consists of a policy operating through an external harness. Improving the harness requires deciding which edits to retain; improving the policy requires selecting training tasks. In a paired harness comparison, agreement cannot favor either configuration. In GRPO, an all-equal reward group supplies no centered reward advantage. DISCORD therefore measures the contrast relevant to retention, and this motivates BRACE to forecast the availability of contrast before generating a training group. We present a design principle applied to these two development decisions, without claiming a joint optimizer or a measured benefit from composing the methods. Sections 3 and 4 distinguish their evidence sources and guarantees.

When is DISCORD worth its evaluation cost? Its intended use is validation of edits that will be reused, such as changes to prompts, tool descriptions, or routing. Section 5.2 now distinguishes two comparisons. Against completing the same 100-pair exact test, the additional stopping regimes use 60/80/100 pairs, saving 40%/20%/0% of evaluation episodes; curtailment preserves that terminal test's decision. Against a cheaper deployed retention rule, stronger validation can cost more. We report this explicitly: episode use increases by 8.3%/108.8% in the shallow/deep live settings, while final objectives are lower by 0.62/1.09 points. The benefit is protection against unsupported retention, not universal cost superiority. In the matched harmful-proposal experiment, degradation occurs in 0/17 DISCORD runs versus 9/17 strict-improvement runs (exact paired p=0.0039). Appendix F separates fixed-test savings from search and downstream reuse costs.

Text-only, instruction-tuned models. The original comparison examines scale within the Qwen3-VL family using text/tool inputs. We now add Qwen3-4B-Instruct-2507, a text-only instruction-tuned model, with BRACE's configuration and bank unchanged (Table 4, Section 5.5). Mean held-out AWM solve rates are 13.6% for BRACE, 12.4% for PLR, 12.1% for uniform GRPO, and 11.4% for TRACE. BRACE yields 34.8 reconstructed informative groups per 1,000 attempts, versus 21.1/18.3/15.6 for those comparators. At the final checkpoint, its BFCL/NESTFUL gains over the untrained model are +10.0/+3.1 points, without further training. The task-level AWM test is deterministic, and only TRACE differs significantly after four-comparison Holm correction (adjusted p=0.004); we do not claim significance for the other gaps. This provides direct evidence beyond vision-language backbones while keeping the conclusion limited to the tested Qwen3 models.

General methods versus MCP-specific design. We agree that the statistical mechanisms do not require MCP messages. DISCORD requires paired binary outcomes, while BRACE requires task identities and outcome histories. Tool-use agents provide a consequential setting because each evaluation or rollout may incur multiple model and tool interactions. The setting-specific component is eligibility checking with executable verifiers and reachability information. The abstract, Introduction, and Section 4 now distinguish these general mechanisms from the experimental substrate. We also revised the title to “Measure First, Then Allocate: Reward Contrast for Harness Retention and Policy Training.”

We hope they address the concerns about motivation, applicability, and validation.

Reviewer_p5xXOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budget allocation problems in tool-using agents. Specifically, the authors propose DISCORD, which evaluates harness edits using exact paired inference and decision-preserving early stopping. They further introduce BRACE, which predicts the likelihood of mixed-reward rollout groups to guide task selection before generation. The proposed methods are evaluated in tool-use environments, including experiments with Qwen3-VL models.

Critical strengths

BRACE provides a practical approach to improving rollout allocation efficiency by predicting the probability of mixed-reward groups before allocating the generation budget. This helps reduce potentially uninformative rollouts.

Critical weaknesses
  1. Lack of a unified core contribution. The paper combines two largely independent methods, DISCORD and BRACE. Although the authors attempt to connect them through the concept of reward contrast, I find this connection rather weak. The two methods address different problems, and their combination does not form a sufficiently coherent contribution.

  2. Limited methodological novelty. Both proposed methods are relatively straightforward and mainly build upon existing statistical techniques and allocation strategies. The methodological advances appear incremental, while the empirical improvements are not sufficiently substantial to establish a strong contribution.

Questions for authors
  1. BRACE uses a Beta distribution to model uncertainty in task success probabilities. Have the authors considered alternative distributions or estimation methods? Could more flexible models provide better estimates and improve allocation performance?

  2. Could the prospective allocation idea of BRACE also be applied to harness retention? For example, could historical evidence be used to predict which evaluation tasks are more likely to produce discordant outcomes between two harness configurations, thereby reducing the evaluation cost of DISCORD?

1 comment

Thank you for the review and the two questions. Please also see the general response, which summarizes the revision.

W1 (two methods, weak connection). We agree. The revision makes BRACE the paper's single contribution and moves the harness-retention study (DISCORD) to Appendix E as a companion result. Figure 1 and the new title, Measure First, Then Allocate: Calibrated Rollout Budgets for Group-Relative Reinforcement Learning, describe BRACE only.

W2 (novelty; size of the gains). We agree that the statistical primitives are classical; Section 3.5 now says so and separates what is new from what is borrowed. The new element is a model of dependence among a group's rollouts, which the closest pre-generation selectors (MoPPS, BOTS, GRESO, Knapsack RL, TRACE, VIP) do not have; Table 1 contrasts the mechanisms. In Section 3.3 (Proposition 1) we show what this dependence changes: it keeps the ordering of variance-based scores, reshapes the independent forecast, and supplies the calibrated level of degenerate groups that a variance score lacks. In Section 4.4 and Appendix D.1 we show that the correlated forecast is calibrated before generation while an independent forecast from the same evidence is not, and in Section 4.3 that this holds on a text-only model without refitting. Table 6 shows that BRACE obtains substantially more informative groups per rollout than uniform GRPO. We agree that the solve-rate gains are modest, and the revision says so: Section 4.2 and Appendix B.7 show that BRACE has the highest mean held-out solve rate at every scale and is significantly better than several baselines, while its margins over uniform GRPO and TRACE are within run-to-run variation. We therefore present the calibrated forecast and the yield as the main evidence.

Q1 (alternatives to the Beta model). BRACE already goes one step beyond a Beta posterior over each task's success rate: Section 3.2 models a group's shared success propensity with a Beta-Binomial, so the rollouts of one group are correlated. We kept the Beta family because it is conjugate for binary rewards: banked and new outcomes enter as counts, and the score is computed exactly from Beta moments without sampling or model calls (Section 3.2, Appendix C.3). In Section 4.4 (Table 5) we compare alternatives within this family and show that the group-level correlation is the part that matters: removing it, which gives the plain independent model, loses calibration and lowers the solve rate at every scale, while replacing the posterior with a plug-in point estimate or refitting the correlation per scale changes little. Appendix D.1 shows where a more flexible model could still help: the forecast's level is calibrated, but its task-by-task discrimination among the selected tasks is weak. Models that share evidence across related tasks, use task features, or track the policy's drift during training could improve that ranking. We have not tested them and consider this a promising direction.

Q2 (prospective allocation for harness retention). Yes, and DISCORD's test makes this unusually safe. Appendix E.3 shows that its early stopping (exact curtailment) always returns the decision that full evaluation would have returned, so the order in which tasks are evaluated cannot change the decision or its error control. Ordering tasks by a forecast of discordance built from earlier evaluations could therefore only make evaluation stop sooner: concordant pairs carry no evidence, and evaluating likely-discordant tasks first accumulates the deciding evidence faster. This targets DISCORD's main cost, discussed in Appendix E.6. Skipping tasks forecast to be concordant would save more but is a different procedure, because the decision would then refer to the selected subset rather than to the full task suite. We have not run either variant and offer it as future work.

We hope this clarifies the contribution and would be grateful if you would reconsider your assessment in light of the revision.

Comments on the paper

1 comment

Thank you to the reviewers for the constructive feedback. We have revised the manuscript to make the connection between the methods, their technical contribution, and the empirical scope easier to assess. The principal changes are summarized below.

1. A clearer organizing principle and scope. The Introduction and Figure 1 connect two decisions in developing the same agent: retaining changes to its external harness and training the policy operating within it. DISCORD uses observed paired disagreement to make a retention decision; this motivates BRACE's prediction of within-group reward variation before allocating training rollouts. We distinguish this shared design principle from a jointly optimized pipeline. The revised title, “Measure First, Then Allocate: Reward Contrast for Harness Retention and Policy Training,” reflects the methods' general scope; MCP is the experimental setting.

2. New text-only evidence in the main paper. Table 4 adds Qwen3-4B-Instruct-2507 under the same training protocol. BRACE achieves 13.6% mean held-out AWM solve rate, compared with 12.4% for PLR, 12.1% for uniform GRPO, and 11.4% for TRACE. It also obtains +10.0 percentage points on BFCL and +3.1 on NESTFUL relative to the untrained model. The deterministic task-level AWM test uses four-comparison Holm correction; only the comparison with TRACE is significant after correction (adjusted p=0.004). Transfer gains are descriptive.

3. Stronger component and mechanism evidence. Section 5.5 adds a 4B eligibility-matched comparison: BRACE reaches 14.9% versus 13.2% for VIP-style allocation, with the 1.7-point gap persisting when the eligibility mask is removed. The main text now also reports realized mixed-group yield in a common early log window: 55.8/50.2/43.3 groups per 1,000 attempts for BRACE at 2B/4B/8B, versus 26.2/21.9/21.9 for uniform GRPO. We distinguish these allocation diagnostics from full-training compute savings. Table 3 and Proposition 1 clarify the roles of historical evidence, uncertainty, and correlation calibration.

4. DISCORD's use case and cost trade-off. Section 5.2 explains screening harness edits before repeated reuse. The added fixed-test comparison evaluates 60/80/100 pairs in the reported stopping regimes, versus 100 for the full exact test, yielding 40%/20%/0% evaluation savings. This isolates curtailment relative to the same terminal test. We separately report the live comparison with the cheaper deployed retention rule, including its cost and objective trade-offs.

5. Reliability and reporting. We explicitly discuss why improved solve rate does not imply uniformly fewer invalid calls or turns.

We provide point-by-point replies below and would appreciate the reviewers' assessment of these clarifications and additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.