acceptodds
Under review as a conference paper at ICLR 2027

CARE-PO: Controllable Tool-Call Efficiency for LLM Agents

Abstract

External tool calls are a major running cost of an LLM agent, while its training objectives align almost exclusively with task completion. In a step-level diagnosis of Qwen3.5 trajectories at two model sizes on GAIA, a long-horizon tool-use benchmark, we find that most contain removable calls, at a similar rate at both sizes. Some methods add an efficiency reward to the objective to reduce the call count, but introduce problems of their own: the anchor they price calls against is computed from the policy's own rollouts (e.g., the group's shortest correct rollout) and so moves with the policy; training can drift toward a collapsed policy that issues no tool call at all, a degeneracy seen in published reports and in our 27B baseline; and a training run delivers a single operating point, so the user cannot trade efficiency against capability. We present CARE-PO (Controllable Anchored Relative Efficiency Policy Optimization) to mitigate these problems. The efficiency reward prices calls against a reference budget anchored to the task's own difficulty and fixed before training. Injecting a successful trajectory from the training data into each online GRPO rollout group and charging for evidence a trajectory never gathers ease the drift toward a collapsed call count. Shaping the cost curve by an effort tag in the prompt yields an efficiency and a capability operating point from a single training run. On GAIA, five of the six operating points across three model sizes improve on their base model in success rate and call count at once. On four held-out multi-hop benchmarks the result is size-dependent: at 27B and 4B success rate is held or raised at a lower call count, while at 9B both operating points trade success rate for fewer calls. The call count remains well above zero throughout training at all three sizes.

Primary area:
reinforcement learning
Keywords:
LLM agents, tool use, reinforcement learning, GRPO, inference-time control, efficiency, reward design, agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4RgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CARE-PO trains tool-using agents to adjust their execution effort with a single set of model weights. Its reward combines correctness and evidence coverage with a bounded cost multiplier based on a task-specific reference budget estimated before training. A successful source trajectory is added to each GRPO group with a nonnegative advantage, while an effort tag in the prompt adjusts the cost assigned to tool calls. The evaluation covers three model sizes on GAIA and four held-out multi-hop benchmarks, alongside analyses of token costs and training dynamics.

Critical strengths
  1. The method addresses a practical deployment concern by substantially reducing tool calls and accounting for token costs. Its bounded reward preserves the stated ordering between successful and unsuccessful trajectories. This design is useful even without a stronger guarantee at the policy level.

  2. The evaluation covers three model sizes and several tasks, includes ablations of effort conditioning and evidence coverage, and reports both improvements and the transfer losses at 9B. The supplementary records support recomputation of the main tables and reported ablations, making the results easier to verify.

Critical weaknesses
  1. The experiments do not isolate which components prevent collapse. The comparison with OTC-PO changes the reference budget, evidence shaping, and injection of successful source trajectories simultaneously. Source injection also provides a demonstration update with positive advantage, which could stabilize training independently of the fixed budget. The existing effort and coverage ablations do not distinguish these effects. This limits the conclusions about why collapse is avoided at 27B, even if the combined method performs well.

  2. The evidence for controlling effort is stronger than the evidence for controlling capability. On GAIA, the 27B model achieves higher success with fewer calls under low effort than under high effort: 57.4% success with 5.20 calls versus 54.7% with 6.12 calls. At 4B, both settings achieve the same mean success rate despite substantially different call counts. Both 9B variants reduce success on the held-out multi-hop tasks. These results support control over computational expenditure, but do not consistently establish the claimed high-effort capability setting or a transferable trade-off between quality and cost.

Questions for authors
  1. Could you isolate the contributions of the fixed reference budget and the nonnegative advantage floor for source trajectories while keeping the remaining reward design and training procedure unchanged? A CARE-PO ablation without source injection, or an OTC-PO baseline with the same source injection, would help clarify the collapse result. If these comparisons are unavailable, please limit the attribution to the combined method.

  2. What do paired comparisons on individual questions reveal about low and high effort? Do the three GAIA runs evaluate the same trained checkpoint repeatedly, or do they include independent training runs? Please distinguish these sources of uncertainty and clarify when users should expect higher effort to improve quality.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4Rgd

We thank the reviewer for the clear and precise questions.

Q1. Isolating the fixed budget and the source-trajectory floor. We ran both controls named in your review at 27B under matched settings (Tables R1 and R2 in the general comment; Appendix E, Tables 5–6): CARE-PO without source injection, and OTC-PO with the same source injection. Removing injection from CARE-PO leads to partial collapse later in training (training success 0.58, 18.8% zero-call rollouts in steps 901–1000), whereas adding the same injection to OTC-PO prevents collapse altogether (training success 0.89, 0.5% zero-call rollouts vs. 99.8% without it). Because these controls toggle the injected member as a unit—combining the successful demonstration and its zero-clipped advantage floor—we scope our anti-collapse attribution strictly to the source-trajectory floor mechanism as a whole (Sections 3.4 and 6.3). Above that floor, the fixed per-task budget with evidence pricing governs the quality–cost equilibrium: replacing the fixed budget inside CARE-PO with a moving group-minimum anchor still trains stably under injection, yet compresses further at a substantial accuracy cost on GAIA (low tier: 41.6% SR at 1.96 calls vs. 53.3% at 3.22 calls; Table R2).

Q2. Uncertainty in GAIA runs, paired per-question tests, and when higher effort improves quality. Each configuration in the main tables uses a single training run (one training seed), and the three GAIA evaluation rounds repeatedly evaluate that same checkpoint across three sessions (Table 1 and Section 4 captions updated for clarity); the reported ± s.d. therefore reflects evaluation variance (live web search non-stationarity and decoding) rather than cross-seed training variance, which Section 7 states explicitly as a limitation. Paired per-question comparisons between high and low effort (Appendix E, Table 8; detailed in our response to Reviewer hSat, item 2) separate expenditure control from accuracy gains:

  • Expenditure: On individual questions, high effort uses more calls than low effort across all scales on GAIA (+2.59 calls at 4B, +1.02 at 9B, +0.92 at 27B; paired bootstrap p ≤ 0.011) and across all 12 size-benchmark cells on the held-out sets (79.2%–100% of differing questions).
  • Quality: On GAIA, per-question SR differences between high and low are not statistically significant at any size (0.0 points, 95% CI [−5.1, +5.1], p = 0.98 at 4B; +4.0, [−1.8, +9.7], p = 0.16 at 9B; −2.6, [−6.3, +0.8], p = 0.15 at 27B; and +4.4, p = 0.12 on the 1000-step 27B checkpoint in Table R2). On the held-out multi-hop benchmarks, high effort significantly improves accuracy over low effort in all 8 of 8 cells at 4B and 9B (exact McNemar, p < 0.04) and in 0 of 4 cells at 27B (p > 0.5).

Users should therefore expect the low tag to deliver substantial call savings when the model has sufficient capacity for the task, and select the high tag on smaller models or retrieval-critical multi-hop queries to mitigate—though at 9B not fully eliminate (60.8% vs. 43.2% on Bamboogle, vs. 76.0% for the base model)—accuracy loss from skipping a required hop.

Reviewer_hSatOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents CARE-PO, an RL framework for improving tool-call efficiency in LLM agents while maintaining task performance. Built upon GRPO, CARE-PO introduces a task-relative efficiency reward based on fixed reference budgets, evidence-aware reward shaping, and the injection of successful reference trajectories into rollout groups to mitigate tool-call collapse. The framework further supports inference-time control of the efficiency–capability trade-off through low- and high-effort conditioning within a single trained model. Experiments fine-tune Qwen3.5 models at 4B, 9B, and 27B scales on 300 tasks sampled from MiroVerse, with evaluations on GAIA and four held-out multi-hop QA benchmarks. Results show substantial reductions in tool calls with generally maintained or improved task success rates, although performance gains vary across model sizes and benchmarks.

Critical strengths
  • The proposed task-relative efficiency reward, evidence-aware reward shaping, and source-trajectory injection provide a reasonable approach to mitigating tool-call collapse while encouraging more efficient agent behaviors.
  • The experiments cover three model sizes (4B, 9B, and 27B), multiple baselines, and several held-out benchmarks. The reported reductions in tool calls are substantial, and the training dynamics provide useful evidence that CARE-PO can mitigate the tool-call collapse observed in the OTC-PO baseline.
Critical weaknesses
  • The paper introduces several mechanisms, including fixed task-relative reference budgets, evidence-aware rewards, and source-trajectory injection. However, the experiments do not sufficiently isolate their individual contributions, particularly the fixed reference budget and source-trajectory injection. Since CARE-PO differs from OTC-PO in multiple aspects, the observed training stability and performance improvements cannot be clearly attributed to specific design choices.

  • Although the paper claims to support controllable efficiency-capability trade-offs, the results are not consistently aligned with this objective. On GAIA, the high-effort setting at 27B achieves lower success rates while using more tool calls than the low-effort setting, and at 4B both settings achieve the same success rate despite substantially different call counts. Moreover, both operating points at 9B exhibit accuracy degradation across all four held-out benchmarks, raising concerns about whether the learned efficiency preferences generalize reliably without sacrificing task performance.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer hSat

We thank the reviewer for the constructive and insightful comments.

1. Isolating fixed budgets, evidence-aware rewards, and source injection. We ran a five-run factorial study at 27B under identical optimization settings (Tables R1–R2 in the general comment; Appendix E, Tables 5–6), complementing the reward-term ablations in Section 6.2:

  • Source-trajectory injection prevents collapse: Removing the source trajectory from CARE-PO leads to partial collapse later in training (18.8% zero-call rollouts at steps 901–1000; GAIA low-tier SR 34.9%), whereas adding the same injection to OTC-PO eliminates its collapse (0.5% zero-call rollouts vs. 99.8% without it).
  • The fixed per-task budget preserves accuracy above that floor: With injection active in both, replacing CARE-PO's fixed budget with a moving group-minimum anchor compresses further at an 11.7-point SR cost under the low tag (41.6% SR at 1.96 calls, HV 0.375 vs. 53.3% at 3.22 calls, HV 0.448), and OTC-PO + injection likewise settles lower (49.3% SR at 3.17 calls, HV 0.415).
  • Evidence-aware cost shaping curbs futile exploration: Removing the (1 − Cov) coverage term while keeping partial credit (Section 6.2, 9B GAIA) raises average calls from 3.42 to 4.00 with unchanged SR (39.6% vs. 39.8%), and the extra calls occur exclusively on incorrect trajectories (3.97 to 4.90 calls, vs. 2.61 on correct ones under both).

2. Effort control and capability trade-offs. Paired per-question bootstrap tests on the Table 1 GAIA runs (three-round means) show that the SR difference (high minus low) is 0.0 points at 4B (95% CI [−5.1, +5.1], p = 0.98), +4.0 at 9B ([−1.8, +9.7], p = 0.16), and −2.6 at 27B ([−6.3, +0.8], p = 0.15)—none statistically significant—whereas the call-count increase under high effort is positive and significant at every scale (+2.59, +1.02, and +0.92 calls; p ≤ 0.011). On the 1000-step 27B checkpoint in Table R2, high minus low is +4.4 SR points (p = 0.12) at +1.01 calls (p < 1e-4), showing no statistically significant inversion there either. Together with the four held-out multi-hop benchmarks—where the high tier significantly outperforms the low tier in all 8 of 8 size-benchmark cells at 4B and 9B (exact McNemar, p < 0.04) and in 0 of 4 cells at 27B (p > 0.5)—the current results support reliable control over tool expenditure across all settings, while statistically significant accuracy gains from high effort appear on retrieval-heavy tasks where the base model leaves headroom (4B and 9B held-out) rather than on GAIA or at 27B. Sections 6.2 and 7 and Appendix E (Table 8) calibrate this claim explicitly.

3. Held-out accuracy degradation at 9B. Per-question failure analysis on Bamboogle shows that 9B CARE-PO (low) answers 55% of questions (69/125) with zero calls, getting 23/69 right vs. 58/69 for the base model and 43/69 for CARE-PO (high) (1.16 calls), while 26 of the 28 questions CARE-PO (high) still loses are single-call answers on two-hop queries (by contrast, 27B answers 42% with zero calls yet holds 72.0% vs. 74.4% SR). This is consistent with premature skipping of external retrieval where 9B parametric capacity cannot supply the missing hop, which high effort mitigates partially (60.8% vs. 43.2%, vs. 76.0% for the base model; see our response to Reviewer 49P4, item 4, and Appendix E).

Reviewer_49P4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paperm the authors propose CARE-PO, a reinforcement learning method for reducing unnecessary tool calls in LLM agents while prioritizing task completion. It evaluates efficiency against fixed, task-specific reference budgets and discourages collapse into making no calls through evidence-aware rewards and successful reference trajectories. By conditioning training on an effort tag, a single model learns low- and high-effort modes, allowing users to adjust the efficiency–capability trade-off at inference time. Experiments show simultaneous improvements in success rate and call efficiency for five of six configurations on GAIA, although performance on other multi-hop benchmarks varies with model size.

Critical strengths
  1. The efficiency yardstick is anchored to each task’s own difficulty and fixed before training, which is the paper’s most fundamental advance over existing online-RL methods. OTC-PO prices calls against the group’s shortest correct rollout, an anchor that drops as the policy compresses so the target keeps receding, while TEPO has no task-level reference at all; CARE-PO instead estimates a per-task reference budget offline and defines cost as the number of budgets spent (u = S_charged / B_ref, Eq. 3). This single design choice makes the pressure scale with difficulty (one extra call costs 1/3 under a 3-call budget but only 1/10 under a 10-call budget), keeps the standard from tightening as trajectories shorten, and tolerates estimation error since the bounded, monotone tanh factor only changes the magnitude of the pressure, never its direction (Sec. 3.3, App. D). The evidence demonstration in experimental section bears this out as well.

  2. The paper provides rigorous and candid experiments across model sizes, with cost measured where it is actually paid. Baselines span all three classes of prior work (inference-time, offline, online RL) on a shared base model, toolset and system prompt; GAIA results are reported as mean ± s.d. over three runs with paired-bootstrap p-values, multi-hop benchmarks use McNemar tests, an 8-gram contamination check is included, and the HV ranking is shown to hold for T_max ∈ {30, 50, 100}. Five of the six operating points beat the base model on both SR and AC at once (27B low: SR +6.1, calls −51.2%), one of the two points tops OTC-PO on HV at every size, and Table 4 confirms the call reduction carries through to tokens (9B: −45% generated, −67% context processed), ruling out compensation by longer reasoning. The limitations (SR loss at 9B on two-hop held-out benchmarks, a single model family, QA-only tasks) are stated plainly in Sec. 7 rather than glossed over.

  3. Fine-grained behavioral analysis strengthens the evidence for efficiency gains. Beyond reporting aggregate tool-call counts, the paper diagnoses redundant behaviors and analyzes the types of calls removed. On the 37 questions that both the base model and the efficiency mode answer correctly in every run, tool calls still decrease by 57.5%, supporting the interpretation that the savings reflect more efficient execution rather than merely earlier termination of unsuccessful trajectories.

Critical weaknesses
  1. The contributions of the core components are not clearly isolated. The ablations examine effort conditioning and the coverage term, but do not separately evaluate the fixed reference budget or source-trajectory injection. Because CARE-PO changes both reward design and training supervision relative to OTC-PO, the current comparisons make it difficult to identify which components drive the improvements in performance and training stability. The authors should compare fixed versus rollout-derived budgets, with and without source-trajectory injection, while keeping the other training settings unchanged.

  2. The obvious training-free baseline is missing, and the included baselines are weak. By the paper’s own accounting, 58% of the calls the efficiency point removes are repeats and 58.5% of the reduction comes from trajectories that hit the 20-call cap (Sec. 6.1), and since the reward already detects repeats with a deterministic exact-match rule (Eq. 3, App. A), a simple inference-time guard that blocks identical calls or breaks tail loops could plausibly capture much of the gain with no training, yet it is never tried; meanwhile the inference-time class is represented by a self-designed “Budget-Prompt” that lowers SR by 4.8 points and increases calls by 2.9% rather than any cited method (Liu 2026b; Chen 2026b; Fang 2026), TEPO is discussed but never run, and DEPO falls below the untrained base, raising doubts about whether the reimplemented baselines were properly tuned. It would be nice if the authors can add a repeat-blocking/loop-breaking guard on the base model as a baseline, and to either run at least one published inference-time method (or TEPO) or report the tuning done for Budget-Prompt and DEPO.

  3. Cross-benchmark generalization does not consistently preserve task performance. For the 9B model, both operating points reduce success rates on all four held-out benchmarks; on Bamboogle, success drops from 76.0% to 43.2% (low) and 60.8% (high). Ensuring that successful trajectories receive higher rewards than failed ones does not guarantee that the optimized policy maintains its overall success rate. Evaluation on a single model family and exclusively on question-answering tasks further limits the generality of the findings. The authors should analyze the 9B failure cases, and extend evaluation to additional model families and task types.

Questions for authors

Please see the question proposed in weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 49P4 (1/2)

We thank the reviewer for the thoughtful evaluation and for highlighting the fixed, task-anchored reference budget as a key advance over rollout-derived anchors.

1. Isolating the fixed budget and source-trajectory injection. Please see Tables R1 and R2 in the general comment for the full 27B factorial study. Holding all other settings fixed, we cross the efficiency reward (fixed budget with evidence pricing vs. OTC-PO's moving group-minimum anchor) with source-trajectory injection, and also test replacing only the fixed budget by the moving anchor inside CARE-PO. In addition, at 9B—where training is already stable without injection—we evaluated CARE-PO without source injection on GAIA (low effort): it reaches 38.0% SR vs. 39.8% for the full method (+1.8 points for full, paired bootstrap p = 0.44) at 4.44 calls vs. 3.42 (1.02 fewer calls per question for full, p < 1e-4). Source injection thus yields further compression where training is stable and prevents collapse where it is not, while the fixed budget preserves accuracy above that floor (Table R2).

2. Training-free repeat-blocking guard. Following your suggestion, we implemented a deterministic wrapper that blocks any call identical to an earlier call in the same trajectory (same tool and normalized arguments), returns a brief "already queried" notice without executing the tool, and after k = 3 consecutive blocked repeats instructs the model to stop calling tools and output its final answer. All other settings (prompts, tools, grading, 20-call cap, greedy decoding) match the main evaluation. Because GAIA uses live web search, each guarded configuration is evaluated alongside its unguarded counterpart in the same session (three rounds each), and we report changes relative to that paired run. A call is attempted when issued by the model (counting toward the 20-call cap) and executed when sent to the tool.

ModelConfigurationBlocked, share of attempted callsExecuted calls, change vs. unguarded pairSuccess, change vs. unguarded pair
9BVanilla + guard11.5%−33%−1.6 points
9BCARE-PO (low) + guard4.3%−24%+1.8 points
27BVanilla + guard11.7%−15%0.0 points

(Round-to-round SR standard deviations are 0.7 to 2.4 points, exceeding the observed SR shifts.)

On the untrained models the guard blocks ~12% of attempted calls at both sizes (firing on 28–30 of 165 questions at 9B, 0.61 blocked calls per question) and reduces executed calls by 33% at 9B and 15% at 27B—exceeding the blocked share itself because the stop notice terminates tail repeat loops early. It is therefore a genuinely useful training-free baseline that captures a substantial share of the savings at 9B. At the same time, it is complementary rather than a substitute for CARE-PO: relative to the same-session unguarded base model, CARE-PO (low) removes more than half of the calls at both sizes, reduces executed calls by >50% vs. 15% for the guard at 27B, and composes cleanly with the guard—when combined with CARE-PO (low) at 9B, the guard removes a further 24% of executed calls with no SR loss. This aligns with Section 6.1: exact repeats represent 58% of the calls removed by the efficiency point, whereas low-yield non-repeat queries (20%) and compressed essential steps (21%) are not targeted by exact-match blocking and are instead compressed by policy training.

AuthorsOpenReview ↗

Response to Reviewer 49P4 (2/2)

3. Inference-time and training baselines. Among cited inference-time approaches, Chen et al. (2026b) steer internal activations, while Liu et al. (2026b) and Fang et al. (2026) inject step-by-step remaining-budget state (e.g., BATS's Budget Tracker) or pair it with dedicated planning and verification controllers rather than conditioning only the initial prompt of an unmodified ReAct loop. Following the second option suggested in your review, we run a post-hoc sensitivity sweep of Budget-Prompt and report the exact training configuration for DEPO (both added to Appendix E).

As a post-hoc diagnostic check on Budget-Prompt at 9B on GAIA (not used to select Table 1), we sweep explicit call targets (2 and 6) alongside a stricter wording (target 3), three rounds each, comparing every variant against the unprompted base model in the same session as in item 2:

Budget-Prompt variant (9B)Calls vs. unprompted pairSuccess vs. unprompted pair
Target 2−12%+0.2 points
Target 3, stricter wording−21%+1.3 points
Target 6−7%+0.7 points

Even the strongest prompt (target 3, stricter wording) removes about a fifth of the calls with SR changes (+0.2 to +1.3 points) smaller than the round-to-round s.d. (1.0–2.3 points). Across variants, explicit numerical targets are only weakly followed (−7% to −21% regardless of target), far short of CARE-PO (low), which removes over half of the calls under the same comparison. For DEPO, we train on 27,426 trajectory examples (13,713 compressed desirable trajectories paired with their 13,713 original long trajectories from the same candidate pool) using the original paper's KTO objective and hyperparameters (β = 0.2, efficiency coefficients α₁ = α₂ = 2), LoRA rank 8 on attention projections, effective batch size 4, and one epoch at lr = 2e-5 with 5% warm-up and cosine decay. Because we keep no GAIA validation split for checkpoint or hyperparameter selection, we report DEPO under this fixed literature recipe and final end-of-epoch checkpoint—without dedicated hyperparameter tuning—matching the single-recipe, final-checkpoint protocol used for CARE-PO itself (Appendices C–E).

4. Held-out results at 9B. We agree with your observation that trajectory-level reward ordering (scoring every successful rollout above every failed one) does not by itself prevent policy mass from shifting toward zero-call answers when parametric guesses occasionally succeed at zero call cost. Per-question inspection supports this explanation. On Bamboogle at 9B, CARE-PO (low) answers 55% of questions (69/125) without issuing any call: on those 69 questions the base model answers 58 correctly, CARE-PO (low) 23, and CARE-PO (high) (1.16 calls on average) 43; moreover, 26 of the 28 questions where CARE-PO (high) still trails the base model are single-call answers on two-hop queries. By contrast, at 27B CARE-PO (low) answers 42% of Bamboogle without a call yet retains nearly full accuracy (72.0% vs. 74.4%), because stronger parametric knowledge covers many first hops. These patterns are consistent with premature skipping of retrieval on two-hop questions where the 9B model still needs external evidence—which CARE-PO (high) mitigates substantially (60.8% vs. 43.2%) though without fully recovering the base model's 76.0%—as now detailed in Section 7 and Appendix E.

5. Additional model families and task types. We fully agree that extending evaluation beyond Qwen3.5 and QA benchmarks to open-ended interactive tasks will further test generality. Given the compute cost of multi-scale live-web RL runs and building verifiable non-QA tool sandboxes during the discussion window, we state these scope boundaries explicitly in Section 7 as primary directions for future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): 27B Factorial Controls — Training Dynamics

We thank all three reviewers for their thorough and constructive feedback. A central question raised across all three reviews is how to isolate the fixed per-task reference budget from source-trajectory injection (with Reviewer 4Rgd naming the exact controls). We have run these factorial controls at 27B—the scale where collapse occurs—alongside the other requested experiments, and report them below. Individual responses follow in separate comments.

Setup of the 27B factorial controls. Following the 1000-step extended-horizon setup at 27B (Appendix C), all five runs share the base model, training tasks and task order, learning rate (1e-5), 1000-step cosine schedule, warm-up and seed. They differ in one factor at a time: the efficiency reward (CARE-PO's fixed per-task budget with evidence pricing, or OTC-PO's moving group-shortest-correct anchor) and whether the source trajectory is injected into each group as a floor. A fifth run retains CARE-PO's reward and injection while replacing the fixed budget with the moving group-shortest-correct anchor.

Table R1. Training dynamics at 27B (optimizer steps; calls and success are means over the 8 policy rollouts per group, injected trajectory excluded).

Efficiency rewardSource floorCalls / rollout, steps 401–500601–700901–1000Train success, 901–1000Zero-call rollouts, 601–700901–1000
CARE-PO (fixed budget, evidence pricing)on3.263.092.530.901.7%1.1%
CARE-POoff2.350.850.920.5838.8%18.8%
OTC-PO (moving group-min anchor)on2.512.522.050.890.3%0.5%
OTC-POoff2.620.000.000.38100%99.8%
CARE-PO with moving group-min anchoron2.221.701.350.890.9%2.7%

(All statistics cover the 8 policy rollouts of each group; the injected trajectory, averaging ~10.6 calls, is excluded. One seed per row.)

Key takeaways from Table R1.

  1. The source-trajectory floor prevents call-count collapse at 27B. Removing it from CARE-PO leads to partial collapse later in training (calls fall below 1 per rollout from around step 650; up to 46% of rollouts in a 100-step window issue no call, and 19% at the end). Conversely, adding the same floor to OTC-PO eliminates its collapse (0.5% zero-call rollouts at the end vs. 99.8% without it). This confirms the reviewers' hypothesis that source injection stabilizes training independently of the fixed budget, and we state this attribution explicitly in Section 6.3 and Appendix E.
  2. With the floor active, both the fixed budget and the moving anchor train stably, so stability alone does not separate them; instead, they converge to different policies because the moving anchor keeps tightening the target as rollouts shorten. Table R2 evaluates the resulting accuracy–cost trade-offs on GAIA.
AuthorsOpenReview ↗

General Response (2/2): 27B GAIA Evaluation & Summary of Revisions

Table R2. GAIA-165 evaluation of the 27B controls (mean ± s.d. over three evaluation rounds of one checkpoint; SR in %, AC = tool calls per question, HV = SR · (1 − AC / T_max) with T_max = 20).

Run (checkpoint)SR lowAC lowHV lowSR highAC highHV high
CARE-PO + source floor (ck1000)53.3 ± 2.63.220.44857.8 ± 0.94.230.456
CARE-PO, no source floor (ck500, pre-collapse)46.5 ± 0.42.360.41055.4 ± 0.93.910.445
CARE-PO, no source floor (ck1000)34.9 ± 1.41.720.31925.7 ± 1.81.870.233
CARE-PO, moving anchor + source floor (ck1000)41.6 ± 2.51.960.37550.3 ± 0.62.580.438
OTC-PO + source floor (ck1000)†49.3 ± 0.93.170.415
OTC-PO, no source floor (ck1000)†16.4 ± 0.60.680.158

† OTC-PO has no effort tag and yields a single adaptive operating point.

Reading of Table R2.

  1. Without the source-trajectory floor, the 1000-step policy degrades at both tiers and the high tag no longer helps (25.7 vs. 34.9 under low; tier separation breaks down), matching the partial collapse in Table R1. At ck500, shown to illustrate the pre-collapse state, the run without injection is still near the 1000-step full run under the high tag (55.4 vs. 57.8) while already 6.8 points lower under the low tag (46.5 vs. 53.3, at 2.36 vs. 3.22 calls).
  2. OTC-PO with the source floor trains stably yet reaches a weaker operating point than CARE-PO (49.3 SR at 3.17 calls, vs. 53.3 at 3.22 under low and 57.8 at 4.23 under high).
  3. Replacing the fixed budget with the moving anchor under the same floor over-compresses at the expense of accuracy: under the low tag it uses 1.96 calls vs. 3.22 and loses 11.7 points of SR (HV 0.375 vs. 0.448); under the high tag the HV gap narrows (0.438 vs. 0.456) because the 7.5-point SR loss is partly offset by 1.65 fewer calls. Preserving accuracy above the anti-collapse floor—especially at the efficiency operating point—is therefore the distinct contribution of the fixed per-task budget.

Summary of revisions in the uploaded PDF. Appendix E adds the 27B factorial controls (Tables 5–6, matching Tables R1–R2), the repeat-blocking guard (Table 7), the Budget-Prompt sweep and DEPO training specification, the paired per-question tests of the two effort tiers (Table 8), and the per-question analysis of 9B held-out failures; Sections 3.4, 4, 6.2, 6.3 and 7 incorporate the corresponding clarifications and cross-references. The main text remains within the 10-page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.