acceptodds
Under review as a conference paper at ICLR 2027

CALIPER: Compute-Aware Local Intervention and Targeted Policy Updates for Tool-Augmented Agents

Abstract

Agentic RL commonly samples, compares, and updates full trajectories. Under sparse terminal rewards, more rollouts need not yield more useful supervision: full-trajectory generation consumes the budget, one outcome conflates multiple behavioral changes, and trajectory-level advantages may update decisions unrelated to the observed outcome change. This problem is pronounced in tool-integrated reasoning (TIR), where local changes arise both from autonomous decisions and from responses to newly received tool feedback. The challenge is to choose where to spend computation, which local changes merit comparison, and how to turn their evidence into localized training credit. We propose CALIPER. Under a fixed rollout budget, CALIPER first allocates a portion to full-trajectory exploration, with resulting trajectories serving as training samples and online probes of the current policy. From failed trajectories, CALIPER identifies two types of candidate positions, autonomous model decisions and tool-feedback-driven responses, and estimates their per-compute value from historical success rates, uncertainty, and generation cost. It then adaptively allocates the remaining budget between full exploration and local validation. For selected positions, CALIPER preserves the preceding interaction history and tool state while resampling only subsequent behaviors, yielding strictly verifiable same-state comparisons. During training, CALIPER combines global trajectory advantages with reliability-calibrated local Leave-One-Out (LOO) credit, restricts local credit to the regions where behavior actually changes, and uses negative local evidence to selectively strengthen policy updates. Even when all full trajectories fail, local comparisons can provide non-zero and spatially localized training signals. Across tool-augmented reasoning benchmarks, CALIPER overall outperforms GRPO and other strong baselines, demonstrating improved training-information efficiency under limited rollout budgets.

Primary area:
reinforcement learning
Keywords:
Augments trajectory-level RL with local interventions, adaptively allocating rollout budget and aligning credit and updates to changed decisions.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2PuoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CALIPER, a method for improving credit assignment in tool-augmented agentic RL under a fixed rollout budget. It adaptively allocates samples between complete trajectories and local interventions at decision points in failed trajectories. Local continuations preserve the same preceding execution state, and their comparisons provide credit restricted to post-fork behavior. Experiments cover five benchmarks with Qwen3-8B and Qwen3-4B. CALIPER achieves the highest average score among the evaluated methods with Qwen3-8B, while component ablations support the proposed design.

Critical strengths

The paper addresses an important problem with a reasonable approach. Sparse terminal rewards provide limited guidance about which decisions in an agent trajectory deserve credit. Combining local comparisons with compute-aware sampling offers a practical way to obtain more useful supervision from a limited rollout budget. This is particularly relevant when trajectories involve multiple tool interactions.

The main results and ablations provide encouraging evidence. CALIPER improves over Plain GRPO across all five benchmarks with Qwen3-8B, raising the average score from 68.44 to 71.38. The Qwen3-4B results show a similar overall improvement. Removing individual components consistently lowers average performance, supporting the contribution of the different design choices.

Critical weaknesses

The presentation and allocation of space weaken the paper’s focus. Several figures have labels that are difficult to read, especially Figures 5 and 6. References are also missing where they are needed, including the discussion of prior approaches in the second introduction paragraph and several baselines in the experimental setup. More broadly, the method combines many mechanisms, making its central contribution harder to identify. The lengthy method description leaves limited room for deeper experimental analysis. Narrowing the scope and moving secondary implementation details to the appendix would improve the paper.

The robustness of the method to its many hyperparameters remains unclear. Important choices include the initial full-rollout count (B_0=4), the 32-step history window, the efficiency margin of 1.10, and the contrast threshold of 0.40. The paper specifies these values but does not report sensitivity analyses. The held-out calibration also fixes (B_0=4), so it does not establish the suitability of the initial allocation. The adaptive-routing ablation compares against fixed 4/4 sampling, but does not show whether the advantage holds against other fixed allocations. These gaps limit my confidence that the reported gains are stable across reasonable settings.

Questions for authors

Could you clarify which mechanisms are the core contributions and provide the missing references for the prior-work discussion and baselines?

How was (B_0=4) chosen? Was it compared with other initial allocations or a fixed 6/2 baseline?

Do you have sensitivity results for the history window, efficiency margin, and contrast threshold that would establish the robustness of the reported gains?

26 comments
AuthorsOpenReview ↗

Main Response | Part 01/15

Main Response | Part 01/15

Main Response

In this rebuttal, we have added extensive controlled experiments, ablation studies, and in-depth analyses and redrawn the paper's main and analytical figures, further improving CALIPER's experimental validation, technical justification, and presentation. We believe the new evidence makes the work more complete, rigorous, and convincing. We sincerely hope the reviewer will consider these improvements comprehensively and reconsider the paper's rating. Your recognition would also give us greater motivation to continue conducting more rigorous, interesting, and valuable research in the future.

Thank you for recognizing the problem of credit assignment in tool-use trajectories under sparse terminal rewards and for appreciating CALIPER's consistently positive results across five Qwen3-8B benchmarks, Qwen3-4B, and component ablations. We agree that, although the original manuscript included a complete implementation, it did not clearly distinguish core contributions from secondary mechanisms, nor did it provide sufficiently direct robustness evidence for fixed allocations and routing parameters. In response, we recast CALIPER as a loop of "opportunity selection -- same-state intervention -- credit attribution to changed regions," adding a fixed 6/2 training control, actual allocation-distribution statistics, and routing-sensitivity analysis, and completing the prior-work and baseline citations. We first answer your three sets of Questions for Authors in turn. The paper's focus, figure readability, and appendix restructuring are addressed together under the corresponding weaknesses.

Because the new experiments and analyses were jointly conducted in response to multiple reviewers, experiment identifiers are assigned globally. The identifiers cited in an individual response may therefore be nonconsecutive, but all refer to the same set of experiments and analyses.

1. Clarification 1: CALIPER's Three-Link Loop Is the Core Contribution, and Related-Work and Baseline Citations Have Been Completed

Your two questions: (1) Which mechanisms in CALIPER constitute the core contribution? (2) Can the missing citations in the prior-work discussion and experimental baselines be added?

1.1 Brief Summary of the Conclusions for Clarification 1

Prior work on intermediate-state branching, suffix resampling, and local credit provides important foundations for fine-grained agentic RL. CALIPER advances the problem beyond simply constructing local branches and local credit: it uses existing full trajectories as online probes to discover and validate local opportunities. Specifically, CALIPER addresses three interrelated questions in one framework: (1) how to discover intervenable decision points in autonomous decisions and tool-feedback responses, and dynamically allocate the remaining budget between further full-trajectory exploration and local validation that reuses reached states, according to recovery value and generation cost; (2) how to ensure that local outcome differences genuinely result from behavioral changes after the same tool execution state; and (3) how to update only the tokens genuinely supported by that evidence. CALIPER's central technical innovation is to unify these three links under a fixed rollout budget into a complete loop of "compute-value assessment -- execution-consistent comparison -- credit-scope alignment."

AuthorsOpenReview ↗

Main Response | Part 02/15

** Main Response | Part 02/15**

1.1.1 CALIPER's Three Core Innovations

  1. From generic prefix branching to typed interventions with explicit behavioral semantics. CALIPER does not indiscriminately retry intermediate prefixes. It distinguishes autonomous model decisions from responses to tool feedback and constructs feasible rethink, finalize, or tool-repair interventions according to the remaining tool, interaction, and generation budgets. Thus, it asks not merely "should this prefix be retried?" but "is a particular executable behavioral change worthwhile at this failed state?"

  2. From selecting a branch location to online compute decisions between full exploration and local validation. CALIPER does not select branches based only on entropy or predicted value. It combines recent strict rescue rates, uncertainty, and additional generation costs to estimate the verified success-per-compute of each typed opportunity relative to a full trajectory, then adaptively allocates the budget among 8/0, 6/2, and 4/4. Local retries therefore compete directly with fresh full rollouts under the same fixed budget instead of being automatically added branch overhead.

  3. From observing local outcome differences to learning only behavior supported by those differences. CALIPER restores the same conversation history, tool outputs, and consumed budgets for each selected fork and resamples only post-fork behavior. It then uses reliability-calibrated signed LOO credit to update only changeable post-fork policy tokens, and strengthens negative updates only for locally inferior continuations in same-state comparisons. Local sampling, outcome comparison, and token-level actor updating thus share one attribution boundary.

These are not arbitrary modules placed side by side, but an end-to-end aligned evidence chain: opportunity valuation determines where compute is spent, execution-consistent interventions determine whether the resulting differences are comparable, and scope-aligned credit determines what the evidence is allowed to update. This also explains how CALIPER can recover nonzero and spatially localized training signals from local strict rescues or bounded failure-quality contrasts even when the global GRPO advantage of an all-root-fail group is zero.

1.1.2 Novelty Boundaries Relative to Representative Related Work

To clarify CALIPER's substantive advances relative to representative local-sampling, budget-allocation, and credit methods, we added a technical-boundary analysis of related work in this rebuttal.

AuthorsOpenReview ↗

Main Response | Part 03/15

Main Response | Part 03/15

Method/paradigmMain designLinks not jointly addressed for our problemCALIPER's design and relative advantage
Plain GRPO / DAPOUse full trajectories and terminal rewards as primary sampling, comparison, and update units; DAPO further improves trajectory-level optimizationHomogeneous groups have no relative signal; heterogeneous groups cannot identify which intermediate decision causes outcome differencesReuse reached states in failed trajectories, put local comparisons and full exploration under one budget, and confine new evidence to actually changed behavior
ARPO / AEPOUse entropy to choose branch positions or allocate rolloutsEntropy reflects uncertainty but not an intervention's verifiable recovery probability, additional cost, or value relative to a full rolloutEstimate per-compute value of opportunities from strict recovery, uncertainty, and additional token cost, deciding both whether and how many local suffixes to sample
VinePPO / GiGPOResample intermediate prefixes or construct fine-grained credit for same-state actionsProvide a basis for local comparisons but do not jointly resolve local-versus-full compute allocation, tool-state restoration, and post-fork update scopeJointly align comparable local evidence with budget routing, execution state, and token-level scope rather than merely adding intermediate branches
TRACEUses a learned value predictor to allocate tree-structured budgets between roots and prefix continuationsPredictor-guided tree allocation does not directly express a typed intervention's verified recovery-per-token or naturally require joint alignment of complete tool execution state and local update scopeAvoid an additional allocation predictor; directly estimate online typed recovery evidence and cost, grounding samples in restored states and post-fork credit
ELPOUses binary-search-like rollouts to locate the first irrecoverable error and assigns hierarchical credit to surrounding descendantsLocal sampling centers on the first irrecoverable error rather than directly choosing the real-time value of autonomous/feedback-driven interventions relative to full explorationNot limited to one error location; directly selects rethink/finalize/tool-repair from both types of decision opportunities and jointly allocates full/local budget
CBPOEntropy candidate scanning, exact-prefix branching, and CBV-modulated hierarchical creditStrong branching and outcome-contrastive credit, but no unified typed recovery-per-compute, complete tool-state restoration, and adaptive 8/0--6/2--4/4 trade-offUse actual rescue value to decide whether local compute is worthwhile and attribute outcomes through same-execution-state comparisons and signed post-fork LOO credit

The foundational ideas of these methods are all related to CALIPER. CALIPER-specific novelty is that it neither equates "can branch" with "worth branching," nor equates "local outcome differences exist" with "the whole trajectory may be updated"; instead, it aligns compute value, comparison basis, and credit scope simultaneously.

1.1.3 New Experiments: E02--E06 Controls Support the Additional Value of the Core Mechanisms

To clarify the sources of CALIPER's gains and its technical boundaries relative to existing local-sampling and credit-assignment methods, we added five controlled training experiments E02--E06 in this rebuttal, testing auxiliary reward, matched TRACE/CBPO baselines, fixed 6/2, and budget-matched random routing. Their settings, results, and analyses will be included in the revised main text or appendix.

Reviewer_9ojPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CALIPER to address the problems of sparse terminal rewards, underutilization of failed trajectories, and imprecise credit assignment in tool-augmented RL. The method first identifies key decision steps in failed trajectories and uses historical information together with generation costs to adaptively allocate sampling budgets between full-trajectory generation and local regeneration. It then uses same-state local comparisons and localized credit assignment to improve RL training. Finally, CALIPER outperforms the baselines on five benchmarks.

Critical strengths
  1. The motivation and method focus on a key concern within the research community: the underutilization of supervisory signals in traditional trajectory-level RL. The method integrates computational budget allocation, state-consistent local comparisons, and fine-grained credit assignment.

  2. The experiments show consistent improvements, yielding strong performance gains across a range of reasoning and coding benchmarks and at two model scales.

Critical weaknesses
  1. Limited Novelty and Insufficient Comparisons. The core design of CALIPER closely overlaps with recent works such as TRACE, ELPO, and CBPO. The method's contributions mainly involve integrating existing techniques, but the paper does not sufficiently clarify its key technical distinctions or provide experimental comparisons with the related works (TRACE and CBPO).

  2. Insufficient Evidence for the Claimed Improvements in Training Signals and Compute Efficiency. First, when both full trajectories and local resamples fail, strict rewards and local LOO advantages remain zero. Thus, CALIPER cannot resolve the homogeneous group problem in this case. Second, failure-quality rewards are not unique to CALIPER, yet the paper lacks a baseline equipped with the same auxiliary rewards. Finally, the two-stage sampling process introduces additional sequential dependencies, but the paper only reports token-generation efficiency without evaluating actual training throughput or wall-clock efficiency.

Questions for authors
  1. Could the authors explain that what are the key technical innovations of CALIPER compared to TRACE, ELPO, and CBPO? Could the authors provide more direct experimental comparisons?

  2. Could the authors explain that how does the two-stage rollout process affect end-to-end training time and GPU throughput?

24 comments
AuthorsOpenReview ↗

Main Response | Part 01/11

Main Response | Part 01/11

Main Response

In this rebuttal, we have added extensive controlled experiments, ablation studies, and in-depth analyses, and redrawn the paper's main and analytical figures, further improving CALIPER's experimental validation, technical justification, and presentation. We believe this new evidence makes the work more complete, rigorous, and convincing. We sincerely hope you will consider these improvements comprehensively and reconsider the paper's rating. Your recognition would also motivate us to continue conducting more rigorous, interesting, and valuable research in the future.

Thank you for recognizing the importance of our investigation into the underutilization of failed trajectories under sparse terminal rewards, and for appreciating CALIPER's overall approach of combining compute-budget allocation, state-consistent local comparisons, and fine-grained credit assignment, as well as its consistently positive results across reasoning and code benchmarks and the two Qwen3-4B/8B model sizes. We agree that the original manuscript did not sufficiently clarify CALIPER's technical boundaries relative to TRACE, ELPO, and CBPO, and that it characterized computational efficiency only by token counts without fully presenting the end-to-end system cost of two-wave rollouts.

To address your two direct questions, we have added two controlled training experiments with TRACE-matched and Full CBPO-matched baselines and completed an end-to-end efficiency analysis covering full training and strictly paired same-prompt comparisons. The results further clarify the paper's contributions and boundaries: CALIPER does not merely append local branches or auxiliary credit. Instead, under a fixed rollout-slot budget, it unifies opportunity-value assessment, same-execution-state interventions, and credit attribution restricted to changed regions. This loop outperforms both new strong baselines in overall results across five benchmarks. At the same time, two-wave routing introduces a measurable cost from sequential generation. We will therefore accurately describe our efficiency result as better task performance under a fixed sampling budget with transparent, auditable system costs, rather than wall-clock acceleration. Below, we first answer your two Questions for Authors in turn; the new experiments and analyses will be added to the revised main text or appendix.

1. Clarification 1: CALIPER's Joint Loop Constitutes a Substantive Innovation Supported by Direct Comparisons

Your two questions: (1) What exactly is CALIPER's key technical innovation relative to TRACE, ELPO, and CBPO? (2) Can more direct experimental comparisons be provided?

1.1 Brief Summary of Our Response

TRACE, ELPO, and CBPO provide important foundations for intermediate-state branching, error localization, tree-structured budget allocation, and local credit. Building on these foundations, CALIPER addresses a complete online decision problem: how to use already generated failed trajectories as both training samples and opportunity probes, discovering intervention points with explicit behavioral semantics in autonomous decisions and tool-feedback responses; how to dynamically allocate budget between continued full exploration and local validation reusing reached execution states, based on verifiable recovery value and additional generation cost; and how to ensure evidence from local comparisons updates only the post-fork behavior it genuinely supports.

CALIPER's core contribution consists of three end-to-end aligned links:

  1. Typed interventions with explicit behavioral semantics. CALIPER does not indiscriminately resample arbitrary prefixes. Instead, it distinguishes autonomous model decisions from post-tool-feedback responses and constructs executable rethink, finalize, or tool-repair interventions whenever the remaining tool, interaction, and generation budgets permit. It estimates whether applying this particular behavioral change at a failed state is worthwhile, rather than measuring generic prefix uncertainty.
AuthorsOpenReview ↗

Main Response | Part 02/11

** Main Response | Part 02/11**

  1. Online compute decisions between full exploration and local validation. CALIPER combines recent strict recovery evidence, uncertainty, and additional token cost to estimate verified recovery-per-compute of a typed opportunity relative to a fresh full rollout. It selects 8/0, 6/2, or 4/4 for each prompt within a fixed 8-slot budget. A local retry is thus not automatically added branch overhead; it competes directly with new full trajectories for the remaining budget.

  2. Local credit aligned jointly with execution state and update scope. CALIPER restores the same conversation history, tool outputs, and consumed budgets, and resamples only post-fork behavior. It then uses reliability-calibrated signed LOO credit to update only changeable post-fork policy tokens, and strengthens negative updates only for continuations that perform worse in a same-state comparison.

Together, these links form the loop of "opportunity discovery and compute-value assessment -- execution-consistent intervention comparison -- credit-scope alignment." The key is not merely to observe local outcome differences, but to let the sampling decision, comparison basis, and actor update share the same budget and attribution boundary. CALIPER's advance beyond existing work is therefore not stacking modules in parallel, but unifying "where additional compute is worthwhile, whether the observed difference is credible, and where the evidence permits updating" into an executable training procedure.

1.2 Technical Boundaries Relative to TRACE, ELPO, and CBPO

To explain more clearly what CALIPER shares with related methods and where it differs substantively, we have added a method-boundary analysis in this rebuttal. The table below and corresponding discussion will be included in the revised main text or appendix.

MethodMain designPrimary problem addressedCALIPER's advance
TRACEUses a learned value predictor to allocate a tree-structured budget between roots and prefix continuations, together with Tree-GRPO creditGuides tree exploration and budget allocation with predicted valuesInstead of an additional allocation predictor, directly compares local intervention against fresh full rollout using online typed rescue evidence and additional costs; grounds the sampling results in restored execution states and post-fork credit
ELPOUses binary-search-like rollouts to locate the first irrecoverable error and assigns hierarchical credit around corresponding descendantsError localization and hierarchical credit assignmentGoes beyond the first irrecoverable error by discovering both autonomous and feedback-driven opportunities, selecting rethink/finalize/tool-repair, and jointly deciding whether the remaining budget goes to full exploration or local validation
CBPOEntropy candidate scanning, exact-prefix branching, and CBV-modulated hierarchical credit; uses a 4-root/4-branch structure in the matched implementationUncertainty-based branch selection and outcome-contrastive hierarchical creditUses actual rescue evidence, cost, and uncertainty to assess whether a typed opportunity is more worthwhile than full exploration and combines complete tool-state restoration, adaptive 8/0--6/2--4/4 routing, and signed post-fork credit
CALIPERTyped opportunity posterior, adaptive full/local routing, restored-state intervention, and signed local LOO creditConverts failed trajectories into verifiable, attributable decision-level supervision under a fixed budgetCloses compute value, same-state comparability, and token-level update scope within one training semantics

These distinctions explain why CALIPER does not equate "being able to branch" with "being worth branching," or equate "different local outcomes" with "crediting the entire trajectory."

1.3 New Experiments and Analyses

To test more directly whether the above technical distinctions translate into better overall performance, we added two controlled training experiments with TRACE-matched and Full CBPO-matched baselines in this rebuttal. Both use the same Qwen3-8B, training settings, and five-benchmark evaluation protocol as CALIPER. The full configurations, results, and analyses will be included in the revised main text or appendix. ELPO was already included among the original baselines, and the revision will add its accurate citation and clarify its mechanism boundary.

AuthorsOpenReview ↗

Main Response | Part 03/11

Main Response | Part 03/11

Because the new experiments and analyses in this rebuttal were conducted jointly in response to multiple reviewers, experiment identifiers are assigned globally. As a result, the identifiers cited in an individual response may be nonconsecutive, but they all refer to the same set of experiments and analyses.

1.3.1 New Experiment E03: The TRACE-Matched Baseline Is Effective but Remains Below CALIPER Overall

To directly test whether TRACE's tree-structured budgeting and credit mechanisms can explain CALIPER's overall gains, we added E03: TRACE CALIPER-matched Baseline.

Experimental objective. Under fully matched model, data, optimizer steps, rollout budget, and evaluation protocol, test whether TRACE-style learned-predictor-guided tree allocation and Tree-GRPO credit are sufficient to explain CALIPER's overall improvement.

Experimental setup.

SettingE03 configuration
Model and trainingQwen3-8B; same training settings as CALIPER
Sampling budgetFixed 8 terminal slots per prompt
Retained TRACE mechanismsOnline value predictor, cross-prompt root allocation, within-prompt prefix continuation, Tree-GRPO credit, copied-prefix masking
Disabled CALIPER mechanismsTyped adaptive router, bounded auxiliary credit, matched local credit, and ELC
EvaluationSame five benchmarks and pass@8/2/2/4/2 as in the paper

Experimental procedure. Complete TRACE-matched training and evaluation on all five benchmarks, and compare with Plain GRPO and full CALIPER using the same paper-level metrics. This setup retains TRACE's core tree-budgeting and credit semantics while removing CALIPER-specific routing and local-credit pathways, thereby providing a direct controlled comparison.

1.3.2 New Experiment E04: The Full CBPO-Matched Baseline Is Below CALIPER on All Five Benchmarks

To directly test whether CBPO's branching and hierarchical credit can reproduce CALIPER, we added E04: Full CBPO CALIPER-matched Baseline.

Experimental objective. Test whether entropy-guided branching and outcome-contrastive hierarchical credit are already sufficient to reproduce CALIPER's overall improvements.

Experimental setup.

SettingE04 configuration
Model and trainingQwen3-8B; same training settings as CALIPER
Sampling budgetFixed 8 terminal slots per prompt; 4 roots / 4 branches according to CBPO
Retained CBPO mechanismsEntropy candidate scanning, priority/path/node decay and caps, exact-prefix branching, CBV-modulated hierarchical credit, copied-prefix masking, non-overlapping segmentation
Disabled other mechanismsTRACE predictor and CALIPER-specific adaptive routing and typed intervention-credit pathway
EvaluationSame five benchmarks and pass@8/2/2/4/2 as in the paper

Experimental procedure. Complete Full CBPO-matched training and all five evaluations, and compare it with Plain GRPO, TRACE-matched, and full CALIPER under the same budget and metrics. This tests how much of the gain balanced branch exploration and hierarchical contrastive credit can explain, and whether CALIPER's unified loop adds value.

1.3.3 Experimental Results

Plain GRPO and CALIPER use the paper's final aggregate values from multiple experiments; E03 and E04 are newly conducted protocol-matched controlled results.

MethodAIME25 p@8MATH500 p@2GSM8K p@2GPQA p@4LCB p@2Five-benchmark macro-average
Plain GRPO73.3388.6092.6559.6028.0068.44
E03: TRACE-matched70.0089.0093.1864.6528.5769.08
E04: Full CBPO-matched73.3389.6093.5664.6529.7170.17
CALIPER76.6790.0093.6365.1531.4371.38

1.3.4 Analysis of Results

ComparisonRelative to Plain GRPOMacro-average gap to CALIPERExplanation
E03: TRACE-matched+0.64 pp-2.30 ppPredictor-guided tree allocation and Tree-GRPO credit form an effective strong baseline but cannot fully explain CALIPER's overall gains
E04: Full CBPO-matched+1.73 pp-1.21 ppEntropy-guided balanced branching and hierarchical credit explain part of the gain, but CALIPER's unified loop still adds overall value
Reviewer_1PT9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper targets sparse terminal rewards in tool-integrated reasoning (TIR). In this setting GRPO assigns one trajectory-level advantage to every decision, and homogeneous groups give no signal at all. CALIPER spends 4 of 8 rollout slots per prompt on full trajectories. From the failed ones, it extracts two kinds of candidate forks: autonomous forks at sentence, formula, or code-block boundaries, and feedback-driven forks at tool returns. It pairs each fork with an intervention (rethink, finalize, or tool-repair). A Beta-posterior estimate of strict success per generated token then routes the remaining 4 slots either to more full trajectories or to suffixes resampled from a restored execution state, giving an 8/0, 6/2, or 4/4 split. Training combines the global GRPO advantage with three additions: a reliability-weighted leave-one-out local advantage applied only to post-fork tokens, a capped auxiliary failure-quality signal, and a wider lower PPO clip for locally inferior continuations. The authors train Qwen3-8B and Qwen3-4B with LoRA for 130 steps on 8,320 Open-AgentRL examples, using a single Python interpreter tool. They evaluate on AIME25, MATH500, GSM8K, GPQA-Diamond, and LiveCodeBench v6. CALIPER reaches a 71.38 average on Qwen3-8B, compared with 68.44 for GRPO and 70.08 for the strongest baseline (VinePPO). It improves over GRPO on all five benchmarks, and removing any single component lowers the average.

Critical strengths

S1. The paper addresses a real and relevant problem. In long TIR traces, a single terminal reward credits good and bad intermediate steps equally, and all-fail groups (25.8% of GRPO groups in Figure 1a) contribute no gradient. Two core design choices follow directly from this problem. Restoring the full execution state (prior tool outputs and consumed budgets) before resampling makes the local comparison valid. Masking local credit to post-fork tokens keeps it off the shared prefix. Both ideas are sound and reusable beyond this specific method, and they are the main reason I found the paper informative.

S2. The empirical comparison is broad and controlled. The authors compare against GRPO and six other RL baselines covering trajectory-level optimization, adaptive branching, and fine-grained credit assignment. All methods share the same 8-slot rollout budget, with branch samples counted against it. CALIPER improves over GRPO on all five benchmarks and at both model scales, with the largest gains on GPQA (+5.55) and LiveCodeBench (+3.43). The gains point in the same direction across benchmarks, and the 2.94-point average gain over GRPO appears to exceed evaluation noise, although the paper reports a single training run (see Q3).

Critical weaknesses

W1. The presentation obscures the core algorithm. The high-level idea is easy to grasp, but the main text does not make clear how CALIPER selects forks and applies interventions.

  • Section 4 introduces many symbols (the five-part type c(ω), E^U, E^L, S, V, r^probe, r^hist, λ_a, β_aux, β_loc) and defers the concrete routing rules and slot counts to the appendix.
  • The appendix then uses three different confidence multipliers: 1.5 for calibration, 0.5 for selection, and 0.84 for routing.
  • The paper describes each of the three interventions in a single sentence and never shows a concrete fork with its pivot and suffixes.
  • Figure 2 is too small to read, as are the axis labels in Figures 3 to 6. This matters because the contribution is a specific combination of mechanisms. Readers cannot evaluate or reuse that combination without understanding how its parts fit together.

W2. The experiments do not test the setting the paper motivates, and they give limited evidence for how the method helps. On scope, training uses one dataset, one model family, and a short LoRA run, and the 4B experiments compare only against GRPO. The TIR setting is also shallow:

  • a single Python interpreter as the only tool,

  • at most six tool calls per training trajectory,

  • tool outputs truncated to 256 characters,

  • a mean of 3.8 feedback cycles among failed trajectories (Figure 1c). Current agentic workloads involve many heterogeneous tools and tens of calls per trajectory, which is exactly where trajectory-level credit should hurt most. The paper’s own Figure 3a shows gains growing with tool use. The absence of tool-heavy benchmarks therefore leaves the central claim untested in the regime where it matters most. On mechanism, the paper does not show how CALIPER-trained policies behave differently from GRPO-trained ones:

  • It includes no qualitative trajectory comparisons and no measurements of tool-call error rate, duplicate calls, or tool-repair success.

  • The strict rescue rate stays flat at around 11.6% throughout training (Figure 5b), so it is unclear whether the policy improves at the decisions it is trained on.

  • The ablations only toggle components on and off, and two of them fall below the untrained base model: MATH500 drops to 69.00 without matched local credit, and AIME25 drops to 60.00 without scope alignment. This pattern suggests unstable runs rather than clean component contributions.

Questions for authors

Clarifications that could change my assessment:

  1. How do you enforce finalize and tool-repair during generation, for example by masking tool-call tokens? If constrained decoding produces the suffix but the PPO ratio uses unconstrained π_θold probabilities, the suffix tokens come from a different distribution than the ratio assumes. How do you handle this, and does it matter in practice?
  2. Removing matched local credit drops MATH500 to 69.00, about 19 points below the base model, while other benchmarks barely move. Local credit is capped at 12% of the advantage mass with λ_a ≤ 0.5, so why does removing it cost 7.2 average points? In that ablation, what happens to the suffix slots: are they discarded? How is “w/o opportunity prioritization” implemented, and why does the routing ablation use a fixed 4/4 split rather than a fixed 6/2 split or random routing?
  3. Can you report multiple seeds or confidence intervals, plus mean accuracy over samples (avg@k) alongside pass@k? Why does k differ per benchmark? On AIME25, the gain over GRPO is one problem out of 30, and CALIPER ties VinePPO.
  4. What fraction of prompts receives each of the 8/0, 6/2, and 4/4 allocations over the course of training? If most prompts receive 8/0, CALIPER reduces to GRPO for most of training.
  5. Appendix A.4.9 disables auxiliary credit when the global strict signal is zero. In all-fail groups, then, only strict rescues (11.6% of probes) produce local credit. How many all-fail prompts actually receive a nonzero gradient? Figures 4 and 5 count auxiliary contrast before gating, so they seem to overstate the signal that reaches the update.
  6. In Figure 3a, CALIPER underperforms GRPO by 2.6 points on no-tool instances. What causes this? Why is LiveCodeBench excluded from the figure?

Suggestions that would strengthen the paper (believe that these are strongly needed):

  1. Train and evaluate on tool-heavy, multi-tool settings with long horizons, such as τ-bench, BFCL multi-turn, or search and MCP-based agents. Add at least one more model family.
  2. Show side-by-side trajectories from GRPO and CALIPER on the same problems. Report tool-use metrics for both, including tool-call error rate, duplicate-call rate, and tool-repair success rate.
  3. Add an algorithm box and a figure that walks one failed trajectory through the full pipeline: fork extraction, intervention choice (rethink, finalize, tool-repair), suffix resampling, and credit assignment.
  4. Report sensitivity to the main hand-set constants, such as the 12% local-credit cap, ε_l^loc = 0.315, and the routing thresholds.
44 comments
AuthorsOpenReview ↗

Main Response | Part 01/34

Document 06/06 | Main Response | Part 01/34

Main Response

In this rebuttal, we have added extensive controlled experiments, ablation studies, and in-depth analyses, and redrawn the paper's main and analytical figures, further strengthening CALIPER's experimental validation, technical justification, and presentation. We believe this new evidence makes the work more complete, rigorous, and convincing. We sincerely hope the reviewer will consider these improvements comprehensively and reconsider the paper's rating. Your recognition would also give us greater motivation to continue conducting more rigorous, interesting, and valuable research in the future.

We thank the reviewer for recognizing the research problem addressed in this paper and, in particular, for appreciating the two central designs of restoring complete execution states and assigning post-fork local credit. We agree that the original manuscript did not present sufficient evidence on the algorithmic procedure, experimental stability, and implementation details. Below, we respond point by point using new experiments, implementation audits, and statistical analyses. We will also add a complete algorithmic example, parameter explanations, and improved figures to the revision.

Because the new experiments and analyses in this rebuttal were conducted jointly in response to multiple reviewers, experiment identifiers are assigned globally. Identifiers cited in an individual response may therefore be nonconsecutive, but all refer to the same set of experiments and analyses.

1. Clarification 1: finalize/tool-repair Are Environment-Level Structural Constraints and Do Not Create Support Mismatch in GRPO Actor Updates

Your three questions: (1) How are finalize and tool-repair enforced during generation, e.g., are tool-call tokens masked? (2) If constrained decoding generates suffixes but the PPO ratio uses the unrestricted old-policy probabilities, is the suffix behavior distribution inconsistent with the ratio assumptions? (3) How do we handle this issue, and does it have practical effects?

1.1 Brief Conclusions for Clarification 1

We use GRPO in this paper. The term "PPO ratio" specifically refers to the PPO-style clipped likelihood ratio used by the GRPO actor update.

CALIPER does not use token-level constrained decoding. Both finalize and tool-repair first sample an assistant action from the full model distribution, and then the executor constrains whether that action can be executed and the remaining interaction horizon. Thus, our implementation has no support mismatch of the form "restricted behavior-token distribution versus unrestricted GRPO old-policy probability." E18 verifies this using the authoritative code snapshot, full training trajectories, and all 130 training steps.

1.2 New Experiment E18: Intervention Implementation Is Consistent with GRPO Update Accounting

To answer whether structural interventions cause mismatches in token support, masks, or GRPO probability accounting, we added E18: Consistency Verification between Constrained Decoding and GRPO.

1.2.1 Verification Objective and Principles for Canonical-Run Selection

We added E18: Consistency Verification between Constrained Decoding and GRPO Policy Updates in this rebuttal. E18 uses the pre-frozen official CALIPER Qwen3-8B canonical run, with complete code, configurations, and training logs, ensuring an authoritative and traceable audit target.

E18 audits the entire training path of this run rather than a few cases. Since the concern is whether interventions, masks, and GRPO updates are internally consistent, a full implementation audit is more direct than comparing random training outcomes. Specifically, it checks:

  1. Whether finalize and tool-repair constrain token support or the environment's executable actions and remaining horizon;

  2. Which tokens are sampled by the model and enter GRPO actor loss, and which are only deterministic context;

  3. Whether behavior generation, actor recomputation of old log-probabilities, and the PPO-style likelihood ratio in GRPO use the same action support and mask.

AuthorsOpenReview ↗

Main Response | Part 02/34

Document 06/06 | Main Response | Part 02/34

1.2.2 Experimental Setup

E18 uses the authoritative code and related artifacts archived after training was completed.

ArtifactScopePurpose
Complete CALIPER codeFull snapshotDetermine the actual implementation of interventions, masks, and log-probabilities
Official Hydra configurationCanonical Qwen3-8B runDetermine sampling support, temperature, and GRPO actor-update/TIS pathway
Training trajectoriesAll training trajectoriesMeasure actual coverage of copied prefixes, tool observations, and interventions
Canonical training logs130/130 stepsCheck mask alignment, prefix leakage, and behavior/actor probability consistency

1.2.3 Full Audit Across Four Independent Evidence Chains

We examine four independent evidence chains:

  1. Static implementation audit: Trace generation, parsing, execution, and termination of finalize and tool-repair from the archived source code, and check for grammar, token allow-lists, logit bias, logits processors, or predetermined assistant-token injection.

  2. Token-accounting audit: Separately count newly sampled assistant tokens, copied suffix prefixes, tool/environment observations, and padding; check their response_mask, log-probabilities, and policy-loss participation.

  3. GRPO actor-update probability-chain audit: Check the role of vLLM behavior log-probabilities, the actor's recomputation of old log-probabilities for the same input_ids/responses/response_mask/position_ids, and the actual denominator of the PPO-style likelihood ratio.

  4. 130-step empirical audit: Check path/span mask alignment and copied-prefix train-token leakage at each step, and compare token probabilities from behavior generation and actor recomputation.

1.2.4 How finalize and tool-repair Are Actually Executed

InterventionHow the model generatesExecutor/environment handlingChanges token support?
finalizeRestore context from a safe anchor, then normally sample one assistant turn from the full model distributionIf the sampled action still requests a tool, do not execute it and terminate; otherwise check final-answer formattingNo
tool-repairNormally sample an assistant action from the full model distribution after the anchorExecutor permits only one new, non-duplicate tool call; after injecting its observation into context, the next assistant turn is still normally sampled but enters a final-only transitionNo

Thus, finalize does not forcibly insert a final-answer token into the sequence, and tool-repair does not replace incorrect output with a predetermined correction. Both constrain whether an already sampled action can be executed and the remaining interaction horizon, not the sampling vocabulary.

1.2.5 Experimental Results

1. Tokens and loss participation

Token categoryActual coverageMask/log-probabilityGRPO actor-loss participation rate
Hard-forced assistant token0 tokensNone exists0%
Newly sampled assistant token100% sampled by the modelresponse_mask=1, generated log-probabilities retainedMay participate
Copied suffix prefixCovers 18.80% of trajectory branchesresponse_mask=0, log-probability is the 0 sentinel0%
Tool/environment observationCovers 48.76% of trajectory branchesresponse_mask=0, log-probability is the 0 sentinel0%
PaddingVaries with sequence lengthZero after multiplication with attention mask0%

2. Generation support and GRPO actor-update probability pathway

Configuration/computationActual settingMeaning
Training rollout temperature1.0Behavior sampling and actor recomputation use the same temperature
Training rollout top-p / top-k1.0 / -1No truncation of nucleus or top-k support
TIS cap-1TIS disabled; saved rollout log-probabilities are only for diagnosis
PPO-style likelihood ratio in GRPOexp(logp_current-logp_old_recomputed)Denominator is the old-policy probability recomputed by the actor on the same sequence and mask
AuthorsOpenReview ↗

Main Response | Part 03/34

Document 06/06 | Main Response | Part 03/34

3. Empirical consistency across 130 steps

CheckResult
Valid auxiliary rollout/actor probability diagnoses130/130 steps
Complete path/span-mask alignment130/130 steps
Copied-prefix train-token leakage0/130 steps
Pearson correlation of behavior vs. actor token probabilitiesMean 0.99882; minimum 0.99835
Batch mean of absolute differences in token probabilitiesOverall mean 0.00472; P95 0.00528

1.2.6 Analysis of Results

  1. No support mismatch caused by a restricted vocabulary. The generation side has no grammar, allow-list, logit bias, or logits processor. Every trainable assistant token comes from the full model distribution. Therefore, the reviewer's concern about a "restricted behavior policy versus unrestricted GRPO old-policy probability" does not arise in our implementation.

  2. Deterministic context is not mistakenly trained as policy actions. Copied prefixes, tool observations, and padding all have response_mask=0. They preserve restored interaction history and execution state but contribute exactly zero to policy loss. No copied-prefix leakage was found in the 130 training steps.

  3. GRPO's PPO-style likelihood ratio does not use the 0 sentinel or rollout log-probabilities as its denominator. The official configuration disables TIS. Saved vLLM log-probabilities are only for consistency diagnosis; the GRPO actor update uses old log-probabilities recomputed by the actor on the same sequence and the same response_mask=1 support.

  4. Empirical measurements agree with the code audit. vLLM and actor forward are not bitwise identical, but the lowest correlation over the 130 steps is still 0.99835, and the overall mean batch-mean probability difference is 0.00472. This correlation is only an auxiliary diagnostic of the generation engine versus actor forward, not the primary proof of GRPO update correctness. The decisive evidence is that generation does not restrict token support and GRPO uses actor-recomputed old log-probabilities on the same sequence and response_mask=1 support.

1.3 Point-by-Point Responses to the Three Questions in Clarification 1

1.3.1 finalize and tool-repair Are Enforced as Environment-Level Structural Constraints

They do not mask tool-call tokens or forcibly insert any assistant tokens. Both normally sample an assistant action from the full model distribution, then the executor constrains whether it may be executed and the remaining interaction horizon. finalize stops executing newly sampled tool calls. tool-repair executes at most one new, non-duplicate tool call and enters a final-only transition after injecting its observation. The key evidence is zero hard-forced assistant tokens and that all newly generated assistant tokens originate from the full model distribution.

1.3.2 Behavior Generation and Actor Recomputation Share the Same Token Support and Mask

No such inconsistency exists. The reviewer's concern presupposes suffixes sampled from a restricted token distribution, but CALIPER's behavior side uses no grammar, token allow-list, logit bias, or logits processor and does not alter token support. All newly sampled assistant tokens have response_mask=1. GRPO's PPO-style likelihood ratio uses actor-recomputed old log-probabilities on the same sequence and action mask. Thus, behavior generation and GRPO actor updating use the same action support.

1.3.3 Full Audit Finds No Support or Probability-Accounting Mismatch

If tokens generated from a restricted distribution were indeed treated as samples from an unrestricted old policy and entered directly into likelihood-ratio calculations, the importance ratio could in principle cease to correspond to the real behavior policy and lead to biased updates. CALIPER avoids this at the implementation level: copied prefixes, tool observations, and padding are deterministic context with response_mask=0 and do not enter policy loss. Only newly sampled assistant tokens enter the update as policy actions. E18 further verifies mask alignment across the entire official run, zero copied-prefix training leakage, and the use of old-policy probabilities recomputed on the same sequence and mask in GRPO. Thus, this theoretical risk warrants checking but did not occur during CALIPER's actual training and did not introduce the support/normalization bias of concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.