- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CALIPER, a method for improving credit assignment in tool-augmented agentic RL under a fixed rollout budget. It adaptively allocates samples between complete trajectories and local interventions at decision points in failed trajectories. Local continuations preserve the same preceding execution state, and their comparisons provide credit restricted to post-fork behavior. Experiments cover five benchmarks with Qwen3-8B and Qwen3-4B. CALIPER achieves the highest average score among the evaluated methods with Qwen3-8B, while component ablations support the proposed design.
Critical strengths
The paper addresses an important problem with a reasonable approach. Sparse terminal rewards provide limited guidance about which decisions in an agent trajectory deserve credit. Combining local comparisons with compute-aware sampling offers a practical way to obtain more useful supervision from a limited rollout budget. This is particularly relevant when trajectories involve multiple tool interactions.
The main results and ablations provide encouraging evidence. CALIPER improves over Plain GRPO across all five benchmarks with Qwen3-8B, raising the average score from 68.44 to 71.38. The Qwen3-4B results show a similar overall improvement. Removing individual components consistently lowers average performance, supporting the contribution of the different design choices.
Critical weaknesses
The presentation and allocation of space weaken the paper’s focus. Several figures have labels that are difficult to read, especially Figures 5 and 6. References are also missing where they are needed, including the discussion of prior approaches in the second introduction paragraph and several baselines in the experimental setup. More broadly, the method combines many mechanisms, making its central contribution harder to identify. The lengthy method description leaves limited room for deeper experimental analysis. Narrowing the scope and moving secondary implementation details to the appendix would improve the paper.
The robustness of the method to its many hyperparameters remains unclear. Important choices include the initial full-rollout count (B_0=4), the 32-step history window, the efficiency margin of 1.10, and the contrast threshold of 0.40. The paper specifies these values but does not report sensitivity analyses. The held-out calibration also fixes (B_0=4), so it does not establish the suitability of the initial allocation. The adaptive-routing ablation compares against fixed 4/4 sampling, but does not show whether the advantage holds against other fixed allocations. These gaps limit my confidence that the reported gains are stable across reasonable settings.
Questions for authors
Could you clarify which mechanisms are the core contributions and provide the missing references for the prior-work discussion and baselines?
How was (B_0=4) chosen? Was it compared with other initial allocations or a fixed 6/2 baseline?
Do you have sensitivity results for the history window, efficiency margin, and contrast threshold that would establish the robustness of the reported gains?
26 comments
Main Response | Part 02/15
** Main Response | Part 02/15**
1.1.1 CALIPER's Three Core Innovations
-
From generic prefix branching to typed interventions with explicit behavioral semantics. CALIPER does not indiscriminately retry intermediate prefixes. It distinguishes autonomous model decisions from responses to tool feedback and constructs feasible
rethink,finalize, ortool-repairinterventions according to the remaining tool, interaction, and generation budgets. Thus, it asks not merely "should this prefix be retried?" but "is a particular executable behavioral change worthwhile at this failed state?" -
From selecting a branch location to online compute decisions between full exploration and local validation. CALIPER does not select branches based only on entropy or predicted value. It combines recent strict rescue rates, uncertainty, and additional generation costs to estimate the verified success-per-compute of each typed opportunity relative to a full trajectory, then adaptively allocates the budget among 8/0, 6/2, and 4/4. Local retries therefore compete directly with fresh full rollouts under the same fixed budget instead of being automatically added branch overhead.
-
From observing local outcome differences to learning only behavior supported by those differences. CALIPER restores the same conversation history, tool outputs, and consumed budgets for each selected fork and resamples only post-fork behavior. It then uses reliability-calibrated signed LOO credit to update only changeable post-fork policy tokens, and strengthens negative updates only for locally inferior continuations in same-state comparisons. Local sampling, outcome comparison, and token-level actor updating thus share one attribution boundary.
These are not arbitrary modules placed side by side, but an end-to-end aligned evidence chain: opportunity valuation determines where compute is spent, execution-consistent interventions determine whether the resulting differences are comparable, and scope-aligned credit determines what the evidence is allowed to update. This also explains how CALIPER can recover nonzero and spatially localized training signals from local strict rescues or bounded failure-quality contrasts even when the global GRPO advantage of an all-root-fail group is zero.
1.1.2 Novelty Boundaries Relative to Representative Related Work
To clarify CALIPER's substantive advances relative to representative local-sampling, budget-allocation, and credit methods, we added a technical-boundary analysis of related work in this rebuttal.
Main Response | Part 03/15
Main Response | Part 03/15
| Method/paradigm | Main design | Links not jointly addressed for our problem | CALIPER's design and relative advantage |
|---|---|---|---|
| Plain GRPO / DAPO | Use full trajectories and terminal rewards as primary sampling, comparison, and update units; DAPO further improves trajectory-level optimization | Homogeneous groups have no relative signal; heterogeneous groups cannot identify which intermediate decision causes outcome differences | Reuse reached states in failed trajectories, put local comparisons and full exploration under one budget, and confine new evidence to actually changed behavior |
| ARPO / AEPO | Use entropy to choose branch positions or allocate rollouts | Entropy reflects uncertainty but not an intervention's verifiable recovery probability, additional cost, or value relative to a full rollout | Estimate per-compute value of opportunities from strict recovery, uncertainty, and additional token cost, deciding both whether and how many local suffixes to sample |
| VinePPO / GiGPO | Resample intermediate prefixes or construct fine-grained credit for same-state actions | Provide a basis for local comparisons but do not jointly resolve local-versus-full compute allocation, tool-state restoration, and post-fork update scope | Jointly align comparable local evidence with budget routing, execution state, and token-level scope rather than merely adding intermediate branches |
| TRACE | Uses a learned value predictor to allocate tree-structured budgets between roots and prefix continuations | Predictor-guided tree allocation does not directly express a typed intervention's verified recovery-per-token or naturally require joint alignment of complete tool execution state and local update scope | Avoid an additional allocation predictor; directly estimate online typed recovery evidence and cost, grounding samples in restored states and post-fork credit |
| ELPO | Uses binary-search-like rollouts to locate the first irrecoverable error and assigns hierarchical credit to surrounding descendants | Local sampling centers on the first irrecoverable error rather than directly choosing the real-time value of autonomous/feedback-driven interventions relative to full exploration | Not limited to one error location; directly selects rethink/finalize/tool-repair from both types of decision opportunities and jointly allocates full/local budget |
| CBPO | Entropy candidate scanning, exact-prefix branching, and CBV-modulated hierarchical credit | Strong branching and outcome-contrastive credit, but no unified typed recovery-per-compute, complete tool-state restoration, and adaptive 8/0--6/2--4/4 trade-off | Use actual rescue value to decide whether local compute is worthwhile and attribute outcomes through same-execution-state comparisons and signed post-fork LOO credit |
The foundational ideas of these methods are all related to CALIPER. CALIPER-specific novelty is that it neither equates "can branch" with "worth branching," nor equates "local outcome differences exist" with "the whole trajectory may be updated"; instead, it aligns compute value, comparison basis, and credit scope simultaneously.
1.1.3 New Experiments: E02--E06 Controls Support the Additional Value of the Core Mechanisms
To clarify the sources of CALIPER's gains and its technical boundaries relative to existing local-sampling and credit-assignment methods, we added five controlled training experiments E02--E06 in this rebuttal, testing auxiliary reward, matched TRACE/CBPO baselines, fixed 6/2, and budget-matched random routing. Their settings, results, and analyses will be included in the revised main text or appendix.
Main Response | Part 01/15
Main Response | Part 01/15
Main Response
In this rebuttal, we have added extensive controlled experiments, ablation studies, and in-depth analyses and redrawn the paper's main and analytical figures, further improving CALIPER's experimental validation, technical justification, and presentation. We believe the new evidence makes the work more complete, rigorous, and convincing. We sincerely hope the reviewer will consider these improvements comprehensively and reconsider the paper's rating. Your recognition would also give us greater motivation to continue conducting more rigorous, interesting, and valuable research in the future.
Thank you for recognizing the problem of credit assignment in tool-use trajectories under sparse terminal rewards and for appreciating CALIPER's consistently positive results across five Qwen3-8B benchmarks, Qwen3-4B, and component ablations. We agree that, although the original manuscript included a complete implementation, it did not clearly distinguish core contributions from secondary mechanisms, nor did it provide sufficiently direct robustness evidence for fixed allocations and routing parameters. In response, we recast CALIPER as a loop of "opportunity selection -- same-state intervention -- credit attribution to changed regions," adding a fixed 6/2 training control, actual allocation-distribution statistics, and routing-sensitivity analysis, and completing the prior-work and baseline citations. We first answer your three sets of Questions for Authors in turn. The paper's focus, figure readability, and appendix restructuring are addressed together under the corresponding weaknesses.
Because the new experiments and analyses were jointly conducted in response to multiple reviewers, experiment identifiers are assigned globally. The identifiers cited in an individual response may therefore be nonconsecutive, but all refer to the same set of experiments and analyses.
1. Clarification 1: CALIPER's Three-Link Loop Is the Core Contribution, and Related-Work and Baseline Citations Have Been Completed
1.1 Brief Summary of the Conclusions for Clarification 1
Prior work on intermediate-state branching, suffix resampling, and local credit provides important foundations for fine-grained agentic RL. CALIPER advances the problem beyond simply constructing local branches and local credit: it uses existing full trajectories as online probes to discover and validate local opportunities. Specifically, CALIPER addresses three interrelated questions in one framework: (1) how to discover intervenable decision points in autonomous decisions and tool-feedback responses, and dynamically allocate the remaining budget between further full-trajectory exploration and local validation that reuses reached states, according to recovery value and generation cost; (2) how to ensure that local outcome differences genuinely result from behavioral changes after the same tool execution state; and (3) how to update only the tokens genuinely supported by that evidence. CALIPER's central technical innovation is to unify these three links under a fixed rollout budget into a complete loop of "compute-value assessment -- execution-consistent comparison -- credit-scope alignment."