acceptodds
Under review as a conference paper at ICLR 2027

CIPO: Counterfactual Imagination Policy Optimization for Adaptive Tool Granularity Selection

Abstract

Large language model (LLM) agents solve complex tasks through multi-step interactions with external tools. These interactions often contain recurring local tool sequences. Treating such sequences as composite "Skills" can shorten tool-use trajectories and reduce repeated low-level decisions. However, when atomic tools and composite skills coexist, skill use becomes a policy problem: the agent must decide whether the current state requires atomic fine control or skill-level abstraction. In this paper, we argue that effective skill use should be studied as adaptive tool granularity selection. The most direct training signal for this problem is to compare the consequences of atomic and skill choices available from the same state. Based on this view, we propose CIPO, a Counterfactual Imagination Policy Optimization framework for adaptive tool granularity. CIPO constructs executable skills through budget-constrained mining of successful tool-use trajectories and trains granularity decisions with counterfactual branch rollouts. For each base rollout, CIPO branches at the first eligible granularity decision and replaces the chosen action with a feasible atomic or skill alternative. The paired outcome difference serves as a supplementary reward for policy optimization. Experiments across multiple benchmarks and model backbones show that CIPO improves task success and decision efficiency over baselines. Further analyses show that CIPO learns effective skill use by improving the choice between atomic tools and composite skills based on the current state, without simply increasing skill frequency.

Primary area:
reinforcement learning
Keywords:
Language Agents, Tool-Use, Reinforcement Learning, Skill Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mhbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A BPE-like approach is used to synthesize a dataset from successful tool execution trajectories. CIPO is then employed to construct two trajectories and amplify the reward difference between them, enabling the model to better adaptively select between skills and atomic actions to address the adaptive tool granularity selection problem.

Critical strengths

1.The choice between atomic tools and skills is clearly modeled as an adaptive tool granularity selection problem. And a pipeline was constructed from data synthesis to the completion of training(CIPO).

2.The average TSR of CIPO increased from 29.30% of GRPO + Skills to 36.48%, and the average Tool F1 rose from 41.20 to 48.23. After removing the counterfactual bonus, the average TSR dropped to 31.65%, indicating that this reward component has a real contribution.

Critical weaknesses

1.The source of the success trajectory is not specified as being from a strong model or from the model itself. No information was provided regarding the specific success screening criteria adopted or the number of successful trajectories in each dataset.

2.The paper's training rewards and some evaluation metrics rely on reference tool trajectories or an LLM judge. Additionally, the paper lacks evaluations of actual inference latency and computational cost.

Questions for authors

1.Provide more details on the construction of successful trajectories.

2.Provide execution-based evaluations beyond TOOLATHLON and report actual inference latency, token consumption, and computational cost.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mhbc (1/2)

We sincerely thank the reviewer for the constructive feedback and for recognizing CIPO's adaptive granularity formulation and counterfactual optimization design. We appreciate the concerns regarding data provenance, evaluation reliability, and computational efficiency, and address them below.


W1 & Q1. What are the sources, success criteria, and quantities of successful trajectories?

CIPO constructs its skill library from the publicly released trajectories of the three original benchmarks, without collecting additional trajectories from a separate teacher model.

Specifically, we use TOOLATHLON (https://arxiv.org/abs/2510.25726), TOUCAN (https://arxiv.org/abs/2510.01179), and TRAJECT-Bench (https://arxiv.org/abs/2510.04550). All three processed datasets contain a binary success field, and we retain trajectories satisfying success == 1. Importantly, the underlying success criteria differ across benchmarks:

BenchmarkSuccess CriterionRetained / TotalAvg. Tool CallsProcessed Tool Entries
TOOLATHLONTask completed and official evaluator passed1,153 / 5,62813.51,250
TOUCANAnnotated target tools matched501 / 2,0003.215,294
TRAJECT-BenchAll tool calls executed successfully811 / 1,6706.4715
Total—2,465 / 9,298——

The success labels follow dataset-specific criteria: TOOLATHLON requires both task completion and a positive result from its official evaluator (task_status.running == 'done' and task_status.evaluation == True); TRAJECT-Bench requires all tool calls in the trajectory to execute successfully (num_tools_used == num_successful_tools); and TOUCAN requires the selected tools to match the annotated target tools. These criteria capture different aspects of tool-use correctness and are used to select positive trajectories for skill discovery within each dataset.

As detailed in Section 3.2 and Appendix A, we subsequently exclude sequences shorter than two tool calls and abstract task-specific argument values to identify reusable tool patterns. Budget-constrained BPE then extracts a compact set of candidate skills.

These trajectories provide reusable tool sequences, while CIPO independently learns when invoking a composite skill is preferable to retaining atomic control.

We have expanded Appendix K to clarify these dataset-specific screening criteria and trajectory statistics.

AuthorsOpenReview ↗

Response to Reviewer mhbc (2/2)

W2 & Q2. Can CIPO demonstrate execution-based improvements beyond TOOLATHLON, and what are its actual inference and training costs?


1. Evaluation reliability and existing execution-based evidence.

We acknowledge that CIPO's training reward uses reference-tool coverage (Appendix E). However, the evaluation also examines task completion beyond reference-based reward optimization.

EvaluationGRPO + SkillsCIPOImprovement
Strict yes-only success (%)21.4027.68+6.28 pp
TOOLATHLON execution success (%)17.6725.17+7.50 pp

The strict evaluation excludes partial completion but still uses an LLM judge. More importantly, TOOLATHLON's official evaluator verifies the final environment state independently of reference-trajectory matching and LLM judgments, showing a 7.50-percentage-point improvement for CIPO.


2. Additional evaluation on TRAJECT-Bench and TOUCAN.

For consistent cross-benchmark comparison, our main experiments adopt a unified TSR protocol. We additionally evaluate TRAJECT-Bench using its native Solution Accuracy metric, which judges final-answer correctness against ground-truth solutions, rather than including partial trajectory completion.

For TOUCAN, no equivalent official task-state evaluator is available. We therefore supplement TSR with Reference-Chain Exact Match, comparing the expanded atomic tool sequence against the annotated target sequence after tool-name normalization.

Both additional comparisons use Qwen2.5-7B and the same 100 evaluation episodes per benchmark.

Additional Benchmark-Specific Evaluation :

Benchmark / MetricGRPO + SkillsCIPO
TRAJECT-Bench / Solution Accuracy (%) ↑19.0025.00
TOUCAN / Reference-Chain Exact Match (%) ↑14.0021.00

These metrics complement the main TSR results by examining final-answer accuracy and strict reference-tool agreement, respectively. Meanwhile, TOOLATHLON's environment-based evaluation provides independent evidence of actual task completion without relying on reference-trajectory matching or LLM judgments.


3. Computational cost and inference efficiency.

We further compare training overhead and inference efficiency under matched hardware and evaluation settings.

Additional Efficiency Analysis:

MethodTokens/Episode ↓LLM Latency (s) ↓End-to-End (s) ↓RL GPU-h ↓
GRPO + Skills3,41018.633.484
CIPO3,15017.131.790

The comparison corresponds to 7.6% fewer generated tokens, 8.1% lower LLM latency, and 5.1% lower end-to-end latency, with 7.1% additional RL training GPU-hours.

Crucially, counterfactual branching is used only during training. At inference, CIPO directly selects atomic tools or composite skills without generating alternative continuations. Thus, the matched counterfactual mechanism does not introduce additional branch-generation overhead at deployment.

Overall, CIPO's improvements are supported by both strict task evaluation and independently verified execution outcomes on TOOLATHLON. The additional benchmark-specific metrics provide complementary evidence of solution quality, while the computational analysis characterizes the trade-off between training overhead and inference efficiency.


Thank you again for your constructive comments, which have helped us improve the clarity and completeness of our work. We hope our detailed responses and supplementary analyses have resolved your concerns, and we would greatly appreciate your reconsideration of the paper's rating.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal Responses

Dear Reviewer mhbc,

We sincerely thank you again for your careful review and constructive feedback. We understand that this is a busy period, and we would be very grateful if you could take a moment to review our rebuttal responses.

We have provided additional analyses and clarifications addressing your main concerns, including the sources and screening criteria of successful trajectories, evaluation reliability beyond the main LLM-judged TSR, and computational costs and inference efficiency. We have also incorporated the corresponding clarifications and analyses into the revised manuscript, with changes highlighted in blue.

As the discussion period draws to a close, we would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be more than happy to address them.

Thank you again for your time and thoughtful consideration.

Best regards,
The Authors

Reviewer_PWnJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies tool-augmented LLM agents under an action space where atomic tools and composite skills coexist. The authors formulate tool use as an adaptive granularity selection problem: deciding per state whether atomic control or macro-skill abstraction is preferable. To address this, the authors propose CIPO (Counterfactual Imagination Policy Optimization). CIPO mines recurring tool subsequences via budget-constrained BPE merging and instantiates them with parameter piping and soft interruption. During policy training, it samples paired counterfactual branch rollouts at the first eligible decision point by swapping atomic and skill actions from the identical prefix, using the clipped outcome difference as a supplementary signal for GRPO. Evaluated on TOOLATHLON, TOUCAN, and TRAJECT-Bench across Qwen2.5-7B and Llama3.1-8B, CIPO achieves superior tool trajectory alignment, higher task success rates, and about 15.5% decision reduction.

Critical strengths

Viewing skill invocation as an adaptive granularity policy problem rather than static macro execution addresses a genuine gap in agent learning. Grounding training on matched counterfactual branches provides a direct, low-variance credit assignment signal without increasing inference latency. Experiments cover three diverse benchmarks with varying trajectory lengths and tool pool sizes across two open backbones. The authors thoroughly evaluate both LLM-judge-based TSR and strict execution-based environment verification, complemented by informative ablation studies on branch selection and mining budgets.

Critical weaknesses

Branching exclusively at the first eligible step restricts counterfactual interventions to the early part of an episode. In long-horizon tasks where cascading tool dependencies occur late in the interaction, this heuristic may under-explore granularity trade-offs in deeper decision states. Mining relies purely on offline successful trajectories. When target domains lack abundant gold traces or exhibit low initial success rates, skill discovery and downstream instantiation may suffer from coverage bottlenecks.

Questions for authors

Did you observe significant performance degradation or sample inefficiency when the first eligible granularity decision was relatively trivial compared to later, high-stakes branch points in long-horizon episodes? How easily can CIPO's offline BPE mining pipeline be adapted into an online or iterative discovery setting where the policy autonomously uncovers new skills during RL exploration?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PWnJ (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of CIPO's motivation, counterfactual training design, and empirical effectiveness. Below, we address your concerns regarding branch-point selection and offline skill discovery, supported by methodological clarifications and additional experimental analyses.


Q1. Does first-eligible branching overlook important granularity decisions later in an episode?

First-eligible branching is designed to balance training efficiency and performance, rather than assuming that early decisions are always more important.

Branching mechanism. As described in Section 3.4, an eligible decision is a state where the selected action has a feasible alternative at the other granularity. For an atomic action, CIPO considers an executable skill containing that tool; for a skill, it considers its first internal atomic tool. CIPO branches at the first such state, preserving the preceding history and comparing two continuations under the same policy. Thus, the first eligible decision is not necessarily the first tool call.

Existing ablations. Appendix H (Tables 4–5) compares CIPO-first with three alternatives: CIPO-random randomly selects an eligible decision, CIPO-late selects a later eligible decision, and CIPO-all branches at every eligible decision with additional computation.

StrategyTool F1 ↑TSR (%) ↑Comp./Raw ↓
CIPO-late45.7634.100.861
CIPO-random46.7234.900.858
CIPO-all47.6035.000.850
CIPO-first48.2336.480.845

CIPO-first achieves the best average performance. Even CIPO-all, which requires additional rollouts, does not outperform it, supporting first-eligible branching as an effective default under a limited training budget.

Additional late-critical analysis. To directly examine cases where later decisions are more consequential, we consider TOOLATHLON with Qwen2.5-7B. Using a fixed SFT policy, we identify a task subset satisfying: (1) at least two eligible decisions; (2) a small counterfactual outcome difference at the first eligible decision (|Δ_first| ≤ 0.10); and (3) a larger difference at a decision in the second half of the trajectory (|Δ_late| ≥ 0.20). Here, Δ denotes the base-reward difference between matched atomic and skill continuations. The subset is fixed before evaluating the trained policies.

We additionally introduce CIPO-first+late, which branches at the first eligible decision in one base rollout and a later eligible decision in the other. Each base rollout produces at most one counterfactual branch, preserving the same budget of G=4.

Late-Critical Task Analysis :

StrategyOverall TSR (%) ↑Late-Critical TSR (%) ↑
CIPO-late28.5026.43
CIPO-random28.9625.00
CIPO-first30.7826.43
CIPO-first+late30.5227.86

Overall TSR is evaluated on the original 100 TOOLATHLON tasks; Late-Critical TSR uses only the independently identified subset, under the same evaluation protocol. The late-critical subset contains 28 tasks selected from the original 100 TOOLATHLON evaluation episodes using a fixed SFT policy.

This comparison illustrates the trade-off we aim to examine: first-eligible branching provides strong overall performance, while combining early and later branches can better cover tasks where critical granularity choices arise late in the trajectory, without increasing the number of rollouts.

Overall, first-eligible branching provides a strong balance between performance and training efficiency, rather than imposing a fundamental restriction on CIPO. The counterfactual mechanism can readily incorporate later decisions when the task structure calls for them.

AuthorsOpenReview ↗

Response to Reviewer PWnJ (2/2)

Q2. Does offline skill mining depend on abundant successful trajectories, and can CIPO support iterative skill discovery?

CIPO uses successful trajectories to discover reusable skills, but does not require manually designed workflows or a complete skill library. As described in Section 3.2 and Appendix A, our BPE-style mining represents each tool call by its name and parameter-key signature, removing task-specific values so that recurring tool sequences can be identified and reused across different tasks. We further constrain mining with a maximum of 50 merges, a minimum pair frequency of 3, and a maximum skill length of 6.

Importantly, CIPO does not depend on complete skill coverage to execute tasks. Atomic tools remain available throughout training and inference. Even when a particular tool sequence has not been discovered as a composite skill, the agent can still execute its constituent atomic operations step by step. Counterfactual branching is performed only when a feasible alternative granularity exists; otherwise, ordinary rollouts are used. Thus, limited successful trajectories primarily reduce the opportunities for skill-based acceleration and granularity optimization, rather than preventing the agent from attempting tasks through atomic tools.

Additional Analysis: Sensitivity to Successful-Trajectory Availability. We focus on TOOLATHLON with Qwen2.5-7B, varying the proportion of successful trajectories available for skill mining while keeping downstream training data and RL budgets unchanged. We additionally consider an iterative variant that starts with 25% and periodically updates its skill library using newly successful RL rollouts, without introducing additional training tasks or rollout samples. All variants are evaluated on the same 100 episodes, with one LLM-judge assessment per episode.

Additional Skill-Mining Analysis :

Skill-Mining SettingTool F1 ↑TSR (%) ↑Comp./Raw ↓
25% successful trajectories32.8627.000.901
50% successful trajectories33.9429.000.879
100% successful trajectories35.1731.000.855
25% + iterative discovery34.5831.000.868

The comparison illustrates how reduced initial skill coverage can affect both trajectory quality and decision compression. The iterative variant is designed to test whether newly successful interactions can restore some of this performance without increasing the rollout budget.

Extension to iterative skill discovery. CIPO's modular pipeline allows newly successful trajectories to be periodically mined, validated, and incorporated as executable skills for subsequent granularity training, without changing the counterfactual optimization objective.

Overall, successful-trajectory availability determines the coverage of reusable skills, not whether the agent can execute the underlying atomic operations. CIPO therefore combines a stable offline skill library with the flexibility to expand it through newly accumulated successful experience.

Reviewer_gcHvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper solves the challenge of adaptive tool granularity selection for LLM agents. In this problem, an agent must dynamically decide when to use fine-grained atomic tool or a set of tools (skills). To solve this, the authors propose CIPO a framework that first constructs a compact library of executable, trace-aware skills from successful trajectories, with the insight from BPE. During GRPO extended RL stage, CIPO generates counterfactual branch rollouts and calculates the difference as a reward. Experiments demonstrate that CIPO consistently outperforms existing memory, workflow, and skill-learning baselines in both task success rate and Tool F1, while reducing policy-level decision steps.

Critical strengths
  • This paper address an important problem: tool selection in this field. Ideally, agents should target at the most efficient and effective tool choices on different tasks. But such diversity often poses challenges on existing LLM agents. The authors propose to use BPE inspired RL methods to adaptively choose tools, which is interesting.

  • The RL design is clean and the engineering is good. The RL problem is well-defined and solved. Experiments well support the method design. The skill engineering is appreciated from a BPE inspired method.

Critical weaknesses
  • Motivation: The benefits behind tool granularity are not well stated. Basically, if a model is strong enough, tool selection is not a problem as it should organize tools on the roll well. A good granularity can only save steps. So there pose the following problems:
  1. Is course tool selection reduces task difficulty? This is the probably the only explanation for the benefits.
  2. Is the improvement comes from more reasoning, instead of reduced task difficulty? More tools may introduce more reasoning, which is not reflected in the paper. And such increased reason explains the performance gain.
  3. Will larger or stronger base models reduce the benefits of tool granularity selection?
  • Experiments: Basically for RL, which did increase the performance of models but introduced extra training resources (compute and data). I am concerned about the fairness of the experiments.
  • Methods: The reward design involves task success. Therefore, the improvement may comes directly from increased task success instead of tool selection. It does mean tool granularity choices are useless but the contributions are unknown.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gcHv (1/2)

We thank the reviewer for recognizing the importance of adaptive tool granularity and the clean design of CIPO. We address the central concerns about why tool granularity improves task success and whether the improvements come from task rewards rather than granularity learning.


Q1. Why does adaptive tool granularity improve task success beyond reducing decision steps?

Adaptive tool granularity improves task success by reducing unnecessary decision-making and the associated risk of errors, rather than merely shortening trajectories. Even capable agents must repeatedly select tools, maintain execution order, and handle parameter dependencies throughout long trajectories. Encapsulating recurring procedures into executable skills reduces these repeated decisions while preserving the underlying atomic operations. However, skills may also hide useful intermediate observations when fine-grained control is needed. The key advantage of CIPO is therefore not simply using more skills, but learning when to invoke a skill and when to retain atomic control.

This distinction is supported by our pilot analysis in Figure 1(d): skills perform better in 37% of examined states, atomic tools in 21%, and both are comparable in 42%. These findings show that the preferred granularity depends on the current state, motivating explicit granularity selection rather than a fixed preference for composite actions.

Our experimental results further support this motivation (Table 2):

MethodTool F1 ↑TSR (%) ↑Comp./Raw ↓
Standard GRPO38.9226.741.000
GRPO + Skills41.2029.300.912
CIPO48.2336.480.845

While GRPO + Skills already has access to the same atomic tools and composite skills, CIPO further improves Tool F1 by 7.03 points and TSR by 7.18 percentage points, while achieving stronger decision compression. This shows that providing composite skills alone is insufficient; learning when to use them is equally important.

In essence, CIPO does not simplify the underlying task. It simplifies the agent's decision process by selecting the appropriate action granularity for each state, improving both task completion and decision efficiency.


Q2. If the reward depends on task success, how can we attribute the gains to granularity learning rather than stronger outcome-based optimization?

The key distinction is not where the reward comes from, but how it guides the policy. Both standard GRPO and CIPO use task outcomes as the primary training signal. However, standard GRPO evaluates complete trajectories without explicitly distinguishing whether an atomic tool or a composite skill was preferable at a particular decision state. CIPO converts the same outcome feedback into a state-matched comparison between alternative action granularities.

Specifically, as described in Section 3.4, CIPO constructs two rollouts sharing the same prefix, replacing only one eligible atomic-versus-skill decision before continuing execution. Both rollouts are evaluated using the same base reward. Their outcome difference is then scaled, clipped, and applied symmetrically: the better branch is reinforced relative to its counterpart. This does not introduce a new task-success objective or increase the pair's total reward before non-negativity clipping; instead, it strengthens the relative training signal associated with the granularity choice.

We further isolate this contribution through controlled ablations (Appendix H, Table 4):

MethodF1 ↑TSR (%) ↑
GRPO + Skills41.2029.30
w/o CF Reward43.2531.65
CIPO48.2336.48

The w/o CF Reward variant retains counterfactual branches and the original task reward but removes the paired reward adjustment. Its TSR drops by 4.83 percentage points compared with full CIPO, while F1 decreases by 4.98 points. This demonstrates that the improvement cannot be explained by task-level rewards or additional branch trajectories alone.

In essence, task success defines which granularity choice is beneficial, while CIPO provides the comparative supervision needed to learn that choice. Our contribution is therefore not a stronger task-success reward, but a more targeted way to train the policy to select between atomic tools and composite skills.

AuthorsOpenReview ↗

Response to Reviewer gcHv (2/2)

Q3. Is the experimental comparison fair, given the additional training data and computational costs introduced by RL?

Our ablation studies (Section 4.4, Table 2) distinguish the gains from conventional RL and skill augmentation from those introduced by CIPO's counterfactual granularity optimization. While the main table compares different tool-use paradigms under a unified evaluation protocol, GRPO + Skills serves as our primary controlled baseline, ensuring that CIPO's improvement is not simply due to additional RL training or access to executable skills.

As detailed in Section 3.4 (Eq. 3) and Appendices C, I, and O, both methods use the same training data, skill library, skill-enabled SFT initialization, action space, base reward, and RL hyperparameters. Crucially, both generate exactly four rollouts per training instruction (G=4G=4). GRPO + Skills samples four independent rollouts, whereas CIPO constructs two base rollouts and up to two matched counterfactual branches, filling unused slots with ordinary rollouts. Thus, CIPO introduces no additional training instructions or rollout samples, but uses the same sampling budget to provide more targeted granularity supervision.

Under this controlled setting, CIPO improves Tool F1 from 41.20 to 48.23 and TSR from 29.30% to 36.48% (Table 2), demonstrating clear gains beyond conventional skill-enabled GRPO.

We further quantify the computational overhead and inference efficiency under identical hardware and evaluation settings.

Additional Efficiency Analysis:

MethodTokens/Episode ↓LLM Latency (s) ↓End-to-End (s) ↓Training GPU-h ↓
GRPO + Skills3,41018.633.484
CIPO3,15017.131.790

The comparison indicates 7.6% fewer generated tokens, 8.1% lower LLM latency, and 5.1% lower end-to-end latency, with a 7.1% increase in training GPU-hours. This additional training cost is associated with constructing matched counterfactual continuations, while counterfactual branching is entirely disabled at inference.

Overall, CIPO achieves substantially higher task success without additional training instructions or rollout samples, and its modest training overhead is accompanied by improved inference efficiency. The performance gains therefore cannot be attributed simply to more RL training data or a larger rollout budget; they reflect the benefit of using the available training samples to learn more effective granularity decisions.


Q4. Would stronger backbone models reduce the benefits of adaptive granularity selection?

Adaptive granularity selection remains valuable for stronger models because choosing the correct tools and choosing the appropriate level of tool abstraction are two related but distinct capabilities. Even a capable model must decide whether to execute a recurring procedure through a composite skill or retain atomic control when intermediate observations are necessary.

To investigate this concern, we extend our experiments to Qwen3.6-27B, a substantially stronger and more recent backbone. We use 4 NVIDIA H200 GPUs to train GRPO + Skills and CIPO on Qwen3.6-27B, following the TOOLATHLON setup in Section 4. Both methods use the same training tasks, skill library, skill-enabled SFT initialization, action space, reward configuration, and rollout group size (G=4G=4). Due to the limited rebuttal evaluation budget, we evaluate 100 episodes with a single LLM-judge assessment per episode, using the same success criteria as the main experiments.

Additional Results on Qwen3.6-27B :

MethodTool F1 ↑TSR (%) ↑Comp./Raw ↓
GRPO + Skills51.8445.000.904
CIPO56.9251.000.841
Improvement+5.08+6.00 pp−0.063

The comparison corresponds to a 5.08-point improvement in Tool F1 and a 6.00-percentage-point improvement in TSR, alongside stronger decision compression (15.9% vs. 9.6%). This pattern is consistent with the benefit of adaptive granularity selection extending to a substantially stronger backbone.

These results address the your concern from a stronger-backbone perspective: improved model capability does not by itself eliminate the distinction between atomic control and composite execution. CIPO provides explicit supervision for this decision, rather than relying solely on the backbone's general tool-use ability.

Ultimately, our goal is not to make agents invoke more skills, but to teach them when to use atomic tools and when to invoke composite skills according to the current decision state.


We sincerely appreciate your thoughtful feedback and hope that our clarifications and additional experimental evidence have adequately addressed your concerns. If so, we would be grateful if you could reconsider your assessment and score in light of these responses.

Thank you for your rebuttal, which largely addressed my soundness concerns. I will raise my soundness and overall rating (with a cautious 3, probably 2.5 is better). The main limitation is still lack of insights. The authors solidly prove CIPO performs well. However, the mechanism behind the method, how contrastive rollouts improve tool selection beyond lowering task diffculty, etc are still unknown.

Thank you for your thoughtful follow-up and, especially, for your willingness to raise both the soundness and overall ratings. We particularly appreciate the remaining concern regarding the insights behind CIPO's effectiveness, especially why counterfactual rollouts improve tool selection beyond simply reducing task difficulty. We would like to clarify two important insights.


1. CIPO learns when additional reasoning and control are useful, rather than simply making tasks easier.

Figure 1(d) shows that skills are preferable in 37% of the examined states, while atomic tools are preferable in 21%, with the remaining states showing comparable outcomes. However, SFT and GRPO + Skills do not reliably capture these state-dependent preferences.

This reveals an important trade-off: composite skills reduce redundant decisions when a reusable procedure is appropriate, whereas atomic tools preserve intermediate observations and opportunities for further reasoning. CIPO does not aim to uniformly increase or decrease reasoning; it learns which level of control is more beneficial in the current context.


2. Paired counterfactual feedback provides more targeted supervision than additional rollouts alone.

Unlike standard GRPO, CIPO compares feasible atomic and skill choices from the same decision history, allowing their downstream consequences to be evaluated under the same policy. This provides a more direct signal about the granularity decision, rather than attributing performance differences across unrelated trajectories to tool selection.

Importantly, our w/o CF Reward ablation retains the counterfactual rollouts but removes their paired reward adjustment. Restoring this signal improves average TSR from 31.65% to 36.48% (+4.83 pp). This supports the importance of learning from relative outcomes, beyond merely exposing the policy to alternative trajectories.

Together, these insights suggest that CIPO's benefit lies in learning when to exploit a composite skill and when to retain atomic control for better downstream decisions, rather than simply increasing reasoning or reducing task difficulty.

We hope this further clarifies the underlying mechanism, and we sincerely appreciate your valuable feedback and reconsideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for overseeing our submission and the three reviewers (gcHv, PWnJ, and mhbc) for their careful assessment and constructive feedback. During the rebuttal, we provided detailed responses, additional analyses, and manuscript revisions, with all changes highlighted in blue. To facilitate the discussion, we briefly summarize several key concerns and how we have addressed them.

Main Concerns and Our Responses

1. Methodological Contribution and Comparison Fairness · gcHv

Concern. Does CIPO improve upon skill-augmented GRPO through its counterfactual learning mechanism, rather than simply benefiting from additional skills or computation?

  • Controlled comparison. CIPO and GRPO + Skills share the same initialization, augmented action space, and rollout budget (G=4G=4). CIPO improves average TSR by 7.18 pp (29.30% → 36.48%) and Tool F1 by 7.03 pp (41.20% → 48.23%) across six settings.
  • Counterfactual contribution. Removing the counterfactual reward reduces TSR from 36.48% to 31.65%, supporting the benefit of matched granularity-specific feedback.

Supporting evidence: Section 3.4, Table 2, and Appendix H.

2. Branching Strategy and Skill Discovery · PWnJ

Concern. Why choose the first eligible branching point, and how does the method depend on successful trajectories for skill discovery?

  • Branching analysis. First-eligible branching achieves 36.48% average TSR, outperforming random branching (34.90%) and late branching (34.10%). We further analyzed cases where later decisions may be more consequential.
  • Skill availability. We clarified how successful trajectories are used to construct the skill library, discussed the limitations of offline mining, and examined iterative refinement as a potential extension.

Supporting evidence: Appendices A, H, and M; additional analysis in our response to PWnJ.

3. Data Provenance and Reproducibility · mhbc

Concern. What are the sources, screening criteria, and quantities of successful trajectories used for skill mining?

  • Transparent data construction. The three public benchmark datasets contain 9,298 processed trajectories, including 2,465 positive-labeled trajectories before minimum-length filtering.
  • Explicit screening criteria. We clarified the dataset-specific success definitions and how the selected trajectories are subsequently processed for skill discovery.

In the revised paper: Appendices A and K, including a new trajectory statistics table.

4. Evaluation Reliability and Computational Efficiency · gcHv, mhbc

Concern. Do the improvements reflect actual task completion, and what are the practical computational costs?

  • Stronger evaluation evidence. CIPO improves strict yes-only success by 6.28 pp (21.40% → 27.68%). Under TOOLATHLON's official environment-based evaluation, execution success improves by 7.50 pp (17.67% → 25.17%), independently of LLM judgments.
  • Efficiency analysis. We clarified the relationship between decision compression and actual inference costs and provided a comparison of token usage, latency, and RL training overhead. Counterfactual branching is used only during training, with no additional branch generation at inference.

Supporting evidence: Appendix L.1 and the computational cost analysis added to Appendix I.


We sincerely appreciate the reviewers' valuable feedback, which has helped us strengthen the clarity, reproducibility, and empirical support of our work. We hope our responses and revisions have addressed the main concerns and would be grateful for their consideration in the final assessment.

As the rebuttal period draws to a close, we remain very happy to address any additional questions or concerns the reviewers may have. Thank you again for your time and thoughtful consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.