- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies adaptation of tool-using agents to solution-preserving API replacements. It introduces DynaToolBench, comprising 64 short-chain tasks over 50 APIs and 30 longer-chain tasks over 34 APIs, evaluated at three timesteps. Versioned adapters change agent-visible tool names, arguments, and response fields while preserving an executable MirrorAPI backend. The proposed CATE method builds a SkillBank of atomic and pairwise composite skills through documentation-grounded expansion, interaction, and reflection. After an environment transition, it identifies affected skills, retrieves candidate replacement APIs, revises interface-dependent guidance, and transfers supported experience from retired atomic skills while retaining unaffected knowledge. Experiments use Qwen3-4B and Qwen3-8B, three repetitions, and an independent requirement-level LLM evaluator. CATE has the highest three-timestep average in all four model/subset settings, improves substantially over leaving skills stale, and uses fewer adaptation tokens and logical calls than ContDa in the reported longer-chain comparison.
Critical strengths
- The motivation is reasonable and receives direct experimental support. DynaToolBench keeps tasks executable while changing agent-visible API names, arguments, and outputs, so stale-interface failures can be attributed to adaptation. Table 3 shows that on 8B/Dyna-Long, leaving skills unchanged after replacement yields 68.61% and 67.78% at T1/T2, below current Raw-Doc's 77.22% and 73.33%. This concretely demonstrates the problem the paper proposes to address. The benchmark's short and longer dependency chains also test changes beyond isolated tool selection.
- The method responds plausibly to localized interface invalidation and demonstrates recovery. CATE identifies affected skill positions, grounds replacements and inherited notes in current documentation, and preserves unaffected entries. Full CATE recovers to 78.47% and 79.17% in the same Table 3 setting. Raw-Doc, EasyTool, and the continual ContDa baseline are relevant comparisons, and lower transition-time update cost than ContDa supports the efficiency motivation. The case study makes local workflow repair concrete, although these results do not isolate experience inheritance itself.
The inspected code substantiates this local-update mechanism: composite repair replaces only retired API positions while retaining general skill fields, atomic inheritance appends selected transferable notes, and frozen evaluation checks artifact hashes. Deterministic fixtures confirm these implementation properties. This strengthens confidence that the described mechanism is implemented, while leaving its incremental performance contribution to be established experimentally.
Critical weaknesses
- The implemented inheritance mechanism is clear, but its incremental contribution is not isolated.
core/atomic_replacement_migration.pyactually appends selected old notes to new atomic skills, whilecore/replacement_adaptation.pyuses current documents to replace affected composite positions. This confirms that reuse is implemented. However, document-guided replacement could account for much of the recovery without useful old experience. Table 3 removes all evolution or all adaptation; it does not compare inheritance with the same skill structure and update procedure after old notes are cleared. Raw-Doc/EasyTool lack that structure, and ContDa changes representation and adaptation together. Separate matched controls for atomic inheritance, composite skills, and reflection remain necessary to attribute the gains to selective reuse of accumulated experience. - Implemented retention and success indicators are not yet sufficient experimental evidence for the two stated objectives. The code preserves unaffected skill entries and protected fields, which is meaningful implementation evidence; it does not imply unchanged task behavior after retrieval or environment updates. The 94 tasks span two solution-preserving transitions, but DL has 28/30 affected tasks at T1 and all 30 at T2, leaving little direct evidence for retention. I also refine the earlier metric concern:
execution/evaluator.pycomputes both mean requirement accuracy and success only when all requirement scores equal 1, and the runner aggregates complete-task success and invocation diagnostics. The issue is therefore whether the reported comparative conclusions hold on these stricter measures, not whether they are implemented. Aggregate requirement accuracy and a three-timestep average can conceal incomplete downstream goals or adaptation regressions; for example, 8B/DS at T2 remains 72.66% for CATE versus 74.48% for Raw-Doc. Separate affected/unaffected pre/post outcomes and the already available complete-task diagnostics are needed to show that the method solves both recovery and retention. - Writing quality: frequent defensive statements and excessive qualification of the method and evaluation. The paper repeatedly emphasizes preserved backend functionality and task solvability, then stresses that composite skills encode local relations instead of complete task solutions, retrieved skills are guidance “rather than rigid programs”, and adaptation receives “neither benchmark ground-truth trajectories nor formal evaluation outcomes” (Sections 4–6). Additional passages repeatedly separate learning tasks, training-time judgments, and formal evaluation. The cumulative presentation is overly defensive: successive explanations of what the method does not assume or use can make the prose sound like preemptive rebuttal and create an impression of protesting too much. This reduces concision and obscures the central explanation of why inherited experience improves adaptation. This is the consequence of the author's excessive reliance on GPT writing, and the author should optimize these writing sections to make their paper statements more persuasive, thereby improving the quality of their writing.
Questions for authors
- Using the supplied implementation, does inheritance outperform clearing old notes and rewriting from current documents when skill structure, retrieval, executor, and update budget are held fixed? What remains after separately disabling atomic-note transfer, composite skills, or reflection? These controls should isolate the reuse mechanism rather than only compare whole systems.
- What are the pre/post-adaptation outcomes separately for affected and unaffected tasks? Does preservation of unrelated skill entries in the code translate into preserved task success, and do previously successful affected tasks recover?
- The evaluator already computes
benchmark_success_rate/end_to_end_success_rateand ground-truth invocation diagnostics. Do the reported CATE gains persist on these complete-task and identifier/dependency measures? How much disagreement is there between averaged requirement accuracy and complete completion, and which failure types explain remaining regressions?
8 comments
Author Response to Reviewer 7ncc (Part 2/4): Task Recovery and Retention
W2 & Q2. Affected-Task Recovery and Unaffected-Task Retention
You correctly point out that preserving Skill entries does not necessarily mean preserving task performance. We therefore examined what happens to individual tasks before and after adaptation, using one complete CATE run for each of Qwen3-4B and Qwen3-8B on Dyna-Short (DS) and Dyna-Long (DL).
Pre-/Post-Adaptation Accuracy. We first measured requirement-level accuracy on affected and unaffected tasks within the same updated environment, immediately before and after adaptation.
Table 2. Pre-/post-adaptation Accuracy (%) on affected and unaffected tasks.
| Model | Subset | Stage | Affected (Pre → Post) | Unaffected (Pre → Post) |
|---|---|---|---|---|
| 4B | DS | T1 | 70.00 → 74.17 (+4.17) | 79.17 → 79.17 (±0.00) |
| 4B | DS | T2 | 72.50 → 80.00 (+7.50) | 61.11 → 61.11 (±0.00) |
| 4B | DL | T1 | 76.49 → 84.23 (+7.74) | 100.00 → 75.00 (−25.00) |
| 4B | DL | T2 | 77.38 → 78.27 (+0.89) | 100.00 → 100.00 (±0.00) |
| 8B | DS | T1 | 71.88 → 80.62 (+8.74) | 64.58 → 64.58 (±0.00) |
| 8B | DS | T2 | 71.88 → 75.00 (+3.12) | 70.14 → 68.75 (−1.39) |
| 8B | DL | T1 | 69.20 → 78.12 (+8.92) | 83.33 → 83.33 (±0.00) |
| 8B | DL | T2 | 76.79 → 77.68 (+0.89) | 50.00 → 100.00 (+50.00) |
Changes in parentheses are percentage points (pp). Affected/unaffected tasks are identified separately for each environment transition. Each transition contains 40 affected and 24 unaffected tasks on DS, and 28 affected and only 2 unaffected tasks on DL. Thus, both the −25.00 pp and +50.00 pp changes on DL concern groups of just two tasks, rather than reflecting broad performance changes. Although all 30 DL tasks are affected cumulatively across both transitions, two remain unaffected by each individual transition.
Affected-task accuracy increases after adaptation in all eight settings. For unaffected tasks, accuracy remains unchanged in five settings. The other three show changes, including relatively large percentage fluctuations in the two-task DL groups. We report these cases explicitly rather than treating unchanged Skill entries as a guarantee of unchanged task outcomes.
Complete-Task Recovery and Retention. We further checked whether individual tasks returned to full completion after adaptation. Here, full completion requires every evaluated requirement to receive a score of 1.
Table 3. Recovery and retention of fully completed tasks.
| Model | Subset | Stage | Affected Recovery | Unaffected Retention |
|---|---|---|---|---|
| 4B | DS | T1 | 2/5 (40.0%) | 15/15 (100%) |
| 4B | DS | T2 | 5/7 (71.4%) | 8/11 (72.7%) |
| 4B | DL | T1 | 3/3 (100%) | 1/2 (50.0%) |
| 4B | DL | T2 | 5/6 (83.3%) | 2/2 (100%) |
| 8B | DS | T1 | 4/6 (66.7%) | 11/14 (78.6%) |
| 8B | DS | T2 | 4/5 (80.0%) | 13/14 (92.9%) |
| 8B | DL | T1 | 4/4 (100%) | 1/1 (100%) |
| 8B | DL | T2 | 5/6 (83.3%) | 1/1 (100%) |
Affected Recovery measures the fraction of previously fully completed affected tasks that became incomplete following API replacement but returned to full completion after adaptation. Unaffected Retention measures the fraction of unaffected tasks fully completed immediately before adaptation that remained fully completed afterward.
Across both transitions, Qwen3-4B recovers 15/21 (71.4%) and Qwen3-8B recovers 17/21 (81.0%) of previously successful but disrupted task-completion events. For unaffected tasks, 26/30 (86.7%) previously successful task-transition cases remain fully completed for each model. These counts refer to task-transition cases, not necessarily distinct tasks.
The two measurements address different aspects of adaptation: requirement-level accuracy captures partial recovery, while complete-task outcomes show how often previously successful tasks are fully restored or retained. The results also show that retention is not perfect, particularly in some individual transitions.
Author Response to Reviewer 7ncc (Part 3/4): Writing Revisions and Complete-Task Success
W3. Writing and Presentation
We agree with your criticism that parts of Sections 4–6 are repetitive and occasionally defensive. Several qualifications were intended to make the experimental setting precise, but they made the main argument harder to follow. We will revise these sections accordingly.
In Section 4, we will remove repeated explanations of backend stability and task solvability, keeping the focus on how DynaToolBench constructs controlled API changes.
In Section 5, we will remove unnecessary qualifications such as “rather than rigid programs” and describe Skill composition, interaction-guided refinement, and experience inheritance more directly.
In Section 6, we will reduce repeated explanations of the distinction between Skill learning and evaluation, while retaining the necessary protocol details and placing longer technical clarifications in the appendix.
The aim is to make the method and experimental setting easier to follow without removing information needed for reproducibility.
Q3. Complete-Task Success and Tool-Call Dependency Diagnostics
We also examined complete-task success, adherence to reference tool dependencies, and the remaining failure cases. The analyses cover Qwen3-4B and Qwen3-8B, including the additional AWM and ReasoningBank baselines.
Complete-Task Success. Our primary metric is requirement-level Accuracy because it distinguishes partial completion from complete failure. In a dependent workflow, an agent may correctly perform several operations but fail on a later step. Binary success assigns such cases the same outcome as tasks with no completed requirements. We therefore report complete-task success as an additional, stricter metric.
Table 4. Three-timestep average complete-task success rate (%).
| Method | 4B/DS | 4B/DL | 8B/DS | 8B/DL |
|---|---|---|---|---|
| Raw-Doc | 60.94 | 44.44 | 61.98 | 50.00 |
| EasyTool | 53.13 | 42.12 | 57.81 | 43.33 |
| ContDa | 58.85 | 45.56 | 58.85 | 51.78 |
| AWM | 56.77 | 42.22 | 55.21 | 47.78 |
| ReasoningBank | 61.46 | 36.67 | 59.90 | 45.56 |
| CATE | 58.85 | 50.00 | 63.02 | 55.56 |
CATE ranks first in three of the four settings, exceeding the strongest baseline by 4.44 pp on 4B/DL, 1.04 pp on 8B/DS, and 3.78 pp on 8B/DL. On 4B/DS, ReasoningBank achieves higher complete-task success, although CATE has higher requirement-level Accuracy.
The gaps between requirement-level Accuracy and complete-task success are also substantial across all methods: 20.15 pp for Raw-Doc, 19.60 pp for EasyTool, 21.24 pp for ContDa, 22.71 pp for AWM, 21.95 pp for ReasoningBank, and 20.61 pp for CATE. The gaps are generally larger on Dyna-Long, where completing every requirement in a dependent workflow is more difficult. This also explains why requirement-level Accuracy and binary complete-task success can give different rankings.
Author Response to Reviewer 7ncc (Part 1/4): Skill Components and Ablations
Thank you for your detailed feedback! We address the questions about CATE's individual components, task-level recovery and retention, and evaluation metrics below. We also outline the planned writing revisions.
W1 & Q1. Skill Composition, Experience Inheritance, and Reflection
You asked how much composite Skills, atomic experience inheritance, and reflection each contribute to CATE. We clarify their roles and report two additional ablations.
Role of Composite Skills. Composition is performed during Skill Evolution at T0. Composite Skills do more than represent inter-tool dependencies: they also provide the interactions from which atomic Skills learn. CATE generates diagnostic tasks based on discovered tool compositions, executes them in the environment, and propagates reusable tool-specific experience back to the participating atomic Skills.
Simply removing composition would therefore also remove the main source of interaction evidence for atomic Skill learning. The remaining atomic Skills would largely rely on documentation, making this setting similar to the Raw-Doc baseline.
Additional Ablations. We also evaluated w/o Atomic Inheritance and w/o Reflection to isolate the contributions of experience transfer and interaction-guided refinement.
Table 1. Ablation results on Qwen3-8B / Dyna-Long (Accuracy, %).
The identical T0 scores (81.67%) for Full CATE, w/o Adaptation, and w/o Atomic Inheritance reflect a shared initial T0 SkillBank and evaluation, since these ablations affect only subsequent adaptation stages.
The w/o Atomic Inheritance variant retains the SkillBank structure and document-guided composite adaptation but disables the transfer of accumulated atomic experience. Accuracy drops by 5.41 pp at T1 and 3.48 pp at T2, indicating that inherited experience contributes beyond document-guided interface replacement.
Removing Reflection reduces T0 accuracy by 6.11 pp, showing the benefit of refining Skills through interaction feedback.
These results show that reflection improves the acquired experience, while atomic inheritance carries it across API replacements.