acceptodds
Under review as a conference paper at ICLR 2027

CAFTBench: A Benchmark for Continuous Fine-Tuning of Tool-Calling Agents

Abstract

LLM-based agents increasingly rely on external tools and APIs, yet most deployed systems assume a static tool ecosystem. In practice, APIs evolve: new tools appear, schemas change, and existing parameters are modified. Such changes require agents not only to acquire new tool-use behavior, but also to retain earlier tools, adapt to schema mutations, and compose skills learned at different times. We introduce **CAFTBench**, a static and reproducible benchmark for evaluating continuous fine-tuning of tool-calling agents. CAFTBench consists of **CAFT-30k**, a tool-ordered training corpus of 29,573 tool-calling instances spanning more than 260 tools, and CAFT-Eval, a probe suite covering single-tool routing, cross-stage composition, API upgrades, no-tool abstention, and structured-output robustness. We further define five metrics—Historical Routing Failure (HRF), Cross-Stage Composition Rate (CSCR), Negative Transfer and Historical Interference (NTHI), Tool Over-reliance Index (TOI), and Syntax Collapse Rate (SCR)—to disentangle retention, transfer, interference, over-reliance, and output-format degradation. Experiments with Qwen3-8B and LLaMA-3.1-8B-Instruct show that existing continual learning methods exhibit strong backbone-dependent trade-offs: EWC-style regularization improves stability and upgraded-API adaptation on Qwen, while semantic replay improves composition, but the same methods can substantially degrade LLaMA and induce tool over-reliance. These results highlight the need for continual learning methods designed specifically for evolving tool-calling agents.

Primary area:
datasets and benchmarks
Keywords:
Continuous Learning, Agent, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_b4tpOpenReview ↗
Rating
3(+2)
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CAFTBench, a static benchmark for continual fine-tuning of tool-calling LLMs. A model is first fine-tuned on a stream of about 30K examples over about 260 tools, presented tool by tool. It is then fine-tuned again on "upgraded" APIs, where each function is renamed with a v1/v2 suffix and gets a new required parameter. Fixed probes then score routing error (HRF), unparseable outputs (SCR), composition quality from an LLM judge (CSCR), and spurious tool calls on no-tool prompts (TOI) after the first phase, and errors on the upgraded APIs (NTHI) after the second. Comparing sequential SFT, EWC, RecAdam, Experience Replay and CorpusBrain++ on Qwen3-8B and LLaMA-3.1-8B, plus two more backbones in the appendix, the authors conclude that the trade-offs depend on the backbone and that existing continual-learning methods are not sufficient.

Critical strengths

The problem of evolving APIs is real, and the failure taxonomy is useful. Splitting errors into format collapse, wrong tool, missing upgraded arguments and spurious calls (Figure 4) is more informative than a single accuracy number. The static protocol and the released code also make the benchmark cheap to run.

Critical weaknesses
  1. The paper never shows that fine-tuning is needed. Updating the schema in the prompt may be enough. The motivation is that tool changes force retraining. Yet at test time, CAFTBench already puts the schemas in the prompt: the target tool plus three distractors, and the new schema for upgraded probes (Section 4). The upgrade itself is mild, a rename plus one required parameter. This is exactly the setting in-context function calling handles, and it is what BFCL evaluates. Even so, the paper reports no result without fine-tuning, neither for the base backbones nor for a frontier commercial model given the updated schema. The introduction dismisses prompting because of context length and cost, but never tests it. The paper's own numbers suggest the prompt does most of the work:

    • On Olmo and Ministral, plain SFT has under 1% HRF and about 3% NTHI.
    • On Qwen, almost all errors are parse failures of the custom Call: tool(args) format (Figure 4). Fine-tuning mainly teaches, or breaks, a format that the base model's native tool template might not need.

    Prior work on evolving tools (ToolEVO / ToolQA-D) studies the harder case, where the model sees only outdated documentation. There, fine-tuning on static APIs is what made models brittle. The key test is missing: does a model fine-tuned on the old tools ignore the new schema in its prompt? Without it, the benchmark may be measuring a problem that a schema update already solves.

  2. The setup does not measure forgetting, and the method comparison is confounded.

    • HRF is an endpoint error rate. There is no pre-fine-tuning reference, no joint-training upper bound, and no intermediate checkpoints. The original probes are never re-scored after the upgrade phase, so whether upgrading overwrites the original tools, the most direct forgetting case here, is never measured.
    • Each result comes from a single run, and the methods use different learning rates and batch sizes. The settings for the baseline SFT are not given, and hyperparameters were tuned only on Qwen. The headline backbone-dependence finding may just reflect an EWC λ\lambda tuned for Qwen that does not suit LLaMA.
Questions for authors
  1. What are the five metrics for the base backbones (with their native tool templates) and for a frontier commercial model, given the same schemas in the prompt with no fine-tuning?
  2. On the upgraded probes, with the new schema in the prompt, how does the model fine-tuned only on the old tools (θ1\theta_1) score? Is there any evidence that fine-tuned knowledge overrides an updated in-context schema?
  3. Is there a realistic deployment setting where the schema cannot be given in the prompt? If so, why does the benchmark provide it?
  4. What are HRF and SCR on the original probes after the upgrade phase (θ2\theta_2), and under joint training?
  5. Table 4 reports very different numbers from Table 2 for the same Qwen configurations, for example EWC TOI of about 0.9 against 0. Which runs are correct, and how many seeds were used?
8 comments

We appreciate the time and effort Reviewer b4tp devoted to reviewing our manuscript. However, we respectfully note that the current review reflects a certain degree of misunderstanding of our work.We earnestly ask the reviewer to read our clarifications below and reconsider the evaluation of our paper.

We would like to emphasize that the primary objective of our work is to evaluate continual fine-tuning methods in training-based scenarios. In this context, we use the term "fine-tuning" in a broad sense to refer to methods that involve a training process, rather than aiming to evaluate the broader spectrum of continual learning methods, which encompasses both training-based and training-free approaches.We fully agree with the reviewer that continual learning includes both fine-tuning-based and fine-tuning-free methods. However, as explicitly stated in the title of our paper, our benchmark is specifically designed to evaluate continual fine-tuning methods, rather than continual learning methods in the broader sense. We respectfully ask the reviewer to reconsider their assessment in light of this clarification regarding the intended scope and objectives of our benchmark.

[W1-W2]: The paper does not establish the necessity of fine-tuning or demonstrate that old tool knowledge overrides updated schemas.

[A1-A2]: We thank the reviewer for the comments on our motivation and evaluation scope. We would like to clarify that CAFTBench is designed to evaluate continual fine-tuning methods through sequential training, as reflected in our paper's title. Our dataset and evaluation pipeline assess how different continual fine-tuning methods affect tool-calling behavior across original tool learning and subsequent updates. Our benchmark's objective is to evaluate such training procedures, rather than to establish that fine-tuning is necessary or superior to training-free adaptations. As we stated in the introduction, fine-tuning is positioned as a complementary approach alongside prompt-based methods. The reported training performance also does not identify whether errors arise from old knowledge overriding updated schemas. We distinguish the questions from the benchmark's stated evaluation objective below.

Regarding Q1 (untuned and frontier-model baselines): The low error rates cited by the reviewer were obtained after fine-tuning. They therefore do not establish the claim that "the prompt does most of the work". Our comparisons focus on how different continual fine-tuning methods perform under a shared sequential tool-learning protocol. We would like to point out that methods without weight updates do not undergo the sequential training process studied in our work. Thus untuned and frontier-model baselines would provide additional context, but would not substitute for this comparison.

Regarding Q2 (the pre-upgrade model with updated schemas): CAFTBench provides a unified sequential tool-learning protocol for comparing continual fine-tuning methods across original-tool learning and subsequent API-upgrade training.The API-upgrade phase extends the sequential learning process through further weight updates, allowing us to evaluate continual fine-tuning methods as tool specifications evolve. This comparison focuses on the outcome of the evaluated continual fine-tuning procedures; it neither assumes that schema updates alone must fail nor independently identifies the cause of calling errors. Our current upgrade evaluation focuses on comparing methods after upgrade training and does not include the setting of evaluating the previous phase with updated schemas but without further training.

Regarding Q3 (schema availability): In our benchmark's setting, we provide current schemas as part of the evaluation conditions and compare continual fine-tuning methods after sequential training. Under this setting, schema availability and continued training are compatible. Our motivation does not rely on a deployment setting in which schemas cannot be provided.

[W3]: Qwen results suggest prompts drive tool use, while fine-tuning mainly affects formatting.

[A3]: We respectfully distinguish the observed distribution of errors from the attribution of tool-use performance to prompting versus fine-tuning.In Section 5.2, we identify format preservation as a major source of performance differences across methods on Qwen. Reporting SCR and the error decomposition in Figure 4 makes this distinction clear, rather than attributing all improvements to better tool selection or reasoning. However, this observation comes from fine-tuned models. The predominance of parse failures does not isolate the contribution of the supplied schemas from that of training.Likewise, the fact that formatting accounts for most differences across methods does not establish that fine-tuning mainly teaches or breaks a format. Other reported failures, including wrong-tool selections among parseable LLaMA outputs and tool over-reliance, further illustrate the benchmark’s broader diagnostic scope.Distinguishing these effects is part of the benchmark's diagnostic contribution: it shows how the observed effects of continual fine-tuning methods differ across metrics and backbones. Regarding the native-template comparison in Q1, we adopt the same canonical call format across backbones to provide a consistent output and scoring convention.Our comparison evaluates continual fine-tuning methods under this shared format, rather than under each model's native tool template.

[W4]: The benchmark setup does not measure forgetting,only endpoint errors.

[A4]: As explicitly defined in the paper, HRF measures historical-tool routing performance after sequential tool learning.This endpoint evaluation answers how reliably the resulting model can invoke previously encountered tools after sequential learning.Together with SCR and the error decomposition, it provides a multidimensional and diagnostic basis for comparing continual fine-tuning methods.

[W5]: Potential confounding from training configurations and randomness

[A5]: We thank the reviewer for raising this concern.We would like to clarify that the Qwen hyperparameter sweep reported in Appendix D.1 is a supplementary sensitivity analysis. In the main experiments, our settings follow the recommendations in the original baseline papers; for those without explicit guidance, we used commonly adopted values from prior experience.Crucially, we kept the same configurations for both backbones.The SFT baseline uses the same learning rate(2×10−52 \times 10^{-5}) and effective batch size(512) as the replay methods, together with the shared training settings in Table 3.We will explicitly list the SFT settings in Table 3.The relevant settings are detailed below:

Learning Rates and Batchsize: Based on prior training experience with the respective methods, we set 5×10−55 \times 10^{-5} for regularization methods(EWC, RecAdam) and 2×10−52 \times 10^{-5} for replay methods (Experience Replay, CorpusBrain++) to ensure our evaluations remain fair by allowing each paradigm to operate within an appropriate learning rate interval.The smaller effective batch size(128) for EWC and RecAdam was chosen to accommodate the chunk-based training and consolidation schedule described in Appendix B, while SFT and the replay baselines used 512.

Regularization Strength λ\lambda=400:We set λ\lambda = 400 by aligning with the recommended hyperparameter settings established in the original EWC work[1].

Buffer capacity (N=1500):This setup directly adheres to the empirical buffer sizing principles established by the original work[2].

We therefore interpret the observed interactions as differences across backbones under the reported method-specific configurations.

[W6]: Table 2 and Table 4 report different Qwen TOI results.

[A6]: Regarding Q5 (Table 2/Table 4 consistency and training seeds):We apologize for the unclear reporting of the experimental environments. Evaluation results and their reproducibility should be interpreted in the context of the hardware and software environment. Table 2 was obtained on A800 GPUs, while resource constraints required the supplementary Table 4 experiments to run on H20 GPUs.Due to infrastructure incompatibilities, we switched to hiyouga/llamafactory:0.9.5, although the training and evaluation code remained unchanged. These environment differences may have contributed to the discrepancy. However, the exact original environment was not fully preserved, preventing definitive attribution. We retain Table 2 as the main reported results and will clarify that Table 4 contains separate sensitivity runs on a different platform. Our supplementary multi-seed experiments provide additional evidence in the tested reduced-data setting. The data and code are publicly available so that others can reproduce the protocol in their own environments.

Reference

[1]Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017

[2]Experience Replay for Continual Learning. NeurIPS 2019

Reviewer_DHBcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAFTBench is a static benchmark for continual fine-tuning of tool-calling agents as APIs evolve. It pairs a tool-ordered training corpus with five disentangled metrics — routing, composition, schema-mutation adaptation, abstention, syntax — and finds that the same continual-learning methods behave very differently across backbones: on Qwen3-8B, EWC sharply cuts upgraded-API errors, while on LLaMA-3.1-8B the same methods worsen routing and tool over-reliance. Among the methods tested, none is a universal winner.

Critical strengths
  1. APIs that add mandatory parameters or rename functions force the model to overwrite previously learned routing rather than memorize a new class label (§1), and the 50-upgraded-tool probe with its dedicated NTHI metric (§4) catches exactly what BFCL-style static benchmarks do not.
  2. The five metrics separate failure modes that prior benchmarks conflate, and Figure 4 further decomposes errors into wrong-tool / syntax / missing-args / parse, so a reader can tell whether a method's gain comes from format stabilization or real routing improvement.
  3. The protocol is static, temperature-0 and open (§4, p.5; footnote p.3). Closest Huang et al. 2024 studies continual tool use on synthetic NLP tasks; LifelongAgentBench and BFCL cover interactive agents and static function calling, but neither isolates sequential weight updates under schema drift.
Critical weaknesses
  1. Every reported number is a single temperature-0 run, with no seeds, no CIs and no paired tests (Table 2). On Qwen the EWC–RecAdam HRF gap is only ~0.006, so small comparisons cannot be treated as settled.
  2. EWC/RecAdam use LR 5e-5, batch 128, λ=400, while replay methods use LR 2e-5, batch 512, 1,500-sample reservoir (Table 3); the reported method×backbone interaction may partly be an optimization-schedule effect rather than a method effect.
  3. Table 2 and Table 4 disagree on Qwen TOI. Table 2 reports 0.0000 for all five methods; Table 4 reports EWC TOI = 0.855–0.901. Which no-tool set does Table 4 actually use? Until that is stated, a reader cannot tell whether Qwen EWC causes tool over-reliance.
  4. CSCR is scored by Qwen3-30B, while all Qwen-side methods train on the same backbone family (§5.1); without human agreement or a structural-match CSCR alongside the rubric score, judge calibration is insufficient to separate composition quality from possible family-aligned leniency.
  5. Only two endpoint checkpoints are evaluated, 50 of 260+ tools are upgraded, and the appendix adds two backbones without the same five-method grid; multi-round API evolution and real tool execution remain untested.
Questions for authors

Refer to above weaknesses

2 comments

We thank Reviewer DHBc for the constructive and insightful feedback. We address your specific points directly below.

[W1]: Limited statistical validation.Single-run results may lack uncertainty estimates and significance tests, making small performance (eg., EWC and RecAdam on Qwen) differences inconclusive.

[A1]: We thank the reviewer for this thoughtful suggestion to strengthen the statistical rigor of our evaluation.We would like to clarify that our primary contribution is the construction of the CAFTBench dataset and the evaluation pipeline, rather than fine-grained method rankings.We agree that reporting results over multiple seeds would strengthen the stability of our benchmark.Due to our limited computational resources and time, we conducted additional runs of SFT and EWC on both backbones.We apologize that we were unable to repeat the full experimental grid in Table 2.We trained each selected configuration with two seeds (42 and 2027), using a 10% subset of the training data.The supplementary results are summarized below:

Table 1: Supplementary results using 10% training data with seeds 42 and 2027.

BackboneMethodHRF ↓SCR ↓NTHI ↓CSCR ↑TOI ↓
Qwen3-8BSFT0.0073 ± 0.00050.0011 ± 0.00050.0307 ± 0.00220.5669 ± 0.01570.0000 ± 0.0000
Qwen3-8BEWC0.0138 ± 0.00430.0069 ± 0.00330.0360 ± 0.00540.6210 ± 0.00680.0000 ± 0.0000
LLaMA-3.1-8B-InstructSFT0.0119 ± 0.00050.0057 ± 0.00050.0406 ± 0.00760.4406 ± 0.00760.0000 ± 0.0000
LLaMA-3.1-8B-InstructEWC0.0625 ± 0.01350.0502 ± 0.01570.0632 ± 0.01900.3497 ± 0.03950.6675 ± 0.0884

Across the two seeds in the 10% training data setting, the supplementary results provide a targeted robustness check of CAFTBench evaluations, with relatively small variation in most reported metrics.On TOI, LLaMA continues to exhibit substantial tool over-reliance under EWC(0.6675 versus 0 for SFT), while both methods remain at 0 on Qwen.Relative to SFT, EWC increases HRF, SCR, and NTHI on both backbones, with larger increases on LLaMA.CSCR improves on Qwen but declines on LLaMA in the supplementary setting.

[W2]: Potential confounding from training configurations. Different learning rates and batch sizes may contribute to the observed method–backbone interactions. [A2]: We thank the reviewer for the question on training configurations.Our settings follow the recommendations in the original baseline papers; for those without explicit guidance, we used commonly adopted values from prior experience.Crucially, for each method, we kept the same configurations across both backbones.Our comparison concerns performance under these predefined configurations, rather than independently optimized performance for a given method–backbone pair.We summarize the relevant settings below:

Learning Rates and Batchsize: Based on prior training experience with the respective methods, we set 5×10−55 \times 10^{-5} for regularization methods(EWC, RecAdam) and 2×10−52 \times 10^{-5} for replay methods (Experience Replay, CorpusBrain++) to ensure our evaluations remain fair by allowing each paradigm to operate within an appropriate learning rate interval.The smaller effective batch size(128) for EWC and RecAdam was chosen to accommodate the chunk-based training and consolidation schedule described in Appendix B, while SFT and the replay baselines used 512.

Regularization Strength λ\lambda=400:We set λ\lambda = 400 by aligning with the recommended hyperparameter settings established in the original EWC work[1].

Buffer capacity (N=1500):This setup directly adheres to the empirical buffer sizing principles established by the original work[2].

We therefore interpret the observed interactions as differences across backbones under the reported method-specific configurations.

[W3]: Inconsistent TOI results. Tables 2 and 4 report substantially different TOI values for Qwen EWC without clarifying the evaluation settings.

[A3]: We apologize for the unclear reporting of the experimental environments. Evaluation results and their reproducibility should be interpreted in the context of the hardware and software environment. Table 2 was obtained on A800 GPUs, while resource constraints required the supplementary Table 4 experiments to run on H20 GPUs.Due to infrastructure incompatibilities, we switched to hiyouga/llamafactory:0.9.5, although the training and evaluation code remained unchanged. These environment differences may have contributed to the discrepancy. However, the exact original environment was not fully preserved, preventing definitive attribution. We retain Table 2 as the main reported results and will clarify that Table 4 contains separate sensitivity runs on a different platform. Our supplementary multi-seed experiments provide additional evidence in the tested reduced-data setting. The data and code are publicly available so that others can reproduce the protocol in their own environments.

[W4]: Insufficient CSCR judge calibration. The Qwen-family judge may introduce same-family bias, and human agreement or structural-match validation is missing.

[A4]: We agree that a single LLM judge for CSCR may raise potential bias issues.We chose LLM judge rather than exact matching against a single reference trajectory because compositional tasks allow flexible execution paths in our benchmark.And we would like to point out that LLM-as-Judge has been established as a reliable evaluation paradigm in the tool-calling setting.In ToolLLM [3], an automated GPT-based evaluator is proposed (ToolEval) for assessing tool-use quality and demonstrates 87.1% agreement with human evaluations on pass rate and 80.3% on win rate.These results provide precedent for using LLM-based evaluation in tool-calling tasks.To directly examine the concern about Qwen-family bias, we additionally re-evaluated outputs from our supplementary two-seed experiments using GPT6.1 Sol as an additional non-Qwen judge.The results are reported below:

Table 2: Supplementary CSCR evaluation using GPT6.1 Sol as a non-Qwen judge on the two-seed experiments.

BackboneMethodCSCR (seed 42) ↑CSCR (seed 2027) ↑Mean CSCR ↑
Qwen3-8BSFT0.39100.40300.3970
Qwen3-8BEWC0.49300.44600.4695
LLaMA-3.1-8B-InstructSFT0.32700.35200.3395
LLaMA-3.1-8B-InstructEWC0.28300.28500.2840

The cross-backbone ordering remains consistent with the Qwen judge: Qwen scores higher than LLaMA for both baseline methods.And within each backbone, EWC scores higher than SFT on Qwen but lower on LLaMA, with such patterns holding across the two seeds under GPT6.1 Sol as an llm judge.We believe this consistency could partially alleviate concerns about Qwen-family judge bias and strengthen confidence in interpreting CSCR comparisons.

[W5]: Limited evaluation coverage.The evaluation uses only two endpoints and 50 upgraded tools, with concerns about additional-backbone comparisons, multi-round API evolution, and real tool execution.

[A5]: We sincerely thank the reviewer for the suggestions on broadening the evaluation. We would like to clarify that Appendix D.2, Table 5 reports results for all five methods on both Olmo3-7B and Ministral-3-14B-Instruct, including all five evaluation metrics.We agree that additional evaluation settings would provide a more comprehensive understanding of continual tool-calling behavior.And multi-round API evolution would better reflect ongoing changes in real deployments, and we will consider it for future community releases.We would like to clarify that real tool execution may involve external factors beyond our current scope of isolating model-side continual learning.The current benchmark remains focused on providing a controlled and realistic evaluation of continual fine-tuning methods through sequential tool learning.Within this scope, the current protocol provides a common basis for comparing various continual fine-tuning methods.

Reference

[1]Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017

[2]Experience Replay for Continual Learning. NeurIPS 2019

[3]ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs ICLR 2024

Comments on the paper

5 comments
AuthorsOpenReview ↗

Clarifications From Authors

We thank all reviewers for their feedback. We believe that some concerns may stem from different interpretations of our contributions, research scope, and experimental settings. We respectfully ask reviewers to consider the following clarifications when assessing our work.

  1. Our primary contribution is the benchmark and its evaluation framework. CAFTBench provides carefully constructed training and evaluation datasets, together with a unified protocol for assessing continual fine-tuning of tool-calling agents. It addresses an underexplored evaluation setting by bringing sequential tool learning, API-schema adaptation, and complementary diagnostic metrics into a common framework. Its central contribution is to enable systematic evaluation and comparison of continual fine-tuning methods.

  2. Our research focuses on continual fine-tuning methods that involve training. We recognize that continual learning encompasses both training-based and training-free approaches. However, as reflected in the paper’s title, CAFTBench specifically studies methods that update model parameters through sequential training (What Continous Fine-tuning means). Our experimental design and comparisons should therefore be understood within this stated research scope.

We thank Reviewer RfdJ for the constructive and insightful feedback. We address your specific points directly below.

[W1]: Limited backbone coverage and lack of multi-seed validation of the backbone-dependent findings.

[A1]: We thank the reviewer for recognizing the value of our findings and the constructive suggestions to strengthen the empirical support.We agree that broader backbone coverage and multi-seed validation will strengthen the evidence supporting our findings.We would like to clarify that Table 5 in Appendix D.2 reports results for the five methods on Olmo3-7B and Ministral-3-14B-Instruct, which extends our evaluation to additional model families and a larger model scale.Due to our limited computational resources and time, we conducted additional runs of SFT and EWC on both backbones.We apologize that we were unable to repeat the full experimental grid in Table 2.We trained each selected configuration with two seeds (42 and 2027), using a 10% subset of the training data.The supplementary results are summarized below:

Table 1: Supplementary results using 10% training data with seeds 42 and 2027.

BackboneMethodHRF ↓SCR ↓NTHI ↓CSCR ↑TOI ↓
Qwen3-8BSFT0.0073 ± 0.00050.0011 ± 0.00050.0307 ± 0.00220.5669 ± 0.01570.0000 ± 0.0000
Qwen3-8BEWC0.0138 ± 0.00430.0069 ± 0.00330.0360 ± 0.00540.6210 ± 0.00680.0000 ± 0.0000
LLaMA-3.1-8B-InstructSFT0.0119 ± 0.00050.0057 ± 0.00050.0406 ± 0.00760.4406 ± 0.00760.0000 ± 0.0000
LLaMA-3.1-8B-InstructEWC0.0625 ± 0.01350.0502 ± 0.01570.0632 ± 0.01900.3497 ± 0.03950.6675 ± 0.0884

Across the two seeds in the setting, the supplementary results provide a targeted robustness check of CAFTBench evaluations, with relatively small variation in most reported metrics.On TOI, LLaMA still exhibits substantial tool over-reliance under EWC(0.6675 versus 0 for SFT), while both methods remain 0 on Qwen.Relative to SFT, EWC increases HRF, SCR, and NTHI on both backbones, with larger increases on LLaMA.CSCR improves on Qwen while it declines on LLaMA in the supplementary setting.

[W2]: Potential Qwen-family bias and insufficient validation of the CSCR judge.

[A2]: We agree that a single LLM judge for CSCR may raise potential bias issues. We would like to clarify that LLM-as-Judge has been established as a reliable evaluation paradigm in the tool-calling setting.In ToolLLM [1], an automated GPT-based evaluator is proposed (ToolEval) for assessing tool-use quality and demonstrates 87.1% agreement with human evaluations on pass rate and 80.3% on win rate.These results provide precedent for using LLM-based evaluation in tool-calling tasks.To directly examine the concern about Qwen-family bias, we additionally re-evaluated outputs from our supplementary two-seed experiments using GPT6.1 Sol as an additional non-Qwen judge.The results are reported below:

Table 2: Supplementary CSCR evaluation using GPT6.1 Sol as a non-Qwen judge on the two-seed experiments.

BackboneMethodCSCR (seed 42) ↑CSCR (seed 2027) ↑Mean CSCR ↑
Qwen3-8BSFT0.39100.40300.3970
Qwen3-8BEWC0.49300.44600.4695
LLaMA-3.1-8B-InstructSFT0.32700.35200.3395
LLaMA-3.1-8B-InstructEWC0.28300.28500.2840

The cross-backbone ordering remains consistent with the Qwen judge:Qwen scores higher than LLaMA for both baseline methods.And within each backbone, EWC scores higher than SFT on Qwen but lower on LLaMA, with such patterns holding across the two seeds under GPT6.1 Sol as an llm judge.We believe this consistency could partially alleviate concerns about Qwen-family judge bias and strengthen confidence in interpreting CSCR comparisons.

[W3]: HRF includes parse failures counted by SCR, limiting the disentanglement of routing and syntax errors.

[A3]: We thank the reviewer for the comment on HRF and SCR.We would like to clarify their distinct roles and the additional information provided by HRF.HRF is designed to provide information beyond SCR by additionally counting tool-selection errors among parseable outputs.Reporting both therefore distinguishes models with similar parsing reliability but different routing performance. The suggested conditional HRF measures routing errors among parseable outputs, while the current HRF includes parse failures. We will clarify this overlap and replace "disentangled metrics" with "complementary diagnostic metrics" throughout the paper and explicitly describe their dependencies.Following the reviewer's suggestion, we report conditional HRF, computed as: HRFparseable=HRF−SCR1−SCR.\mathrm{HRF}_{\mathrm{parseable}} = \frac{\mathrm{HRF} - \mathrm{SCR}}{1 - \mathrm{SCR}}. Together with SCR, it separates parsing failures from routing errors among parseable outputs more directly. The tables below report conditional HRF derived from our current results:

Table 3: Conditional routing error derived from the reported main-table results.

BackboneMethodHRF ↓SCR ↓HRF | parseable ↓
Qwen3-8BSFT0.16550.15940.0073
Qwen3-8BEWC0.03680.00920.0279
Qwen3-8BRecAdam0.03070.01530.0156
Qwen3-8BExp. Replay0.12180.11420.0086
Qwen3-8BCorpusBrain++0.21380.20690.0087
LLaMA-3.1-8B-InstructSFT0.28890.15020.1632
LLaMA-3.1-8B-InstructEWC0.41920.23220.2436
LLaMA-3.1-8B-InstructRecAdam0.46280.26130.2728
LLaMA-3.1-8B-InstructExp. Replay0.29040.21150.1001
LLaMA-3.1-8B-InstructCorpusBrain++0.34100.25590.1144

Table 4: Supplementary results using 10% training data with seeds 42 and 2027.

BackboneMethodHRF ↓SCR ↓HRF | parseable ↓
Qwen3-8BSFT0.0073 ± 0.00050.0011 ± 0.00050.0061 ± 0.0011
Qwen3-8BEWC0.0138 ± 0.00430.0069 ± 0.00330.0069 ± 0.0011
LLaMA-3.1-8B-InstructSFT0.0119 ± 0.00050.0057 ± 0.00050.0062 ± 0.0000
LLaMA-3.1-8B-InstructEWC0.0625 ± 0.01350.0502 ± 0.01570.0129 ± 0.0021

This analysis refines the interpretation of the main-table results: Qwen's EWC gains mainly reflect format stability, while LLaMA's EWC and RecAdam still exhibit higher routing errors than SFT. LLaMA's replay methods improve conditional routing despite worse overall HRF. And in the two-seed experiments, SFT and EWC have similar conditional errors on Qwen, but EWC shows worse results on LLaMA. We will note in the main text that this conditional analysis can complement the existing metrics.

Reference

[1]ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs ICLR 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.