- Rating
- 3(+2)
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces CAFTBench, a static benchmark for continual fine-tuning of tool-calling LLMs. A model is first fine-tuned on a stream of about 30K examples over about 260 tools, presented tool by tool. It is then fine-tuned again on "upgraded" APIs, where each function is renamed with a v1/v2 suffix and gets a new required parameter. Fixed probes then score routing error (HRF), unparseable outputs (SCR), composition quality from an LLM judge (CSCR), and spurious tool calls on no-tool prompts (TOI) after the first phase, and errors on the upgraded APIs (NTHI) after the second. Comparing sequential SFT, EWC, RecAdam, Experience Replay and CorpusBrain++ on Qwen3-8B and LLaMA-3.1-8B, plus two more backbones in the appendix, the authors conclude that the trade-offs depend on the backbone and that existing continual-learning methods are not sufficient.
Critical strengths
The problem of evolving APIs is real, and the failure taxonomy is useful. Splitting errors into format collapse, wrong tool, missing upgraded arguments and spurious calls (Figure 4) is more informative than a single accuracy number. The static protocol and the released code also make the benchmark cheap to run.
Critical weaknesses
-
The paper never shows that fine-tuning is needed. Updating the schema in the prompt may be enough. The motivation is that tool changes force retraining. Yet at test time, CAFTBench already puts the schemas in the prompt: the target tool plus three distractors, and the new schema for upgraded probes (Section 4). The upgrade itself is mild, a rename plus one required parameter. This is exactly the setting in-context function calling handles, and it is what BFCL evaluates. Even so, the paper reports no result without fine-tuning, neither for the base backbones nor for a frontier commercial model given the updated schema. The introduction dismisses prompting because of context length and cost, but never tests it. The paper's own numbers suggest the prompt does most of the work:
- On Olmo and Ministral, plain SFT has under 1% HRF and about 3% NTHI.
- On Qwen, almost all errors are parse failures of the custom
Call: tool(args)format (Figure 4). Fine-tuning mainly teaches, or breaks, a format that the base model's native tool template might not need.
Prior work on evolving tools (ToolEVO / ToolQA-D) studies the harder case, where the model sees only outdated documentation. There, fine-tuning on static APIs is what made models brittle. The key test is missing: does a model fine-tuned on the old tools ignore the new schema in its prompt? Without it, the benchmark may be measuring a problem that a schema update already solves.
-
The setup does not measure forgetting, and the method comparison is confounded.
- HRF is an endpoint error rate. There is no pre-fine-tuning reference, no joint-training upper bound, and no intermediate checkpoints. The original probes are never re-scored after the upgrade phase, so whether upgrading overwrites the original tools, the most direct forgetting case here, is never measured.
- Each result comes from a single run, and the methods use different learning rates and batch sizes. The settings for the baseline SFT are not given, and hyperparameters were tuned only on Qwen. The headline backbone-dependence finding may just reflect an EWC tuned for Qwen that does not suit LLaMA.
Questions for authors
- What are the five metrics for the base backbones (with their native tool templates) and for a frontier commercial model, given the same schemas in the prompt with no fine-tuning?
- On the upgraded probes, with the new schema in the prompt, how does the model fine-tuned only on the old tools () score? Is there any evidence that fine-tuned knowledge overrides an updated in-context schema?
- Is there a realistic deployment setting where the schema cannot be given in the prompt? If so, why does the benchmark provide it?
- What are HRF and SCR on the original probes after the upgrade phase (), and under joint training?
- Table 4 reports very different numbers from Table 2 for the same Qwen configurations, for example EWC TOI of about 0.9 against 0. Which runs are correct, and how many seeds were used?
8 comments
[W3]: Qwen results suggest prompts drive tool use, while fine-tuning mainly affects formatting.
[A3]: We respectfully distinguish the observed distribution of errors from the attribution of tool-use performance to prompting versus fine-tuning.In Section 5.2, we identify format preservation as a major source of performance differences across methods on Qwen. Reporting SCR and the error decomposition in Figure 4 makes this distinction clear, rather than attributing all improvements to better tool selection or reasoning. However, this observation comes from fine-tuned models. The predominance of parse failures does not isolate the contribution of the supplied schemas from that of training.Likewise, the fact that formatting accounts for most differences across methods does not establish that fine-tuning mainly teaches or breaks a format. Other reported failures, including wrong-tool selections among parseable LLaMA outputs and tool over-reliance, further illustrate the benchmark’s broader diagnostic scope.Distinguishing these effects is part of the benchmark's diagnostic contribution: it shows how the observed effects of continual fine-tuning methods differ across metrics and backbones. Regarding the native-template comparison in Q1, we adopt the same canonical call format across backbones to provide a consistent output and scoring convention.Our comparison evaluates continual fine-tuning methods under this shared format, rather than under each model's native tool template.
[W4]: The benchmark setup does not measure forgetting,only endpoint errors.
[A4]: As explicitly defined in the paper, HRF measures historical-tool routing performance after sequential tool learning.This endpoint evaluation answers how reliably the resulting model can invoke previously encountered tools after sequential learning.Together with SCR and the error decomposition, it provides a multidimensional and diagnostic basis for comparing continual fine-tuning methods.
[W5]: Potential confounding from training configurations and randomness
[A5]: We thank the reviewer for raising this concern.We would like to clarify that the Qwen hyperparameter sweep reported in Appendix D.1 is a supplementary sensitivity analysis. In the main experiments, our settings follow the recommendations in the original baseline papers; for those without explicit guidance, we used commonly adopted values from prior experience.Crucially, we kept the same configurations for both backbones.The SFT baseline uses the same learning rate() and effective batch size(512) as the replay methods, together with the shared training settings in Table 3.We will explicitly list the SFT settings in Table 3.The relevant settings are detailed below:
Learning Rates and Batchsize: Based on prior training experience with the respective methods, we set for regularization methods(EWC, RecAdam) and for replay methods (Experience Replay, CorpusBrain++) to ensure our evaluations remain fair by allowing each paradigm to operate within an appropriate learning rate interval.The smaller effective batch size(128) for EWC and RecAdam was chosen to accommodate the chunk-based training and consolidation schedule described in Appendix B, while SFT and the replay baselines used 512.
Regularization Strength =400:We set = 400 by aligning with the recommended hyperparameter settings established in the original EWC work[1].
Buffer capacity (N=1500):This setup directly adheres to the empirical buffer sizing principles established by the original work[2].
We therefore interpret the observed interactions as differences across backbones under the reported method-specific configurations.
[W6]: Table 2 and Table 4 report different Qwen TOI results.
[A6]: Regarding Q5 (Table 2/Table 4 consistency and training seeds):We apologize for the unclear reporting of the experimental environments. Evaluation results and their reproducibility should be interpreted in the context of the hardware and software environment. Table 2 was obtained on A800 GPUs, while resource constraints required the supplementary Table 4 experiments to run on H20 GPUs.Due to infrastructure incompatibilities, we switched to hiyouga/llamafactory:0.9.5, although the training and evaluation code remained unchanged. These environment differences may have contributed to the discrepancy. However, the exact original environment was not fully preserved, preventing definitive attribution. We retain Table 2 as the main reported results and will clarify that Table 4 contains separate sensitivity runs on a different platform. Our supplementary multi-seed experiments provide additional evidence in the tested reduced-data setting. The data and code are publicly available so that others can reproduce the protocol in their own environments.
Reference
[1]Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017
[2]Experience Replay for Continual Learning. NeurIPS 2019
We appreciate the time and effort Reviewer b4tp devoted to reviewing our manuscript. However, we respectfully note that the current review reflects a certain degree of misunderstanding of our work.We earnestly ask the reviewer to read our clarifications below and reconsider the evaluation of our paper.
We would like to emphasize that the primary objective of our work is to evaluate continual fine-tuning methods in training-based scenarios. In this context, we use the term "fine-tuning" in a broad sense to refer to methods that involve a training process, rather than aiming to evaluate the broader spectrum of continual learning methods, which encompasses both training-based and training-free approaches.We fully agree with the reviewer that continual learning includes both fine-tuning-based and fine-tuning-free methods. However, as explicitly stated in the title of our paper, our benchmark is specifically designed to evaluate continual fine-tuning methods, rather than continual learning methods in the broader sense. We respectfully ask the reviewer to reconsider their assessment in light of this clarification regarding the intended scope and objectives of our benchmark.
[W1-W2]: The paper does not establish the necessity of fine-tuning or demonstrate that old tool knowledge overrides updated schemas.
[A1-A2]: We thank the reviewer for the comments on our motivation and evaluation scope. We would like to clarify that CAFTBench is designed to evaluate continual fine-tuning methods through sequential training, as reflected in our paper's title. Our dataset and evaluation pipeline assess how different continual fine-tuning methods affect tool-calling behavior across original tool learning and subsequent updates. Our benchmark's objective is to evaluate such training procedures, rather than to establish that fine-tuning is necessary or superior to training-free adaptations. As we stated in the introduction, fine-tuning is positioned as a complementary approach alongside prompt-based methods. The reported training performance also does not identify whether errors arise from old knowledge overriding updated schemas. We distinguish the questions from the benchmark's stated evaluation objective below.
Regarding Q1 (untuned and frontier-model baselines): The low error rates cited by the reviewer were obtained after fine-tuning. They therefore do not establish the claim that "the prompt does most of the work". Our comparisons focus on how different continual fine-tuning methods perform under a shared sequential tool-learning protocol. We would like to point out that methods without weight updates do not undergo the sequential training process studied in our work. Thus untuned and frontier-model baselines would provide additional context, but would not substitute for this comparison.
Regarding Q2 (the pre-upgrade model with updated schemas): CAFTBench provides a unified sequential tool-learning protocol for comparing continual fine-tuning methods across original-tool learning and subsequent API-upgrade training.The API-upgrade phase extends the sequential learning process through further weight updates, allowing us to evaluate continual fine-tuning methods as tool specifications evolve. This comparison focuses on the outcome of the evaluated continual fine-tuning procedures; it neither assumes that schema updates alone must fail nor independently identifies the cause of calling errors. Our current upgrade evaluation focuses on comparing methods after upgrade training and does not include the setting of evaluating the previous phase with updated schemas but without further training.
Regarding Q3 (schema availability): In our benchmark's setting, we provide current schemas as part of the evaluation conditions and compare continual fine-tuning methods after sequential training. Under this setting, schema availability and continued training are compatible. Our motivation does not rely on a deployment setting in which schemas cannot be provided.