- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on agent fine-tuning and proposes a new data construction strategy. Specifically, it leverages a more capable model to generate correct actions and provide critiques of the base model’s incorrect actions, using both to fine-tune the target model. Experiments conducted on four models across two test sets demonstrate the effectiveness of the proposed approach.
Critical strengths
-
This paper focuses on agent fine-tuning, aiming to better leverage the information provided by stronger models to improve the fine-tuning process.
-
The paper presents a comparative analysis to investigate the contributions of different components of the proposed method to overall performance gains.
Critical weaknesses
-
The experimental setup is not described in sufficient detail. For example, the paper does not specify the size of the training dataset, whether the models' thinking mode is enabled, the maximum number of output tokens, or the maximum number of interaction rounds. The omission of these details hinders reproducibility and makes it difficult to assess the reliability of the reported results.
-
The experimental comparisons are insufficient to establish the effectiveness of the proposed method. In particular, the authors do not compare CAFT against SFT on complete, correct trajectories directly generated by a stronger model using detailed prompts and privileged info. Instead, they use trajectories in which the base model’s sampled actions are corrected by a stronger model, whereas direct trajectory distillation is a more common and practical baseline. According to the results in Table 3, directly distilling trajectories from a stronger model already yields better performance on Gemma 4 than the gains achieved by CAFT. This raises concerns about the added value of the proposed method and whether its advantages over simpler alternatives have been sufficiently demonstrated.
-
The proposed method may have limited generalizability. As discussed in Section 3.3, CAFT’s performance gains appear to stem primarily from the use of privileged information. However, obtaining sufficient privileged information can be challenging in many practical scenarios. Conversely, when such information is readily available, direct distillation may already be sufficient, potentially reducing the need for CAFT. Furthermore, the experiments focus primarily on the Gemma 4 model family, leaving the method’s applicability to other model families insufficiently validated.
-
The practical value of the proposed method remains unclear. As shown in Table 1, the performance gains diminish as model size increases, suggesting that the method may offer limited benefits for larger models. This trend raises concerns about its scalability and cost-effectiveness, as well as whether it can deliver meaningful improvements in real-world applications.
Questions for authors
-
Does the model use thinking mode during training? If not, the experimental setup may not reflect common use cases. If it does, the strong model may incorporate information from privileged data into its reasoning traces that is unavailable to the base model. This introduces potentially misleading supervision during fine-tuning and may lead to undesirable behaviors.
-
In the third row of Table 1, the value “69.4” appears twice, but only one instance is bolded.
3 comments
[W4]: The practical value of the proposed method remains unclear. As shown in Table 1, the performance gains diminish as model size increases. [...] This trend raises concerns about its scalability and cost-effectiveness, as well as whether it can deliver meaningful improvements in real-world applications.
We first note the smaller gains for Gemma 4 31B should not be interpreted as evidence that CAFT becomes less effective with model scale. Our experiments use Gemma 4 31B as the expert across all student models. Therefore, as student capability increases, the teacher–student capability gap narrows and diminishing performance gains is to be expected. In particular, the 31B setting is qualitatively different as the expert and student are the same model, making it a self-critique setting.
We acknowledge that evaluating larger students with correspondingly stronger teachers would further characterize CAFT's scalability and cost-effectiveness. However, more broadly, the main value of our work is in showing that an agent can be improved by training it on natural-language critiques of its own mistakes, beyond what corrected actions alone provide – which we believe our experiments do convincingly show. Furthermore, unlike approaches that rely on generating complete expert trajectories, CAFT can directly leverage existing student-generated trajectories to construct targeted supervision for the errors the student actually encounters. This is particularly relevant to agent training, where large amounts of interaction trajectories from the student model may already be available (e.g., an enterprise setting that wants to improve an internal model that is already deployed).
[W1]: The experimental setup is not described in sufficient detail. [...] The paper does not specify the size of the training dataset, whether the models' thinking mode is enabled, the maximum number of output tokens, or the maximum number of interaction rounds.
Additional experiment details are now present in Appendix C.2. To clarify the crucial points, thinking mode is enabled and we use 1,664 trajectories and 1,600 trajectories respectively for and WebShop respectively. Also we plan to release the full code for reproducibility once we receive approval from our institution.
[Q1]: Does the model use thinking mode during training? If it does, the strong model may incorporate information from privileged data into its reasoning traces that is unavailable to the base model.
Yes thinking is enabled, we have clarified this in the appendix now. For the possibility of privileged info leakage, we have checked this by randomly subsampling 60 corrected steps from each and Webshop setting over each model. We then used Claude 5.5 Opus to manually audit each sampled step for leakage of gold privileged information. It flagged 3.3% and 1.1% steps respectively, suggesting that leakage rarely occurs. Furthermore, we want to emphasize that no privileged information is available at inference time, thus any leakage would teach the student to rely on unavailable information, which should hurt evaluation performance, not inflate it.
[Q2]: In the third row of Table 1, the value “69.4” appears twice, but only one instance is bolded.
Thanks for catching this, we have fixed it in the paper.
Final Note: We appreciate the in-depth feedback and questions about the practical value of our method. We hope our comments and additional experiments address the questions and would appreciate the reviewer reconsidering their assessment.
Review Response Remainder
Dear Reviewer,
Thank you again for the time reviewing our paper. Given the discussion period ends soon, we would greatly appreciate hearing whether you have any remaining concerns. And If our responses have addressed your concerns, we would be really grateful if you would consider updating your rating accordingly.
Thanks,
The Authors
We thank the reviewer for their detailed feedback and for recognizing we give a comparative analysis of the different components of our proposed methods.
Following your suggestion, Appendix E.4 now includes an additional experiment on the τ2 dataset to test whether using privileged information during expert trajectory generation can further improve performance. Specifically, we provided the generator with the exact same privileged information given to the CAFT critique expert via the system prompt. Our results show that this addition does not noticeably change the performance of Expert Distillation. We hypothesize this is because privileged information is more useful for verifying an action than for generating one. In Expert Distillation, the expert may rely on information unavailable to the student, producing trajectories that are difficult for the student to learn from. In CAFT, the privileged information is instead used to judge an action the student actually took, and therefore stays grounded in the student's own context.
While Expert Distillation slightly outperforms CAFT on the τ2 domain for Gemma 4, as noted in the paper, CAFT can still have complementary benefits. Namely, raining using a mixture of both CAFT and Expert Distillation achieve the best performance for both Gemma 4 models (cf. Table 3). Furthermore, CAFT does in fact outperform expert distillation on the WebShop domain, and also provides broader utility by providing a stronger initialization for downstream reinforcement learning (RL) compared to Expert Distillation.
Finally, CAFT offers a key advantage over Expert Distillation: lower training-data generation costs when using API-based expert models. Model providers typically price output tokens much more expensive than input tokens (e.g., around five times the price for Anthropic). While both methods pay the same per-token input price, they differ substantially in their output-token costs. In Expert Distillation, every token of the expert-generated trajectories, including actions and reasoning, is billed at the higher output-token rate. In contrast, CAFT incurs output-token costs only for short correctness verdicts at each step, and critiques and corrections for steps flagged as incorrect (20–35% in our setting). By requiring fewer expensive output tokens, CAFT has a reduced cost for generating training data.
We first want to underscore that even without privileged information our method can still improve upon the base model for all tested settings (e.g., for Nemotron Nano 4B we see Base 47.8 → Unprivileged 57.7 → Privileged 60.3), thus suggesting the method can still be used when privileged information is not present. Furthermore, for our main experiment of Table 1, we note that the privileged information is available to the baselines too. Therefore it does not explain the further benefits of CAFT beyond those methods.
We similarly argue that obtaining sufficient privileged information does not pose a significant practical limitation. In fact, the availability of such information is a well-established assumption underlying numerous existing methods for LLM training (e.g., OPSD, SDFT, LEAP and related approaches), which explicitly leverage training-time information such as expert demonstrations, privileged environment states, or task-specific evaluation criteria that are unavailable to the student at inference time.