acceptodds
Under review as a conference paper at ICLR 2027

Beyond Corrected Actions: Training LLM Agents on Textual Critiques

Abstract

Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain *why* specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose **C**ritique-Augmented **A**gentic **F**ine-**T**uning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action *and* a textual critique explaining the student's mistake—which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains -Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning (RL) than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agents, Conversational Agents, Training, Critique Models, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RPVPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on agent fine-tuning and proposes a new data construction strategy. Specifically, it leverages a more capable model to generate correct actions and provide critiques of the base model’s incorrect actions, using both to fine-tune the target model. Experiments conducted on four models across two test sets demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. This paper focuses on agent fine-tuning, aiming to better leverage the information provided by stronger models to improve the fine-tuning process.

  2. The paper presents a comparative analysis to investigate the contributions of different components of the proposed method to overall performance gains.

Critical weaknesses
  1. The experimental setup is not described in sufficient detail. For example, the paper does not specify the size of the training dataset, whether the models' thinking mode is enabled, the maximum number of output tokens, or the maximum number of interaction rounds. The omission of these details hinders reproducibility and makes it difficult to assess the reliability of the reported results.

  2. The experimental comparisons are insufficient to establish the effectiveness of the proposed method. In particular, the authors do not compare CAFT against SFT on complete, correct trajectories directly generated by a stronger model using detailed prompts and privileged info. Instead, they use trajectories in which the base model’s sampled actions are corrected by a stronger model, whereas direct trajectory distillation is a more common and practical baseline. According to the results in Table 3, directly distilling trajectories from a stronger model already yields better performance on Gemma 4 than the gains achieved by CAFT. This raises concerns about the added value of the proposed method and whether its advantages over simpler alternatives have been sufficiently demonstrated.

  3. The proposed method may have limited generalizability. As discussed in Section 3.3, CAFT’s performance gains appear to stem primarily from the use of privileged information. However, obtaining sufficient privileged information can be challenging in many practical scenarios. Conversely, when such information is readily available, direct distillation may already be sufficient, potentially reducing the need for CAFT. Furthermore, the experiments focus primarily on the Gemma 4 model family, leaving the method’s applicability to other model families insufficiently validated.

  4. The practical value of the proposed method remains unclear. As shown in Table 1, the performance gains diminish as model size increases, suggesting that the method may offer limited benefits for larger models. This trend raises concerns about its scalability and cost-effectiveness, as well as whether it can deliver meaningful improvements in real-world applications.

Questions for authors
  1. Does the model use thinking mode during training? If not, the experimental setup may not reflect common use cases. If it does, the strong model may incorporate information from privileged data into its reasoning traces that is unavailable to the base model. This introduces potentially misleading supervision during fine-tuning and may lead to undesirable behaviors.

  2. In the third row of Table 1, the value “69.4” appears twice, but only one instance is bolded.

3 comments

We thank the reviewer for their detailed feedback and for recognizing we give a comparative analysis of the different components of our proposed methods.

[W2]: The experimental comparisons are insufficient to establish the effectiveness of the proposed method. [...] The authors do not compare CAFT against SFT on complete, correct trajectories directly generated by a stronger model using detailed prompts and privileged info.

Following your suggestion, Appendix E.4 now includes an additional experiment on the τ2\tau^2 dataset to test whether using privileged information during expert trajectory generation can further improve performance. Specifically, we provided the generator with the exact same privileged information given to the CAFT critique expert via the system prompt. Our results show that this addition does not noticeably change the performance of Expert Distillation. We hypothesize this is because privileged information is more useful for verifying an action than for generating one. In Expert Distillation, the expert may rely on information unavailable to the student, producing trajectories that are difficult for the student to learn from. In CAFT, the privileged information is instead used to judge an action the student actually took, and therefore stays grounded in the student's own context.

StudentStandard expertPrivileged expert
Gemma 4 E2B50.448.1
Gemma 4 E4B53.854.7
Nemotron 3 Nano 4B53.353.1

[W2,W3]: According to the results in Table 3, directly distilling trajectories from a stronger model already yields better performance on Gemma 4 than the gains achieved by CAFT. This raises concerns about the added value of the proposed method. [...] Direct distillation may already be sufficient, potentially reducing the need for CAFT.

While Expert Distillation slightly outperforms CAFT on the τ2\tau^2 domain for Gemma 4, as noted in the paper, CAFT can still have complementary benefits. Namely, raining using a mixture of both CAFT and Expert Distillation achieve the best performance for both Gemma 4 models (cf. Table 3). Furthermore, CAFT does in fact outperform expert distillation on the WebShop domain, and also provides broader utility by providing a stronger initialization for downstream reinforcement learning (RL) compared to Expert Distillation.

Finally, CAFT offers a key advantage over Expert Distillation: lower training-data generation costs when using API-based expert models. Model providers typically price output tokens much more expensive than input tokens (e.g., around five times the price for Anthropic). While both methods pay the same per-token input price, they differ substantially in their output-token costs. In Expert Distillation, every token of the expert-generated trajectories, including actions and reasoning, is billed at the higher output-token rate. In contrast, CAFT incurs output-token costs only for short correctness verdicts at each step, and critiques and corrections for steps flagged as incorrect (20–35% in our setting). By requiring fewer expensive output tokens, CAFT has a reduced cost for generating training data.

[W3]: CAFT’s performance gains appear to stem primarily from the use of privileged information. However, obtaining sufficient privileged information can be challenging in many practical scenarios.

We first want to underscore that even without privileged information our method can still improve upon the base model for all tested settings (e.g., for Nemotron Nano 4B we see Base 47.8 →\to Unprivileged 57.7 →\to Privileged 60.3), thus suggesting the method can still be used when privileged information is not present. Furthermore, for our main experiment of Table 1, we note that the privileged information is available to the baselines too. Therefore it does not explain the further benefits of CAFT beyond those methods.

We similarly argue that obtaining sufficient privileged information does not pose a significant practical limitation. In fact, the availability of such information is a well-established assumption underlying numerous existing methods for LLM training (e.g., OPSD, SDFT, LEAP and related approaches), which explicitly leverage training-time information such as expert demonstrations, privileged environment states, or task-specific evaluation criteria that are unavailable to the student at inference time.

[W4]: The practical value of the proposed method remains unclear. As shown in Table 1, the performance gains diminish as model size increases. [...] This trend raises concerns about its scalability and cost-effectiveness, as well as whether it can deliver meaningful improvements in real-world applications.

We first note the smaller gains for Gemma 4 31B should not be interpreted as evidence that CAFT becomes less effective with model scale. Our experiments use Gemma 4 31B as the expert across all student models. Therefore, as student capability increases, the teacher–student capability gap narrows and diminishing performance gains is to be expected. In particular, the 31B setting is qualitatively different as the expert and student are the same model, making it a self-critique setting.

We acknowledge that evaluating larger students with correspondingly stronger teachers would further characterize CAFT's scalability and cost-effectiveness. However, more broadly, the main value of our work is in showing that an agent can be improved by training it on natural-language critiques of its own mistakes, beyond what corrected actions alone provide – which we believe our experiments do convincingly show. Furthermore, unlike approaches that rely on generating complete expert trajectories, CAFT can directly leverage existing student-generated trajectories to construct targeted supervision for the errors the student actually encounters. This is particularly relevant to agent training, where large amounts of interaction trajectories from the student model may already be available (e.g., an enterprise setting that wants to improve an internal model that is already deployed).

[W1]: The experimental setup is not described in sufficient detail. [...] The paper does not specify the size of the training dataset, whether the models' thinking mode is enabled, the maximum number of output tokens, or the maximum number of interaction rounds.

Additional experiment details are now present in Appendix C.2. To clarify the crucial points, thinking mode is enabled and we use 1,664 trajectories and 1,600 trajectories respectively for τ2\tau^2 and WebShop respectively. Also we plan to release the full code for reproducibility once we receive approval from our institution.

[Q1]: Does the model use thinking mode during training? If it does, the strong model may incorporate information from privileged data into its reasoning traces that is unavailable to the base model.

Yes thinking is enabled, we have clarified this in the appendix now. For the possibility of privileged info leakage, we have checked this by randomly subsampling 60 corrected steps from each τ2\tau^2 and Webshop setting over each model. We then used Claude 5.5 Opus to manually audit each sampled step for leakage of gold privileged information. It flagged 3.3% and 1.1% steps respectively, suggesting that leakage rarely occurs. Furthermore, we want to emphasize that no privileged information is available at inference time, thus any leakage would teach the student to rely on unavailable information, which should hurt evaluation performance, not inflate it.

[Q2]: In the third row of Table 1, the value “69.4” appears twice, but only one instance is bolded.

Thanks for catching this, we have fixed it in the paper.


Final Note: We appreciate the in-depth feedback and questions about the practical value of our method. We hope our comments and additional experiments address the questions and would appreciate the reviewer reconsidering their assessment.

AuthorsOpenReview ↗

Review Response Remainder

Dear Reviewer,

Thank you again for the time reviewing our paper. Given the discussion period ends soon, we would greatly appreciate hearing whether you have any remaining concerns. And If our responses have addressed your concerns, we would be really grateful if you would consider updating your rating accordingly.

Thanks,
The Authors

Reviewer_8xUwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether textual critiques of an agent's mistakes provide useful training supervision beyond corrected actions. CAFT collects student rollouts and uses a privileged expert to identify mistaken steps, explain them, and supply corrected actions. The student is fine-tuned on these annotations and acts without the expert at test time. Across four student models and the τ²-Bench Airline+Retail and WebShop settings, CAFT outperforms the paper's action-distillation baseline in all eight comparisons.

Critical strengths

The comparison with Action Distillation is well chosen for the paper's central question: it uses the same student trajectories and expert annotations while changing how mistaken steps are supervised. The critique-only and correction-only variants add useful evidence that the critique contributes to the observed gains (Sections 3.1–3.2, Table 1). The experiments on privileged information, further CAFT training, and RL initialization also show where the approach appears useful and where its benefits vary.

Critical weaknesses
  1. The comparison with prior correction-based methods could be more complete. Action Distillation provides a controlled comparison with corrected-action supervision, but methods such as LEAP also train on expert-corrected reasoning. Table 1 therefore does not establish whether textual critiques offer benefits beyond this richer form of supervision. A controlled baseline using corrected reasoning and actions would help distinguish the value of explaining an error from that of providing improved reasoning. If such a comparison is not feasible, the scope of the conclusions should be made explicit (Sections 3.1 and 4).

  2. The smaller gains in Table 1 need an indication of uncertainty. For example, the WebShop result for Gemma 4 31B rises from 59.6% with Action Distillation to 61.0% with CAFT. Sixteen rollouts per task help account for variation between rollouts, but the paper does not show how sensitive these differences are to the evaluated tasks. Paired, task-level confidence intervals would help readers assess the claim that CAFT improves performance across all eight settings (Section 3.1, Table 1).

  3. The WebShop expert-distillation result is difficult to interpret without more detail about the baseline. Expert Distillation lowers the success rate of all three students below their base performance (Table 3). The authors report observing repeated searches without a purchase, which is a plausible explanation, but the paper does not clearly specify the number of expert trajectories used or whether unsuccessful trajectories were excluded. These choices matter for interpreting the comparison. Reporting the data construction and a simple statistic such as purchase rate or episode length would make the result easier to assess (Section 3.4, Appendix C.1).

  4. A training-data filter should be included in the main method description. Algorithm 1 describes constructing samples from every annotated turn, whereas Appendix B.1 states that the final τ²-Bench runs retain only steps judged critical. Since this changes which actions contribute to training, the paper should state where the filter enters the procedure and report how many correct and mistaken steps remain. It would also help to clarify whether the same filter is applied to the Action Distillation baseline (Section 2, Appendix B.1).

Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for the feedback and for acknowledging that our experiments are well chosen and show where the approach is useful.

[W1]: The comparison with prior correction-based methods could be more complete. [...] A controlled baseline using corrected reasoning and actions would help distinguish the value of explaining an error from that of providing improved reasoning.

We have added your suggested LEAP-inspired corrected reasoning baseline in Appendix E.4. While this method improves over the base model, we find that its performance in general fluctuates around that of Action Distillation, outperforming it in some configurations and underperforming in others, and is uniformly inferior to CAFT. This suggests that providing a critique of the incorrect action as in CAFT has benefits beyond that of simply correcting the reasoning.

EnvironmentStudentAction DistillationCorrected reasoningCAFT
τ² ARGemma 4 E2B35.929.146.6
τ² ARGemma 4 E4B49.749.152.1
τ² ARNemotron 3 Nano 4B52.153.160.3
WebShopGemma 4 E2B54.645.365.3
WebShopGemma 4 E4B59.259.862.6
WebShopNemotron 3 Nano 4B31.651.258.9

[W2]: The smaller gains in Table 1 need an indication of uncertainty. [...] Paired, task-level confidence intervals would help readers assess the claim that CAFT improves performance across all eight settings (Section 3.1, Table 1).

We now report 95% paired confidence intervals for the main Table 1 experiment in Appendix E.4 using n=32 rollouts freshly sampled from each checkpoint. Across nearly all configurations, CAFT has a statistically significant improvement over Action Distillation. While the Gemma 31B models do not reach statistical significance, they remain directionally consistent and aligned with the broader findings. We have updated the main text to more accurately contextualize the performance margins at the 31B scale.

EnvironmentStudent95% CI (CAFT − AD)
τ² ARGemma 4 E2B[4.3, 11.9]
τ² ARGemma 4 E4B[0.2, 3.7]
τ² ARGemma 4 31B[-0.8, 3.2]
τ² ARNemotron 3 Nano 4B[2.5, 10.8]
WebShopGemma 4 E2B[4.6, 14.8]
WebShopGemma 4 E4B[1.0, 6.7]
WebShopGemma 4 31B[-1.6, 4.2]
WebShopNemotron 3 Nano 4B[22.4, 30.4]

[W3]: The WebShop expert-distillation result is difficult to interpret without more detail about the baseline.[...] The paper does not clearly specify the number of expert trajectories used or whether unsuccessful trajectories were excluded.

We use a total of 1,600 trajectories for expert distillation (100 tasks, 16 rollouts per task), which is the same as the number of trajectories used over our other experiments involving CAFT and Action Distillation. In our original experiments, we included unsuccessful trajectories. Table 8 now includes results from additional experiments where only successful trajectories are used.

For WebShop specifically, upon further tuning the Expert Distillation baseline (namely by relaxing a forced thinking mode we had enforced for Gemma 31B when generating trajectories over WebShop specifically) and applying success filtering, we find that its performance can increase over that of the base model. Nevertheless performance across all settings still underperforms CAFT, so our main conclusions still hold true (e.g., for Gemma E2B: Base 43.1 →\to ED 52.9 →\to CAFT 65.3). We have updated Table 3 and the main text accordingly.

StudentBaseExpert DistillationCAFT
Gemma 4 E2B43.152.965.3
Gemma 4 E4B42.153.362.6
Nemotron 3 Nano 4B24.354.158.9

[W4]: A training-data filter should be included in the main method description. [...] It would also help to clarify whether the same filter is applied to the Action Distillation baseline (Section 2, Appendix B.1).

The training-data filter is now included in the main text. And to clarify, the same filter is also applied to Action Distillation.


Final Note: We thank the reviewer again for the review, and for suggesting useful baselines that strengthen our paper. We would appreciate the reviewer reconsidering their assessment given the new results.

AuthorsOpenReview ↗

Review Response Remainder

Dear Reviewer,

Thank you again for the time reviewing our paper. Given the discussion period ends soon, we would greatly appreciate hearing whether you have any remaining concerns. And If our responses have addressed your concerns, we would be really grateful if you would consider updating your rating accordingly.

Thanks,
The Authors

Reviewer_DwSqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Critique-Augmented Agentic Fine-Tuning (CAFT). An expert model reviews student-generated trajectories and provides a textual critique and a corrected action for each identified mistake. The student is then trained to generate these targets, while keeping the original mistake in context. The expert and critique prompts are only used during training. Experiments cover four student models on τ²-Bench Airline+Retail and WebShop, with additional studies on privileged information, repeated training, and RL initialization.

Critical strengths
  1. The main comparison is well controlled. The methods share student trajectories and expert annotations. CAFT-Correction keeps the mistaken action in context but removes the critique target, while CAFT-Critique removes the explicit corrected-action target. These comparisons provide useful evidence for the value of critique supervision.

  2. The improvements are consistent across the tested settings. CAFT outperforms action distillation in all eight student–environment pairs, with an average gain of 8.35 percentage points. Critique-only training also outperforms correction-only training in every pair, by about 4.96 points on average.

  3. The method improves ordinary agent performance without an inference-time expert. The trained policy acts directly during evaluation. The experiments also show that a model can improve using critiques from the same model with privileged information, and provide encouraging initial results for using CAFT before RL.

Critical weaknesses
  1. Independent training-seed variation and confidence intervals are not reported. The 16 rollouts per task do not establish stability across independently trained models. This matters for smaller gains, such as 59.6% to 61.0% on WebShop. The RL warm-start result is also limited to one model–environment setting.

  2. Expert distillation reduces WebShop success to 15.0–19.1%, below the base models (Table 3). The reported repeated-search behavior provides a partial explanation, but further robustness checks and clarification of successful-trajectory filtering are needed to assess this baseline comparison.

Questions for authors
  1. Does CAFT retain its advantage when target-token budgets and loss weights are matched? How does it compare with training on explanations of the correct action?

  2. Can you report results across independent training seeds and task-level confidence intervals, especially for the smaller gains and the RL warm-start experiment?

  3. Were only successful expert trajectories used for distillation? Does the WebShop performance drop remain after tuning the baseline and checking its training-data composition?

3 comments

We thank the reviewer for the thoughtful feedback and for recognizing that (i) our experiments are well controlled and (ii) our method can improve the agent without requiring additional experts at inference-time.

[W1,Q2]: Independent training-seed variation and confidence intervals are not reported. [...] The RL warm-start result is also limited to one model–environment setting.

We now report 95% paired confidence intervals for the main Table 1 experiment in Appendix E.4. Across nearly all configurations, CAFT has a statistically significant improvement over Action Distillation. While the Gemma 31B models do not reach statistical significance, they remain directionally consistent and aligned with the broader findings.

EnvironmentStudent95% CI (CAFT − AD)
τ² ARGemma 4 E2B[4.3, 11.9]
τ² ARGemma 4 E4B[0.2, 3.7]
τ² ARGemma 4 31B[-0.8, 3.2]
τ² ARNemotron 3 Nano 4B[2.5, 10.8]
WebShopGemma 4 E2B[4.6, 14.8]
WebShopGemma 4 E4B[1.0, 6.7]
WebShopGemma 4 31B[-1.6, 4.2]
WebShopNemotron 3 Nano 4B[22.4, 30.4]

Towards the concern regarding training seed variation, we re-trained Nemotron 4B on WebShop with CAFT using an additional random seed. The new seed yielded an accuracy of 58.1, closely matching our original result of 58.9 suggesting that cross-seed variance is minimal relative to the performance gaps between methods. Due to compute constraints, we have not run additional RL experiments, but have updated the main text of the paper to more strongly emphasize that the broader RL experiments reflect a single-seed study.

[W2,Q3]: Were only successful expert trajectories used for distillation? Does the WebShop performance drop remain after tuning the baseline and checking its training-data composition?

In our original experiments, we keep all trajectories rather than only filtering to the successful ones. We made this design choice as over experiments on τ2\tau^2 we find that success-filtering generally degrades performance slightly. We have added Table 8 which details these results along with the corresponding additionally run experiments on WebShop.

For WebShop specifically, upon further tuning the Expert Distillation baseline (namely by relaxing a forced thinking mode we had enforced for Gemma 31B when generating trajectories over WebShop specifically) and applying success filtering, we find that its performance can increase over that of the base model. Nevertheless performance across all settings still underperforms CAFT, so our main conclusions still hold true (e.g., for Gemma E2B: Base 43.1 →\to ED 52.9 →\to CAFT 65.3). We have updated Table 3 and the main text accordingly.

StudentBaseExpert DistillationCAFT
Gemma 4 E2B43.152.965.3
Gemma 4 E4B42.153.362.6
Nemotron 3 Nano 4B24.354.158.9

[Q1]: Does CAFT retain its advantage when target-token budgets and loss weights are matched? How does it compare with training on explanations of the correct action?

We now include both experiments in Appendix E.4 (cf. Table 10 for the results). To match the loss weighting, we upweight the originally flagged incorrect turns in Action Distillation so they receive the same share of the training signal as the corresponding longer targets in CAFT. To test training on explanations of the correct actions, we implement a LEAP-inspired corrected reasoning baseline (as proposed by reviewer 8xUw). As can be seen in the results, while both methods improve on the base model, their performance is still below that of CAFT.

EnvironmentStudentAction DistillationCorrected reasoningBalanced weightingCAFT
τ² ARGemma 4 E2B35.929.135.546.6
τ² ARGemma 4 E4B49.749.144.252.1
τ² ARNemotron 3 Nano 4B52.153.154.560.3
WebShopGemma 4 E2B54.645.350.465.3
WebShopGemma 4 E4B59.259.850.262.6
WebShopNemotron 3 Nano 4B31.651.256.158.9

Final Note: We thank the reviewer again, and hope these results clarify concerns about statistical significance and soundness of the baselines. If our responses have addressed your concerns, please kindly consider raising the score.

Reviewer_DwSqOpenReview ↗

maintain orignial rating

Thank you to the authors for the clarifications and additional experiments. After considering the response, I will maintain my original rating.

Reviewer_DwSqOpenReview ↗

maintain original rating

Thank you to the authors for the clarifications and additional experiments. After considering the response, I will maintain my original rating.

Comments on the paper

3 comments

Dear reviewers,

The authors have submitted their rebuttals. I would be grateful if you could take a moment to review them carefully and respond as appropriate. Thank you for your ongoing engagement and support.

Your AC

AuthorsOpenReview ↗

Gentle Remainder to Respond before Discussion Period Deadline (Tomorrow, 11/18)

Dear Reviewers,

Thank you for the time reviewing our paper. Given the discussion period ends tomorrow (11/18) and as we still haven't received your replies, we would be grateful if you could respond at your earliest convenience.

Thanks,
The Authors

AuthorsOpenReview ↗

Summary of New Rebuttal Experiments and Clarifications

Dear Area Chair and Reviewers,

As the discussion period ends, here is a summary of our contributions and the new experiments added during the rebuttal period to aid with your assessment of our paper.

We introduce CAFT, which trains multi-turn LLM agents on natural language critiques explaining why student actions are incorrect alongside corrected actions. Reviewers highlighted our well-chosen controlled experimental design (DwSq, 8xUw, RPVP), consistent gains over Action Distillation (DwSq, 8xUw), and experiments for showing potential effectiveness for self-critique and RL warm-starting (DwSq, 8xUw).

Key experiments and clarifications from our rebuttal:

  • Richer Action Distillation Baselines (DwSq, 8xUw): We added the Balanced Weighting (matching CAFT's loss share on flagged turns) and Corrected Reasoning (LEAP-style) Action Distillation baselines as suggested by the reviewers. CAFT consistently outperforms both across all tested models and environments.
  • Statistical Significance and Training Seed Stability (DwSq, 8xUw): We added 95% paired confidence intervals, confirming statistically significant gains over Action Distillation across all smaller student configurations (and directionally positive results in the 31B self-critique regime). An additional training seed on WebShop (Nemotron 4B) scored 58.1% vs. 58.9% originally, indicating low cross-seed variance.
  • WebShop Expert Distillation Baseline Tuning (DwSq, 8xUw): After relaxing a forced thinking mode on Gemma 31B and filtering for successful trajectories (Table 3 & Table 8), Expert Distillation now improves over the base models. However CAFT still outperforms it across all settings and so our main conclusion still holds.
  • Privileged Information Expert Distillation (RPVP): We test whether privileged information can help Expert Distillation on τ2\tau^2 by providing the trajectory-generating expert with the exact same privileged information used in CAFT. Giving the expert privileged information yielded marginal gains over the non-privileged expert distillation counterpart. This suggests showing privileged information has a larger effect when used for verifying student actions than when used directly for trajectory generation.
  • Scalability of CAFT (RPVP): We clarify that the diminishing gains of CAFT as we increase the student model size reflects a narrowing teacher–student gap (since Gemma 31B is the fixed teacher across all runs) rather than a scaling limitation.
  • Practical Value of CAFT (RPVP): While the primary focus of our paper is to establish the benefits of using natural-language critiques during training to improve multi-turn agents, from a practical standpoint we note that CAFT (i) cuts expensive API output-token costs compared to full-trajectory Expert Distillation by generating full critiques and corrections only on the 20–35% of steps flagged as mistakes, (ii) can be jointly trained with Expert Distillation trajectories in a data mixture to achieve even stronger performance than either method alone, and (iii) can directly leverages and utilize existing student trajectories that are commonly available in deployed enterprise settings.

Thank you for your time and consideration,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.