acceptodds
Under review as a conference paper at ICLR 2027

Training Advisors for LLM Agents from Task Outcomes

Abstract

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference demonstrations, Caddie learns from whether the agent ultimately succeeds after receiving the critic’s feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

Primary area:
reinforcement learning
Keywords:
llm, reinforcement learning, critic, advisor, agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GVABOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CADDIE, a generative critic trained to advise a frozen tool-using agent during an ongoing trajectory. Training samples an intermediate prefix, generates a group of critiques, evaluates each by the terminal success of the agent's continuation, and updates only the critic using DAPO. No reference critiques or step-level correctness labels are required, although a feedback-format constraint is enforced. At inference, the base agent can invoke the critic through a self-call interface. Separate Qwen3-4B critics trained for MuSiQue-attached, MuSiQue-wiki, and ALFWorld improve in-domain success; the MuSiQue critics are also evaluated with unseen base models and on DeepDive and customer-service tasks. The strongest improvements are approximately 25 percentage points in MuSiQue-attached and ALFWorld, while cross-model Wikipedia and some out-of-domain gains are considerably smaller or mixed.

Critical strengths

A simple training signal produces substantial improvements over useful controls. The method directly rewards critique-conditioned continuations rather than requiring reference critique data. Table 1 compares the trained critic against its prompted initialization and a much larger prompted critic, showing 46.88% versus 21.56% without a critic on MuSiQue-attached and 49.72% versus 23.79% on ALFWorld. Appendix E additionally shows improvement when the acting model has already been task-trained. These controls make the contribution more convincing than an improvement over a no-feedback baseline alone. Transfer is tested along distinct axes and the limitations are openly characterized. Table 2 evaluates three unseen acting models; Table 3 tests new task families, including joint model/task transfer. The attached-passage transfer gains of 9.31–13.19 points are meaningful. The paper also reports cases of degradation, timing-sensitive interventions, the continuation/restoration cost, and the possibility of direct solution generation (Secs. 4–5; Appendices F and K). This helps identify both where the approach works and where its interpretation should remain limited.

Critical weaknesses

The substantive methodological advance over the closest trained-advisor approach is insufficiently established. Advisor Models already trains advice for a frozen student from task rewards and includes multi-turn advice conditioned on interaction histories (Sec. 3 and Appendix B.2); tool-call consultation is also described, although reported to be used rarely (B.2.1). Thus the paper's fixed-schedule distinction applies to that work's reported SWE setup, not its entire formulation. CADDIE's sampled-prefix branching and successful deployment are potentially useful additions, but the experiments do not compare them with a matched outcome-trained Advisor Models variant or isolate the training change. Prompted critics establish the benefit of training, not the contribution beyond the closest trained method. This missing evidence is consequential to the method's novelty and significance. Advisor Models, v3 The remaining adaptive-consultation claim is not supported by an adequately matched protocol comparison. Appendix G requires at least one ALFWorld review and strongly encourages one in MuSiQue; timing can still be adaptive, but optional need-based consultation is not isolated. Appendix F uses one self-call while the main evaluation allows three, and no prompt-matched neutral-feedback/self-reflection or consultation-budget control is reported. Consequently, the evidence does not establish that adaptive consultation is a meaningful improvement over a comparable scheduled advisor. Transfer results are not a substitute for this comparison: several Wikipedia gains are only 0.81 points, and the stronger model's DeepDive effects are +0.97 points with one critic and −4.30 with the other. These boundaries do not negate the large in-domain or attached-transfer gains, but leave the distinctive contribution insufficiently demonstrated.

Questions for authors

Closest-prior comparison affecting acceptance: Which existing result or controlled comparison establishes the benefit of sampled-prefix training beyond an outcome-trained, trajectory-conditioned Advisor Models variant using the same critic/base checkpoints and comparable training resources? Please distinguish your recipe from the prior multi-turn and tool-call formulations, rather than only its reported fixed-schedule SWE configuration. Protocol clarification affecting selective-consultation claims: What exactly is the no-critic base prompt, particularly on ALFWorld, and which task-planning instructions are shared with critic-assisted conditions? Does the effect persist when the same call opportunities and token allowance are given to a neutral-feedback or self-reflection control? How do self-call and fixed/probabilistic intervention compare at the same three-call allowance or matched average consultation cost? Transfer evidence affecting the empirical increment: Can you provide paired task-level uncertainty for the Table 2 Wikipedia and Table 3 OOD gains, and clarify sensitivity to critic-training randomness? Appendix J also contains direct answers/actions; what fraction of transfer cases supplies a solution rather than guidance requiring further observations? These details would help assess whether the new cross-task evidence establishes a reliable advance despite the modest methodological difference.

6 comments

We thank the reviewer for his comments and suggestions to include additional analysis into our experimental results.

Weakness 1/Question 1

The submission lacked a matched comparison with Advisor Models (AM) and our fixed-schedule distinction describes only AM’s reported SWE setup, not its tool-call formulation. We will revise the Related Work section accordingly.

The main methodological difference is how training samples are built, not when advice is requested. In AM’s multi-turn formulation, a training sample is a whole advising episode, and its final outcome supplies the reward for the advice sequence. Our aim is to train a process critic, similar in role to a PRM, that assesses intermediate agent states and provides natural-language analysis and guidance. Caddie samples a prefix of an episode in progress, generates 16 critiques for that same state, and scores each by its own continuation’s final outcome. The critic can therefore learn which advice works in that particular situation, without having the reward depend on a whole sequence of other advice.

Observed improvements differ as well. AM’s multi-turn result (SWE) reduces steps at an unchanged resolve rate, whereas Caddie improves the success rate (+25 pp on MuSiQue-Attached, +26 pp on ALFWorld). Caddie also shows zero-shot gains on unseen task families, where AM reports only no degradation.

We reimplemented Advisor Models and trained it on both MuSiQue settings with the same critic model, base model, training budget, DAPO objective and optimizer. “AM periodic” denotes its fixed-schedule advice protocol. Success rates (%, mean ± SEM) are:

SettingBase modelNo criticAM multi-turn (periodic)AM tool-call (self-call)Caddie
Attached, in-domainQwen3-4B21.56 ± 2.0739.50 ± 2.6936.00 ± 2.6846.88 ± 2.68
Wiki, in-domainQwen3-4B10.38 ± 1.5918.75 ± 2.3418.81 ± 2.2821.13 ± 2.38
Attached, transferQwen3-30B-A3B28.44 ± 2.4546.88 ± 2.9338.75 ± 2.8941.63 ± 2.83
Attached, transferQwen3.8-27B46.63 ± 2.9354.12 ± 2.9852.12 ± 2.9355.94 ± 2.93
Attached, transferKimi K338.56 ± 2.8749.38 ± 2.8645.75 ± 2.7551.00 ± 2.76
Wiki, transferQwen3-30B-A3B19.94 ± 2.3822.50 ± 2.4720.62 ± 2.4320.75 ± 2.41
Wiki, transferQwen3.8-27B26.69 ± 2.6927.19 ± 2.7625.94 ± 2.6327.50 ± 2.79
Wiki, transferKimi K320.06 ± 2.2926.31 ± 2.5824.94 ± 2.3726.25 ± 2.55

In-domain, Caddie improves over AM periodic by +7.4 pp on Attached and +2.4 pp on Wiki. Across all eight settings, Caddie has the higher mean in five settings and is +0.8 pp higher on average. Transfer results are mixed: AM periodic performs better in three of the six transfer settings, by up to 5.3 pp. The clearest advantage of Caddie’s training formulation is therefore in-domain, while both methods retain useful transfer across base models. However, our AM reimplementation shows no gain on τ³ (out-of-domain transfer) contrary to Caddie (table in Weakness 1 in our reply to Reviewer th1X).

Weakness 2/Question 2

Thank you for highlighting this distinction. We clarify the original prompts and the scope of the published comparisons below.

The original ALFWorld no-critic system prompt was:

You are an agent acting in a household text environment. Choose exactly one currently admissible environment action at a time. You have at most 50 environment actions to finish the task. At every turn, identify the earliest unfinished subgoal and take the shortest action that advances it. Never repeat an action that did not change the state, revisit a location without a task reason, or explore after the needed object or receptacle is known.
Use these compact task recipes:
- place: find and take the object, go to the destination, open it if needed, then put it.
- clean: take the object, use the sink and faucet, retrieve it, then place it.
- heat: take the object, use the microwave correctly, retrieve it, then place it.
- cool: take the object, use the fridge correctly, retrieve it, then place it.
- examine under light: take the object, go to a lamp, and switch the lamp on.
- place two: complete both placements without undoing the first.
Before every action, reason briefly in one to three short sentences about the current unfinished subgoal and best admissible action. Keep the reasoning under 60 words. A critic may be consulted after one of your actions. If critic feedback is provided, check it against the current state and use it when helpful. Reply with exactly these two tags and no other text: <reason>one to three short sentences, at most 60 words</reason> <action>one currently admissible action</action>

The assisted prompt shared these planning instructions and the six task recipes. It added critic-call instructions, including “Use at least one review while there is still enough action budget to apply it.”

The original MuSiQue no-critic system prompt was:

You are an information-retrieval agent. Answer the multi-hop question by searching hidden evidence passages. Use one action per turn. Search for specific entities or relations and combine evidence across results. Do not answer until you have enough evidence. You have unlimited actions. A critic may be consulted after an action. Treat its feedback as advice, verify it against retrieved evidence, and continue the same trajectory. Reply in exactly one of these forms and nothing else: <reason>brief reasoning, at most 60 words</reason> <search>concise search query</search> OR <reason>brief reasoning, at most 60 words</reason> <answer>short final answer only</answer>

We add a neutral-feedback control that uses the same review interface and call budget but provides no trajectory-specific assessment or advice.

Success rate (%), mean ± SEM across tasks:

ConditionMuSiQue-AttachedALFWorld
No critic21.56 ± 2.0723.79 ± 3.01
Neutral feedback (new)26.25 ± 2.5233.96 ± 3.59
Prompted Qwen3-4B critic23.25 ± 2.3720.24 ± 2.89
Prompted Kimi K3 critic29.13 ± 2.4630.22 ± 2.80
Trained critic, self-call46.88 ± 2.6849.72 ± 3.42

We do not claim that self-call is more accurate. Section 2.1 already states that a scheduled critic “can perform better in some settings”, and Appendix F lists such cases. We use self-call because it removes a hyperparameter that it is required to optimize for a specific model and task. On MuSiQue-Attached, moving the step from 3 to 4 changes success from 17.75% to 30.88% for the same critic and base model, and the best step differs between base models (5 for Qwen3-4B, 4 for Qwen3-30B-A3B; Table 7).

Question 3

Can you provide paired task-level uncertainty for the Table 2 Wikipedia and Table 3 OOD gains?

To address the reviewer’s concern, we provide paired task-level uncertainty estimates on the MuSiQue dataset for Table 2. We observe strong improvements in most settings, specifically Kimi K3 improves with Caddie by 6.5 percentage points over prompted Qwen3-4B, and Qwen3.8-27B also improves over the prompted critic by 3.81 points on Wiki, with paired 95% confidence intervals excluding zero, providing statistical support for these gains.

ModeFrozen generatorΔ vs prompted Qwen3-4BPaired 95% confidence intervalΔ vs No criticPaired 95% confidence interval
AttachedQwen3-30B-A3B+6.88[+4.31, +9.56]+13.19[+10.38, +16.06]
AttachedQwen3.8-27B+10.63[+6.94, +14.38]+9.31[+5.56, +13.13]
AttachedKimi K3+16.56[+12.88, +20.38]+12.44[+8.56, +16.38]
WikiQwen3-30B-A3B+0.25[−1.63, +2.06]+0.81[−1.06, +2.75]
WikiQwen3.8-27B+3.81[+1.44, +6.31]+0.81[−1.38, +3.06]
WikiKimi K3+6.50[+4.31, +8.81]+6.19[+3.75, +8.69]

Table 3 OOD: paired gains versus no critic. As detailed in our reply to Reviewer th1X, we now evaluate on the full τ³ Retail and Airline benchmarks pooled (164 tasks) and an expanded DeepDive set (256 tasks), with eight rollouts per task. Entries report percentage-point differences versus no critic, with paired task-level 95% confidence intervals. Q4 and Q38 denote Qwen3-4B and Qwen3.8-27B, respectively.

MethodQ4 Retail + AirlineQ4 DeepDiveQ38 Retail + AirlineQ38 DeepDive
No critic————
Prompted Qwen3-4B+0.99 [−2.90, +4.95]+4.00 [+1.32, +6.69]+6.78 [+2.06, +11.66]−2.64 [−4.93, −0.29]
CADDIE Attached+4.19 [+0.08, +8.38]+5.13 [+1.86, +8.50]+10.37 [+5.56, +15.32]+1.73 [−0.51, +3.98]
CADDIE Wiki+4.27 [+1.07, +7.47]+7.62 [+4.39, +10.84]+9.45 [+4.80, +14.25]+0.49 [−1.56, +2.59]

And clarify sensitivity to critic-training randomness?

Under time and resource constraints during rebuttal, we compare the two checkpoints trained in Wiki and Attached modes on the original τ³ Retail test subset. We have not measured variation across repeated training seeds with the same configuration.

Frozen generatorNo criticAttached-trained criticWiki-trained critic
Qwen3-4B44.69%45.00% (+0.31 pp)48.75% (+4.06 pp)
Qwen3.8-27B50.31%59.06% (+8.75 pp)58.12% (+7.81 pp)

Appendix J also contains direct answers/actions; what fraction of transfer cases supplies a solution rather than guidance requiring further observations?

Following the reviewer’s request, we analyze Caddie’s advice on the τ³ Retail and Airline benchmarks by the first actionable instruction in each message. We observe that the distribution varies depending on the type of task: the frequency of advice to provide an answer (i.e., to supply an immediate solution) ranges from 21.6% for the subset related to air travel (Airline) to 34.9% for the subset pertaining to retail (Retail), among the analyzed critique messages. The remaining messages fall into the Search, Verify, Ask, or Plan and Execute categories.

BenchmarkMessagesAnswerSearchVerifyAskPlan and Execute
τ³ Retail6334.9%1.6%17.5%6.3%39.7%
τ³ Airline3721.6%8.1%2.7%2.7%64.9%

A complementary analysis of actionable advice types is provided in Appendix H.2 (Figure 8b), which shows how they vary across tasks and generators.

Reviewer_kWt2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CADDIE, training critics to provide natural language analysis and advice to enhance the agent's response quality, the critics is trained via RL with the task's success as reward, and use DAPO to train the language model-based critic.

Critical strengths

This paper introduces critics to provide language analysis or advice, and language guidance is useful to improve LLM agents' responses.

Critical weaknesses

The novelty of this paper is limited, as there already exists work on using RL to train LMs to generate natural-language critiques, such as: [1] ReMA: Learning to Meta-Think for LLMs with Multi-Agent Reinforcement Learning.

The experimental validation is weak. The results show that using a critic is better than using no critic, and that the fine-tuned critic is better than a directly prompted Kimi K3 critic. However, under the same RL training budget, would training the critic, the generator (action policy), or both yield better performance? The benchmarks are relatively simple, and Qwen3-4B is too small to support solid insights. A stronger generator (larger than 4B) may reduce or eliminate the benefit of the critic.

Questions for authors

For each benchmark, what do the critiques look like? Is there any intuitive explanation of why a stateful critic is useful? During inference and when sampling trajectories to estimate rewards, does the critic need to generate critiques at every step, or only at one step?

4 comments

We thank the reviewer for thoughtful comments and valuable feedback.

Novelty

We agree that the idea of training language models with RL to produce natural-language feedback is not new, and we do not claim it. Related Work already discusses how we differ from RL4F, CTRL, Critique-RL and Advisor Models.

ReMA is relevant but addresses a different setting. It trains a high-level meta-thinking agent and a low-level reasoning agent, including a multi-turn formulation in which the agents interact repeatedly. It is evaluated on math and LLM-as-a-judge problems, without tools or an environment.

Caddie trains a critic that can be called at any step and reviews the trajectory so far, similar in role to a process reward model (PRM) but with natural-language output. The base model stays frozen, Caddie samples an intermediate state, generates 16 alternative critiques, and evaluates a separate base model continuation for each. The critic can therefore learn which advice works in that particular situation, without sharing one reward across a whole sequence of advice. The trained critic can also be applied to other base models without updating their parameters.

Similar to ReMA, Advisor Models learns natural-language guidance from downstream outcomes, but keeps the receiving model frozen, making it a direct baseline for our setting.

We reimplemented Advisor Models and trained it on both MuSiQue settings with the same critic model, base model, training budget, DAPO objective and optimizer. “AM periodic” denotes its fixed-schedule advice protocol. With Qwen3-4B as the base model, success rates (%, mean ± SEM) are:

SettingNo criticAM periodicCaddieCaddie − AM
MuSiQue-Attached21.56 ± 2.0739.50 ± 2.6946.88 ± 2.68+7.4 pp
MuSiQue-Wiki10.38 ± 1.5918.75 ± 2.3421.13 ± 2.38+2.4 pp

These results support the value of Caddie’s approach beyond the shared paradigm of outcome-trained natural-language advisors in these in-domain settings. We provide the full table and further analysis in the reply to Reviewer th1X.

Critic, generator, or both?

We report this comparison in Section 3.2 (paragraph “Critic feedback after generator training”). Table 1 and Appendix E (Table 5) report this comparison on MuSiQue with Qwen3-4B:

Trained componentAttachedWiki
None21.56 ± 2.0710.38 ± 1.59
Critic only46.88 ± 2.6821.13 ± 2.38
Generator only (GRPO)49.69 ± 2.9819.69 ± 2.36
Generator (GRPO) + critic trained with GRPO generator63.56 ± 2.9823.38 ± 2.63

The generator, and the critic in the last row, were trained in Attached mode only.

Training only the critic or only the generator gives comparable results, while combining generator RL with a trained critic produces the highest means in both settings.

Both runs use batch size 128, context length 8,192 and the same DAPO objective and optimizer settings. These shared settings do not establish equal total training compute. The results therefore demonstrate that critic training can complement generator RL. Critic training has several practical uses: improving closed or large models that cannot readily be fine-tuned, and providing further gains after generator optimization. We do not claim that critic training is generally more compute-efficient than directly optimizing the generator.

Stronger generator comparison

We report the performance of a critic trained with Qwen3-4B with larger generator base-models in Table 2. We additionally evaluate GPT-6.1-Sol with high reasoning as the base model:

MuSiQue-Attached

Base modelNo criticCaddieGain (pp)
Qwen3-4B (used in training)21.56 ± 2.0746.88 ± 2.68+25.32
Qwen3-30B-A3B28.44 ± 2.4541.63 ± 2.83+13.19
Qwen3.8-27B46.63 ± 2.9355.94 ± 2.93+9.31
Kimi K338.56 ± 2.8751.00 ± 2.76+12.44
GPT-6.1-Sol, high reasoning43.50 ± 3.2356.50 ± 3.22+13.00

MuSiQue-Wiki

Base modelNo criticCaddieGain (pp)
Qwen3-4B (used in training)10.38 ± 1.5921.13 ± 2.38+10.75
Qwen3-30B-A3B19.94 ± 2.3820.75 ± 2.41+0.81
Qwen3.8-27B26.69 ± 2.6927.50 ± 2.79+0.81
Kimi K320.06 ± 2.2926.25 ± 2.55+6.19
GPT-6.1-Sol, high reasoning30.88 ± 3.0834.00 ± 3.14+3.12

We additionally made an experiment with a critic trained with Qwen3.8-27B as a base model. This critic also improves all four evaluated targets, including the smaller Qwen3-4B. We report its results here:

Target (Attached)No criticCaddie trained with Qwen3-4BCaddie trained with Qwen3.8-27B (new)
Qwen3-4B21.56 ± 2.0746.88 ± 2.6837.88 ± 2.69
Qwen3-30B-A3B28.44 ± 2.4541.63 ± 2.8340.19 ± 2.86
Qwen3.8-27B46.63 ± 2.9355.94 ± 2.9353.69 ± 2.90
Kimi K338.56 ± 2.8751.00 ± 2.7649.81 ± 2.80

Benchmark difficulty

We respectfully disagree that the benchmarks are simple. The τ³-bench family is a standard benchmark for evaluating frontier agent models, and it is far from solved: without a critic, Qwen3.8-27B reaches 56.7% on τ³ and 53.2% on DeepDive (we recalculated the numbers on a larger set, see our reply to a reviewer th1X). MuSiQue is also hard even for large models: without a critic, results stay below 50% on MuSiQue-Attached (Kimi K3 38.6%, Qwen3.8-27B 46.6%) and below 31% on MuSiQue-Wiki. (GPT-6.1-Sol: 30.9%.)

What do the critiques look like?

The critic provides a brief diagnosis followed by advice for the next steps, using the format:

Analysis: <brief diagnosis>
Advices for next steps: <concise guidance>

Feedback can address missing evidence, unfinished subgoals, revised user requests, or action ordering. The Analysis section and the appendix “Qualitative OOD Trajectories” provide concrete examples with generated advice and the agent’s subsequent actions.

For example, in τ³ Retail, the user abandons a camera upgrade and asks to cancel the pending order. Caddie gives the following advice:

Call the cancel_order function with order_id “#W9284598” and cancel_reason “ordered by mistake”.

The agent then confirms and cancels the order, refunding the original payment method and completing the task.

Why is a stateful critic useful?

Seeing the trajectory lets the critic account for the current state and, for example, changes in user intent. In the τ³ example above, it recommends cancellation after the user abandons the upgrade, instead of continuing the outdated plan. Similarly, on DeepDive, it can suggest further search when evidence is missing or recommend answering once sufficient evidence has been retrieved.

Does the critic need to generate critiques at every step, or only at one step?

During training the critic intervenes at most once per trajectory, at a randomly sampled step.

During inference, the frequency depends on the protocol: once at a fixed step, probabilistically at each step, or when the base model requests feedback (self-call). Our main evaluations use self-call, with at most three critiques per trajectory. Usually, feedback is not required at every step.

Dear Reviewer, As the rebuttal period comes to an end, we wanted to reach out to see if you had any remaining concerns we could address. Thanks again for your time and feedback.

Dear Reviewer, with only a few hours left in the rebuttal period, could you please let us know whether our response addresses your concerns? If anything remains unclear, we’d be happy to clarify it before the deadline.

Thank you for your time.

Reviewer_th1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CADDIE, a method for training a critic to provide natural-language advice during an LLM agent’s execution. The method samples intermediate trajectory states, generates candidate critiques, and optimizes the critic with DAPO using the final rewards of the frozen agent’s subsequent continuations, without step-level annotations or reference critiques. At inference time, the agent can decide when to consult the critic. Experiments cover MuSiQue and ALFWorld, alongside transfer evaluations across base models and to DeepDive and τ³. The authors report improvements over both no-critic and prompted-critic baselines in the evaluated in-domain settings.

Critical strengths
  • Training the critic directly from task outcomes avoids the need for step-level labels or reference critiques, reducing reliance on specialized supervision.
  • The evaluation includes multiple task settings and examines transfer across base models and tasks, providing useful empirical evidence about the approach’s applicability.
Critical weaknesses

1. Limited and inconsistent generalization across task domains.

The practical value of a trained critic largely depends on whether it can generalize to unseen tasks and domains. Otherwise, training a separate critic for every task family would introduce substantial overhead and significantly limit the applicability of the proposed approach.

However, the strongest improvements in Table 1 are obtained under in-domain settings, where a separate Caddie critic is trained for each task configuration (MuSiQue-Attached, MuSiQue-Wiki, and ALFWorld). Although Table 3 provides out-of-domain evaluations on DeepDive and τ³, the improvements are considerably less consistent. For example, on DeepDive with Qwen3.8-27B, the Wiki-trained critic improves the success rate only from 48.83% to 49.80% (+0.97 percentage points), while the Attached-trained critic decreases performance from 48.83% to 44.53% (-4.30 points). These observations raise concerns about the robustness of cross-domain transfer.

Moreover, the OOD results are evaluated on relatively small fixed task sets, and the reported SEM is computed across seed-level aggregate accuracies rather than independently sampled task sets. Therefore, the evidence for reliable generalization to new task distributions remains limited.

2. Insufficient investigation of critic transferability across target models.

The effectiveness of a critic may depend substantially on the capabilities, behavior, and failure modes of the target agent. Although Table 2 evaluates transfer from a critic trained with Qwen3-4B to several other models, the experiments do not systematically study how the choice of the source model used for critic training affects transferability.

There is direct evidence in the paper suggesting that this dependency can be substantial. In Appendix Table 5, when advising the same GRPO-trained Qwen3-4B agent on MuSiQue-Attached, the critic trained with the original generator achieves 51.69% success, whereas a critic trained specifically with the GRPO generator achieves 63.56%, an additional 11.87 percentage points. This suggests that critic effectiveness can be highly sensitive to the agent used during training, even within the same model family.

Furthermore, transfer gains are not uniformly substantial. For example, on MuSiQue-Wiki, applying the trained critic to Qwen3.8-27B improves success from only 26.69% to 27.50% (+0.81 points).

A more systematic source-model × target-model evaluation is needed, particularly involving critics trained with stronger agents and transferred to weaker or architecturally different agents. Without this analysis, it remains unclear whether the proposed approach learns generally transferable feedback strategies or primarily specializes to the behavior of the agent used during training.

3. Lack of comparisons against prompted critics based on proprietary frontier models.

The paper compares Caddie against prompted Qwen3-4B and Kimi K3 critics. While these are useful baselines, they do not establish whether training an additional critic is preferable to directly using a stronger proprietary frontier model, such as GPT or Claude, to generate feedback.

This distinction is especially important because the paper's primary motivation is to improve frozen agents through an auxiliary critic. A natural practical alternative is to invoke a highly capable existing model as the critic without any additional training.

The current results do not conclusively establish the superiority of trained critics over strong prompted critics. For instance, on MuSiQue-Wiki in Table 1, Caddie achieves 21.13% success compared with 19.19% using prompted Kimi K3, a difference of only 1.94 percentage points, with overlapping reported SEM intervals.

Comparisons with proprietary frontier critics, ideally under both matched inference budgets and unconstrained API-based settings, would help establish whether the proposed critic training offers a meaningful advantage over simply using stronger models with well-designed critique prompts.

4. Limited methodological novelty compared with existing Advisor Models.

The main idea of training a lightweight model to provide natural-language guidance to a frozen agent, using downstream task outcomes as the reinforcement learning signal, substantially overlaps with the existing How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models (ICML 2026).

That work already introduced outcome-based reinforcement learning for natural-language advisors, demonstrated transfer across student models, and considered multi-turn agent interactions. It also discussed student-initiated advisor requests through tool calls.

The main distinctions of Caddie appear to be its intermediate-state branching procedure, the specific integration into agent trajectories, and the use of DAPO rather than GRPO. However, the paper does not provide sufficiently targeted experiments to isolate the contributions of these design choices. In particular, there is no direct comparison with Advisor Models under the same tasks, target agents, and training budgets.

Consequently, it is difficult to determine how much of the reported improvement comes from the proposed methodological changes rather than the already established paradigm of outcome-trained natural-language advisors.

5. Missing compute-matched baselines and insufficient evidence of computational efficiency.

Training Caddie requires sampling multiple critiques from an intermediate agent state and running separate agent continuations to obtain terminal rewards. The main configuration uses 16 candidate critiques per sampled state, with one agent continuation per critique. This introduces substantial training overhead, particularly for long-horizon tasks involving expensive tool calls and environment interactions.

However, the paper does not provide a systematic compute-matched comparison against alternatives such as additional agent rollouts, stronger prompted critics, self-refinement, or direct reinforcement learning of the target agent.

This is particularly relevant given Appendix Table 5: directly applying GRPO to Qwen3-4B improves its MuSiQue-Attached success rate from 21.56% to 49.69%, which is higher than the 46.88% obtained by the original agent with Caddie in Table 1. Although these approaches have different training budgets and should not be treated as directly comparable, the result raises an important question about the relative efficiency of training a separate critic.

The paper should provide training and inference costs, including agent rollout tokens, GPU-hours, additional model calls, and end-to-end latency, alongside performance under matched budgets. Without such comparisons, the results demonstrate that trained critics can improve task success, but do not establish that this is a computationally efficient or practically preferable approach.

Overall, while the paper reports encouraging in-domain improvements, its practical significance depends on the extent to which critics generalize across task domains and target models. The current experiments provide only partial evidence for these properties, while the methodological novelty and compute-efficiency advantages over existing approaches remain insufficiently established. These limitations substantially weaken the paper's claimed contributions.

Questions for authors

Refer to Critical Weaknesses.

7 comments

We thank the reviewer for thoughtful comments and valuable feedback.

Weakness 1

Our central claim is that outcome-only training yields a critic that improves a base policy without modifying it. This is useful in practice even without cross-domain transfer:

  • Models that cannot be fine-tuned (closed or too large). A 4B critic trained with a small model as a base policy improves all three larger, unseen base models (Table 2: +9 to +13 pp on MuSiQue-Attached, smaller on Wiki).

  • Policies that are already trained. On top of a GRPO-trained generator, the critic adds +13.9 pp on MuSiQue-Attached without the need of an additional data split (Appendix E, Table 5).

Cross-domain transfer is an additional zero-shot result, and we agree it is less uniform than the in-domain results. Our results demonstrate useful zero-shot transfer on the evaluated benchmarks, while also showing that its effectiveness depends on the training and target domain. The cited result is the weakest in the original Table 3: Caddie improves over no critic in 11 of 12 cases, and the Wiki-trained critic in all 6 (mean +4.9 pp).

Table 3 (OOD results) indeed reported the SEM across seeds. We will update the paper and we attach the new corrected results with SEM across tasks below. We now also evaluate on the full τ³ Retail and Airline benchmarks pooled (164 tasks vs only a subset of 40 + 20 tasks originally) and an expanded DeepDive set (256 tasks vs original 64 tasks).

MethodQ4 Retail + AirlineQ4 DeepDiveQ38 Retail + AirlineQ38 DeepDive
No Critic38.95 ± 2.7635.60 ± 2.3056.71 ± 2.7353.22 ± 2.28
Prompted Qwen3-4B39.94 ± 2.9139.60 ± 2.3963.49 ± 2.4850.59 ± 2.18
Prompted Kimi K338.19 ± 2.8741.94 ± 2.3763.34 ± 2.4050.73 ± 2.22
Agent-RRM40.55 ± 2.9338.92 ± 2.3462.73 ± 2.5449.46 ± 2.22
Advisor Models37.65 ± 2.8140.82 ± 2.5056.71 ± 2.4453.17 ± 2.27
CADDIE Attached43.14 ± 2.8740.72 ± 2.4567.07 ± 2.4354.96 ± 2.23
CADDIE Wiki43.22 ± 2.8343.21 ± 2.4766.16 ± 2.4553.71 ± 2.24

Success rates (%), mean ± SEM across tasks, 8 rollouts per task. Advisor Models is our reimplementation (see Weakness 4).

On these larger sets, both Caddie critics improve over no critic in all 8 cases (mean +5.4 pp), and one of them has the highest mean in every column while every other method falls below no critic at least once.

Weakness 2

Table 2 indeed varies only the target model to check whether a critic trained on one model can produce meaningful feedback to unseen ones. To address your concern regarding source model used during critic training, we trained a critic in attached mode with Qwen3.8-27B as the base model and evaluated it with all four target models:

Target (Attached)No criticCaddie trained with Qwen3-4BCaddie trained with Qwen3.8-27B (new)
Qwen3-4B21.56 ± 2.0746.88 ± 2.6837.88 ± 2.69
Qwen3-30B-A3B28.44 ± 2.4541.63 ± 2.8340.19 ± 2.86
Qwen3.8-27B46.63 ± 2.9355.94 ± 2.9353.69 ± 2.90
Kimi K338.56 ± 2.8751.00 ± 2.7649.81 ± 2.80

In some cases a critic may work better with the model it was trained with, since it has seen that model's failure modes, pitfalls and successes. Table 5 shows this dependence. GRPO fine-tuning on MuSiQue raises the score of the generator without critic from 21.56% to 49.69%, strongly changing the behaviour policy of the base model. The original critic therefore adds only +2.0 pp, while a critic trained with the new policy adds +13.9 pp. When a matched critic is not an option, a critic trained with another model still helps, as Table 2 and the table above show.

The critic thus learns feedback that transfers across models, while its effectiveness can also depend on the model used during training. In some cases, however, the transferred gains are smaller than in others, as with the cited +0.81 pp for Qwen3.8-27B on MuSiQue-Wiki.

Weakness 3

The paper already tests a strong prompted critic: Kimi K3 was among the strongest models available at submission, and Caddie outperforms the prompted Kimi K3 critic by +17.8 pp on MuSiQue-Attached and +19.5 pp on ALFWorld (Table 1). Nevertheless, we ran the requested comparison with a proprietary frontier model: GPT-6.1-Sol (high reasoning) as a prompted critic, with the same prompt and call budget as the other critics, for all four base models.

SettingBase modelNo criticPrompted Qwen3-4BPrompted Kimi K3Prompted GPT-6.1-Sol (new)Caddie (ours)
AttachedQwen3-4B21.56 ± 2.0723.25 ± 2.3729.13 ± 2.4634.13 ± 2.8846.88 ± 2.68
AttachedQwen3-30B-A3B28.44 ± 2.4534.75 ± 2.8236.00 ± 2.8637.13 ± 2.9541.63 ± 2.83
AttachedQwen3.8-27B46.63 ± 2.9345.31 ± 2.8441.13 ± 2.7742.13 ± 3.0755.94 ± 2.93
AttachedKimi K338.56 ± 2.8734.44 ± 2.6330.19 ± 2.3632.63 ± 2.6851.00 ± 2.76
WikiQwen3-4B10.38 ± 1.5916.06 ± 2.1919.19 ± 2.2321.88 ± 2.4621.13 ± 2.38
WikiQwen3-30B-A3B19.94 ± 2.3820.50 ± 2.4320.06 ± 2.4220.63 ± 2.4520.75 ± 2.41
WikiQwen3.8-27B26.69 ± 2.6923.69 ± 2.5321.44 ± 2.3622.63 ± 2.6027.50 ± 2.79
WikiKimi K320.06 ± 2.2919.75 ± 2.2818.75 ± 2.1818.88 ± 2.4026.25 ± 2.55

With Qwen3-4B as a base model Caddie is 12.8 pp above the GPT-6.1-Sol critic on Attached (46.9 vs 34.1). On Wiki, the results are close (21.1 vs 21.9).

Across all four base models Caddie has the highest mean in 7 of 8 settings and never falls below no critic. The GPT-6.1-Sol critic lowers success for the two strongest base models, Qwen3.8-27B and Kimi K3, in both settings.

To see how our critic performs with a closed frontier model as the agent, we also ran GPT-6.1-Sol (high reasoning) as the base model, with and without Caddie:

SettingNo criticCaddie (ours)Difference
Attached43.50 ± 3.2356.50 ± 3.22+13.00
Wiki30.88 ± 3.0834.00 ± 3.14+3.12

Caddie improves GPT-6.1-Sol by +13.0 pp on Attached and by +3.1 pp on Wiki. A trained critic is therefore complementary to frontier closed-source models. In attached mode, a 4B agent with Caddie also achieves higher mean success than GPT-6.1-Sol alone (46.9 vs 43.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.