- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CADDIE, a generative critic trained to advise a frozen tool-using agent during an ongoing trajectory. Training samples an intermediate prefix, generates a group of critiques, evaluates each by the terminal success of the agent's continuation, and updates only the critic using DAPO. No reference critiques or step-level correctness labels are required, although a feedback-format constraint is enforced. At inference, the base agent can invoke the critic through a self-call interface. Separate Qwen3-4B critics trained for MuSiQue-attached, MuSiQue-wiki, and ALFWorld improve in-domain success; the MuSiQue critics are also evaluated with unseen base models and on DeepDive and customer-service tasks. The strongest improvements are approximately 25 percentage points in MuSiQue-attached and ALFWorld, while cross-model Wikipedia and some out-of-domain gains are considerably smaller or mixed.
Critical strengths
A simple training signal produces substantial improvements over useful controls. The method directly rewards critique-conditioned continuations rather than requiring reference critique data. Table 1 compares the trained critic against its prompted initialization and a much larger prompted critic, showing 46.88% versus 21.56% without a critic on MuSiQue-attached and 49.72% versus 23.79% on ALFWorld. Appendix E additionally shows improvement when the acting model has already been task-trained. These controls make the contribution more convincing than an improvement over a no-feedback baseline alone. Transfer is tested along distinct axes and the limitations are openly characterized. Table 2 evaluates three unseen acting models; Table 3 tests new task families, including joint model/task transfer. The attached-passage transfer gains of 9.31–13.19 points are meaningful. The paper also reports cases of degradation, timing-sensitive interventions, the continuation/restoration cost, and the possibility of direct solution generation (Secs. 4–5; Appendices F and K). This helps identify both where the approach works and where its interpretation should remain limited.
Critical weaknesses
The substantive methodological advance over the closest trained-advisor approach is insufficiently established. Advisor Models already trains advice for a frozen student from task rewards and includes multi-turn advice conditioned on interaction histories (Sec. 3 and Appendix B.2); tool-call consultation is also described, although reported to be used rarely (B.2.1). Thus the paper's fixed-schedule distinction applies to that work's reported SWE setup, not its entire formulation. CADDIE's sampled-prefix branching and successful deployment are potentially useful additions, but the experiments do not compare them with a matched outcome-trained Advisor Models variant or isolate the training change. Prompted critics establish the benefit of training, not the contribution beyond the closest trained method. This missing evidence is consequential to the method's novelty and significance. Advisor Models, v3 The remaining adaptive-consultation claim is not supported by an adequately matched protocol comparison. Appendix G requires at least one ALFWorld review and strongly encourages one in MuSiQue; timing can still be adaptive, but optional need-based consultation is not isolated. Appendix F uses one self-call while the main evaluation allows three, and no prompt-matched neutral-feedback/self-reflection or consultation-budget control is reported. Consequently, the evidence does not establish that adaptive consultation is a meaningful improvement over a comparable scheduled advisor. Transfer results are not a substitute for this comparison: several Wikipedia gains are only 0.81 points, and the stronger model's DeepDive effects are +0.97 points with one critic and −4.30 with the other. These boundaries do not negate the large in-domain or attached-transfer gains, but leave the distinctive contribution insufficiently demonstrated.
Questions for authors
Closest-prior comparison affecting acceptance: Which existing result or controlled comparison establishes the benefit of sampled-prefix training beyond an outcome-trained, trajectory-conditioned Advisor Models variant using the same critic/base checkpoints and comparable training resources? Please distinguish your recipe from the prior multi-turn and tool-call formulations, rather than only its reported fixed-schedule SWE configuration. Protocol clarification affecting selective-consultation claims: What exactly is the no-critic base prompt, particularly on ALFWorld, and which task-planning instructions are shared with critic-assisted conditions? Does the effect persist when the same call opportunities and token allowance are given to a neutral-feedback or self-reflection control? How do self-call and fixed/probabilistic intervention compare at the same three-call allowance or matched average consultation cost? Transfer evidence affecting the empirical increment: Can you provide paired task-level uncertainty for the Table 2 Wikipedia and Table 3 OOD gains, and clarify sensitivity to critic-training randomness? Appendix J also contains direct answers/actions; what fraction of transfer cases supplies a solution rather than guidance requiring further observations? These details would help assess whether the new cross-task evidence establishes a reliable advance despite the modest methodological difference.
6 comments
Weakness 2/Question 2
Thank you for highlighting this distinction. We clarify the original prompts and the scope of the published comparisons below.
The original ALFWorld no-critic system prompt was:
You are an agent acting in a household text environment. Choose exactly one currently admissible environment action at a time. You have at most 50 environment actions to finish the task. At every turn, identify the earliest unfinished subgoal and take the shortest action that advances it. Never repeat an action that did not change the state, revisit a location without a task reason, or explore after the needed object or receptacle is known.
Use these compact task recipes:
- place: find and take the object, go to the destination, open it if needed, then put it.
- clean: take the object, use the sink and faucet, retrieve it, then place it.
- heat: take the object, use the microwave correctly, retrieve it, then place it.
- cool: take the object, use the fridge correctly, retrieve it, then place it.
- examine under light: take the object, go to a lamp, and switch the lamp on.
- place two: complete both placements without undoing the first.
Before every action, reason briefly in one to three short sentences about the current unfinished subgoal and best admissible action. Keep the reasoning under 60 words. A critic may be consulted after one of your actions. If critic feedback is provided, check it against the current state and use it when helpful. Reply with exactly these two tags and no other text: <reason>one to three short sentences, at most 60 words</reason> <action>one currently admissible action</action>
The assisted prompt shared these planning instructions and the six task recipes. It added critic-call instructions, including “Use at least one review while there is still enough action budget to apply it.”
The original MuSiQue no-critic system prompt was:
You are an information-retrieval agent. Answer the multi-hop question by searching hidden evidence passages. Use one action per turn. Search for specific entities or relations and combine evidence across results. Do not answer until you have enough evidence. You have unlimited actions. A critic may be consulted after an action. Treat its feedback as advice, verify it against retrieved evidence, and continue the same trajectory. Reply in exactly one of these forms and nothing else: <reason>brief reasoning, at most 60 words</reason> <search>concise search query</search> OR <reason>brief reasoning, at most 60 words</reason> <answer>short final answer only</answer>
We add a neutral-feedback control that uses the same review interface and call budget but provides no trajectory-specific assessment or advice.
Success rate (%), mean ± SEM across tasks:
| Condition | MuSiQue-Attached | ALFWorld |
|---|---|---|
| No critic | 21.56 ± 2.07 | 23.79 ± 3.01 |
| Neutral feedback (new) | 26.25 ± 2.52 | 33.96 ± 3.59 |
| Prompted Qwen3-4B critic | 23.25 ± 2.37 | 20.24 ± 2.89 |
| Prompted Kimi K3 critic | 29.13 ± 2.46 | 30.22 ± 2.80 |
| Trained critic, self-call | 46.88 ± 2.68 | 49.72 ± 3.42 |
We do not claim that self-call is more accurate. Section 2.1 already states that a scheduled critic “can perform better in some settings”, and Appendix F lists such cases. We use self-call because it removes a hyperparameter that it is required to optimize for a specific model and task. On MuSiQue-Attached, moving the step from 3 to 4 changes success from 17.75% to 30.88% for the same critic and base model, and the best step differs between base models (5 for Qwen3-4B, 4 for Qwen3-30B-A3B; Table 7).
Question 3
Can you provide paired task-level uncertainty for the Table 2 Wikipedia and Table 3 OOD gains?
To address the reviewer’s concern, we provide paired task-level uncertainty estimates on the MuSiQue dataset for Table 2. We observe strong improvements in most settings, specifically Kimi K3 improves with Caddie by 6.5 percentage points over prompted Qwen3-4B, and Qwen3.8-27B also improves over the prompted critic by 3.81 points on Wiki, with paired 95% confidence intervals excluding zero, providing statistical support for these gains.
| Mode | Frozen generator | Δ vs prompted Qwen3-4B | Paired 95% confidence interval | Δ vs No critic | Paired 95% confidence interval |
|---|---|---|---|---|---|
| Attached | Qwen3-30B-A3B | +6.88 | [+4.31, +9.56] | +13.19 | [+10.38, +16.06] |
| Attached | Qwen3.8-27B | +10.63 | [+6.94, +14.38] | +9.31 | [+5.56, +13.13] |
| Attached | Kimi K3 | +16.56 | [+12.88, +20.38] | +12.44 | [+8.56, +16.38] |
| Wiki | Qwen3-30B-A3B | +0.25 | [−1.63, +2.06] | +0.81 | [−1.06, +2.75] |
| Wiki | Qwen3.8-27B | +3.81 | [+1.44, +6.31] | +0.81 | [−1.38, +3.06] |
| Wiki | Kimi K3 | +6.50 | [+4.31, +8.81] | +6.19 | [+3.75, +8.69] |
Table 3 OOD: paired gains versus no critic. As detailed in our reply to Reviewer th1X, we now evaluate on the full τ³ Retail and Airline benchmarks pooled (164 tasks) and an expanded DeepDive set (256 tasks), with eight rollouts per task. Entries report percentage-point differences versus no critic, with paired task-level 95% confidence intervals. Q4 and Q38 denote Qwen3-4B and Qwen3.8-27B, respectively.
| Method | Q4 Retail + Airline | Q4 DeepDive | Q38 Retail + Airline | Q38 DeepDive |
|---|---|---|---|---|
| No critic | — | — | — | — |
| Prompted Qwen3-4B | +0.99 [−2.90, +4.95] | +4.00 [+1.32, +6.69] | +6.78 [+2.06, +11.66] | −2.64 [−4.93, −0.29] |
| CADDIE Attached | +4.19 [+0.08, +8.38] | +5.13 [+1.86, +8.50] | +10.37 [+5.56, +15.32] | +1.73 [−0.51, +3.98] |
| CADDIE Wiki | +4.27 [+1.07, +7.47] | +7.62 [+4.39, +10.84] | +9.45 [+4.80, +14.25] | +0.49 [−1.56, +2.59] |
And clarify sensitivity to critic-training randomness?
Under time and resource constraints during rebuttal, we compare the two checkpoints trained in Wiki and Attached modes on the original τ³ Retail test subset. We have not measured variation across repeated training seeds with the same configuration.
| Frozen generator | No critic | Attached-trained critic | Wiki-trained critic |
|---|---|---|---|
| Qwen3-4B | 44.69% | 45.00% (+0.31 pp) | 48.75% (+4.06 pp) |
| Qwen3.8-27B | 50.31% | 59.06% (+8.75 pp) | 58.12% (+7.81 pp) |
Appendix J also contains direct answers/actions; what fraction of transfer cases supplies a solution rather than guidance requiring further observations?
Following the reviewer’s request, we analyze Caddie’s advice on the τ³ Retail and Airline benchmarks by the first actionable instruction in each message. We observe that the distribution varies depending on the type of task: the frequency of advice to provide an answer (i.e., to supply an immediate solution) ranges from 21.6% for the subset related to air travel (Airline) to 34.9% for the subset pertaining to retail (Retail), among the analyzed critique messages. The remaining messages fall into the Search, Verify, Ask, or Plan and Execute categories.
| Benchmark | Messages | Answer | Search | Verify | Ask | Plan and Execute |
|---|---|---|---|---|---|---|
| τ³ Retail | 63 | 34.9% | 1.6% | 17.5% | 6.3% | 39.7% |
| τ³ Airline | 37 | 21.6% | 8.1% | 2.7% | 2.7% | 64.9% |
A complementary analysis of actionable advice types is provided in Appendix H.2 (Figure 8b), which shows how they vary across tasks and generators.
We thank the reviewer for his comments and suggestions to include additional analysis into our experimental results.
Weakness 1/Question 1
The submission lacked a matched comparison with Advisor Models (AM) and our fixed-schedule distinction describes only AM’s reported SWE setup, not its tool-call formulation. We will revise the Related Work section accordingly.
The main methodological difference is how training samples are built, not when advice is requested. In AM’s multi-turn formulation, a training sample is a whole advising episode, and its final outcome supplies the reward for the advice sequence. Our aim is to train a process critic, similar in role to a PRM, that assesses intermediate agent states and provides natural-language analysis and guidance. Caddie samples a prefix of an episode in progress, generates 16 critiques for that same state, and scores each by its own continuation’s final outcome. The critic can therefore learn which advice works in that particular situation, without having the reward depend on a whole sequence of other advice.
Observed improvements differ as well. AM’s multi-turn result (SWE) reduces steps at an unchanged resolve rate, whereas Caddie improves the success rate (+25 pp on MuSiQue-Attached, +26 pp on ALFWorld). Caddie also shows zero-shot gains on unseen task families, where AM reports only no degradation.
We reimplemented Advisor Models and trained it on both MuSiQue settings with the same critic model, base model, training budget, DAPO objective and optimizer. “AM periodic” denotes its fixed-schedule advice protocol. Success rates (%, mean ± SEM) are:
In-domain, Caddie improves over AM periodic by +7.4 pp on Attached and +2.4 pp on Wiki. Across all eight settings, Caddie has the higher mean in five settings and is +0.8 pp higher on average. Transfer results are mixed: AM periodic performs better in three of the six transfer settings, by up to 5.3 pp. The clearest advantage of Caddie’s training formulation is therefore in-domain, while both methods retain useful transfer across base models. However, our AM reimplementation shows no gain on τ³ (out-of-domain transfer) contrary to Caddie (table in Weakness 1 in our reply to Reviewer th1X).