- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a new warm-up method that can be followed by supervised learning (SFT) and reinforcement learning (RL) to improve the agentic performance. The proposed agentic critical training (ACT) pairs an expert action with an alternative sampled from the initial policy and employs reinforcement learning with verifiable rewards (RLVR) to train models to judge actions directly. It shows consistent improvements compared to the training baselines on various agentic benchmarks, and the results also show that the ACT warm-up improves agentic OOD performance and transfers across model sizes.
Critical strengths
- The proposed ACT warm-up method can be combined with SFT and RL training methods, and the proposed method shows consistent improvements over the baselines on the agentic tasks.
- The data construction in the ACT method could reuse the demonstrations, and doesn't require any reference rationales annotated by human experts, making the ACT training method can be easily applied in practice.
Critical weaknesses
- My major concern is constructing the negative pair by sampling the alternatives from the initial policy to compare with the expert actions. Although it applies filtering to remove duplicates, I think the sampling process can be quite noisy. How does the author enforce the complexity of contrastive learning to prevent sampling obviously inferior alternatives from the initial policy? Also, the authors simply mentioned, "the majority of sampled alternatives differ from the expert action and are identifiably suboptimal given the task context", but this assumption might not be guaranteed for every task in practice.
- The proposed ACT method is only evaluated on three benchmarks, including ALFWorld, WebShop, and ScienceWorld; it is still unclear to me whether the ACT method's performance improvement can be generalized to other tasks and model sizes (i.e., whether the performance gains scale with the base model size).
Questions for authors
- When constructing the pairs, the authors draw candidate actions and pair the expert action with each alternative to form the contrastive pairs. In the appendix, the authors mention that they set . Did the authors perform empirical studies to show whether affects the training performance?
6 comments
Response to Reviewer S68L (2)
| Benchmark | Audit Method | Alternative Suboptimal | Equivalent / Tie | Ambiguous / Unscored | Alternative Better |
|---|---|---|---|---|---|
| ALFWorld | Independent blind eval (gpt-6.1-sol) | 72.73% | 9.29% | 14.14% | 3.84% |
| WebShop | Independent blind eval (gpt-6.1-sol) | 76.69% | 5.13% | 11.67% | 6.51% |
| ScienceWorld | Independent blind eval (gpt-6.1-sol) | 94.13% | 2.81% | 0.52% | 2.54% |
| ALFWorld | Simulator rollout check | 68.10% | 13.50% | 12.57% | 5.83% |
| WebShop | Simulator rollout check | 88.83% | 6.44% | 0.37% | 4.36% |
| ScienceWorld | Simulator rollout check | 74.42% | 12.45% | 1.76% | 11.37% |
Both audits confirm that suboptimal alternatives dominate across all three environments (72.73% to 94.13% in blind evaluation; 68.10% to 88.83% in simulator rollouts), while functional ties (2.81% to 13.50%) and better alternatives (2.54% to 11.37%) form a small minority, with 93.21% directional agreement between the two audits.
4. Strict Pair-Filtering Ablation (Added to Section 4.4, Table 4): To test how sensitive ACT is to removing the remaining 17.41% of non-suboptimal pairs in the full training set, we removed all Equivalent / Tie (1,388 pairs), Ambiguous / Unscored (1,840 pairs), and Alternative Better (842 pairs) samples from all 23,384 training pairs and retrained both Qwen3-8B and Olmo-3-7B-Instruct on the 19,314 strictly suboptimal pairs (82.59%):
Qwen3-8B Filtering Ablation (Section 4.4, Table 4)
| Method | ALFWorld ID | ALFWorld OOD | WebShop | ScienceWorld |
|---|---|---|---|---|
| ACT stage | 62.14 ± 7.72 | 59.45 ± 9.23 | 21.53 ± 4.58 | 30.04 ± 2.13 |
| ACT stage (Filtered) | 59.29 ± 4.08 | 60.70 ± 7.01 | 25.53 ± 6.91 | 29.93 ± 0.74 |
| ACT -> IL | 89.76 ± 1.88 | 83.58 ± 3.80 | 32.00 ± 1.18 | 32.85 ± 0.15 |
| ACT -> IL (Filtered) | 91.91 ± 2.88 | 87.56 ± 7.71 | 33.33 ± 3.02 | 32.01 ± 0.66 |
| Full ACT pipeline | 96.19 ± 0.67 | 91.54 ± 0.35 | 42.80 ± 0.59 | 34.63 ± 0.33 |
| Full ACT pipeline (Filtered) | 95.95 ± 1.87 | 91.79 ± 1.83 | 43.93 ± 0.62 | 33.09 ± 0.76 |
Olmo-3-7B-Instruct Filtering Ablation (Section 4.4, Table 4)
| Method | ALFWorld ID | ALFWorld OOD | WebShop | ScienceWorld |
|---|---|---|---|---|
| ACT stage | 16.90 ± 1.69 | 11.94 ± 3.71 | 11.53 ± 1.31 | 12.03 ± 0.92 |
| ACT stage (Filtered) | 18.57 ± 2.34 | 17.17 ± 3.39 | 10.33 ± 1.81 | 12.99 ± 0.58 |
| ACT -> IL | 74.52 ± 2.21 | 67.41 ± 4.06 | 14.80 ± 7.27 | 10.40 ± 0.89 |
| ACT -> IL (Filtered) | 76.19 ± 5.54 | 73.88 ± 7.03 | 14.87 ± 2.37 | 12.27 ± 0.73 |
| Full ACT pipeline | 87.14 ± 3.08 | 83.83 ± 0.35 | 24.13 ± 0.41 | 21.67 ± 0.61 |
| Full ACT pipeline (Filtered) | 83.81 ± 2.43 | 81.09 ± 6.71 | 25.20 ± 1.85 | 21.45 ± 1.47 |
Across the 24 comparisons, 62.50% (15/24) show modest improvements and 37.50% (9/24) show slight decreases, demonstrating that ACT is highly robust to minor pair noise and that default exact-match deduplication is sufficient in practice.
Response to Reviewer S68L (3)
For Weakness 2
Weakness 2: The proposed ACT method is only evaluated on three benchmarks, including ALFWorld, WebShop, and ScienceWorld; it is still unclear to me whether the ACT method's performance improvement can be generalized to other tasks and model sizes (i.e., whether the performance gains scale with the base model size).
Response:
To test ACT's generalization to a new agentic task domain and its scaling behavior on a much larger base model, we conducted two new sets of experiments (reporting the mean and standard deviation across 3 independent end-to-end training runs):
1. Generalization to a New Task Domain: BFCL v4 Multi-Turn (Added to Section 4.1 and 4.2, Tables 1 and 2, Appendix A.3, A.7, A.9) We evaluated ACT on the Berkeley Function Calling Leaderboard (BFCL) v4 Multi-Turn benchmark (Patil et al., ICML 2025) using 1,224 training pairs from the official gold trajectories, evaluating on the in-distribution (ID) Base split (77 episodes) and three out-of-distribution (OOD) splits (Miss-Func, Miss-Param, and Long-Context, 600 episodes in total):
| Method | Qwen3-8B ID (%) | Qwen3-8B OOD (%) | Olmo-3-7B-Instruct ID (%) | Olmo-3-7B-Instruct OOD (%) |
|---|---|---|---|---|
| Prompt w/o CoT thinking | 22.51 ± 1.62 | 16.78 ± 0.78 | 1.73 ± 0.61 | 0.94 ± 0.42 |
| Prompt w/ CoT thinking | 28.57 ± 2.12 | 18.72 ± 8.64 | 6.49 ± 2.12 | 2.67 ± 0.68 |
| ACT stage | 28.14 ± 10.88 | 23.11 ± 1.32 | 8.66 ± 0.61 | 3.94 ± 0.64 |
| Imitation Learning (IL) | 46.75 ± 2.12 | 47.56 ± 1.16 | 42.43 ± 1.62 | 39.50 ± 1.43 |
| Early Experience | 46.32 ± 2.67 | 45.56 ± 0.42 | 40.26 ± 3.18 | 35.06 ± 2.94 |
| SAND | 44.16 ± 3.18 | 42.89 ± 2.99 | 38.53 ± 3.41 | 37.05 ± 1.78 |
| Agent-R | 48.92 ± 1.62 | 44.61 ± 1.64 | 41.13 ± 2.67 | 38.00 ± 2.79 |
| ACT -> IL | 50.21 ± 4.90 | 49.61 ± 0.28 | 43.29 ± 0.61 | 40.22 ± 0.95 |
| IL -> RL w/o ACT | 49.35 ± 2.12 | 48.06 ± 1.06 | 44.59 ± 3.41 | 40.00 ± 0.89 |
| Full ACT pipeline | 52.81 ± 2.67 | 50.33 ± 1.60 | 48.05 ± 2.12 | 41.39 ± 0.78 |
Across both model families, ACT improves the corresponding downstream IL and IL→RL pipelines on both BFCL v4 ID and OOD splits. The full ACT pipeline achieves the highest performance across all four settings and consistently outperforms the supervised reflection baselines.
2. Model Size Scaling: Scaling Olmo from 7B to 32B (Added to Section 4.2 & Appendix A.12, Table 14)
Combining our existing Qwen3-4B cross-size transfer results (Appendix A.11, Table 13) with a larger model scale, we scaled the Olmo backbone over from 7B (Olmo-3-7B-Instruct) to 32B (Olmo-3.1-32B-Instruct) on ALFWorld and WebShop:
Method (Olmo-3.1-32B-Instruct) | ALFWorld ID (%) | ALFWorld OOD (%) | WebShop (%) |
|---|---|---|---|
| Prompt w/o CoT thinking | 22.62 ± 5.29 | 18.16 ± 3.36 | 15.33 ± 0.66 |
| Prompt w/ CoT thinking | 27.14 ± 1.17 | 26.87 ± 0.61 | 9.27 ± 0.93 |
| ACT stage | 45.95 ± 6.63 | 37.32 ± 7.61 | 16.73 ± 1.75 |
| Imitation Learning (IL) | 91.91 ± 2.36 | 88.06 ± 1.22 | 27.87 ± 1.54 |
| ACT -> IL | 92.38 ± 1.22 | 91.04 ± 0.61 | 31.67 ± 1.81 |
| IL -> RL w/o ACT | 93.33 ± 1.78 | 87.56 ± 3.35 | 34.40 ± 0.75 |
| Full ACT pipeline | 94.52 ± 0.89 | 92.54 ± 0.61 | 35.27 ± 1.86 |
As the base model scales from 7B to 32B, ACT's performance gains remain clear and consistent across both downstream pipelines (ACT -> IL and Full ACT pipeline), boosting ALFWorld-OOD by +2.98 points over IL and +4.98 points over IL->RL w/o ACT, and boosting WebShop by +3.80 points over IL and +0.87 points over IL->RL w/o ACT.
Response to Reviewer S68L (1)
We sincerely thank Reviewer S68L for the careful and positive review, and for noting that our paper is clear and easy to follow. We are glad that you recognized that ACT can be combined with SFT and RL training methods to achieve consistent improvements over baselines on agentic tasks, and that reusing demonstrations without human-annotated reference rationales makes ACT easy to apply in practice.
For Weakness 1
Response:
1. How ACT prevents sampling obviously inferior (trivial) alternatives:
Each alternative action is sampled directly from the pretrained initial policy πθ0(⋅∣si) conditioned on the exact same task context and admissible action set, rather than sampled uniformly at random from the action space. Because the initial LLM policy already possesses basic instruction-following and task comprehension capabilities, obviously nonsensical or trivial actions have very low probability under πθ0 and are rarely sampled. Instead, the sampled alternatives represent plausible, high-probability mistakes that the initial policy itself is prone to making at state si. This is precisely why we sample negatives from the initial policy rather than using random actions: it automatically calibrates the difficulty of the contrastive pairs to the model's current capability.
2. Why the assumption holds & algorithmic robustness to occasional ties:
3. Quantitative Audits of Pair Quality Across Benchmarks (Added to Section 2.2 & Appendix A.5, Table 8): To verify empirically that the majority of sampled alternatives are suboptimal, we conducted two complementary audits on our ACT training pairs across ALFWorld, WebShop, and ScienceWorld:
gpt-6.1-sol) to perform a double-blind evaluation on 23,384 pairs (10,240 on ALFWorld, 2,904 on WebShop, and 10,240 on ScienceWorld). Hiding the expert label and randomizing candidate order (50/50), the judge classified each pair based on the task goal, action history, current observation, and admissible actions into Alternative Suboptimal (expert better), Equivalent / Tie, Ambiguous / Unscored, or Alternative Better.