acceptodds
Under review as a conference paper at ICLR 2027

Agentic Critical Training

19 views

Abstract

Imitation learning (IL) teaches language-model agents to reproduce expert actions but not to distinguish them from plausible mistakes. Self-reflection methods expose models to alternatives yet use supervised fine-tuning (SFT) to imitate fixed rationales and actions. We introduce Agentic Critical Training (ACT), which uses reinforcement learning with verifiable rewards (RLVR) to train models to judge actions directly. At each expert-trajectory state, ACT pairs an expert action with an alternative sampled from the initial policy and randomizes their order. The model generates its own reasoning but is rewarded only for selecting the expert action. ACT reuses demonstrations, requires no reference rationales, and allows pair reuse across model sizes. ACT is a warm-up before IL, optionally followed by RL; inference requires no candidate comparison. Across Qwen3-8B and Olmo-3-7B-Instruct on ALFWorld-ID, WebShop, and ScienceWorld, ACT yields average gains of 5.85 points over IL and 4.12 points over ILRL without ACT, while also improving ALFWorld-OOD. With Olmo on ScienceWorld, the full pipeline gains 15.36 points over CoT prompting and 9.23 points over ILRL without ACT. Both ACTIL and the full pipeline outperform supervised reflection baselines. Controls with fixed pairs or matched training durations show that the gains stem from the ACT objective rather than additional data or training. Without reasoning-specific post-training, the standalone ACT checkpoint achieves the highest mean among evaluated models on MATH-500 and GPQA-Diamond, showing that action comparison complements generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agents, Reasoning, Agentic RL, Imitation Learning, LLM Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_S68LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new warm-up method that can be followed by supervised learning (SFT) and reinforcement learning (RL) to improve the agentic performance. The proposed agentic critical training (ACT) pairs an expert action with an alternative sampled from the initial policy and employs reinforcement learning with verifiable rewards (RLVR) to train models to judge actions directly. It shows consistent improvements compared to the training baselines on various agentic benchmarks, and the results also show that the ACT warm-up improves agentic OOD performance and transfers across model sizes.

Critical strengths
  • The proposed ACT warm-up method can be combined with SFT and RL training methods, and the proposed method shows consistent improvements over the baselines on the agentic tasks.
  • The data construction in the ACT method could reuse the demonstrations, and doesn't require any reference rationales annotated by human experts, making the ACT training method can be easily applied in practice.
Critical weaknesses
  • My major concern is constructing the negative pair by sampling the alternatives from the initial policy to compare with the expert actions. Although it applies filtering to remove duplicates, I think the sampling process can be quite noisy. How does the author enforce the complexity of contrastive learning to prevent sampling obviously inferior alternatives from the initial policy? Also, the authors simply mentioned, "the majority of sampled alternatives differ from the expert action and are identifiably suboptimal given the task context", but this assumption might not be guaranteed for every task in practice.
  • The proposed ACT method is only evaluated on three benchmarks, including ALFWorld, WebShop, and ScienceWorld; it is still unclear to me whether the ACT method's performance improvement can be generalized to other tasks and model sizes (i.e., whether the performance gains scale with the base model size).
Questions for authors
  • When constructing the pairs, the authors draw KK candidate actions and pair the expert action with each alternative to form the contrastive pairs. In the appendix, the authors mention that they set K=1K=1. Did the authors perform empirical studies to show whether KK affects the training performance?
6 comments
AuthorsOpenReview ↗

Response to Reviewer S68L (1)

We sincerely thank Reviewer S68L for the careful and positive review, and for noting that our paper is clear and easy to follow. We are glad that you recognized that ACT can be combined with SFT and RL training methods to achieve consistent improvements over baselines on agentic tasks, and that reusing demonstrations without human-annotated reference rationales makes ACT easy to apply in practice.


For Weakness 1

Weakness 1: My major concern is constructing the negative pair by sampling the alternatives from the initial policy to compare with the expert actions. Although it applies filtering to remove duplicates, I think the sampling process can be quite noisy. How does the author enforce the complexity of contrastive learning to prevent sampling obviously inferior alternatives from the initial policy? Also, the authors simply mentioned, "the majority of sampled alternatives differ from the expert action and are identifiably suboptimal given the task context", but this assumption might not be guaranteed for every task in practice.

Response:

1. How ACT prevents sampling obviously inferior (trivial) alternatives:

Each alternative action is sampled directly from the pretrained initial policy πθ0(⋅∣si)\pi_{\theta_0}(\cdot | s_i) conditioned on the exact same task context and admissible action set, rather than sampled uniformly at random from the action space. Because the initial LLM policy already possesses basic instruction-following and task comprehension capabilities, obviously nonsensical or trivial actions have very low probability under πθ0\pi_{\theta_0} and are rarely sampled. Instead, the sampled alternatives represent plausible, high-probability mistakes that the initial policy itself is prone to making at state sis_i. This is precisely why we sample negatives from the initial policy rather than using random actions: it automatically calibrates the difficulty of the contrastive pairs to the model's current capability.

2. Why the assumption holds & algorithmic robustness to occasional ties:

  • First, ACT only requires expert demonstrations to be better than initial-policy samples on average, which is consistent with using expert demonstrations as useful supervision for IL and with the preference signal used by RL and our comparison methods. Because general-purpose 7B to 8B models have a clear performance gap relative to expert demonstrators on multi-step agent tasks, their self-sampled actions are predominantly suboptimal.
  • Second, in terms of optimization, ACT is naturally robust to occasional benign or equivalent alternatives. Unlike offline contrastive losses that force a fixed log-probability margin on every static pair, GRPO samples a group of GG on-policy responses per prompt and normalizes advantages using the group mean and standard deviation. Because GRPO normalizes advantages within each sampled group rather than enforcing a fixed margin on every static pair, it does not require every pair to induce a large update. The strict-filtering ablation provides the direct empirical evidence: removing the minority of tied, ambiguous, and reversed pairs changes downstream performance only modestly. Subsequent IL and online RL stages then operate on the resulting warm-up checkpoint.

3. Quantitative Audits of Pair Quality Across Benchmarks (Added to Section 2.2 & Appendix A.5, Table 8): To verify empirically that the majority of sampled alternatives are suboptimal, we conducted two complementary audits on our ACT training pairs across ALFWorld, WebShop, and ScienceWorld:

  • Audit 1: Independent external blind evaluation. We used an unrelated frontier model (gpt-6.1-sol) to perform a double-blind evaluation on 23,384 pairs (10,240 on ALFWorld, 2,904 on WebShop, and 10,240 on ScienceWorld). Hiding the expert label and randomizing candidate order (50/50), the judge classified each pair based on the task goal, action history, current observation, and admissible actions into Alternative Suboptimal (expert better), Equivalent / Tie, Ambiguous / Unscored, or Alternative Better.
  • Audit 2: Simulator dual-branch rollout check. For all 12,725 pairs whose underlying simulator states can be uniquely and deterministically reconstructed from the history prefix (9,532 on ALFWorld, 2,454 on WebShop, and 739 on ScienceWorld), we forked the simulator at that state to execute the expert action and alternative action on two branches, and rolled out 5 complete trajectories per branch using the Qwen3-8B IL policy. Comparing the mean official environment return across the 5 trajectories (0/1 success on ALFWorld, 0 to 1 attribute-match reward on WebShop, and 0 to 1 normalized subgoal score on ScienceWorld), we classified each pair using a 0.05 return margin.
AuthorsOpenReview ↗

Response to Reviewer S68L (2)

BenchmarkAudit MethodAlternative SuboptimalEquivalent / TieAmbiguous / UnscoredAlternative Better
ALFWorldIndependent blind eval (gpt-6.1-sol)72.73%9.29%14.14%3.84%
WebShopIndependent blind eval (gpt-6.1-sol)76.69%5.13%11.67%6.51%
ScienceWorldIndependent blind eval (gpt-6.1-sol)94.13%2.81%0.52%2.54%
ALFWorldSimulator rollout check68.10%13.50%12.57%5.83%
WebShopSimulator rollout check88.83%6.44%0.37%4.36%
ScienceWorldSimulator rollout check74.42%12.45%1.76%11.37%

Both audits confirm that suboptimal alternatives dominate across all three environments (72.73% to 94.13% in blind evaluation; 68.10% to 88.83% in simulator rollouts), while functional ties (2.81% to 13.50%) and better alternatives (2.54% to 11.37%) form a small minority, with 93.21% directional agreement between the two audits.

4. Strict Pair-Filtering Ablation (Added to Section 4.4, Table 4): To test how sensitive ACT is to removing the remaining 17.41% of non-suboptimal pairs in the full training set, we removed all Equivalent / Tie (1,388 pairs), Ambiguous / Unscored (1,840 pairs), and Alternative Better (842 pairs) samples from all 23,384 training pairs and retrained both Qwen3-8B and Olmo-3-7B-Instruct on the 19,314 strictly suboptimal pairs (82.59%):

Qwen3-8B Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage62.14 ± 7.7259.45 ± 9.2321.53 ± 4.5830.04 ± 2.13
ACT stage (Filtered)59.29 ± 4.0860.70 ± 7.0125.53 ± 6.9129.93 ± 0.74
ACT -> IL89.76 ± 1.8883.58 ± 3.8032.00 ± 1.1832.85 ± 0.15
ACT -> IL (Filtered)91.91 ± 2.8887.56 ± 7.7133.33 ± 3.0232.01 ± 0.66
Full ACT pipeline96.19 ± 0.6791.54 ± 0.3542.80 ± 0.5934.63 ± 0.33
Full ACT pipeline (Filtered)95.95 ± 1.8791.79 ± 1.8343.93 ± 0.6233.09 ± 0.76

Olmo-3-7B-Instruct Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage16.90 ± 1.6911.94 ± 3.7111.53 ± 1.3112.03 ± 0.92
ACT stage (Filtered)18.57 ± 2.3417.17 ± 3.3910.33 ± 1.8112.99 ± 0.58
ACT -> IL74.52 ± 2.2167.41 ± 4.0614.80 ± 7.2710.40 ± 0.89
ACT -> IL (Filtered)76.19 ± 5.5473.88 ± 7.0314.87 ± 2.3712.27 ± 0.73
Full ACT pipeline87.14 ± 3.0883.83 ± 0.3524.13 ± 0.4121.67 ± 0.61
Full ACT pipeline (Filtered)83.81 ± 2.4381.09 ± 6.7125.20 ± 1.8521.45 ± 1.47

Across the 24 comparisons, 62.50% (15/24) show modest improvements and 37.50% (9/24) show slight decreases, demonstrating that ACT is highly robust to minor pair noise and that default exact-match deduplication is sufficient in practice.

AuthorsOpenReview ↗

Response to Reviewer S68L (3)

For Weakness 2

Weakness 2: The proposed ACT method is only evaluated on three benchmarks, including ALFWorld, WebShop, and ScienceWorld; it is still unclear to me whether the ACT method's performance improvement can be generalized to other tasks and model sizes (i.e., whether the performance gains scale with the base model size).

Response:

To test ACT's generalization to a new agentic task domain and its scaling behavior on a much larger base model, we conducted two new sets of experiments (reporting the mean and standard deviation across 3 independent end-to-end training runs):

1. Generalization to a New Task Domain: BFCL v4 Multi-Turn (Added to Section 4.1 and 4.2, Tables 1 and 2, Appendix A.3, A.7, A.9) We evaluated ACT on the Berkeley Function Calling Leaderboard (BFCL) v4 Multi-Turn benchmark (Patil et al., ICML 2025) using 1,224 training pairs from the official gold trajectories, evaluating on the in-distribution (ID) Base split (77 episodes) and three out-of-distribution (OOD) splits (Miss-Func, Miss-Param, and Long-Context, 600 episodes in total):

MethodQwen3-8B ID (%)Qwen3-8B OOD (%)Olmo-3-7B-Instruct ID (%)Olmo-3-7B-Instruct OOD (%)
Prompt w/o CoT thinking22.51 ± 1.6216.78 ± 0.781.73 ± 0.610.94 ± 0.42
Prompt w/ CoT thinking28.57 ± 2.1218.72 ± 8.646.49 ± 2.122.67 ± 0.68
ACT stage28.14 ± 10.8823.11 ± 1.328.66 ± 0.613.94 ± 0.64
Imitation Learning (IL)46.75 ± 2.1247.56 ± 1.1642.43 ± 1.6239.50 ± 1.43
Early Experience46.32 ± 2.6745.56 ± 0.4240.26 ± 3.1835.06 ± 2.94
SAND44.16 ± 3.1842.89 ± 2.9938.53 ± 3.4137.05 ± 1.78
Agent-R48.92 ± 1.6244.61 ± 1.6441.13 ± 2.6738.00 ± 2.79
ACT -> IL50.21 ± 4.9049.61 ± 0.2843.29 ± 0.6140.22 ± 0.95
IL -> RL w/o ACT49.35 ± 2.1248.06 ± 1.0644.59 ± 3.4140.00 ± 0.89
Full ACT pipeline52.81 ± 2.6750.33 ± 1.6048.05 ± 2.1241.39 ± 0.78

Across both model families, ACT improves the corresponding downstream IL and IL→RL pipelines on both BFCL v4 ID and OOD splits. The full ACT pipeline achieves the highest performance across all four settings and consistently outperforms the supervised reflection baselines.

2. Model Size Scaling: Scaling Olmo from 7B to 32B (Added to Section 4.2 & Appendix A.12, Table 14) Combining our existing Qwen3-4B cross-size transfer results (Appendix A.11, Table 13) with a larger model scale, we scaled the Olmo backbone over 4×4\times from 7B (Olmo-3-7B-Instruct) to 32B (Olmo-3.1-32B-Instruct) on ALFWorld and WebShop:

Method (Olmo-3.1-32B-Instruct)ALFWorld ID (%)ALFWorld OOD (%)WebShop (%)
Prompt w/o CoT thinking22.62 ± 5.2918.16 ± 3.3615.33 ± 0.66
Prompt w/ CoT thinking27.14 ± 1.1726.87 ± 0.619.27 ± 0.93
ACT stage45.95 ± 6.6337.32 ± 7.6116.73 ± 1.75
Imitation Learning (IL)91.91 ± 2.3688.06 ± 1.2227.87 ± 1.54
ACT -> IL92.38 ± 1.2291.04 ± 0.6131.67 ± 1.81
IL -> RL w/o ACT93.33 ± 1.7887.56 ± 3.3534.40 ± 0.75
Full ACT pipeline94.52 ± 0.8992.54 ± 0.6135.27 ± 1.86

As the base model scales from 7B to 32B, ACT's performance gains remain clear and consistent across both downstream pipelines (ACT -> IL and Full ACT pipeline), boosting ALFWorld-OOD by +2.98 points over IL and +4.98 points over IL->RL w/o ACT, and boosting WebShop by +3.80 points over IL and +0.87 points over IL->RL w/o ACT.

Reviewer_rbyBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic Critical Training (ACT), a reinforcement-learning warm-up for LLM agents before imitation learning (IL), optionally followed by downstream RL. At each expert-trajectory context, ACT pairs the demonstrated action with an alternative sampled from the initial policy, filters exact matches, and randomizes candidate order. GRPO trains the model to generate its own rationale and select an action without reference rationales. The composite reward combines expert-action matching with admissibility and formatting terms; the admissibility term is disabled for WebShop. At inference, the policy generates actions directly without candidate comparisons. Across Qwen3-8B and Olmo-3-7B-Instruct on ALFWorld, WebShop, and ScienceWorld, adding ACT improves the reported mean performance of both IL and IL-to-RL pipelines, including ALFWorld-OOD. Same-pair objective controls and additional-training controls help assess these gains. The paper also reports reuse of 8B-generated pairs for a 4B model and preliminary general-reasoning transfer from the standalone ACT checkpoint.

Critical strengths
  1. ACT offers a practical way to add explicit action-comparison supervision to agent training. It reuses expert trajectories, samples alternatives from the initial policy, and requires neither reference rationales nor additional environment rollouts to label the pairs. Randomized candidate order removes a straightforward positional shortcut. The resulting model can subsequently be trained for direct action generation, making the approach useful without requiring candidate comparisons at deployment.

  2. The empirical evaluation provides meaningful support for ACT as a warm-up. ACT-to-IL and the full ACT pipeline improve over their corresponding baselines in all four reported agent settings for both model families. For example, on Qwen3-8B, ACT-to-IL improves WebShop success from 28.00% to 32.00% and ScienceWorld score from 21.94 to 32.85; on Olmo, the full pipeline improves ScienceWorld from 12.44 to 21.67 relative to IL-to-RL.

Critical weaknesses
  1. The central action-quality supervision is insufficiently validated. ACT treats the demonstrated action as preferred, but exact matching verifies agreement with the demonstration rather than independently verifying greater long-term value. The expert's overall advantage over the initial policy does not establish the ordering of each retained pair, and exact-match filtering does not exclude semantically equivalent actions or alternatives with comparable or better outcomes, particularly for open-ended WebShop search queries. With one sampled alternative per expert context and no quantitative audit of pair quality, it remains unclear how often training requires a meaningful choice between two admissible, plausible actions.

  2. The incremental advantage over alternative training objectives is not yet established at comparable computational cost. Table 3 includes useful same-pair SFT-on-choice, DPO, and no-reasoning controls, as well as extended IL/RL training. However, matching epochs does not establish the compute-matched comparison claimed in the conclusion, because GRPO response sampling and rationale generation have different costs. The margins over DPO are 1.40 points on WebShop and 1.07 points on ScienceWorld, while SFT-on-choice and no-reasoning have higher ALFWorld-OOD means than full ACT. These results support a useful training stage, but make its advantage over the existing objective controls at a comparable budget a question.

Overall, the cross-task improvements and existing controls are meaningful, but the unresolved validity of the preference supervision and the comparison at measured computational budgets leave me insufficiently convinced by the current evidence for this contribution. A targeted label-quality audit and a convincing comparison at comparable cost could change my assessment.

Questions for authors
  1. How many pairs remain after exact-match filtering, and what fraction contain two admissible actions? Could a representative audit, using independent judgments or outcome checks, estimate the prevalence of ties, equivalent actions, and alternatives that are better than the demonstration? Performance on the subset requiring a substantive preference would materially strengthen the central interpretation.

  2. Please report actual training cost for ACT and the objective/continuation controls, including generated tokens or accelerator hours, and clarify checkpoint selection and the reward used in downstream RL. Do ACT's gains over DPO and SFT-on-choice persist under comparable computational budgets? Appendix A.8 describes three seeded evaluations per checkpoint; do the error bars also include independent training runs?

  3. In Table 6, incorrect-rationale accuracy plus the fraction following the injected action sums to 84% for the original model and 79% for ACT. What are the remaining outputs, and how are non-candidate actions and parsing failures scored? Would correct or neutral prefills of comparable length support the same interpretation of the intervention?

  4. Table 8 evaluates only the standalone ACT checkpoint, with gains of 0.80 and 1.85 percentage points over CoT prompting on MATH-500 and GPQA-Diamond. Are these gains retained after the subsequent IL and RL stages under the same evaluation and generation budgets?

8 comments
AuthorsOpenReview ↗

Response to Reviewer rbyB (1)

We sincerely thank Reviewer rbyB for the thorough and constructive review, and for clearly stating what evidence would strengthen our work. We appreciate your positive assessment of ACT's practical action-comparison training objective that reuses expert trajectories without reference rationales or environment rollouts, its randomized candidate order that removes positional shortcuts, and the consistent empirical gains of ACT->IL and the full ACT pipeline across all reported settings and both model families.

For Weakness 1 & Question 1

Weakness 1: The central action-quality supervision is insufficiently validated. ACT treats the demonstrated action as preferred, but exact matching verifies agreement with the demonstration rather than independently verifying greater long-term value. The expert's overall advantage over the initial policy does not establish the ordering of each retained pair, and exact-match filtering does not exclude semantically equivalent actions or alternatives with comparable or better outcomes, particularly for open-ended WebShop search queries. With one sampled alternative per expert context and no quantitative audit of pair quality, it remains unclear how often training requires a meaningful choice between two admissible, plausible actions.

Question 1: How many pairs remain after exact-match filtering, and what fraction contain two admissible actions? Could a representative audit, using independent judgments or outcome checks, estimate the prevalence of ties, equivalent actions, and alternatives that are better than the demonstration? Performance on the subset requiring a substantive preference would materially strengthen the central interpretation.

Response:

1. Theoretical & Algorithmic Grounding:

  • First, ACT trains the model to learn the action preferences of expert models or human demonstrators and only requires expert actions to be better on average, which matches the standard assumption of IL, RL, and all compared baselines. Because Qwen3-8B and Olmo-3-7B-Instruct have a clear capability gap relative to the expert demonstrations, sampling from the initial policy naturally produces suboptimal actions in the vast majority of states.
  • Second, GRPO and our multi-stage pipeline are inherently robust to occasional benign or equivalent alternatives. Unlike offline losses that enforce a fixed log-probability margin on every static pair, GRPO samples a group of GG on-policy responses per prompt and normalizes advantages using the group mean and standard deviation. Because GRPO normalizes advantages within each sampled group rather than enforcing a fixed margin on every static pair, it does not require every pair to induce a large update. The strict-filtering ablation below provides the direct empirical evidence: removing the minority of tied, ambiguous, and reversed pairs changes downstream performance only modestly. Subsequent IL and online RL stages then operate on the resulting warm-up checkpoint.

2. Pair Count After Exact-Match Filtering: In our implementation, when a sampled candidate action exactly matches the expert action, we reject it and resample from the initial policy until a distinct alternative action is obtained. Thus, the number of retained training pairs matches the number of expert state-action pairs reported in Appendix A.3 (Table 6): 10,240 on ALFWorld, 2,904 on WebShop, and 10,240 on ScienceWorld (we have updated the wording in Section 2.2 and Algorithm 1 in Appendix A.1 to state this rejection-resampling step explicitly).

3. Quantitative Audits of Pair Quality (Added to Section 2.2 & Appendix A.5, Table 8): To quantify how often sampled alternatives are suboptimal versus functionally equivalent or superior, we conducted two audits on all 23,384 pairs across ALFWorld, WebShop, and ScienceWorld:

  • Audit 1: Independent external blind evaluation. We used an unrelated strong external model (gpt-6.1-sol) to evaluate 23,384 pairs (10,240 on ALFWorld, 2,904 on WebShop, and 10,240 on ScienceWorld) in a double-blind setup. The expert and alternative actions were placed in Candidate A and Candidate B in 50/50 randomized order with labels hidden, given only the task goal, action history, current observation, and admissible actions, and classified into Alternative Suboptimal (expert better), Equivalent / Tie, Ambiguous / Unscored, or Alternative Better.
AuthorsOpenReview ↗

Response to Reviewer rbyB (2)

  • Audit 2: Simulator dual-branch rollout check. For all 12,725 pairs whose underlying simulator states can be uniquely and deterministically reconstructed from the text history prefix (9,532 on ALFWorld, 2,454 on WebShop, and 739 on ScienceWorld), we forked the simulator at that state to execute the expert action on one branch and the alternative action on the other, and then rolled out 5 complete trajectories per branch using the Qwen3-8B IL policy. Comparing the mean official environment return across the 5 rollouts (0/1 success on ALFWorld, 0 to 1 attribute-match reward on WebShop, and 0 to 1 normalized subgoal progress on ScienceWorld), we categorized each pair as Alternative Suboptimal (alternative step invalid or expert branch mean return higher by at least 0.05), Equivalent / Tie (both branches score positively within a 0.05 margin), Ambiguous / Unscored (both branches score near zero), or Alternative Better (alternative branch mean return higher by at least 0.05).
BenchmarkAudit MethodAlternative SuboptimalEquivalent / TieAmbiguous / UnscoredAlternative Better
ALFWorldIndependent blind eval (gpt-6.1-sol)72.73%9.29%14.14%3.84%
WebShopIndependent blind eval (gpt-6.1-sol)76.69%5.13%11.67%6.51%
ScienceWorldIndependent blind eval (gpt-6.1-sol)94.13%2.81%0.52%2.54%
ALFWorldSimulator rollout check68.10%13.50%12.57%5.83%
WebShopSimulator rollout check88.83%6.44%0.37%4.36%
ScienceWorldSimulator rollout check74.42%12.45%1.76%11.37%

Both audits reach the same conclusion: (1) suboptimal alternatives dominate across all three environments (72.73% / 76.69% / 94.13% in blind evaluation; 68.10% / 88.83% / 74.42% in simulator rollouts); (2) functional ties (9.29% / 5.13% / 2.81% blind; 13.50% / 6.44% / 12.45% simulator) and better alternatives (3.84% / 6.51% / 2.54% blind; 5.83% / 4.36% / 11.37% simulator) are a small minority; and (3) on pairs where both audits render a directional preference, they agree without directional reversal on 93.21% of samples.

4. Retraining on the Subset Requiring a Substantive Preference (Added to Section 4.4, Table 4): Using the blind evaluation results, we removed all Equivalent / Tie (1,388 pairs, 5.94%), Ambiguous / Unscored (1,840 pairs, 7.87%), and Alternative Better (842 pairs, 3.60%) samples, removing 4,070 pairs (17.41%) in total, and retained only the 19,314 strictly suboptimal pairs (82.59%) (7,448 on ALFWorld, 2,227 on WebShop, and 9,639 on ScienceWorld). We retrained both Qwen3-8B and Olmo-3-7B-Instruct on this filtered subset ((Filtered) rows below):

Qwen3-8B Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage62.14 ± 7.7259.45 ± 9.2321.53 ± 4.5830.04 ± 2.13
ACT stage (Filtered)59.29 ± 4.0860.70 ± 7.0125.53 ± 6.9129.93 ± 0.74
ACT -> IL89.76 ± 1.8883.58 ± 3.8032.00 ± 1.1832.85 ± 0.15
ACT -> IL (Filtered)91.91 ± 2.8887.56 ± 7.7133.33 ± 3.0232.01 ± 0.66
Full ACT pipeline96.19 ± 0.6791.54 ± 0.3542.80 ± 0.5934.63 ± 0.33
Full ACT pipeline (Filtered)95.95 ± 1.8791.79 ± 1.8343.93 ± 0.6233.09 ± 0.76

Olmo-3-7B-Instruct Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage16.90 ± 1.6911.94 ± 3.7111.53 ± 1.3112.03 ± 0.92
ACT stage (Filtered)18.57 ± 2.3417.17 ± 3.3910.33 ± 1.8112.99 ± 0.58
ACT -> IL74.52 ± 2.2167.41 ± 4.0614.80 ± 7.2710.40 ± 0.89
ACT -> IL (Filtered)76.19 ± 5.5473.88 ± 7.0314.87 ± 2.3712.27 ± 0.73
Full ACT pipeline87.14 ± 3.0883.83 ± 0.3524.13 ± 0.4121.67 ± 0.61
Full ACT pipeline (Filtered)83.81 ± 2.4381.09 ± 6.7125.20 ± 1.8521.45 ± 1.47

Across the 24 comparisons between (Filtered) and unfiltered rows, 62.50% (15/24) improve and 37.50% (9/24) slightly decrease. This confirms that ACT's gains neither rely on nor are bottlenecked by minor pair noise, and that default exact-match deduplication is sufficient without requiring external judges or simulator rollouts.

AuthorsOpenReview ↗

Response to Reviewer rbyB (3)

For Weakness 2 & Question 2 (Part 1: Training Cost and Budget-Calibrated Comparisons)

Weakness 2: The incremental advantage over alternative training objectives is not yet established at comparable computational cost. Table 3 includes useful same-pair SFT-on-choice, DPO, and no-reasoning controls, as well as extended IL/RL training. However, matching epochs does not establish the compute-matched comparison claimed in the conclusion, because GRPO response sampling and rationale generation have different costs. The margins over DPO are 1.40 points on WebShop and 1.07 points on ScienceWorld, while SFT-on-choice and no-reasoning have higher ALFWorld-OOD means than full ACT. These results support a useful training stage, but make its advantage over the existing objective controls at a comparable budget a question.

Question 2 (Part 1): Please report actual training cost for ACT and the objective/continuation controls, including generated tokens or accelerator hours... Do ACT's gains over DPO and SFT-on-choice persist under comparable computational budgets?

Response:

  1. Clarification on Table 3 controls and terminology: We first note that Full pipeline (DPO) and Full pipeline (SFT-on-choice) in Table 3 are also instantiations of our ACT warm-up framework: they use the exact same ACT contrastive pairs before IL->RL, replacing only the stage-1 optimization algorithm (GRPO) with DPO or SFT-on-choice. We also agree that matching epochs does not equate to matching total compute, and we have updated our terminology in the revised manuscript to distinguish epoch-matched and budget-calibrated controls.
  2. Full End-to-End Computational Cost Profile (Added to Appendix A.6, Tables 9 to 11): Below we report generated rollout tokens (0 for offline stages), total processed prompt + response tokens across forward/backward passes, hardware-independent PFLOPs, and wall-clock NVIDIA GH200 GPU-hours for all pipelines in Table 3 on Qwen3-8B (ordered as ALFWorld / WebShop / ScienceWorld):
Training Pipeline (ALFWorld / WebShop / ScienceWorld)Generated TokensProcessed TokensTotal Compute (PFLOPs)GH200 GPU-Hours
IL (3 epochs)0 / 0 / 014.6M / 10.1M / 40.0M715 / 497 / 1,9660.7 / 1.0 / 1.4
IL -> IL (6 epochs)0 / 0 / 029.1M / 20.2M / 80.0M1,430 / 994 / 3,9331.5 / 2.1 / 2.8
IL -> RL w/o ACT15.7M / 3.5M / 8.9M33.0M / 13.8M / 50.1M2,179 / 796 / 2,7749.6 / 6.3 / 6.5
IL -> RL -> RL (extended RL)78.0M / 23.8M / 96.1M240.2M / 342.8M / 929.6M16,740 / 23,250 / 63,510107.0 / 104.0 / 137.7
Full pipeline (SFT-on-choice, 3 epochs)15.7M / 3.5M / 8.9M55.1M / 49.9M / 62.2M3,267 / 2,568 / 3,36511.4 / 8.4 / 7.4
Full pipeline (DPO, 3 epochs)15.7M / 3.5M / 8.9M77.3M / 85.9M / 74.2M5,079 / 5,523 / 4,35013.7 / 11.1 / 7.9
Full pipeline (no-reasoning, 3 epochs)34.7M / 12.7M / 16.2M233.8M / 304.0M / 153.4M15,644 / 19,958 / 9,658114.5 / 86.7 / 43.4
Full ACT pipeline (default 3 epochs, G=8G=8)187.2M / 37.5M / 670.1M375.6M / 124.5M / 1,086.8M27,677 / 8,773 / 82,184134.0 / 13.7 / 385.5
  1. Budget-Calibrated ACT Experiments (Added to Section 4.3, Table 3): We evaluated a budget-calibrated ACT setting with 1 ACT epoch and GRPO group size G=2G=2. This reduces ACT-stage compute by 12×12\times relative to the default ACT stage. The resulting budget is close to 3-epoch DPO on ALFWorld (5,101 vs. 5,079 PFLOPs), about 70% lower than DPO on WebShop (1,673 vs. 5,523 PFLOPs), and higher than DPO on ScienceWorld (6,258 vs. 4,350 PFLOPs). The downstream results are:
Method (Qwen3-8B, Table 3)ALFWorld ID (%)ALFWorld OOD (%)WebShop (%)ScienceWorld
ACT -> IL (3 epochs, G=8G=8)89.76 ± 1.8883.58 ± 3.8032.00 ± 1.1832.85 ± 0.15
ACT -> IL (budget-calibrated, 1 epoch, G=2G=2)94.29 ± 2.3392.04 ± 2.7540.53 ± 3.5131.98 ± 0.78
Full pipeline (SFT-on-choice, 3 epochs)94.76 ± 1.3492.78 ± 1.4137.73 ± 1.4830.10 ± 0.19
Full pipeline (DPO, 3 epochs)93.57 ± 0.5890.05 ± 1.5341.40 ± 0.9933.56 ± 0.27
Full pipeline (no-reasoning, 3 epochs)94.52 ± 0.6792.54 ± 0.6138.47 ± 2.5128.12 ± 2.33
Full ACT pipeline (3 epochs, G=8G=8)96.19 ± 0.6791.54 ± 0.3542.80 ± 0.5934.63 ± 0.33
Full ACT pipeline (budget-calibrated, 1 epoch, G=2G=2)95.95 ± 2.7694.03 ± 2.4445.00 ± 3.0234.99 ± 1.26
Reviewer_NTfQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Agentic Critical Training (ACT), a RL warm-up framework designed to enhance LM agents ability to distinguish expert actions from plausible but suboptimal alternatives. Rather than relying solely on direct supervision via IL, ACT constructs contrastive action pairs combining expert demonstrations with alternative actions sampled from the initial policy. Evaluated on Qwen3-8B and Olmo-3-7B-Instruct across ALFWorld, WebShop, and ScienceWorld, ACT demonstrates consistent performance gains.

Critical strengths
  • ACT effectively addresses a core limitation of conventional imitation learning by explicitly training agents to distinguish expert decisions from plausible alternatives
  • The paper demonstrates consistent gains across two distinct model families and three standard environments. The authors back these results with a broad suite of controls
Critical weaknesses
  • ACT operates under the assumption that any sampled alternative differing from the expert trajectory is suboptimal. In complex, open-ended sequential decision environments. multiple distinct actions can be equally effective. Because exact-match filtering does not catch semantically equivalent or benignly alternative actions, the dataset likely contains noisy contrastive labels
  • While ACT yields overall performance improvements, the margin over strong baselines in specific settings is relatively narrow (e.g., +0.74 percentage points on ALFWorld-OOD for Qwen3-8B and +0.73 percentage points on WebShop for Olmo-3-7B).
  • Missing related hybrid RL related work (e.g., https://openreview.net/forum?id=vO8LLoNWWk, https://openreview.net/forum?id=vXVjbbcuHy)
Questions for authors
  • Have you conducted a qualitative or quantitative test to determine how often policy-sampled alternative actions are suboptimal versus functionally equivalent to the expert action?
  • How sensitive is ACT's performance to filtering out semantically equivalent or ambiguous actions?-
  • Could you provide token-, FLOP-, or GPU-hour-normalized comparisons across ACT, IL, DPO, and the additional-training controls?
6 comments
AuthorsOpenReview ↗

Response to Reviewer NTfQ (1)

We sincerely thank Reviewer NTfQ for the constructive feedback and for recognizing that ACT effectively addresses a core limitation of conventional imitation learning (IL) by training agents to distinguish expert decisions from plausible alternatives, and that our paper demonstrates consistent gains across two distinct model families and multiple environments backed by a broad suite of controls.

Since Weakness 1, Question 1, and Question 2 build on one another progressively, we address them in sequence below.


For Weakness 1

Weakness 1: ACT operates under the assumption that any sampled alternative differing from the expert trajectory is suboptimal. In complex, open-ended sequential decision environments, multiple distinct actions can be equally effective. Because exact-match filtering does not catch semantically equivalent or benignly alternative actions, the dataset likely contains noisy contrastive labels.

Response:

  1. Consistency with standard agent training assumptions: ACT trains the model to learn the action preferences of expert models or human demonstrators. It only assumes that expert actions are better on average, which is consistent with using expert demonstrations as useful supervision for IL and with the preference signal used by RL and our comparison methods. Because pretrained backbones such as Qwen3-8B and Olmo-3-7B-Instruct have a substantial capability gap relative to human or expert demonstrators, actions sampled from the initial policy are much more likely to be suboptimal for the task. Our empirical results across multiple model families and agent benchmarks confirm that this setup already provides a reliable learning signal.
  2. Ties and superior alternatives are rare in practice: As quantified in Question 1 below, our two independent audits show that functionally equivalent ties and superior alternatives account for only a small fraction of pairs across all benchmarks.
  3. Algorithmic robustness of GRPO and warm-up training: Unlike offline pairwise losses (such as DPO) that enforce a fixed log-probability margin on every static pair, GRPO samples a group of GG on-policy responses per prompt and normalizes rewards using the group mean and standard deviation. Because GRPO normalizes advantages within each sampled group rather than enforcing a fixed margin on every static pair, it does not require every pair to induce a large update. More directly, the strict-filtering ablation below shows that removing the minority of tied, ambiguous, and reversed pairs changes downstream performance only modestly. Subsequent IL and online RL stages then operate on the resulting warm-up checkpoint.
AuthorsOpenReview ↗

Response to Reviewer NTfQ (2)

For Question 1

Question 1: Have you conducted a qualitative or quantitative test to determine how often policy-sampled alternative actions are suboptimal versus functionally equivalent to the expert action?

Response:

To quantify the quality distribution of policy-sampled alternative actions relative to expert actions, we conducted two evaluations on all 23,384 ACT pairs (added to Section 2.2 and Appendix A.5, Table 8):

  • Audit 1: Independent external blind evaluation. We used an external strong model (gpt-6.1-sol) completely unrelated to both trained model families to conduct a double-blind evaluation on all 23,384 pairs (10,240 on ALFWorld, 2,904 on WebShop, and 10,240 on ScienceWorld). Hiding the expert label, we placed the expert action and alternative action into Candidate A and Candidate B in a 50/50 randomized order alongside the task goal, action history, current observation, and admissible actions. Each pair was independently classified into one of four categories: Alternative Suboptimal (expert is better), Equivalent / Tie, Ambiguous / Unscored, or Alternative Better.
  • Audit 2: Simulator dual-branch rollout check. Because some environment states cannot be replayed or involve invalid actions that cannot be executed further, we identified all 12,725 pairs whose underlying simulator states can be uniquely and deterministically reconstructed from the text history prefix (9,532 on ALFWorld, 2,454 on WebShop, and 739 on ScienceWorld) and performed a dual-branch rollout check in the real simulator. From the exact same state, we forked the simulator to execute the expert action on one branch and the alternative action on the other, and then rolled out 5 complete trajectories per branch using the Qwen3-8B IL policy (to avoid low baseline completion rates of the untuned model). We compared the mean official environment return across the 5 trajectories (0/1 task success on ALFWorld, 0 to 1 official attribute-match purchase reward on WebShop, and 0 to 1 normalized subgoal progress score on ScienceWorld):
    • Alternative Suboptimal: the alternative action is invalid at that step, or the expert branch achieves a mean return at least 0.05 higher;
    • Equivalent / Tie: both branches achieve positive return within a 0.05 margin;
    • Ambiguous / Unscored: both branches achieve near-zero return;
    • Alternative Better: the alternative branch achieves a mean return at least 0.05 higher than the expert branch.
BenchmarkAudit MethodAlternative SuboptimalEquivalent / TieAmbiguous / UnscoredAlternative Better
ALFWorldIndependent blind eval (gpt-6.1-sol)72.73%9.29%14.14%3.84%
WebShopIndependent blind eval (gpt-6.1-sol)76.69%5.13%11.67%6.51%
ScienceWorldIndependent blind eval (gpt-6.1-sol)94.13%2.81%0.52%2.54%
ALFWorldSimulator rollout check68.10%13.50%12.57%5.83%
WebShopSimulator rollout check88.83%6.44%0.37%4.36%
ScienceWorldSimulator rollout check74.42%12.45%1.76%11.37%

Both audits yield highly consistent findings:

  1. Across ALFWorld, WebShop, and ScienceWorld, suboptimal alternatives dominate, accounting for 72.73% / 76.69% / 94.13% under blind evaluation and 68.10% / 88.83% / 74.42% under simulator rollouts.
  2. Functionally equivalent ties form only a small minority (9.29% / 5.13% / 2.81% in blind evaluation; 13.50% / 6.44% / 12.45% in simulator rollouts), and alternatives better than the expert are even rarer (3.84% / 6.51% / 2.54% in blind evaluation; 5.83% / 4.36% / 11.37% in simulator rollouts).
  3. On pairs where both audits give a clear directional judgment, the two audits agree without directional reversal on 93.21% of samples.
AuthorsOpenReview ↗

Response to Reviewer NTfQ (3)

For Question 2

Question 2: How sensitive is ACT's performance to filtering out semantically equivalent or ambiguous actions?

Response:

To directly measure ACT's sensitivity to semantically equivalent, ambiguous, or reversed pairs, we conducted a strict filtering ablation based on the full blind audit from Question 1 (added to Section 4.4, Table 4). Specifically, out of all 23,384 training pairs, we removed all Equivalent / Tie pairs (1,388 pairs, 5.94%), Ambiguous / Unscored pairs (1,840 pairs, 7.87%), and Alternative Better pairs (842 pairs, 3.60%), removing 4,070 pairs (17.41%) in total, and retained only the 19,314 strictly suboptimal pairs (82.59%): 7,448 pairs (72.73%) on ALFWorld, 2,227 pairs (76.69%) on WebShop, and 9,639 pairs (94.13%) on ScienceWorld.

Keeping all hyperparameters identical, we retrained Qwen3-8B and Olmo-3-7B-Instruct across all three benchmarks on this strictly filtered subset (reported as (Filtered) rows below each corresponding unfiltered method; mean ±\pm standard deviation across three independent training runs):

Qwen3-8B Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage62.14 ± 7.7259.45 ± 9.2321.53 ± 4.5830.04 ± 2.13
ACT stage (Filtered)59.29 ± 4.0860.70 ± 7.0125.53 ± 6.9129.93 ± 0.74
ACT -> IL89.76 ± 1.8883.58 ± 3.8032.00 ± 1.1832.85 ± 0.15
ACT -> IL (Filtered)91.91 ± 2.8887.56 ± 7.7133.33 ± 3.0232.01 ± 0.66
Full ACT pipeline96.19 ± 0.6791.54 ± 0.3542.80 ± 0.5934.63 ± 0.33
Full ACT pipeline (Filtered)95.95 ± 1.8791.79 ± 1.8343.93 ± 0.6233.09 ± 0.76

Olmo-3-7B-Instruct Filtering Ablation (Section 4.4, Table 4)

MethodALFWorld IDALFWorld OODWebShopScienceWorld
ACT stage16.90 ± 1.6911.94 ± 3.7111.53 ± 1.3112.03 ± 0.92
ACT stage (Filtered)18.57 ± 2.3417.17 ± 3.3910.33 ± 1.8112.99 ± 0.58
ACT -> IL74.52 ± 2.2167.41 ± 4.0614.80 ± 7.2710.40 ± 0.89
ACT -> IL (Filtered)76.19 ± 5.5473.88 ± 7.0314.87 ± 2.3712.27 ± 0.73
Full ACT pipeline87.14 ± 3.0883.83 ± 0.3524.13 ± 0.4121.67 ± 0.61
Full ACT pipeline (Filtered)83.81 ± 2.4381.09 ± 6.7125.20 ± 1.8521.45 ± 1.47

Takeaways:

  1. Comparing each (Filtered) row against its unfiltered counterpart across the 24 stage-benchmark pairs, 62.50% (15/24) show modest improvements and 37.50% (9/24) show slight decreases. Thus, the presence of a small fraction of semantically equivalent or ambiguous actions has very little impact on ACT's performance: our reported gains neither come from noisy pairs nor are noticeably limited by them.
  2. Because ACT is a warm-up stage prior to downstream IL and RL, it does not conflict with data filtering; any filtering method can be applied on top of ACT when desired. However, exact-match deduplication is already sufficient to achieve the reported gains, whereas stricter filtering via external LLM judges or simulator rollouts is a high-cost optional refinement. Retaining exact-match deduplication as our default keeps the pipeline simple and self-contained without sacrificing performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.