acceptodds
Under review as a conference paper at ICLR 2027

Agentic-DPO: Rollout-Free Agent Training from the Student's Own Mistakes

Abstract

Expert trajectories are the cheapest supervision for LLM agents, yet supervised fine-tuning (SFT) never confronts the agent with the mistakes it would itself make at the expert's states. Agentic-DPO mines these mistakes without executing any action during training: at each expert state it samples a few one-step actions from the current student, keeps the most probable one whose decision differs from the expert's, and trains the student to prefer the expert action with a length-scaled DPO loss, stabilized by an SFT anchor and by Policy-Preserving Augmentation (PPA), which re-renders trajectories under different action schemas. We evaluate with ten seeds (five at 32B), validation-selected checkpoints, and twelve tuning configurations for every method. On five closed-loop tasks (τ-bench retail and airline with the standard GPT-4o user, ALFWorld, WebShop, and end-to-end StableToolBench), Agentic-DPO improves over SFT with the same augmentation by 5.9–15.2 points at 9B, matches or exceeds the step-level methods IPR and PivotRL, and recovers 93% (95% CI 77–111) of online GRPO's τ-bench gain with 6.7× fewer GPU-hours; GRPO started from Agentic-DPO has the highest mean on all five tasks. Matched controls show that the choice of negative matters more than the loss: on the same states, continued SFT recovers 21% of the gain and a random negative from the student 69%, whereas losses that keep pushing down the student's most probable mistake recover 88–97%. Its advantage over learning from the student's own samples comes mostly from states where the student never samples the expert's decision, where such methods receive no direct signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI Agent, Post-training, SFT, RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VvgeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Agentic-DPO samples one-step actions from the current student at expert states. It removes invalid actions and actions that are equivalent to the expert's, and then contrasts the expert action with the most probable remaining action. Training combines length-scaled DPO, an SFT anchor, and schema/recovery augmentation that does not change the policy. Sampled actions are never executed to get labels. The evaluation covers closed-loop agent tasks, static action prediction and several backbones. The main claim is that picking the student's likely mistake matters more than the specific preference loss. This gives large gains over augmented SFT, costs less than online RL, and gives a good starting point for later GRPO.

Critical strengths

S1. The mechanism study rules out real alternative explanations: Tables 4 and 24-28 separate the effects of mined-state selection, negative selection, the preference loss, the pairing unit, refreshing and stabilizers, including own-mining controls. The local propositions explain a restricted first update and are honest about their limits, including the unexplained shortfall at K=1K=1. So the paper tells us more than a simple DPO-versus-SFT comparison.

S2. The practical evaluation is broad and careful: The paper reports validation-based selection, multiple seeds, task-and-seed intervals, compute-limited GRPO comparisons, negative audits, and analyses of long-horizon and off-expert states. It is careful to say "close to GRPO", not "equal to GRPO". The fixed-compute controls and the GRPO initialization study (Table 31) are useful for real training decisions, and the alternate expert source and the blind negative audit test key data assumptions.

Critical weaknesses

W1. The supplied negative miner does not do the selection and filtering described in §3.2:

  • The miner does not ask the sampler for log probabilities. When scores are missing, it sets them all to zero, so the "most probable" choice just returns the first eligible candidate. Under the default vLLM V1 settings, scores are indeed missing when log probabilities are not requested. So the runtime version matters directly for the most-probable versus random comparison. I tested this fallback in isolation; I did not reproduce neural sampling with the authors' runtime.
  • The filter compares tool names only. So a same-tool action with wrong arguments is thrown away, while a non-empty string that cannot be parsed can survive, because the parser gives it an empty tool name.
  • The parser also cannot read the JSON-array format that the supplied renderer produces. I rendered all 3,896 provided expert pairs in the JSON-based formats and parsed them back. The parser recovered the right tool name for none of them. This tests whether the renderer and parser fit together, not how often the model outputs invalid actions.

These issues hit the core selection rule and the equivalence filter, not an optional feature.

W2. The supplied training code changes the stated SFT anchor and augmentation:

  • Appendix B says the SFT anchor uses action tokens only. The code also keeps the state and history tokens in the SFT labels and averages their loss. A tokenization and gradient check confirmed that the state prefix does contribute to the anchor.
  • Algorithm 1 re-renders each negative with a fresh, independent schema view. The code instead reuses the view chosen during mining. Warm-up still varies the views, so augmentation is not missing completely.
  • The training script uses quantized LoRA, a 10x higher learning rate, half as many updates per round, and zero reference scores. The paper uses full bf16 fine-tuning, the smaller learning rate, more updates, and the warmed-up reference.

The zero reference and the same-view training may matter little on their own; the near-linear-loss analysis and Table 27 already suggest this. The real problem is knowing which algorithm was evaluated, especially the selector and the anchor. The supplied material says it is the pipeline used for the reported STB results. It may be an old or simplified release, but this needs to be shown, not assumed.

Questions for authors

Q1. Which exact miner, and which fixed runtime version, produced the main STB results and the selection/filter ablations? Please share a small trace that can be replayed: all candidates, parse status, normalized tool and arguments, scores, and the chosen index. Please include same-tool/wrong-argument and malformed examples. How are the JSON-format outputs parsed?

Q2. Which SFT mask, schema-view sampling, reference and optimization settings were used? If the attached code is a simplified version, please map it to the real one. If the reported runs did use the behavior in the attached code, a focused, otherwise matched comparison of most-probable versus random negatives, with these issues fixed, would be needed to support the claimed mechanism.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Vvge

We thank the reviewer for running the supplementary code and providing precise, testable observations. We used W1/W2 as a checklist for a substantial supplementary update: executable tests, complete candidate-level traces, training records, and a short guide mapping each concern to its implementation and evidence.

The earlier supplementary snapshot was outdated. The updated package provides an executable implementation and an additional StableToolBench/Qwen3.5-2B verification run, allowing the selection and training rules to be inspected directly.

W1/Q1: Scoring, decision filtering, and JSON parsing.

1. Explicit action scoring and replayable selection. The miner parses each candidate into a canonical decision, re-renders it under the mining view, and explicitly computes its summed student action log-probability using a separate vLLM scoring pass. Missing or non-finite eligible scores raise an error; they are never replaced by zero.

The included first-round trace contains 2,400 states and all 9,600 candidates. Replaying the records confirms that all 2,009 retained pairs select the highest-scoring eligible candidate. Each record exposes the raw output, parse status, normalized decision, equivalence flag, score, and chosen index. The verification runtime—vLLM 0.19.0, PyTorch 2.10.0+cu128, and Transformers 5.4.0—is recorded in the trace header.

2. Filtering operates on complete decisions. Equivalence compares the canonical tool and normalized arguments. Same-tool actions with different arguments remain eligible, while unparseable outputs are rejected. In the first-round verification trace, 478 selected negatives use the expert’s tool with different arguments, directly illustrating this distinction. All 23 targeted filtering tests pass, including the reviewer’s same-tool/wrong-argument, malformed-output, and expert-equivalent cases.

3. Renderer–parser compatibility is tested across views. The updated parser explicitly supports the JSON-array format produced by the renderer and maps renamed tools back to canonical names. Round-trip checks pass for all 3,892 usable expert targets under each of the four views, with an additional 15,324/15,324 recovery-context checks.

W2/Q2: Action-only anchor, independent views, and training configuration.

1. Action-only supervision. The SFT warm-up and anchor average loss over action tokens only. The supplied checks include 160 tokenizer-sequence checks, a separate logit-gradient check confirming zero direct SFT gradient at state-predicting positions, and agreement between the optimized action-loss computation and a full-forward test calculation.

2. Independent training views. Mined negatives are stored as canonical decisions. At each training draw, the trainer independently samples a new view and re-renders the state, expert action, and negative together.

3. Explicit reference and optimization settings. The verification implementation uses full-parameter training with bf16 computation, learning rate 2e-5, 150 updates per round, effective batch size 16, and the frozen warm-up student as reference. The resolved configuration and first-round training log are included.

For convenience, python3 scripts/check_package.py runs the standard-library checks and complete trace replay; REVIEWER_GUIDE.md maps Q1/Q2 to the relevant files.

These implementation checks complement the paper’s matched selection studies in Table 4. Together, they make the central mechanism concrete: Agentic-DPO identifies and corrects the student’s strongest sampled competing decision at an expert state, extracting targeted corrective supervision from demonstrations without environment rollouts. We appreciate the reviewer’s careful inspection, which helped us make this mechanism substantially easier to verify.

Reviewer_VvgeOpenReview ↗

Follow-up on the updated supplementary material

Thank you for the revised supplement. I have gone through these materials, running the parsers, filters and trace replay, and everything works as described. As far as I can tell, the training scripts also include the action-only SFT losses, the new training views, and the reference models being kept fixed after the warmup period. I consider these points addressed.

One thing I am still unsure about is how the code relates to the reported numbers. From your description, it sounds like you ran additional traces to verify the results. If possible, could you point me to the relevant scripts and hyperparameters for reproducing the STB numbers and the baselines in Table 4? Even a small excerpt of the logs from these runs would be very helpful. I certainly don't expect exhaustive logs or any new runs.

SOURCE_NOTES.txt also seems to describe two options for sampling random negatives. Could you clarify which one was used for the experiments in Table 4?

Thanks again for preparing the additional files. I would simply like to understand a bit better how they connect to the published results.

AuthorsOpenReview ↗

Connecting the Verification Example to Table 4 Experiments

Thank you for taking the time to run the checks. We are glad they resolved the implementation questions, and we appreciate this helpful follow-up.

Your reading is correct: the traces are additional verification runs. We kept the example small so that the core operations could be checked without running the full experimental suite. The included first-round log belongs to that example; Table 4 reports the larger, ten-seed study described in the paper.

To connect the two directly, src/miner.py implements candidate construction and selection, and src/trainer.py implements the preference objective and SFT anchor. The STB columns in Table 4 use Qwen3.5-2B and measure canonical/perturbed next-action accuracy. Their shared preference-training settings are learning rate 2e-5, batch size 16, 150 updates per round, up to five refresh rounds, K=4, β=0.008, α=0.5, and λ=0.5. Training uses full bf16 fine-tuning with AdamW, zero weight decay, and a constant learning rate after 5% linear warm-up; the reference is frozen after SFT warm-up.

The important connection for the baselines is the matched comparison: every Table 4 row starts from PPA+SFT and follows the same augmentation, update budget, and validation-selection protocol. Blocks (a) and (b) also use the same Agentic-DPO-mined states. Block (a) changes the negative, while block (b) changes the objective, with control-specific coefficients selected on validation. This is how the study separates the benefit of correcting a particular student mistake from the effects of additional training or state selection.

On the random-negative question, Table 4 used uniform sampling over distinct valid, expert-different decisions, after parsing and equivalence filtering. Repeated samples of the same decision do not receive extra weight. The candidate-instance option in SOURCE_NOTES.txt refers only to the auxiliary random-choice field in the additional trace, not to the reported Table 4 baseline. We should have made that distinction clearer in our previous response.

We hope this makes the connection easier to follow. Thank you again for helping us make both the implementation and the experimental comparison clearer.

Reviewer_VvgeOpenReview ↗

Follow-up after the clarification

Thank you, this answers my remaining questions. I now understand which training settings you used for Table 4 and how its random-negative baseline differs from the auxiliary choices saved in the verification trace.

I checked the updated package earlier, and its parser, filtering, and trace-replay tests passed. Together with your explanation, this sufficiently addresses my main implementation concerns. Please add these clarifications to the supplementary guide and include the scripts and configurations for the Table 4 variants in the final release.

My original review was positive about the mechanism study and the broader evaluation. Since the implementation concerns were the main reason for my earlier rating, I will raise my rating to 3 (Weak acceptance).

Reviewer_tA8sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agentic-DPO, a demonstration-based training method for LLM agents that does not execute sampled actions during preference-data construction or gradient updates. At expert states, the current student samples candidate actions; after removing unparseable and expert-equivalent candidates, the highest-probability remaining action is paired with the expert action. Training uses a length-scaled DPO-style objective, an SFT anchor, policy-preserving augmentation, and periodically refreshed negatives. The paper evaluates the method across several agent benchmarks and investigates the effects of negative selection, loss formulation, and stabilization.

Critical strengths
  1. Preference pairs are built offline at expert states without executing sampled actions. This avoids the rollout cost of methods such as IPR and ETO, and the paper shows that in agent preference training, choosing the right negative matters more than the choice of loss.

  2. The evaluation is thorough. It spans multiple benchmarks and backbones, uses 10 seeds with confidence intervals and validation-based checkpoint selection, and includes extensive controls: continued SFT, independent re-mining, expert corruption, false negatives, and compute-matched GRPO. The results are practically meaningful. On τ-retail with Qwen3.5-9B, Agentic-DPO nearly matches GRPO (41.0% vs. 42.1%) at about one-seventh of the compute (16.7 vs. 112.2 GPU-hours), clearly outperforms budget-matched GRPO (33.9%), and also serves as an effective initialization for GRPO.

Critical weaknesses
  1. Expert-versus-model preference training and iterative refinement have already been explored in SPIN, DITTO, and IPR. The main new contribution is the decision-aware negative selection and the experiments that validate it.

  2. The implementation details of the closest baselines need clarification for a fair comparison. Appendix B does not make clear whether IPR keeps its original combination of outcome-level DPO, step-level DPO, and SFT. The adapted PivotRL uses the paper's own decision-equivalence checker, which may accept fewer valid alternatives than the original domain-specific verifier.

  3. The mechanistic explanation does not fully account for other reward designs or for actual training behavior.

  4. The results do not show a consistent advantage over competing methods once perturbations similar to training-time PPA are excluded.

Questions for authors

Please see the Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer tA8s - part 1

We thank the reviewer for the careful and constructive assessment, particularly for recognizing the thorough evaluation, practical efficiency, and importance of negative selection. We appreciate the opportunity to clarify the methodological contribution, baseline implementations, and scope of our findings.

W1. Novelty beyond existing iterative preference learning

We agree that expert-versus-student preference training and iterative refinement have established precedents in SPIN, DITTO, and IPR. Our central contribution is a specific principle for selecting informative negatives at agent decision boundaries, together with theoretical and empirical evidence explaining its effectiveness.

The key methodological distinction is the combination of three properties:

  1. Decision-aware selection: We parse candidate actions, remove expert-equivalent decisions, and select the highest-probability remaining action from the current student. This directly targets the strongest competing decision affecting greedy action selection.
  2. A mechanism distinguishing hard-negative selection from ordinary self-generated preferences: Proposition 1 shows that unselected on-policy negatives induce an SFT-aligned expected gradient at initialization. Proposition 2 shows that selecting the strongest competitor maximizes first-order improvement of the expert-versus-competitor margin among the studied sampling rules.
  3. Systematic isolation and validation of this principle: Under matched training conditions (Table 4a), continued SFT recovers 21% of Agentic-DPO's τ-retail gain, a random valid negative recovers 69%, and selecting the most probable negative recovers 100%. Meanwhile, replacing DPO with IPO, CPO, or linear push-down preserves 95–97% of the gain (Table 4b).

Thus, the contribution extends beyond applying known preference optimization techniques to agents: we identify and validate which student-generated negative provides a particularly effective corrective signal, and why. The resulting method requires neither environment execution nor additional reward annotation during preference-pair construction.

W2. Fidelity and fairness of the IPR and PivotRL baselines

We appreciate the request for more precise implementation details.

IPR. The original IPR method combines outcome-level DPO, step-level DPO, and SFT, with Monte Carlo rollouts supplying step-level rewards. Our Appendix B currently emphasizes the step-level reward construction and does not sufficiently specify the full optimization objective. We will clarify the implemented loss components, trajectory-pair construction, reward estimation, and corresponding hyperparameters so that the comparison can be assessed against the original method.

PivotRL. We preserve its core training paradigm: profiling expert states, selecting informative pivots based on reward statistics, and optimizing group-relative advantages with KL regularization. For the verifier, we use the same normalized decision-equivalence checker employed in our method, providing a consistent correctness criterion across the compared methods.

We recognize that this is an adaptation of PivotRL's domain-specific functional verifier and may assign different rewards to some valid alternative actions. We will state this distinction explicitly and clarify its implications for comparison. Our additional controls—particularly the negative-selection ablations in Table 4—are independent of the relative ranking against PivotRL and directly isolate the proposed mechanism.

AuthorsOpenReview ↗

Response to Reviewer tA8s - part 2

W3. Scope of the mechanistic explanation

We agree that the local analysis does not constitute a complete account of neural-network optimization or all possible reward designs. Section 3.4 explicitly states the assumptions: a tabular latent-decision softmax model, analysis at initialization, and a first-order greedy-decision margin.

Importantly, we also investigate whether its predictions correspond to actual training behavior. Appendix F reports that:

  • The most-probable selection rule chooses the greedy competitor in 77% of first-round pairs, compared with 43% for random selection.
  • The predicted ratio of per-pair margin improvement is 1.24, closely matching the measured ratio of 1.26.
  • The contrastive component accounts for approximately 38% of the Adam update norm, despite its relatively small coefficient.

We additionally compare several alternative optimization objectives using the same selected negatives (Table 4b), including group-relative RL, IPO, CPO, KTO, SimPO, hinge, unlikelihood, and linear push-down. The results support the empirical conclusion that informative negative selection is a major driver of improvement under the studied conditions.

We will further distinguish the scope of our local theoretical predictions from the broader empirical findings.

W4. Generalization beyond PPA-like perturbations

We agree with the reviewer's observation regarding Appendix M, Table 38. After excluding the two evaluation operators most similar to training-time PPA, Agentic-DPO obtains 84.1% accuracy, compared with 84.6% for ETO and 84.5% for PivotRL.

This indicates comparable performance to these strong baselines on this particular subset, rather than a consistent advantage over every competing method.

Nevertheless, Agentic-DPO retains a 3.8-point improvement over PPA+SFT (80.3% → 84.1%) on these seven operators. At the more severe level-3 perturbations, it also obtains the highest aggregate mean (79.6%, versus 78.1% for ETO and 77.9% for PivotRL).

More broadly, our primary contribution is supported by closed-loop agent success and controlled training-mechanism comparisons, rather than superiority on every distribution-shift evaluation. Across the five closed-loop tasks in Table 2, Agentic-DPO consistently improves over PPA+SFT, approaches GRPO's performance at substantially lower cost, and provides a strong initialization for subsequent online RL.

We will make this distinction clearer and avoid overstating the generality of our perturbation-robustness results.

We thank the reviewer again for these valuable suggestions, which will help us sharpen the contribution statement and improve the reproducibility and precision of the final manuscript.

Reviewer_pELwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a rollout-free post-training method for LLM-based agents. At each expert state, the method samples candidate actions from the current student model, selects the action with the highest probability that differs from the expert’s decision as a "hard negative," and then employs DPO to steer the model toward preferring the expert action. The core contribution is not the proposal of a new preference loss, but rather the demonstration that training on the most competitive incorrect actions—specifically those chosen by the current student model—is more effective than using random negatives or continuing with standard supervised fine-tuning (SFT). Empirically, this method significantly outperforms SFT across various agent benchmarks and achieves performance comparable to online GRPO at a fraction of the computational cost; it also serves as a superior initialization for GRPO. Its limitations include the fact that its innovation is primarily rooted in the combination of training strategies and negative mining, and it remains constrained by the state-distribution coverage of the expert trajectories.

Critical strengths
  • The paper identifies a distinct flaw in SFT: it reinforces expert actions without explicitly correcting the errors a student is most likely to make. Agentic-DPO addresses this by identifying the action most likely to be chosen by the student in a given expert state and pairing it with the expert’s action to form a preference pair. This approach is intuitive, interpretable, and requires no environment rollouts.
  • Agentic-DPO demonstrates significant performance gains over PPA+SFT across multiple closed-loop agent benchmarks. On the τ-retail benchmark, it achieves a success rate of 41.0 using 16.7 GPU-hours—approaching the 42.1 success rate of GRPO, which requires 112.2 GPU-hours. The paper reports that Agentic-DPO recovers approximately 93% of the gains seen with GRPO at roughly 6.7 times lower computational cost.
  • Beyond reporting final performance, the paper systematically compares various negative selection strategies and preference loss functions. The results show that selecting the "most probable" negative action significantly outperforms random negative sampling. Furthermore, methods such as IPO, CPO, SimPO, and linear push-down—when applied using the same "hard" negatives—all recover the majority of the gains typically associated with DPO. This effectively supports the paper’s core conclusion: that the strategy for negative selection is more critical than the specific loss function employed.
Critical weaknesses
  • In terms of its individual components, the method is primarily a combination of expert trajectories, student-generated hard negatives, DPO, iterative mining, and data augmentation. Techniques such as DPO, hard-negative mining, and self-generated negatives have existing precedents; therefore, the innovation here lies more in the "effective combination and discovery of mechanisms tailored for agentic scenarios" rather than in the proposal of an entirely new optimization framework. The paper itself acknowledges that the concept of "most-violating output selection" shares connections with ideas found in structured prediction. 31736_Agentic_DPO_Rollout_Free
  • Agentic-DPO primarily learns from states covered by expert trajectories. If a student model deviates from an expert trajectory during a long-horizon task and enters a new state, this method cannot directly provide a corrective signal—whereas online RL allows for training on the actual states visited by the student model itself. The paper explicitly positions this approach as an effective initialization for online RL, rather than a complete replacement for it.
Questions for authors
  • Compared to existing preference learning methods—such as those employing hard-negative mining or self-generating samples via the model itself—what are the fundamental methodological differences and novel contributions of Agentic-DPO, beyond its application of these strategies to "expert states"?

  • Given that Agentic-DPO primarily learns from an "expert-state distribution," how does the method handle subsequent errors that arise when a student model deviates from expert trajectories—entering states not covered by the expert data—during long-horizon tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pELw

We thank the reviewer for the thoughtful and constructive assessment, and particularly for recognizing the effectiveness, computational efficiency, and mechanistic insights of Agentic-DPO. We address the two questions below.

Q1. Methodological novelty relative to prior preference learning and hard-negative mining

We agree that DPO, hard-negative mining, and self-generated negatives each have established precedents. Our contribution lies in identifying which negative actions provide useful learning signals for agentic decision-making, why they work, and how to obtain them without environment interaction.

We would like to highlight three distinctions:

(1) Decision-level, student-adaptive negative selection. Rather than contrasting arbitrary model-generated responses or failed trajectories, Agentic-DPO selects the highest-probability decision-different action from the current student at the same expert state, with decision-equivalence filtering and iterative re-mining. This targets the strongest competing action that directly determines whether greedy decoding follows the expert.

(2) A mechanistic explanation supported by controlled experiments. Section 3.4 establishes that unselected on-policy negatives produce an expected gradient aligned with SFT at initialization (Proposition 1), whereas selecting the strongest competitor maximizes the first-order improvement in the expert-versus-competitor margin among the considered sampling rules (Proposition 2). The experiments validate this distinction: on τ-retail, continued SFT on the same mined states recovers only 21% of Agentic-DPO's gain, a random valid negative recovers 69%, and the most probable negative recovers 100% (Table 4a). In contrast, changing DPO to IPO, CPO, or linear push-down retains 95–97% of the gain (Table 4b).

These findings establish a general training insight: the choice of which student mistake to correct can matter substantially more than the particular preference optimization loss.

(3) Direct expert supervision at states where self-sampled rewards are uninformative. Unlike sample-only approaches such as PivotRL, Agentic-DPO always supplies the expert action as the preferred alternative when a valid negative is found. It therefore provides a corrective signal even when none of the student's sampled actions matches the expert. On the 16% of held-out expert states where PPA+SFT samples zero correct actions in eight attempts, Agentic-DPO reduces greedy error from 96% to 52%, compared with 90% for PivotRL (Figure 2b; Appendix G).

Taken together, our contribution is a specific, empirically validated principle for extracting substantially stronger supervision from existing expert trajectories, with no environment rollouts during preference training.

Q2. Errors after deviating from expert trajectories

We agree that expert-state coverage is an important limitation, particularly for long-horizon tasks. We address this at two complementary levels.

(1) Partial robustness through recovery-context augmentation. Our Policy-Preserving Augmentation (PPA) introduces recovery contexts containing earlier failed, side-effect-free tool calls, while preserving the environment state and the correct next decision (Section 3.3; Appendix N). This exposes the student to some locally perturbed histories beyond the original expert trajectory. Removing these recovery contexts reduces τ-retail success from 41.0% to 36.6% (Table 4c), indicating that such augmentation contributes to closed-loop robustness.

(2) Complementarity with online RL for genuinely off-distribution states. Section 4.5 and Appendix G explicitly evaluate this limitation. On states visited during the student's own test episodes, GRPO achieves a lower action-error rate than Agentic-DPO (16.1% vs. 19.8%), while Agentic-DPO performs better on held-out expert states (11.8% vs. 19.8%). The difference is most apparent on tasks requiring at least ten expert actions, where GRPO outperforms Agentic-DPO by 6.2 points (Table 30).

These results motivate a two-stage training strategy: Agentic-DPO first efficiently corrects likely mistakes at expert-covered states, and online RL subsequently improves decisions under the student's own state distribution. Indeed, Agentic-DPO → GRPO improves τ-retail success from 42.1% (GRPO alone) to 46.2%, while reducing total training cost from 112.2 to 70.4 GPU-hours (Table 2). The combination also achieves the lowest measured error on both expert and student-visited states (Appendix G, Table 29).

We therefore position Agentic-DPO as a compute-efficient way to extract stronger supervision from expert trajectories and as a complementary initialization for online RL, particularly when long-horizon exploration and recovery are important.

We appreciate the reviewer's questions and will further clarify these methodological distinctions and scope boundaries in the revised manuscript.

I will remain my score. And good luck for your paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.