acceptodds
Withdrawn from ICLR 2027

Agentic DAgger: Self-Improving Interactive Imitation Learning via Agentic Harness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Interactive imitation learning (IIL), such as DAgger, improves a policy by collecting expert feedback on learner-visited states. However, conventional IIL typically relies on a static feedback-generation pipeline comprising a likely suboptimal expert and training experience with diminishing learning value as the policy improves. Consequently, further policy improvement can be bottlenecked by both the quality of expert feedback and the utility of accumulated experience. We introduce Agentic DAgger, a training harness that jointly improves feedback quality and the learning value of experience alongside the learner. Agentic DAgger employs two specialized agent roles operating through verifier-grounded agentic loops. Within each training round, a feedback-evolution loop diagnoses original expert proposal, propose refined corrective labels, and verifies the labeling quality. Between rounds, an experience-evolution loop analyzes the learner-expert gap on the current training experience and acquires novel training experiences that maximize learning value. Together, these loops transform conventional IIL into a self-improving training process that progressively improves corrective feedback and the learning value of training experience. We evaluate Agentic DAgger by improving pretrained policies on NAVSAFE-∞, a verifiable closed-loop driving benchmark. Our experiments show that Agentic DAgger achieves higher closed-loop driving success than conventional IIL baselines under matched training budgets. It further improves learner performance across successive IIL rounds, demonstrating sustained policy self-improvement. These findings demonstrate the effectiveness of Agentic DAgger and highlight its potential to sustain policy self-improvement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Interactive Imitation Learning, Self-Improving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3tjuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic DAgger, a training harness that improves expert feedback and adapts training experience while retaining the standard DAgger learner update. The feedback-evolution loop refines corrective actions through iterative proposal and verification. The experience-evolution loop uses learner performance and expert–learner gap evidence to propose curriculum changes, which are evaluated through subsequent learner training. The paper evaluates the approach in closed-loop driving on NAVSafe-∞, using DrivoR as the learner and PDM-Closed as the base expert. Comparisons with IIL baselines and single-loop variants show improved driving success, with the combined system achieving further gains over successive training cycles.

Critical strengths

1. A meaningful connection between expert correction and curriculum selection.

Curriculum selection considers both where the learner struggles and whether the expert can provide useful corrections. Improved feedback can therefore make previously unproductive scenarios useful for training. This gives the two loops a clear connection and offers a practical way to address limitations in both supervision quality and training experience.

2. Empirical evidence that the two loops are complementary.

Figure 4 compares standard DAgger, each single-loop variant, and the full system. The combined approach reaches 76.78% success, compared with 65.00% for standard DAgger, and exceeds both single-loop variants. These results provide evidence that jointly adapting feedback and experience benefits the downstream learner in the evaluated driving setup.

Critical weaknesses

1. Limited methodological novelty; the contribution is mainly system integration.

The method follows the straightforward idea of improving expert feedback, selecting more useful training experience, and then updating the learner. Connecting feedback evolution with experience evolution allows curriculum selection to account for both learner performance and the expert’s ability to provide corrections. This has system-level value, but the contribution mainly lies in organizing and automating the training process, with limited methodological novelty.

2. The relative advantages of the agentic design are not sufficiently established.

The experiments support the effectiveness of feedback evolution and experience evolution. However, they do not show what advantages state-specific repair offers over using an agent to improve the base expert itself. They also do not establish whether the experience-evolution agent selects training scenarios more effectively than simple strategies. These missing comparisons, together with limited reporting of total cost, make it difficult to assess whether the gains justify the proposed design’s added complexity and computational overhead.

Questions for authors

Q1. Could using an agent to improve the base expert itself be more effective and less costly than state-specific repair?

Since the agent already diagnoses expert failures and proposes repairs, could it instead modify PDM-Closed’s program logic and retain those improvements for reuse? The current feedback-evolution loop refines corrective actions at individual interventions rather than modifying the expert program. What advantages does this choice offer over agent-driven improvements to the base expert in correction quality, downstream learner performance, runtime, and total cost?

Q2. What does the feedback-evolution loop contribute in practice, and at what cost?

The paper lacks aggregate statistics on intervention frequency, the types of failures addressed, repair success rates, and overall computational cost. This makes it difficult to assess how much useful supervision the loop provides relative to its overhead. It is also unclear how many successful repairs select candidates already in the initial menu and how many require new trajectory proposals and further search. Since candidates are generated and scored before the agent’s first proposal, this distinction matters for understanding the agent’s role.

Q3. Does the experience-evolution agent select training scenarios more effectively than simple strategies?

The current comparison mainly contrasts fixed and adaptive curricula. It does not establish whether the experience-evolution agent outperforms random scenario replacement or selection based on the expert–learner gap. With the same access to scenarios and selection constraints, and comparable total budgets, does its diagnosis and selection improve learner performance or curriculum-search efficiency beyond the benefit of changing training scenarios alone?

Reviewer_si69OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Agentic DAgger, an interactive imitation learning (IIL) framework built around an Agentic Harness, aiming to address two limitations of conventional IIL: suboptimal expert feedback and the diminishing learning value of training experience. The framework retains the standard DAgger learner update while introducing two Diagnose–Propose–Verify loops. Feedback Evolution uses an LLM agent, trajectory refinement tools, and a rule-based verifier to improve suboptimal expert corrections during interaction. Experience Evolution adapts the training curriculum between rounds based on learner–expert performance gaps and previous training outcomes.

The authors evaluate the method on the NAVSAFE-∞ closed-loop driving benchmark, starting from a pretrained DrivoR policy. The results show that Agentic DAgger improves expert feedback quality and achieves better benchmark performance than the compared baselines across multiple training rounds.

Critical strengths

Strength 1. The paper integrates expert feedback refinement and adaptive curriculum selection into a unified IIL training framework, addressing both corrective label quality and the learning value of training experience. Although the underlying ideas of these components are not entirely new, their integration offers some engineering and practical value.

Strength 2. The method shows promising results in closed-loop driving experiments. The authors also provide useful discussions of practical issues, including objective hacking, search stagnation, and objective mismatch in experience selection.

Strength 3. The paper is well motivated and clearly presented, with comparisons against multiple IIL baselines and component-level ablations. Overall, the experimental presentation is reasonably comprehensive.

Critical weaknesses

Weakness 1. The novelty and necessity of the Agentic mechanism remain unclear. Both suboptimal expert feedback and diminishing learning value have been extensively explored in prior work. Although this paper integrates the two directions into an Agentic Harness, it does not sufficiently clarify what new capabilities emerge beyond a straightforward combination of the two components, or why LLM agents are necessary instead of simpler search or curriculum selection methods.

Weakness 2. The lack of comprehensive cost accounting makes it difficult to assess whether the additional overhead is justified. The paper emphasizes matched training budgets, but these budgets mainly cover data collection and learner updates rather than the full cost of the Agentic Harness. The proposed method introduces additional operations, including LLM calls, counterfactual trajectory simulations, verifier evaluations, learner–expert probing, curriculum proposals, candidate learner training, and unsuccessful trials. Without reporting cumulative token usage, simulation and verification costs, and overall training time, it is difficult to assess the trade-off between performance gains and additional computational overhead.

Weakness 3. The evaluation involves adaptive reuse of the benchmark. The same 280 NAVSAFE-∞ scenarios are used for diagnosis and candidate selection in Experience Evolution, as well as for reporting final performance. This may introduce selection bias, making the results more reflective of iterative optimization on a development benchmark than independent generalization. The paper also lacks evaluation on a held-out test set that has not been involved in these decisions, as well as a clear distinction between IID and OOD performance.

Questions for authors

Q1. Core novelty. Prior work on suboptimal expert action correction, verifier-guided trajectory optimization, and data or curriculum selection methods such as DataEnvGym, STAT, and DataMIL is closely related to the two components of this framework. Could the authors clarify what they consider the main contribution of this paper? Compared with a straightforward combination of existing expert action search and adaptive curriculum selection methods, what additional capabilities does Agentic DAgger provide?

Q2. Implementation details and reproducibility of Experience Evolution. Appendix A, Table 3, and Algorithm 1 provide the tool descriptions and overall workflow, but several important implementation details remain unclear. For example, how does search_experience retrieve and rank candidate scenarios? How many rollouts are performed by probe_learner and probe_feedback, and how are the learner–expert gaps aggregated? How does replace_experience determine the final set of scenarios to replace, and how are conflicts among candidates resolved? How are the thresholds in Eq. (3) and the maximum number of replacements configured? It would be helpful to clarify which decisions are made by the LLM agent and which are handled by deterministic rules, and to provide actual configurations or representative tool execution traces.

Q3. Necessity of the Agentic Harness. The current experiments do not sufficiently demonstrate the benefits of introducing computationally expensive LLM agents. For Feedback Evolution, could the authors provide a fair comparison against non-agentic verifier-guided search? For Experience Evolution, could the authors include a simple curriculum selection heuristic that directly ranks and replaces scenarios based on learner–expert gaps? These comparisons would help isolate the benefits of agentic decision-making from those provided by the tools and verifiers themselves. In addition, the framework uses a powerful model, Claude Opus 5, as an additional source of intelligence. Could this raise fairness concerns when comparing against conventional IIL methods? How would the performance and cost change when using smaller or less capable LLMs?

Q4. Significant drop in Comfort. In Table 1, Agentic DAgger achieves a Comfort score of 66.72, substantially lower than the 95.71 reported for standard DAgger. Could the authors explain whether this is an intentional trade-off for improvements in other metrics, or an unintended consequence of the verifier design or changes in the corrective data distribution? In particular, Section 4.4 removes Extended Comfort from the internal optimization objective. Is this design choice related to the final policy's lower Comfort score?

Q5. Computational budget. Could the authors report the total computational cost of Agentic DAgger and standard DAgger over the entire training process, rather than only the budget for individual training attempts? I would particularly like to see cumulative LLM token usage, the number of simulations and verification calls, additional costs from candidate learner training and failed trials, and a corresponding performance–cost comparison.

Q6. Independent evaluation and framework generalization. Given that the same NAVSAFE-∞ scenarios are used for diagnosis, candidate acceptance, and performance reporting, could the authors introduce separate development and test sets and evaluate the framework on scenarios that have never been used for curriculum adaptation or model selection?

I also understand that autonomous driving provides relatively well-defined verifiers and repeatable simulation-based interactions, making it naturally suitable for IIL and particularly well aligned with the design of Agentic DAgger. However, this also raises concerns about how much the framework depends on these favorable properties. In other tasks where reliable verifiers are unavailable, or where counterfactual simulations and repeated interactions are expensive, would the two evolution loops still work effectively? I would appreciate further discussion of these limitations and, if possible, evidence of transfer to other tasks along with the associated cost analysis.

Reviewer_McaEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agentic DAgger, a training harness that addresses the bottlenecks of interactive imitation learning (IIL). Conventional IIL relies on a static feedback pipeline, limited by expert quality and the diminishing utility of accumulated experience. The harness introduces two LLM-driven, verifier-grounded agentic loops: a within-round feedback-evolution loop that diagnoses and corrects expert labels, and a between-rounds experience-evolution loop that retires saturated scenarios and acquires new, high-value experiences, forming a self-evolving curriculum.

Critical strengths

(i)A well-motivated, verifier-gated re-framing of IIL bottlenecks.The two loops map one-to-one onto two concrete DAgger limits — an expert that more queries cannot repair, and curriculum saturation — and every agent proposal is gated by rule-based verification rather than free-form LLM labels. (ii)The authors surface and mitigate three failure modes — verifier reward hacking, proposal-search stagnation, and capability-label/learning-value mismatch — and release the full prompts, tool catalogs, and Algorithm 1.

Critical weaknesses

(i)Privileged information and a stronger expert confound the main comparison, and the controls needed to attribute the gain to the harness are missing.Only Agentic DAgger is marked as having privileged simulator access (Table 1): its supervision comes from a Claude Opus 5 agent with bird's-eye renders, ground-truth objects, and resettable rollouts, whereas baselines use PDM-Closed or humans. There is no information-parity baseline and no LLM-without-harness control. (ii)Baselines, statistics, checkpoint selection, and cost reporting are weak.PPL is reported as “PPL (L = 0)” with predicted-state preference propagation omitted, HG-DAgger drops the original risk-threshold analysis, and supervision comes from only three humans; PVP/PPL underperform the pretrained checkpoint, signaling truncated baselines.Results are single runs with no variance or significance . The best-of-three checkpoint per cycle is reported without its selection set (potential test-set leakage). LLM/simulator cost and rejected-trial overhead are unreported; matched budgets cover only episodes and checkpoints. (iii)Framework-level generality is asserted but shown on a single favorable setting.One learner, one expert, one benchmark (280 verifiable driving scenarios), and one LLM are used.

Questions for authors

(i) Which simulator signals does each actor receive, and how much of the gain survives when the privileged tools are removed (information-parity)? (ii)Can you provide an LLM-without-harness control, and give a strong baseline equal tool/verifier access, to isolate the harness from the LLM's own planning ability? (iii)Are Table 1/Figure 4 single runs? Please provide multi-seed variance/significance and state the set on which the best-of-three checkpoint is selected. (iv) How much of the PVP/PPL deficit is due to the omitted components (PPL L=0; HG-DAgger risk threshold), and can you provide full implementations with more than three supervisors? (v) What is the total LLM/simulator cost over 10 cycles (including rejected trials), and is there independent validation (blind human rating or heterogeneous metrics) given the overlap between the verifier and the reported DS/SR metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.