- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces DriveLab, an autonomous research framework designed for long-horizon, computationally expensive machine learning research, specifically instantiated on autonomous vehicle (AV) motion prediction. It organizes exploration via researcher, coder, and evolution-manager agents around WOMD and reports consistent Soft mAP gains for Wayformer and MotionLM under a fixed 24-hour budget, while identifying the balance between idea novelty and code-execution stability as a key design consideration.
Critical strengths
The paper makes a methodological contribution by shifting focus from purely final benchmark numbers to evaluating the research process itself. It introduces metrics such as Unproductive Attempt Rate (UAR), Active Utilization Rate (AUR), and LLM tokens per valid experiment, as well as an independent 2D rubric (Scope and Novelty). Standard baselines exhibited a clear coupling where higher novelty triggered catastrophic code failures (UAR rising from 22.0% to 35.5%). DriveLab effectively breaks this coupling, achieving both the highest Novelty (1.21-1.35) and the lowest UAR (3.8%-10.7%).
Critical weaknesses
Already stated in the Questions.
Questions for authors
- From the ablation study in Table 2, it appears that the Exploration term contributes more prominently than the other two components. Could the authors clarify what β = 0 represents in this context and how it relates to in Eq.1? Please add a theoretical analysis explaining why the Exploration term yields such a large improvement in UAR.
- If DriveLab were initialized without expert-authored seed skills (specifically without WOMD failure-diagnosis and interactive-kernel-smoke-test), does the automatic skill distillation mechanism alone manage to recover execution stability, or does UAR degrade back to baseline levels?
2 comments
Thank you for your rebuttal. I have read your responses carefully and appreciate the clarifications. After reconsidering the paper in light of the rebuttal, I will maintain my original score.
We thank the reviewer for recognizing our process evaluation, especially the distinction between ambitious ideas and reliable execution.
Q1: What does β = 0 mean, and why does UAR change?
In Eq. 1, the Aggregator ranks each parent–hypothesis pair using normalized parent Soft mAP, the Reviewers' expected gain/risk score, and an exploration bonus β_t·u(h, 𝒜). Here, u measures dissimilarity from previously attempted edits using deterministic token-edit similarity (Eq. 3). Full DRIVELAB uses β_t = 1.5 × 0.92^t, encouraging diverse hypotheses early and gradually shifting toward exploitation. β = 0 removes only this bonus; Reviewers, Coders, skills, and execution safeguards remain unchanged.
Our presentation of Table 2 should have made two results clearer. First, removing exploration lowers UAR from 10.5% to 5.7%, rather than increasing it. The largest UAR increase occurs when skill distillation is removed (10.5% → 27.6%). Second, removing exploration produces the smallest observed drop in final Soft mAP (0.0256), compared with removing skill distillation (0.0282) or the Researcher Team (0.0435). Its most pronounced effect is on Novelty (1.48 → 0.75). These differences are descriptive, not estimates of relative causal contributions.
The explanation follows our design rationale in Sec. 3: expected progress from testing hypothesis h depends on both its execution success probability and its potential gain if successful:
E[Δf | h, c] ≈ P(valid | h, c) · E[Δf | h, c, valid].
The exploration bonus changes which hypotheses are selected, not how they are executed. Without exploration, selection relies on parent performance and the Reviewers' gain/risk assessments, which already penalize implementation risk. This tends to favor familiar, incremental changes that are easier to execute but offer limited improvement. With exploration, structurally different hypotheses receive additional weight. These may be harder to implement but offer opportunities for larger gains.
Table 2 is consistent with this trade-off. Without exploration, 94.3% of attempts produce valid evaluations, compared with 89.5% for full DRIVELAB. Yet the full system achieves approximately 1.5× the improvement over the initial checkpoint (+0.0727 vs. +0.0471 Soft mAP). Skill distillation helps manage the additional implementation risk: retaining exploration but removing distillation raises UAR to 27.6%.
The exploration bonus also discourages repeated edits, which could reduce UAR because duplicate attempts count as failures (App. A). Nevertheless, the observed net UAR is higher with exploration, consistent with the greater difficulty of the selected hypotheses.
This is a mechanistic explanation rather than a theoretical guarantee, and Table 2 contains only one open-ended episode per variant. Sec. 4.3 already notes that β = 0 yields the lowest UAR; we will make the comparison and its interpretation clearer in the revision.
Q2: Can automatic distillation recover without expert seed skills?
We have not run a no-seed-skills experiment, so we cannot determine whether automatic skill distillation alone would recover execution stability. This was a deliberate choice in our experimental design. We provide basic infrastructure workflows to all agents because our goal is to study their ability to conduct autonomous model research, rather than requiring them to first discover how to operate a particular computing or evaluation environment. Specifically, the two expert seed skills serve different purposes:
interactive-kernel-smoke-test(Listing 8) gives Coders a procedure for hot-reloading JAX modules and checking shapes and gradients before expensive training runs.womd-failure-diagnosis(Listing 10) gives Researchers a procedure for inspecting prediction errors through scenario-level diagnostics and BEV plots. It also contains an expert-defined failure taxonomy, which helped identify the problems in Case 2, but did not prescribe the algorithmic solution.We view these primarily as environment-specific procedures rather than research solutions. For example, requiring agents to rediscover how to operate an interactive JAX kernel or which diagnostic tools are available would introduce an additional infrastructure-discovery problem. We chose to standardize these procedures so that differences in research performance would not be confounded by differences in how effectively agents learn to operate the environment.
We acknowledge that the distinction between infrastructure and research is not absolute. Better diagnostic tools can influence which hypotheses agents pursue, and improved execution procedures can make more ambitious experiments feasible. Our matched evaluation controls access to these tools and skills across all baselines, but does not measure their absolute contribution. Allowing agents to develop and improve the research infrastructure alongside the models is an interesting direction for future work.