acceptodds
Under review as a conference paper at ICLR 2027

Closed-Loop Agentic Reinforcement Learning for Vision-Language Driving Agents

Abstract

Vision-language driving policies commonly reason through a fixed chain of structured questions before acting, asking the same questions at every frame. On Bench2Drive-VL, a closed-loop benchmark that scores driving and reasoning on the same episodes, a model answering this chain drives worse than one answering nothing: which questions a frame warrants is itself a decision, and a fixed chain never makes it. Adaptive-reasoning methods do make it, but optimise the accuracy or cost of the answers where they are produced; the value of a question is realised only in the trajectory that follows. We formalise reasoning-chain selection as a closed-loop Markov decision process, which makes this mismatch explicit. On this basis we introduce DynaChain, an agentic driving policy that decides at every frame which questions to ask before committing to an action. We train it with closed-loop agentic reinforcement learning, under a hybrid advantage that combines the episode's driving outcome with a dense per-step reward derived from the benchmark's scoring rule. DynaChain raises the Driving Score of its base model from 58.93 to 79.15 while asking 2.52 questions per decision instead of seven, and outperforms two adaptive-reasoning methods. Our analysis shows that the gain comes from which questions the agent asks rather than how accurately it answers them, and that it requires both learned selection and closed-loop training. We will release our source code.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language models, autonomous driving, reinforcement learning, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_My2SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

DynaChain addresses the failure of fixed question-answering chains in closed-loop vision-language driving. A Qwen2.5-VL-3B agent selects relevant questions or commits to driving actions within a closed-loop decision process, and reinforcement learning combines route-level outcomes with dense step-wise feedback. On the ten-route Bench2Drive-VL evaluation, the method reports a Driving Score of 79.15 versus 58.93 without a reasoning chain and 42.24 with a fixed chain, using 2.52 questions per decision on average.

Critical strengths
  1. A meaningful failure mode motivates the work. Fixed reasoning chains may introduce irrelevant answers that distract a driving policy. Treating question selection as part of decision making addresses a practical limitation of vision-language driving agents.
  2. Closed-loop reasoning selection offers a coherent formulation. Modeling questions and action commitment as sequential decisions connects reasoning choices to later driving outcomes. This formulation moves beyond optimizing answer accuracy or reasoning length in isolation.
  3. Hybrid reinforcement learning provides a practical credit-assignment design. Combining route-level returns with step-wise control and infraction feedback gives useful signals at different time scales. Assigning delayed infraction penalties to preceding steps further targets decisions that contribute to failures.
  4. Reported driving improvements are substantial. Driving Score rises to 79.15, exceeding the no-chain baseline of 58.93 and reproduced AlphaDrive at 66.68. Average question usage also decreases to 2.52 from seven in the fixed-chain setting.
  5. Ablations examine several central modeling choices. Comparisons distinguish open-loop and closed-loop training, individual reward terms, and learned versus random question selection. Reported outcomes support the combined value of learned selection and closed-loop feedback.
Critical weaknesses
  1. Evaluation independence is unclear due to dev10-based model selection. Submitted training scripts describe using dev10 for validation and selecting checkpoints by its mean Driving Score, while the reported results also use dev10. Reuse of evaluation routes for model selection could overestimate generalization and requires an independent held-out test.
  2. A ten-route evaluation is insufficient for robust conclusions. Main driving metrics come from only ten scenarios, with a 50% success rate representing five successful routes. Additional routes, repeated seeds, and uncertainty estimates are needed to establish that the gains are stable.
  3. Privileged training answers introduce a train-test mismatch. Closed-loop reinforcement learning supplies expert answers to selected questions, whereas evaluation relies on the model’s own potentially inaccurate answers. Resulting selection policies may be vulnerable to answer noise or errors in safety-critical scenes.
  4. Benefits cannot be attributed exclusively to question selection. Reinforcement-learning updates also affect reasoning and committed driving actions rather than only the tool-selection policy. A frozen action-decision component would better isolate whether the reported gains come from choosing questions.
  5. More competitive low-cost selection baselines are needed. Existing controls cover random question subsets and selected adaptive reasoning systems but not simple risk-aware or uncertainty-triggered question selection. Such comparisons are important for establishing whether closed-loop RL is necessary to obtain similar decisions.
  6. End-to-end inference cost and deployment feasibility remain unclear. Each selected question requires an additional model call that processes the scene separately, despite the reduced question count. Without end-to-end latency and compute measurements, real-time driving practicality remains uncertain.
Questions for authors
  1. Did checkpoint selection or hyperparameter tuning use dev10? Please clarify the exact evaluation protocol and report results on an untouched route split.
  2. Can authors provide repeated-seed uncertainty and results on a larger, independent collection of driving scenarios?
  3. How does closed-loop training perform when question answers come from the model itself or are perturbed instead of supplied by an expert?
  4. Can the action-generation policy be frozen while training question selection to isolate the source of improvement?
  5. How do risk-aware or uncertainty-triggered selection baselines perform under a matched question budget?
  6. What are the complete per-decision latency and compute costs, including separate question-answering calls?
Reviewer_muEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the limitation of fixed-chain reasoning for vision-language driving agents, where the same predefined questions are asked at every frame regardless of the driving context. The authors formulate the choice of reasoning questions as a sequential decision problem whose value is determined by the subsequent closed-loop driving outcome. Based on this formulation, they propose DynaChain, an agentic policy that adaptively selects which reasoning questions to invoke at each frame. The method uses a three-stage training pipeline combining supervised fine-tuning and closed-loop reinforcement learning with a modified GRPO objective. Its hybrid advantage combines episode-level driving outcomes with dense per-step rewards derived from the benchmark’s scoring rules. Experiments are conducted on the Bench2Drive-VL closed-loop CARLA benchmark. The results show substantial improvements in Driving Score over fixed-chain and adaptive-reasoning baselines, while ablations indicate that both learned question selection and closed-loop reinforcement learning contribute to the reported gains.

Critical strengths
  1. The formalization in Section 3.1 casts reasoning-chain selection as a closed-loop Markov decision process. This provides a clear theoretical perspective for analysing why static fixed-question chains may be suboptimal in driving scenarios, and establishes a useful motivation for adaptive agentic reasoning in VLA driving agents.
  2. The hybrid advantage formulation in Section 3.3 is well motivated. It combines an episode-level driving outcome with dense per-step rewards derived from infraction penalties and the benchmark’s scoring rule. This addresses the mismatch between a sparse route-level driving score and the frame-level question-selection decisions made during closed-loop driving.
  3. The paper includes comprehensive ablation studies in Section 4.3. The experiments isolate the contributions of learned question selection, closed-loop training, and the different reward components. The diagnostic statistics on question-selection behaviour also help interpret which questions the policy activates, going beyond aggregate benchmark scores.
Critical weaknesses
  1. The novelty is limited. The agentic fremework is simple with some simple tools and few steps. And agentic GRPO is widely studied.
  2. This agentic framework has little application value because of latency. 3.The evaluation provides limited evidence for statistical reliability and transferability. The main results are reported on the ten-route dev10 set and use the benchmark’s fixed question catalogue, expert-answer format, controller, and scoring protocol. The paper does not report route-level uncertainty or test whether the learned selection policy transfers to different scenario sets, simulators, controllers, or interfaces. The fixed question catalogue limits the scope of the agentic reasoning claim. DynaChain can only select and order 13 manually curated benchmark questions and cannot formulate a new question when required information is absent from the catalogue. The paper does not test robustness to removing essential questions, adding irrelevant questions, or changing the question wording and answer format. It is therefore unclear whether the learned policy generalizes beyond this specific tool set.
Questions for authors
  1. What happens when an essential question is removed, irrelevant questions are added, or the question wording and answer format are changed? Could you evaluate whether DynaChain learns a transferable question-selection strategy rather than one specialized to the curated catalogue?
  2. Could you report wall-clock latency, generated-token count, and computational cost as a function of the number of selected questions? Could you also provide a cost-aware ablation showing the trade-off between question budget and closed-loop Driving Score?
1 comment

Thanks for the detailed response. The reported 4.41s per decision also remains far from real-time deployment. And comparing to Qwen model makes no sense. Although the method reduces latency relative to the fixed chain, the absolute cost still limits its practical value as a driving agent. I will keep my rating.

Reviewer_UjEEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DynaChain, a driving agent that learns which questions to ask before acting, using closed-loop RL with dense and episode-level rewards. It reports a driving score of 79.15 on dev10, compared with 58.93 for the no-chain base model, while asking fewer questions than the fixed chain.

Critical strengths

The problem is well motivated, and the method is clearly presented. The structured tool interface makes the policy easy to inspect, and deriving infraction weights from the benchmark’s scoring coefficients is a sensible design choice. The reported results are promising.

Critical weaknesses

The contribution of question selection is not sufficiently isolated. The trained policy generates both tool choices and driving actions. Table 2 lacks a no-tool policy trained with the same initialization, reward, and budget. The random-selection ablation shows that the trained system benefits from its choices but does not establish the advantage over an equally trained direct-action policy. Similar average reasoning scores also do not rule out improvements in particular decision-relevant answers.

The closed-loop comparison changes the reward as well. Open-loop RL uses only the dense reward and scores 68.99; the full method adds both closed-loop execution and the episodic term. With dense reward alone, closed-loop RL scores 64.15. These results support the combined recipe, but the claimed contribution of closed-loop training needs a more careful interpretation.

The evaluation leaves the stability of the gains unclear. Results are limited to ten routes and one base model, without reported uncertainty across repeated runs. The evaluation repetition counts and the rule for selecting checkpoint 50 should also be clarified.

Questions for authors

Did you evaluate a no-tool policy trained from the same initialization with the same closed-loop reward and training budget? If so, how does it compare with DynaChain? This comparison would help establish the benefit of adaptive question selection beyond improving the action policy through RL.

How many independent training runs and evaluation repetitions underlie the main results, and how was checkpoint 50 selected? Can you provide the available per-route results and variability across runs, particularly for DynaChain and the strongest baseline?

Comments on the paper

1 comment
Area_Chair_gxwaOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.