acceptodds
Withdrawn from ICLR 2027

Sequential Group-Relative Test-Time Adaptation for LLM-planner on a Closed-loop Single Driving Timeline

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based motion planners still lose substantial closed-loop performance in unseen driving domains, motivating online test-time adaptation from reward feedback. Two structural obstacles stand in the way: per-step gradient updates on a multi-billion-parameter policy are expensive, and critic-free group-relative methods such as GRPO normalize each rollout against other rollouts from the same state, which a deployed planner never observes on its single, irreversible driving timeline. We propose TeTO (Test-time Trajectory-level Optimization). Its core, Sequential Grouping, reformulates group-relative optimization along time, normalizing each rollout against the planner's immediately preceding rollouts in short, non-overlapping windows. The deviation of this baseline from the unobservable same-state baseline is bounded by the local reward drift, which is small under local smoothness, plus a sampling term; empirically, a three-step window matches the reward variance of same-observation groups, and its advantages agree in sign with a same-state reference in 92.3% of logged windows without abrupt time-to-collision changes. Reward Truncation further restricts updates to windows whose weighted per-component minimum reward falls below a threshold, and only LoRA adapters are updated, with the frozen backbone serving as the KL reference. On nuPlan, TeTO improves the supervised planner by up to 16.7% in an out-of-domain city while updating on as few as 7.8% of steps, with consistent gains across two LLM backbones, three unseen cities, and a planner trained on mixed nuPlan–Waymo data. To our knowledge, TeTO is the first critic-free group-relative policy optimization method for a single executed driving timeline whose baseline has a bounded deviation from the same-state baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-based motion planning, closed-loop, domain shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZM8SOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TeTO for test-time adaptation of LLM-based driving planners. Its core is 1) using sequential grouping to estimate relative advantages and 2) selectively updating LoRA via reward truncation. Experiments in nuPlan show improved closed-loop scores with fewer updates than dense adaptation.

Critical strengths
  1. The paper addresses a problem that an LLM-planner is hard to adapt to RL post-training due to sampling and expensive per-step updates.
  2. The evaluation includes two LLM backbones, additional target cities, mixed nuPlan–Waymo supervised training, and controls for update frequency and gating. The analysis of temporal baseline errors also makes the approximation and its limitations explicit.
Critical weaknesses

The method is mainly evaluated in a narrow city-transfer setting (Las Vegas to Singapore), with additional cities reported. However, the paper didn't show more evidence under different OOD settings to support its claims. This missing breadth is a major limitation for the paper’s significance. Details are provided below:

  1. Limited Novelty and Insufficient Validation of Sequential Grouping. Sequential Grouping is a relatively incremental modification of GRPO, and its reliability under changing driving contexts remains a concern. In Appendix Tables 8–9, advantage-sign agreement drops from over 92% to around 62% during TTC reward transitions, with substantially increased context-induced baseline error. While overall planning scores improve, the paper does not establish whether updates based on these less reliable signals benefit subsequent safety-critical decisions.

  2. Cross-city evidence is uneven. The main Test14-Hard evaluation focuses on Las Vegas-to-Singapore transfer. Appendix E adds Pittsburgh and Boston on Test14-Random, so the study is not restricted to one target city. Nevertheless, the detailed comparisons and methodological analyses remain concentrated on Singapore. The mixed nuPlan–Waymo experiment broadens training data but still evaluates in nuPlan on Las Vegas and Singapore. It is not cross-dataset evaluation. More consistent evaluation across the existing target cities would better support broad domain-shift robustness claims.

  3. Limited comparison. Lack of comparison against non-LLM-based methods and LLM-based methods. For non-LLM-based methods, the author should compare those non-LLM-based RL methods, like PlannerRFT[1], and LLM-based methods, like DriveGPT4 and LanguageMPC, mentioned in the paper.
    [1] PlannerRFT: Reinforcing Diffusion Planners through Closed-Loop and Sample-Efficient Fine-Tuning, CVPR'26

Questions for authors

See weakness

Reviewer_YkRcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TeTO, a reward-driven test-time adaptation method for an SFT-trained 7B LLM motion planner running closed-loop in nuPlan. Because GRPO needs several rollouts from the same state, which a deployed planner cannot observe, TeTO uses "Sequential Grouping". Rewards of the N=3 most recent executed plans, collected in non-overlapping windows under a fixed policy, are standardized into advantages. The paper bounds the deviation of this baseline from the same-state mean by local reward drift plus sampling error (Eq. 5). "Reward Truncation" triggers an update only when a weighted per-component minimum reward in the window falls below 0.70. Updates are one clipped, KL-regularized step on a LoRA adapter, with the ratio taken against the frozen reference. The adapter is reset at each scenario. The reward combines comfort, TTC, speed adherence and progress, following Plan-R1. On Test14-Hard, TeTO + RT improves Singapore (OOD) NR-CLS from 62.09 to 72.47 for Llama-2-7B, and from 75.32 to 85.75 for Qwen2.5-7B, updating on 7.8–19.5% of steps. Additional results cover Test14-Random cities, mixed nuPlan–Waymo SFT, matched-update controls, and a same-state cross-evaluation (92.3% advantage sign agreement in windows without TTC transitions).

Critical strengths
  1. Clearly identified, practically relevant problem with a simple solution. The mismatch between same-state group baselines and single-timeline deployment is real and well articulated (Fig. 1). Sequential Grouping is a minimal, inexpensive way to get a locally calibrated baseline. The paper also checks it against a same-state reference (App. B.4) and reports where it breaks down (TTC transitions: 62% sign agreement).
  2. Consistent improvements with sparse updates across several settings. Gains hold across two backbones, four cities, mixed-data SFT, and both reactive and non-reactive traffic. Reward Truncation cuts update cost by >90% relative to dense updates (Sec. 4.4) while matching or improving scores. The matched-update control against a running-mean baseline (Table 4) is a useful attempt to isolate the contribution of the local baseline.
Critical weaknesses
  1. The reward is closely aligned with the evaluation metric and is optimized on the test scenarios themselves, and the natural test-time baseline that uses the same evaluator is missing. The adaptation reward (comfort, TTC, speed-limit adherence, progress) mirrors core sub-scores of the nuPlan CLS, and it is computed from the simulator state by a trajectory evaluator at every step of the evaluated scenario. The authors acknowledge the overlap (Limitations; App. A.2). As a result, the gains (e.g., Qwen Singapore NR 75.3 → 85.8, above the in-domain SFT score) may partly reflect online optimization toward the metric rather than domain adaptation. This is consistent with the fact that the adapter is reset every scenario: nothing about the new city is retained, so the method is per-scenario reward optimization rather than learning the target domain. The paper also shows that the same evaluator can score unexecuted candidate plans at the current state (App. B.1, B.3–B.4). So the obvious baseline is Best-of-N with the same reward: sample N plans and execute the highest-scoring one, possibly with fewer reasoning tokens to match compute. This is the standard "sample and score" approach (cf. rule-based PDM-style scoring in nuPlan) and needs no gradient updates. Without it, the case for gradient-based TTA over reward-guided selection is not made. Its cost (G× decoding) should be weighed against TeTO's at matched wall-clock.
  2. No variance estimates, small effect sizes among methods, and a weak theoretical claim. Decoding uses temperature 0.75 and the simulator is stochastic in reactive mode, yet Tables 2–4 report single numbers without seeds, standard deviations or confidence intervals. Several key differences are around 1 CLS point: TeTO vs running mean, 77.92 vs 76.77; TeTO + RT vs RWR in Singapore NR, 72.47 vs 72.39; vs REINFORCE++, 72.21. The Table 21 per-city study uses one scenario per type (five scenarios per city). Repeated runs with intervals are needed before ranking methods. In addition, Eq. (5) is the triangle inequality, and the Lipschitz bound assumes exactly the local smoothness that fails at the safety-relevant moments (TTC transitions) where updates matter most. Presenting it as "bounded deviation from the same-state baseline" (abstract, contributions) overstates what is guaranteed.
Questions for authors
  1. How does TeTO compare to Best-of-N reranking with the same reward evaluator (N ∈ {2, 4, 8}), at equal and at matched wall-clock compute? And to TeTO with the KL/LoRA update replaced by Best-of-N plus SFT on the selected plans?
  2. Can you report means ± std (or CIs) over at least three runs for Tables 2–4? Which differences are statistically reliable?
  3. To separate metric optimization from domain adaptation: what happens if (a) the adapter is not reset across scenarios within a city, and is evaluated on held-out scenarios after adapting on others; and (b) the reward excludes components that appear in CLS, e.g. progress only plus a learned or heuristic comfort proxy?
  4. The importance ratio is taken against π_ref rather than the sampling policy. Since the sampling policy (current θ) differs from π_ref after the first update, the resulting gradient is not the on-policy clipped objective. Can you discuss the bias this introduces, and whether a ratio against the collection policy with more than one epoch per window behaves differently?
  5. Generation exceeds one second per 0.1 s control step (App. F). How do you envision deployment, and does the advantage of TeTO persist if replanning is done at a realistic frequency (e.g., 2 Hz)?
  6. Table 3: were PlanTF, PLUTO, Diffusion Planner and Plan-R1 trained on the same Las Vegas-only data? Plan-R1 drops to 22–26 in Singapore. Is this a reproduction, and under what setup?
Reviewer_N7CCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies test-time adaptation of an LLM motion planner in nuPlan closed-loop simulation. The base planner is Llama-2-7B, fine-tuned on 0.3M Las Vegas examples in InstructChain format (reasoning followed by 16 waypoints). At test time, a fresh LoRA adapter is trained online inside each 15 s scenario and reset between scenarios. The reward is a Plan-R1-style weighted score over comfort, TTC, speed adherence and route progress. A trajectory evaluator computes it from the current simulator state and the selected 8 s plan.

The method has two parts; Sequential Grouping (SG): Same-state GRPO groups are replaced by non-overlapping windows of N=3N=3 consecutive executed plans. Rewards are standardized within each window. A clipped surrogate is computed against the frozen reference (obtained by disabling the adapter) plus a KL term. Reward Truncation (RT): An update is issued only when the weighted per-component minimum reward in the window is at or below η=0.70\eta = 0.70. The deviation of the window mean from the same-state mean is bounded by a triangle-inequality argument (Eq. 5). Offline diagnostics report shows reward variance at N=3N=3 that matches 20-sample same-observation groups; and 92.3% sign agreement with a same-state plan-scoring reference on windows without a TTC transition.

On Test14-Hard Singapore (OOD), TeTO + RT raises NR-CLS from 62.09 to 72.47 and R-CLS from 72.33 to 77.96, using 11.7 and 28.3 updates per scenario. Further results cover a Qwen2.5-7B backbone, Test14-Random in four cities, and a planner trained on mixed nuPlan+WOMD data. A matched-update control replacing the window mean with a running mean gives 76.77 vs 77.92 R-CLS.

Critical strengths
  1. Broad, fair comparison of adaptation objectives: Six objectives are moved onto the same single-timeline interface with identical backbone, LoRA configuration, reward and decoding, in both dense and sparse variants (Tables 2 and 17). The paper also includes a matched-objective running-mean control (Table 4). This makes the comparison among adaptation objectives much fairer than in most test-time adaptation papers.

  2. The paper measures its own approximation: It tests the temporal baseline against a same-state reference (Appendix B.3–B.5) and reports where it breaks: only 62% sign agreement on windows with a TTC transition. It also states openly that the reward overlaps with the nuPlan score. That candor is useful, and several of my concerns below come straight from these diagnostics.

Critical weaknesses
  1. In this setup, the central obstacle is a cost argument, not an impossibility, and the baselines that follow from it are missing. The abstract says same-state groups are something a deployed planner “never observes” on its single irreversible timeline. But the reward here does not come from executed outcomes. It is a score of a candidate plan against the current simulator state, with TTC from predicted agent motion (Sec. 3.1, App. A.2). Any number of candidates can be scored at the same state without executing them, and the paper does exactly this in Appendix B.3–B.4 to build its same-state reference. Section 3.2 concedes the point and argues only that it costs G×G\times decoding.

That cost is never measured, and there are several reasons it may be much lower: Batched sampling from one prompt shares the prefill and runs well below G×G\times wall-clock on a GPU. Sampling could be limited to the 8–20% of windows that RT selects.

The following comparisons are missing:

  • Same-state GRPO. Run it with evaluator-scored candidates (dense, and RT-gated), reporting wall-clock time.
  • Best-of-GG selection. Use the same evaluator to pick among sampled plans, with no parameter update at all. This is the most important missing baseline. If it matches TeTO, the gain comes from having the evaluator, not from adaptation.
  • Cost of the non-adaptive comparison. As far as I can tell, the reward’s comfort bounds, 0.95 s TTC threshold and 2.23 m/s overspeed tolerance are nuPlan’s own metric thresholds. They are computed with privileged simulator state. Table 3’s claim of “best OOD scores of all compared planners” against PlanTF, PLUTO, Diffusion Planner and Plan-R1, none of which get this signal, is therefore not like-for-like.

Theoretical contribution is thin:

  • Eq. 5 is the triangle inequality. It holds for any average of past rewards, including the running-mean baseline TeTO is compared against, and says nothing about the bias of the update. The claimed novelty (“first … whose baseline has a bounded deviation”) is close to vacuous.
  • The adapter starts at zero, so the first ratio against πref\pi_{ref} is 1. Until the policy drifts, TeTO behaves like REINFORCE with a window-standardized moving-average baseline plus KL, and moving-average baselines go back to Williams (1992).
  1. The gains are mostly within run-to-run noise, and a reward-free control recovers most of the headline improvement.
  • Reward-free TENT nearly matches the headline result. The 16.7% headline is Singapore NR. There, TENT, which uses no reward, goes from 62.09 to 71.64 (+9.55), against +10.38 for TeTO + RT.
  • TeTO without RT is not competitive there. At 71.25, it sits below TENT, REINFORCE (71.61), REINFORCE++ (72.21) and RWR (72.39).
  • The SFT reference may be weak. It samples at temperature 0.75. Any update that sharpens the token distribution could help, which is what TENT’s result suggests. A greedy-decoding or temperature-swept SFT baseline is needed before the size of the gain can be read.
  • Noise is around 2 points, larger than the claimed effects. All results are single runs with stochastic decoding, and the number of scenarios per city subset is not given in the main text. The paper’s own tables show the noise level: Moving η\eta from 0.5 to 0.6 changes R-CLS by 2.2 points at nearly the same update count (10.6 vs 11.7), and adding TTC noise of std 0.1 raises R-CLS from 77.96 to 80.12. Against that, the gains that isolate the contribution are smaller: SG vs running mean is +1.15, the minimum gate vs the mean gate is +0.98, and the window-size study is ±0.25 on five scenarios.
  • The Test14-Random evidence is weaker than presented. “A TeTO variant achieves the highest score in each city” picks the better of two variants per column, and neither is consistently best. TeTO is behind three baselines in Pittsburgh. TeTO + RT is behind TENT in Boston (82.43 vs 83.63). In Singapore, TeTO + RT beats SFT by 1.2 and DPO by 0.03.
  • One result is hard to believe without variance estimates. In Table 21, two updates per scenario lift Singapore from 75.91 to 83.34.
  • η\eta may have been tuned on the test split. Its sweep is on Singapore reactive, the test domain, and 0.70 is the best value.
  • The safety isn’t analyzed. The method is unreliable in exactly the windows that matter most: 62% sign agreement when TTC flips. The trajectory-only ablation collapses to 10.04 R-CLS with zero drivable-area compliance. Yet there is no per-scenario analysis of how often or how badly TeTO is worse than SFT.
Questions for authors
  1. How many scenarios are in each city subset of Test14-Hard? Please report means and standard deviations over at least 3 seeds for Tables 2 and 4.

  2. Were PlanTF, PLUTO, Diffusion Planner and Plan-R1 retrained on Las Vegas only? On how much data? Their Singapore scores (22 to 51) are far below what these methods report with full nuPlan training.

  3. How is TTC computed for the reward in NR mode? In NR mode agents follow logged motion. Does “predicted agent motion” use any logged future, or only the current state (e.g., constant velocity)?

  4. Which split was used to choose η\eta, the reward weights and the learning rate?

  5. For each method, in what fraction of scenarios is it worse than SFT, and what is the largest per-scenario drop?

  6. Please report SFT with greedy decoding and with a temperature sweep. If greedy SFT closes most of the Singapore NR gap, the headline number does not reflect adaptation.

  7. Please run both best-of-GG and same-state GRPO with the same evaluator, with GG in {3, 8}, and report wall-clock cost. This tests whether online learning, and Sequential Grouping in particular, is needed at all.

  8. Please break CLS down into its sub-metrics for SFT, TENT, RWR and TeTO + RT, to show whether gains come from reward-overlapping terms (comfort, TTC, speed, progress) or from terms outside the reward (collision, drivable area, driving direction).

  9. Generation takes more than 1 s per control step while replanning happens every 0.1 s of simulated time, so the simulator is effectively paused. The adaptation saving (945.9 → 64.6 ms) is small next to this, and the deployment framing should be toned down.

  10. Figure 5. The legend contains leftover entries (“TeTO dense/sparse/dynamic”). Figure 6. The reasoning says “Turn right” and then “Uniform to go straight”. The paper credits reasoning tokens for stability (App. D.3), so a reasoning trace that contradicts itself deserves a comment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.