acceptodds
Under review as a conference paper at ICLR 2027

Grounding What Shapes the Plan: Rethinking Groundedness for Physical Intelligence in Autonomous Driving

Abstract

Driving models increasingly ground reasoning in causal relations, spatial structure, perceptual evidence, and predicted futures. These advances make reasoning more faithful to the driving scene, but leave a fundamental question unresolved: *what should groundedness mean when the model ultimately outputs an action?* Correctly grounded reasoning does not, by itself, ensure desirable driving outcomes. We introduce **GroundAct**, which starts from a simple premise: driving unfolds through physical entities and their interactions. Entities therefore become the unit of grounding; a lightweight reference token keeps each selected entity's continuous state addressable through symbolic reasoning; and only the referenced entities' interactions with the evolving proposal correct the plan. The result is an explicit path from what reasoning grounds to what the plan does, which we call *grounded planning*. To assess its practical value, we evaluate GroundAct in both open- and closed-loop settings. GroundAct shows strong open-loop planning across normal, out-of-distribution, and safety-critical scenarios, with closed-loop results extending this evidence to driving in simulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Autonomous Driving, Physical AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ztgNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GroundAct to connect grounded reasoning with trajectory planning in autonomous driving. It maintains continuous entity states and lets the reasoning model refer to them through explicit reference tokens. The selected states are then used to predict interactions with the evolving ego trajectory and guide residual corrections to a flow-matching planner. Experiments cover normal, OOD, and safety-critical PAI-AV samples, together with 913 closed-loop AlpaSim scenes. The results show competitive planning performance, while reference interventions and progressive ablations provide evidence that the proposed pathway affects the generated actions.

Critical strengths

The connection between reasoning and planning is concrete and testable. The same entity state is accessible during both reasoning and trajectory refinement, making it possible to inspect which objects receive explicit interaction modeling. The reference removal and replacement experiments in Table 3 are useful checks that this connection actually matters to the planner. I see this interface as the paper’s main contribution.

The evaluation includes both open-loop prediction and closed-loop driving, with several controls sharing the backbone and training data. Table 4 shows consistent improvements as grounding, referencing, and correction are added. In the closed-loop evaluation, GroundAct achieves the highest progress while maintaining competitive safety performance. This provides reasonable support for the overall design, although it does not outperform Alpamayo 2 Super on every safety metric.

Critical weaknesses

The contribution of reasoning-based entity selection is not fully separated from the additional correction machinery. Removing references also bypasses the correction pathway, while the Act stage receives extra supervision from entity futures and safer alternative actions. Tables 3 and 4 therefore support the combined design, but do not fully establish how much of the gain comes from selecting entities through reasoning rather than from learning a stronger correction module. The authors acknowledge this limitation in Section 4.3. A comparison that keeps the correction training and supervision fixed would make the central contribution more convincing.

Some details needed to assess and reproduce selective correction are still missing. Appendix F explains which modules are trained and where supervision comes from, but does not specify the alternative-action generation procedure, acceptance thresholds, or the final arbitration mentioned in Section 4.3. The safety analysis also needs more context: Table 3 reports that 2.2% of initially safe plans become unsafe, but does not provide the full repair counts or evaluation denominators. Tables 2 and 3 lack uncertainty estimates for the reported differences. These omissions make it difficult to judge how reliably the method corrects unsafe plans without introducing new problems.

I lean toward acceptance because the entity-reference interface offers a useful way to connect reasoning with planning, and the overall design is supported by both open- and closed-loop results. My reservations mainly concern the attribution of the gains and the incomplete specification of the correction mechanism.

Questions for authors

In Table 3, are the correction model, reference count, and trajectory sampling noise held fixed when references are replaced by nearest-entity or highest-risk selections? Were these alternatives trained with their own selection policy, or substituted only at test time?

Could you provide concise pseudocode for generating and accepting the offline alternative actions, and for the final arbitration used at inference? Please clarify which signals the arbitration uses.

For Table 3(b), could you report the evaluation population and the full safe/unsafe transition counts, including unsafe-to-safe repairs? Paired uncertainty estimates from the existing predictions would also help interpret the reference-selection differences.

4 comments

[W1 & Q1] The contribution of reasoning-based entity selection is not fully separated from the additional correction machinery & Experimental Setup of Table 3(a)

A: Thank you for the helpful suggestion on isolating the gain from reasoning-based entity selection. In the original setup, the baseline substitutions were applied only at test time, using the same correction model and trajectory sampling noise without separate training. However, except for 'No reference,' each baseline used a single entity, resulting in a different reference count from GroundAct's. We agree that Table 3(a) did not fully isolate the contribution of entity selection.

Matched comparison. We therefore re-ran the comparison with the number of entities matched to GroundAct's reference count in each scene (Safety-Critical set, 512 scenes × 6 candidates, continuous generation):

Which entities enter the correction (K = GroundAct's reference count per scene)Coll. 2.5 s (%)Coll. 5 s (%)Off-road 2.5 s (%)Off-road 5 s (%)
K entities with the highest attention mass over all valid entity tokens (initially appended to input tokens) ‡0.9811.981.4613.41
K entities nearest to the initial ego position ‡0.7811.881.3713.44
K entities nearest to ego's predicted future trajectory ‡0.6210.771.149.51
Entities referenced by reasoning (GroundAct)0.499.671.048.50
  • ‡: Substituted at test time using the same trained model. All rows use the same trained model and the same sampling noise; only the selected entities differ.

Every alternative selection results in a 5 s collision rate between 10.77% and 11.98%, whereas the entities referenced by reasoning lower it to 9.67%. We revised Table 3(a) and its description accordingly.


[W2-1 & Q2] Could you provide concise pseudocode for generating and accepting the offline alternative actions, and for the final arbitration used at inference? Please clarify which signals the arbitration uses.

A: The "final arbitration" in Section 4.3 is the correction gate: at each solver step, it decides whether the correction is applied or the base field is used unchanged. The "offline alternatives" are speed changes kept only if they are verifiably safer. The high-level logic is:

  • Offline (training targets). For each training scene, we change the base plan's speed along its path, keep the candidates that are safer than the base under ground-truth futures, and verify each by replaying the actual 10-step solver. The kept replays provide the residual targets.
  • Inference (arbitration). At each solver step, if any referenced entity is predicted to contact the current proposal within 4 s, the gate is activated and remains active. Its strength ramps in with solver time, and the correction is bypassed when no referenced entity is valid.
# Offline: generate and accept alternatives (training only; ground-truth futures, first 4 s)

candidates = 21 speed profiles along the base path            # 7 target accelerations x 3 ramp durations
accepted   = {c in candidates : 
              clearance to referenced entities improves by >= 0.5 m
              and existing overlap removed and no new overlap with any entity
              and progress >= 95% of the base plan}
kept       = {c in accepted : replaying the 10-step solver with the inference gate
              produces a nonzero correction and the result still passes the checks above}
store, for every solver step of each kept replay: (flow state, base field, risk flag, residual target)
# Inference: correction gating at each solver step k

if no referenced entity is valid:
    field = base_field                                          # bypass
else:
    clearance, closing_rate = Interact(proposal_k, forecasts)  # per referenced entity and future time
    risk_flag |= any(valid and clearance <= 0 within the first 4 s)   # turns on at predicted contact, then stays on
    strength   = risk_flag * min(1, 2 * s_k**2)                 # 0 applies no correction at this solver step; ramps in over solver time
    field      = base_field + strength * adapter(SetEnc(...), solver context)   # SetEnc pools the referenced entities

Signals. The gate uses the predicted clearance of the referenced entities within the first 4 s. If any clearance is non-positive, the risk latch activates and remains active for subsequent solver steps. The correction strength is the latch multiplied by the solver-time schedule min⁡(1,2sk2)\min(1,2s_k^2). (Closing rate is an input to the adapter, but does not directly control the gate.)

We added the pseudocode to Appendix F (Algorithms 1 and 2).

[W2-2 & Q3] For Table 3(b), could you report the evaluation population and the full safe/unsafe transition counts, including unsafe-to-safe repairs? Paired uncertainty estimates from the existing predictions would also help interpret the reference-selection differences.

A: Correction lowers the unsafe-plan rate from 28.3% to 19.5%, a paired difference of −8.8 pp (95% CI: −10.4 to −7.6).

Evaluation population. We evaluated 3,072 matched plan pairs (512 Safety-Critical scenes × 6 candidates), each generated with correction off and on. A plan is unsafe if it incurs a collision or off-road event within 6.4 s.

Below shows the full transition counts.

Before correction→ Safe after correction→ Unsafe after correctionTotal
Unsafe320 (36.8%)550 (63.2%)870
Safe2,154 (97.8%)48 (2.2%)2,202
Total2,474 (80.5%)598 (19.5%)3,072

Paired uncertainty. As the reviewer suggested, we estimated uncertainty from the existing predictions, using 300 paired cluster-bootstrap resamples of the 512 scenes.

Entities entering the correctionUnsafe plans (over 3,072 plans), before → afterPaired difference (95% CI)
K entities nearest to ego's predicted future trajectory ‡28.3% → 22.1%−6.2 pp (−7.4 to −4.9)
Entities referenced by reasoning (GroundAct)28.3% → 19.5%−8.8 pp (−10.4 to −7.6)
  • ‡: Substituted at test time using the same trained model; the strongest alternative in Q1's answer table.

This shows reasoning-based selection removes 2.6 pp more unsafe plans than the strongest alternative.

Reviewer_ztgNOpenReview ↗

Review for Revised Paper

Thanks for the follow-up. The new comparisons and safety counts help address my concerns, and the pseudocode makes the correction process much clearer.

I still have some concern about the entity-selection comparison. Since the correction model was trained with reasoning-selected entities, switching to another selection rule only at test time may put those alternatives at a disadvantage. Matching the entity counts helps, but it does not fully resolve this issue. I think this limitation should be made clearer in the paper.

So, could you also report a paired confidence interval for the 2.6% gap between reasoning-based selection and the strongest alternative? The current intervals compare each method against correction off, rather than directly comparing the two. It would be useful to know how reliable that gap is.

Overall, the response has addressed most of my concerns, and I will keep my weak-accept rating. I would just be a little more cautious about how much of the improvement can be attributed to reasoning-based selection itself.

Thank you for the careful follow-up.


[#2] Paired confidence interval for the 2.6% gap between reasoning-based selection and the strongest alternative

The paired 95% confidence interval for the 2.6 pp gap is 1.4 to 3.8 pp, computed with 10,000 scene-level bootstrap resamples. For consistency, we recomputed the interval in our earlier reply with 10,000 scene-level bootstrap resamples: correction changes the unsafe-plan rate by −8.8 pp (95% CI −10.0 to −7.7 pp).


[#1] Switching to another selection rule only at test time vs. retraining the alternative with its own selection

Below are the results of retraining the alternatives with their own selection.

Retrained with its own selectionColl. 2.5 s (%)Coll. 5 s (%)Off-road 2.5 s (%)Off-road 5 s (%)
K entities nearest to the initial ego position †0.7511.781.2712.73
K entities nearest to the ego's predicted path †0.6511.231.149.80
Entities referenced by reasoning (GroundAct)0.499.671.048.50
  • K = GroundAct's reference count per scene.
  • †: correction retrained with the alternative's own selection.

Retraining setup: We retrained the residual adapter and SetEnc with the alternative's own selection, keeping the selection-independent modules (base planner and future predictor) unchanged. The alternative's entities enter Interact, SetEnc, and the gate during training and inference, and its training targets are generated by the same offline procedure (Algorithm 1; our Q2 answer). For GroundAct, the entities are those referenced in the reasoning generated by the VLM backbone, in both training and inference. Training steps, initialization, and loss weights are same for both.

Paired CIs: On the same 3,072 plans and sampling noise (unsafe = collision or off-road), we compare reasoning-based selection directly with the strongest alternative (K entities nearest to the ego's predicted path):

Strongest alternativeUnsafe plans (alternative)Unsafe plans (ours)Paired diff. (95% CI)
Substituted at test time22.1%19.5%2.6 pp (1.4 to 3.8)
Retrained with its own selection22.5%19.5%3.0 pp (1.7 to 4.2)

Findings:

  • Retraining does not narrow the gap. Since the training targets depend on the selected entities, this comparison measures the full effect of each selection rule rather than isolating selection itself.
  • Why the strongest alternative stays less safe: its selection (the K entities nearest to the ego's predicted path) often picks the wrong entities. Among the 300 safety-critical scenes whose base plans collide within 4 s under ground-truth futures, it misses the colliding entity in 32.8% of cases, often choosing oncoming vehicles in the adjacent lane instead, versus 5.9% for GroundAct. These misses depend only on the selection rule, so retraining with its own selection cannot remove them.

Paper revisions: We revised Section 4.3 to report the retrained comparison and the limitation above. We will add the retrained rows to Table 3(a) and the miss-rate analysis in the final version.


Please let us know if any questions remain; we would be glad to address them. Thank you again for your thoughtful feedback.

Reviewer_kJUxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

paper investigates the connection between grounded reasoning and trajectory planning in autonomous driving VLAs. The authors argue that although existing methods ground reasoning in language, spatial information, or future states, a gap may still exist between the reasoning process and the resulting driving behavior.To address this issue, the paper proposes GroundAct, which introduces Token-as-Reference to associate physical entities selected during reasoning with their continuous states. An interaction-driven trajectory refinement module further enables the predicted motions of referenced entities and their interactions with the ego vehicle to directly influence the final planning output.The proposed method is evaluated in open-loop, closed-loop, and safety-critical scenarios, demonstrating competitive performance.

Critical strengths

1.The paper addresses a well-motivated and practically meaningful problem. 2.The proposed framework is well-structured and demonstrates a certain degree of novelty. 3.The experimental evaluation is relatively comprehensive and demonstrates competitive results.

Critical weaknesses
  1. The novelty of the proposed method relative to existing latent reasoning and world-model-based VLAs is not sufficiently clarified, such as LCDrive and DriveWorld-VLA. 2.The current ablation studies do not sufficiently isolate the individual contributions of different components, it remains unclear to what extent the performance improvements can be attributed specifically to the explicit referencing mechanism introduced by Token-as-Reference. 3.Minor Comments In Section 4.2 (Page 6), the model size is reported as 3.4B, which is inconsistent with the 3.9B reported in other sections and tables. It is unclear whether this discrepancy reflects a specific model configuration or a typographical error. Some labels and visual elements in Figures 3 and 4 are relatively small. Improving their readability would make the figures easier to follow.
Questions for authors

The main questions are incorporated into the weaknesses above.

2 comments

[W1] The novelty of the proposed method relative to existing latent reasoning and world-model-based VLAs is not sufficiently clarified, such as LCDrive and DriveWorld-VLA.

A: LCDrive and DriveWorld-VLA improve the reasoning; GroundAct makes the reasoning consequential for planning by binding the plan to it. Both LCDrive and DriveWorld-VLA ground the reasoning in a learned foresight of how the scene evolves, which strengthens grounded reasoning. Our position is that grounded reasoning does not guarantee grounded planning, so GroundAct lets the planner directly access the planning-critical entities that the reasoning points to and use their continuous physical states to change how the plan is computed.

GroundAct thus provides an explicit, testable physical connection between reasoning and planning: the same entity state is used in both, so one can inspect which entities receive explicit interaction modeling. The two directions address different questions and are complementary.


[W2] The current ablation studies do not sufficiently isolate the individual contributions of different components, it remains unclear to what extent the performance improvements can be attributed specifically to the explicit referencing mechanism introduced by Token-as-Reference.

A: Each row in the below table changes one component with everything else fixed; the explicit reference helps in both reasoning and planning. Here, the explicit reference is the reasoning token that points to an entity's physical state, which later reasoning and the planner both read.

Component isolatedComparison (all else fixed)Collision (%)Off-road (%)
Prefill-Time Groundingno entity grounding → 3D-grounded entity states (BOTH use original text reasoning)17.74 → 13.6419.86 → 17.42
Token-as-Reference, in reasoneroriginal text reasoning → text reasoning with entity references (no correction)13.64 → 11.9817.42 → 12.99
Residual Correctionno correction → interaction-driven correction (BOTH use prefill-time grounding and text reasoning with entity references)11.98 → 9.6712.99 → 8.50
Token-as-Reference, in planner ‡residual correction on K entities with the highest attention mass over all valid entity tokens → residual correction on the entities the reasoning references11.98 → 9.6713.41 → 8.50
Token-as-Reference, in planner ‡residual correction on K entities nearest to the initial ego position → residual correction on the entities the reasoning references11.88 → 9.6713.44 → 8.50
Token-as-Reference, in planner ‡residual correction on K entities nearest to ego's predicted future trajectory → residual correction on entities the reasoning references10.77 → 9.679.51 → 8.50
  • ‡: Substituted at test time using the same trained model.

  • Values from 512 safety-critical scenes, continuous generation.

Below are the takeaways from this table.

  • In Token-as-Reference, in reasoner, the reference-based reasoning alone (without residual correction in planner) lowers the collision rate from 13.64% to 11.98%.
  • The three planning rows (Token-as-Reference, in planner) use the same trained correction model and differ only in the selected entities, so they suggest that the entities referenced by reasoning are better correction targets. Retraining the correction with the strongest alternative's own selection does not close the gap (5 s collision: 11.23% vs. 9.67%).

[W3] Minor Comments In Section 4.2 (Page 6), the model size is reported as 3.4B, which is inconsistent with the 3.9B reported in other sections and tables. It is unclear whether this discrepancy reflects a specific model configuration or a typographical error.

A: Thank you for catching this. "3.4B" in Section 4.2 is a typo: GroundAct has 3.9B parameters in total (a 2.1B VLM and a 1.8B action expert). We corrected it in Section 4.2 of the revised manuscript.


[W4] Some labels and visual elements in Figures 3 and 4 are relatively small. Improving their readability would make the figures easier to follow.

A: Thank you for the helpful suggestion. We will refine the figures to improve their readability in the next revision.

Thank you. You have adequately addressed my concerns. I will raise my original score, I am now more positive about the paper and lean toward acceptance.

Reviewer_hvH1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes GroundAct, a driving VLA that maintains explicit references from reasoning to continuous entity representations. Referenced entities are used for future prediction and interaction modeling with an evolving ego proposal, and a residual module modifies the vector field of a flow-matching action expert. The paper evaluates open-loop planning, closed-loop driving, reference dependence, and reasoning–action alignment.

Critical strengths

The distinction between correctly grounded reasoning and safe action generation is meaningful. Preserving an explicit connection between an entity mentioned in reasoning and its physical representation is a coherent approach to investigating this problem.

Table 4 shows consistent improvements as grounding, reference-based reasoning, and selective correction are progressively added within the continuous-generation setting. Table 5 further reports higher reasoning–action alignment than Alpamayo 1.5, an RL-post-trained baseline, despite GroundAct not reporting a dedicated alignment-oriented RL stage. These results provide encouraging evidence for connecting entity-grounded reasoning to planning.

Critical weaknesses
  1. Key operations and training procedures are insufficiently specified, hindering understanding and reproducibility. The equations in Section 3.4 express central computations through Interact and SetEnc without adequately defining their internal operations. Although Interact uses quantities such as clearance and relative motion, it is unclear which features are computed from the ego proposal and predicted entity futures, and how. Describing SetEnc as permutation-invariant likewise does not explain how entity states, predicted futures, and interaction features are combined and aggregated. The future predictor architecture and loss, correction gate, safer-action generation procedure, and residual-target construction also require clarification. Appendix F introduces “preservation terms” without specifying their mathematical form or how they are applied during training. These details are essential to understanding how entity information produces trajectory corrections and to reproducing the results.

  2. The stronger discrete results raise questions about the necessity of continuous correction. In Table 1, discrete GroundAct, which does not use selective correction, outperforms the full continuous model on several open-loop metrics, including the safety-critical 5-second collision rate (7.096% vs. 9.668%). Although Table 4 demonstrates improvements within the continuous setting, it does not explain why this pathway is needed over the discrete alternative. A closed-loop D/C comparison is particularly important, yet Table 2 reports only C. If D also matches or exceeds C in closed loop, the necessity of interaction-driven selective correction—a central contribution of the paper—would be substantially weakened.

  3. Table 3(b) is insufficient to assess the usefulness of correction. It reports that 2.2% of initially safe plans become unsafe after correction, but does not show how often unsafe plans become safe. Both rates and overall safety before and after correction are needed to judge whether the benefits outweigh the harm.

Questions for authors
  1. If the detection heads are training-only, how are null entity slots identified at inference?

  2. Table 1 lists 3.9B parameters for GroundAct D, whereas the main text states that discrete generation uses only the 2.1B VLM. Please clarify the reported parameter count.

  3. Is α\alpha a learnable parameter or a fixed hyperparameter? Is interpolation with brefb_{\mathrm{ref}} necessary, and has its benefit been verified through an ablation?

  4. What is the regularizer mentioned in Line 215? Please specify its mathematical form and how it is applied.

3 comments
AuthorsOpenReview ↗

Part 3/3: Q1–Q4

[Q1] If the detection heads are training-only, how are null entity slots identified at inference?

A: The detection heads run at inference; only their 3D supervision is training-only. They run once during prefill: the class head labels each entity slot as an object category or null, and null slots are masked before the reference-pointer softmax (Eq. 1). These predictions are reused throughout reasoning and planning without rerunning the heads. The "training-only" label in Figure 3 refers to the entity-level training supervision; we clarified the wording in Section 3.2 of the revised manuscript.


[Q2] Table 1 lists 3.9B parameters for GroundAct D, whereas the main text states that discrete generation uses only the 2.1B VLM. Please clarify the reported parameter count.

A: Thank you for pointing this out. GroundAct D uses 2.1B parameters. The full model has 3.9B (a 2.1B VLM and a 1.8B action expert), and discrete generation uses only the VLM. Table 1 listed 3.9B because our previous implementation loaded the full model in both modes; to avoid confusion, the table now lists 2.1B for GroundAct D.


[Q3] Is α\alpha a learnable parameter or a fixed hyperparameter? Is interpolation with brefb_{ref} necessary, and has its benefit been verified through an ablation?

A: α\alpha is a fixed hyperparameter (α=0.2\alpha = 0.2), and an ablation over α\alpha shows that the interpolation is necessary. After a reference token, Eq. 2 mixes two inputs: the shared reference embedding brefb_{\text{ref}}, which keeps the reasoning stable after a reference, and the entity input bi+P(Mi)b_i + P(M_i), which carries the selected entity's identity and state into subsequent reasoning and planning. Either one alone fails:

α\alphaInput embeddingTraces with repeated references (of 100 validation scenes)
0brefb_{\text{ref}} only— (no entity info is passed)
0.2 (ours)mixed0
0.4mixed29
0.6mixed67
0.8mixed84
1entity input only (no brefb_{\text{ref}})100

Without brefb_{\text{ref}}, the decoder re-emits the same reference instead of continuing the reasoning:

α = 0.2:  Because of lead vehicle <entity_13>, follow the lead vehicle ...
α = 1:    Because of lead vehicle <entity_13><entity_13><entity_13> ...

We use α=0.2\alpha = 0.2, the largest tested value without repetition, so the decoder receives as much entity information as possible while generation stays stable.


[Q4] What is the regularizer mentioned in Line 215? Please specify its mathematical form and how it is applied.

A: The regularizer limits drift in the entity representations used for reference selection (Eq. 1). The LLM decoder produces both the reasoning and the entity states, so fine-tuning it for reasoning can also change these representations. We therefore save each entity's projected representation WkMiW_k M_i before fine-tuning and penalize the cosine distance between the saved and current values for the same input and entity slot (one of the K=64K = 64 slots). The saved values and the projection WkW_k stay fixed, and the penalty is added with weight 0.1 to the reasoning losses during fine-tuning.

The mathematical form is:

Lanchor=1max⁡(1,∣V∣)∑(x,i)∈V[1−cos⁡(WkMi(x),kˉi(x))],\mathcal{L}_{\mathrm{anchor}} = \frac{1}{\max(1,|\mathcal{V}|)} \sum_{(x,i)\in\mathcal{V}} \Big[1-\cos\big(W_k M_i(x), \bar{k}_i(x)\big)\Big],

where Mi(x)M_i(x) is the current state of entity slot ii for input xx, kˉi(x)\bar{k}_i(x) is WkMi(x)W_k M_i(x) saved before fine-tuning, and V\mathcal{V} contains the input–slot pairs with a valid saved value.

We added this definition to Section 3.3 and Appendix F and clarified the description in Line 215 accordingly.

AuthorsOpenReview ↗

Part 2/3: W1-2, W2, W3

[W1-2] Clarification: "Preservation terms" in Appendix F

A: The preservation terms keep the adapter quiet when no correction is needed. They are two squared penalties on the adapter's output rr, taken before the gate:

\mathcal{L}_{\mathrm{pres}} \= \underbrace{0.1\\,\mathbb{E}_{\mathcal{D}_{\mathrm{off}}}\big[\\|r\\|_{\mathrm{av}}^2\big]}_{\text{no correction when the gate is off}} + \underbrace{0.001\\,\mathbb{E}_{\mathcal{D}}\big[\\|r\\|_{\mathrm{av}}^2\big]}_{\text{keep all corrections small}},

where ∣r∣av2\\|r\\|_{\mathrm{av}}^2 is the mean squared residual over the future times and the (x,y,yaw)(x, y, yaw) channels, D\mathcal{D} contains the solver states recorded offline while the accepted safer alternatives are replayed through the ten-step solver, and Doff\mathcal{D}_{\mathrm{off}} contains the subset in which the correction gate's risk flag is off. Both terms are added to the loss while training the residual adapter, with the base planner and future predictor frozen.


[W2] The stronger discrete results raise questions about the necessity of continuous correction.

A: In closed loop, D falls behind C, and C's safety advantage over D comes from the correction (AlpaSim, 913 scenes, same protocol as Table 2):

ModelProgress (%) ↑At-fault collision (%) ↓Off-road (%) ↓Wrong lane (%) ↓Latency per plan (s) ↓
GroundAct D66.613.919.920.35.69
GroundAct C, correction off77.314.018.521.60.99
GroundAct C, correction on75.210.515.320.71.10

Latency: mean end-to-end time per plan over 200 runs (100 scenes × 2 runs) on one A100 80GB GPU with batch size 1, from data loading to the returned trajectory (model loading excluded).

From this table, we can identify the following three points:

  1. GroundAct D vs. GroundAct C: D's open-loop advantage does not carry over. Small errors in the discrete action tokens accumulate over the rollout and push the ego into states that diverge from the training distribution, so D reaches much lower progress than either continuous variant (66.6% vs. 75.2–77.3%). D's executed path also strays farther from the GT path (2.68 m vs. 2.51 m for GroundAct C with correction on).
  2. Correction off vs. correction on: Turning the correction on lowers at-fault collision, off-road, and wrong lane, at a small cost in progress.
  3. Latency. Discrete generation decodes 128 action tokens sequentially, while the action expert integrates 10 Euler steps, so a plan arrives 5.69 s versus 1.10 s after its observation.

We will include the two additional rows in the next revision.


[W3] Table 3(b) is insufficient to assess the usefulness of correction.

A: Correction repairs 320 unsafe plans while breaking 48 safe ones, lowering the unsafe-plan rate from 28.3% to 19.5% (paired difference −8.8 pp, 95% CI −10.0 to −7.7 pp).

Evaluation population. We evaluated 3,072 matched plan pairs (512 Safety-Critical scenes × 6 candidates), each generated with correction off and on. A plan is unsafe if it incurs a collision or off-road event within 6.4 s. The table below shows the full transition counts.

Before correction→ Safe after correction→ Unsafe after correctionTotal
Unsafe320 (36.8%)550 (63.2%)870
Safe2,154 (97.8%)48 (2.2%)2,202
Total2,474 (80.5%)598 (19.5%)3,072
  • CI: scene-level paired bootstrap over the 512 scenes (10,000 resamples).
AuthorsOpenReview ↗

Part 1/3: W1-1 (key operations)

[W1-1] Clarification on key operations

A: GroundAct predicts the future of each referenced entity (future predictor). At every solver step, it measures how those futures relate to the current ego proposal (Interact), combines the referenced entities’ information into a joint representation (SetEnc), and uses the result to correct the plan.

1) Future predictor: predict each entity's motion once, then reuse it.

: Each entity queries image features from four observed times. A two-layer temporal Transformer aggregates this evidence, and a two-layer cross-entity Transformer incorporates the other currently valid entities. Both use 256 hidden dimensions and eight attention heads. A shared two-layer MLP (256 → 256 → 30) outputs ten future poses (x,y,yaw)(x, y, yaw) at 0.5 s intervals. Predicted XY positions are learned residuals around constant-velocity extrapolation; yaw is predicted as a change from the current heading. These poses are interpolated to 50 poses at 0.1 s intervals, covering 5 s. The forecasts do not depend on the ego proposal, so they are computed once and reused at every solver step. Training uses a Smooth-L1 loss on XY displacement and a 1−cos⁡1-\cos loss on yaw change, averaged over the poses with a valid matched ground-truth label.

2) Interact: measure the clearance and how fast it closes.

: At each solver step kk, Interact compares the ego box on the current proposal with each referenced entity's predicted box at the same future time hh (0.1 s steps over the 5 s forecast):

d^{(k)}_{i,h} \= \mathrm{SignedClearance}\big(B_{\mathrm{ego}}(\hat P^{(k)}_h), \hat B_{i,h}\big), \qquad c^{(k)}_{i,h} \= -\frac{d^{(k)}_{i,h}-d^{(k)}_{i,h-1}}{0.1}, \qquad C^{(k)}_{i,h} = [d^{(k)}_{i,h}, c^{(k)}_{i,h}],

where Bego(P^h(k))B_{\mathrm{ego}}(\hat P^{(k)}_h) is the ego box at the proposal's pose at time hh, and B^i,h\hat B_{i,h} is entity ii's box at its forecast pose. These are the quantities named in Section 3.4:

  • Clearance dd: positive box-to-box distance when separated, zero at contact, and negative penetration depth when overlapping.
  • Closing rate cc (the "relative motion"): how fast clearance decreases.
  • Clearance and closing rate are recomputed whenever the proposal changes, while the forecasts stay fixed.

3) SetEnc: combine each entity's information with an MLP, then pool across referenced entities.

: For each referenced entity and future time, SetEnc concatenates the entity state MiM_i, its current and predicted geometry, the two interaction features from Interact (dd, cc), and the time index, and passes them through one shared two-layer MLP. It then pools over the referenced entities by concatenating the mean and the elementwise maximum of the MLP outputs, giving one 512-dimensional vector per future time that does not depend on entity order. The residual adapter turns this vector, together with the flow state, base vector field, current proposal, solver time, and risk flag, into a joint correction.

4) Correction gate: switch on at predicted contact and stay on.

γk=bkmin⁡(1,2sk2),vk=vbase,k+γkrk,\gamma_k = b_k \min(1, 2s_k^2), \qquad v_k = v_{\mathrm{base},k} + \gamma_k r_k,

where bkb_k is a risk flag that turns on once any valid referenced entity is predicted to contact or overlap the ego box (di,h(k)≤0d^{(k)}_{i,h} \le 0) within the first 4 s, and then stays on for the remaining solver steps; min⁡(1,2sk2)\min(1, 2s_k^2) ramps the correction in over solver time sks_k.

Below items build the training targets offline.

  • Safer-action generation: propose speed changes, keep only verified safer ones.

Along the base path, we generate 21 speed profiles by combining 7 target accelerations {−3, −2, −1, 0, 0.5, 1, 2} m/s² with 3 ramp durations {0.5, 1, 2} s. Using ground-truth futures over 4 s, a candidate is accepted only if it increases the clearance to the referenced entities by at least 0.5 m, removes any existing overlap with them, creates no new overlap with any entity, and keeps at least 95% of the base progress. Each accepted candidate is then replayed via the actual 10-step solver, and only replays that apply a nonzero correction and still pass these checks are kept.

  • Residual-target construction: learn the correction that steers toward the accepted alternative.

At each replayed solver state ss with the risk flag on, the target moves the frozen planner's current endpoint prediction a^s\hat a_s toward the accepted alternative a+a^+:

rs⋆=clip(a+−a^smax⁡(1−s,0.1)min⁡(1,2s2), −0.05, 0.05),r^\star_s = \mathrm{clip}\Big(\frac{a^+ - \hat a_s}{\max(1-s,0.1)\min(1,2s^2)},\ -0.05,\ 0.05\Big),

and is zero otherwise. Dividing by max⁡(1−s,0.1)\max(1-s, 0.1), the remaining solver time, turns the endpoint gap into a velocity, and dividing by min⁡(1,2s2)\min(1, 2s^2) pre-compensates the gate schedule that the solver applies afterwards.

We clarified these parts in Section 3.4, with full details in Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.