- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GroundAct to connect grounded reasoning with trajectory planning in autonomous driving. It maintains continuous entity states and lets the reasoning model refer to them through explicit reference tokens. The selected states are then used to predict interactions with the evolving ego trajectory and guide residual corrections to a flow-matching planner. Experiments cover normal, OOD, and safety-critical PAI-AV samples, together with 913 closed-loop AlpaSim scenes. The results show competitive planning performance, while reference interventions and progressive ablations provide evidence that the proposed pathway affects the generated actions.
Critical strengths
The connection between reasoning and planning is concrete and testable. The same entity state is accessible during both reasoning and trajectory refinement, making it possible to inspect which objects receive explicit interaction modeling. The reference removal and replacement experiments in Table 3 are useful checks that this connection actually matters to the planner. I see this interface as the paper’s main contribution.
The evaluation includes both open-loop prediction and closed-loop driving, with several controls sharing the backbone and training data. Table 4 shows consistent improvements as grounding, referencing, and correction are added. In the closed-loop evaluation, GroundAct achieves the highest progress while maintaining competitive safety performance. This provides reasonable support for the overall design, although it does not outperform Alpamayo 2 Super on every safety metric.
Critical weaknesses
The contribution of reasoning-based entity selection is not fully separated from the additional correction machinery. Removing references also bypasses the correction pathway, while the Act stage receives extra supervision from entity futures and safer alternative actions. Tables 3 and 4 therefore support the combined design, but do not fully establish how much of the gain comes from selecting entities through reasoning rather than from learning a stronger correction module. The authors acknowledge this limitation in Section 4.3. A comparison that keeps the correction training and supervision fixed would make the central contribution more convincing.
Some details needed to assess and reproduce selective correction are still missing. Appendix F explains which modules are trained and where supervision comes from, but does not specify the alternative-action generation procedure, acceptance thresholds, or the final arbitration mentioned in Section 4.3. The safety analysis also needs more context: Table 3 reports that 2.2% of initially safe plans become unsafe, but does not provide the full repair counts or evaluation denominators. Tables 2 and 3 lack uncertainty estimates for the reported differences. These omissions make it difficult to judge how reliably the method corrects unsafe plans without introducing new problems.
I lean toward acceptance because the entity-reference interface offers a useful way to connect reasoning with planning, and the overall design is supported by both open- and closed-loop results. My reservations mainly concern the attribution of the gains and the incomplete specification of the correction mechanism.
Questions for authors
In Table 3, are the correction model, reference count, and trajectory sampling noise held fixed when references are replaced by nearest-entity or highest-risk selections? Were these alternatives trained with their own selection policy, or substituted only at test time?
Could you provide concise pseudocode for generating and accepting the offline alternative actions, and for the final arbitration used at inference? Please clarify which signals the arbitration uses.
For Table 3(b), could you report the evaluation population and the full safe/unsafe transition counts, including unsafe-to-safe repairs? Paired uncertainty estimates from the existing predictions would also help interpret the reference-selection differences.
4 comments
[W2-2 & Q3] For Table 3(b), could you report the evaluation population and the full safe/unsafe transition counts, including unsafe-to-safe repairs? Paired uncertainty estimates from the existing predictions would also help interpret the reference-selection differences.
A: Correction lowers the unsafe-plan rate from 28.3% to 19.5%, a paired difference of −8.8 pp (95% CI: −10.4 to −7.6).
Evaluation population. We evaluated 3,072 matched plan pairs (512 Safety-Critical scenes × 6 candidates), each generated with correction off and on. A plan is unsafe if it incurs a collision or off-road event within 6.4 s.
Below shows the full transition counts.
| Before correction | → Safe after correction | → Unsafe after correction | Total |
|---|---|---|---|
| Unsafe | 320 (36.8%) | 550 (63.2%) | 870 |
| Safe | 2,154 (97.8%) | 48 (2.2%) | 2,202 |
| Total | 2,474 (80.5%) | 598 (19.5%) | 3,072 |
Paired uncertainty. As the reviewer suggested, we estimated uncertainty from the existing predictions, using 300 paired cluster-bootstrap resamples of the 512 scenes.
| Entities entering the correction | Unsafe plans (over 3,072 plans), before → after | Paired difference (95% CI) |
|---|---|---|
| K entities nearest to ego's predicted future trajectory ‡ | 28.3% → 22.1% | −6.2 pp (−7.4 to −4.9) |
| Entities referenced by reasoning (GroundAct) | 28.3% → 19.5% | −8.8 pp (−10.4 to −7.6) |
- ‡: Substituted at test time using the same trained model; the strongest alternative in Q1's answer table.
This shows reasoning-based selection removes 2.6 pp more unsafe plans than the strongest alternative.
Review for Revised Paper
Thanks for the follow-up. The new comparisons and safety counts help address my concerns, and the pseudocode makes the correction process much clearer.
I still have some concern about the entity-selection comparison. Since the correction model was trained with reasoning-selected entities, switching to another selection rule only at test time may put those alternatives at a disadvantage. Matching the entity counts helps, but it does not fully resolve this issue. I think this limitation should be made clearer in the paper.
So, could you also report a paired confidence interval for the 2.6% gap between reasoning-based selection and the strongest alternative? The current intervals compare each method against correction off, rather than directly comparing the two. It would be useful to know how reliable that gap is.
Overall, the response has addressed most of my concerns, and I will keep my weak-accept rating. I would just be a little more cautious about how much of the improvement can be attributed to reasoning-based selection itself.
Thank you for the careful follow-up.
[#2] Paired confidence interval for the 2.6% gap between reasoning-based selection and the strongest alternative
The paired 95% confidence interval for the 2.6 pp gap is 1.4 to 3.8 pp, computed with 10,000 scene-level bootstrap resamples. For consistency, we recomputed the interval in our earlier reply with 10,000 scene-level bootstrap resamples: correction changes the unsafe-plan rate by −8.8 pp (95% CI −10.0 to −7.7 pp).
[#1] Switching to another selection rule only at test time vs. retraining the alternative with its own selection
Below are the results of retraining the alternatives with their own selection.
| Retrained with its own selection | Coll. 2.5 s (%) | Coll. 5 s (%) | Off-road 2.5 s (%) | Off-road 5 s (%) |
|---|---|---|---|---|
| K entities nearest to the initial ego position † | 0.75 | 11.78 | 1.27 | 12.73 |
| K entities nearest to the ego's predicted path † | 0.65 | 11.23 | 1.14 | 9.80 |
| Entities referenced by reasoning (GroundAct) | 0.49 | 9.67 | 1.04 | 8.50 |
- K = GroundAct's reference count per scene.
- †: correction retrained with the alternative's own selection.
Retraining setup: We retrained the residual adapter and SetEnc with the alternative's own selection, keeping the selection-independent modules (base planner and future predictor) unchanged. The alternative's entities enter Interact, SetEnc, and the gate during training and inference, and its training targets are generated by the same offline procedure (Algorithm 1; our Q2 answer). For GroundAct, the entities are those referenced in the reasoning generated by the VLM backbone, in both training and inference. Training steps, initialization, and loss weights are same for both.
Paired CIs: On the same 3,072 plans and sampling noise (unsafe = collision or off-road), we compare reasoning-based selection directly with the strongest alternative (K entities nearest to the ego's predicted path):
| Strongest alternative | Unsafe plans (alternative) | Unsafe plans (ours) | Paired diff. (95% CI) |
|---|---|---|---|
| Substituted at test time | 22.1% | 19.5% | 2.6 pp (1.4 to 3.8) |
| Retrained with its own selection | 22.5% | 19.5% | 3.0 pp (1.7 to 4.2) |
Findings:
- Retraining does not narrow the gap. Since the training targets depend on the selected entities, this comparison measures the full effect of each selection rule rather than isolating selection itself.
- Why the strongest alternative stays less safe: its selection (the K entities nearest to the ego's predicted path) often picks the wrong entities. Among the 300 safety-critical scenes whose base plans collide within 4 s under ground-truth futures, it misses the colliding entity in 32.8% of cases, often choosing oncoming vehicles in the adjacent lane instead, versus 5.9% for GroundAct. These misses depend only on the selection rule, so retraining with its own selection cannot remove them.
Paper revisions: We revised Section 4.3 to report the retrained comparison and the limitation above. We will add the retrained rows to Table 3(a) and the miss-rate analysis in the final version.
Please let us know if any questions remain; we would be glad to address them. Thank you again for your thoughtful feedback.
[W1 & Q1] The contribution of reasoning-based entity selection is not fully separated from the additional correction machinery & Experimental Setup of Table 3(a)
A: Thank you for the helpful suggestion on isolating the gain from reasoning-based entity selection. In the original setup, the baseline substitutions were applied only at test time, using the same correction model and trajectory sampling noise without separate training. However, except for 'No reference,' each baseline used a single entity, resulting in a different reference count from GroundAct's. We agree that Table 3(a) did not fully isolate the contribution of entity selection.
Every alternative selection results in a 5 s collision rate between 10.77% and 11.98%, whereas the entities referenced by reasoning lower it to 9.67%. We revised Table 3(a) and its description accordingly.
[W2-1 & Q2] Could you provide concise pseudocode for generating and accepting the offline alternative actions, and for the final arbitration used at inference? Please clarify which signals the arbitration uses.
A: The "final arbitration" in Section 4.3 is the correction gate: at each solver step, it decides whether the correction is applied or the base field is used unchanged. The "offline alternatives" are speed changes kept only if they are verifiably safer. The high-level logic is:
Signals. The gate uses the predicted clearance of the referenced entities within the first 4 s. If any clearance is non-positive, the risk latch activates and remains active for subsequent solver steps. The correction strength is the latch multiplied by the solver-time schedule min(1,2sk2). (Closing rate is an input to the adapter, but does not directly control the gate.)
We added the pseudocode to Appendix F (Algorithms 1 and 2).