acceptodds
Withdrawn from ICLR 2027

ETM: Event Trigger Module based VLM for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision Language Models (VLMs) have recently demonstrated remarkable semantic reasoning capabilities for autonomous driving and are increasingly incorporated into fast-slow planning frameworks. However, existing approaches invoke the VLM at every planning step even though high-level semantic changes are typically sparse over time, leading to substantial redundant computation. This raises a fundamental question: when should the expensive VLM be invoked? To address this problem, we propose the Event Trigger Module (ETM), a lightweight event-driven policy that selectively activates the VLM only when refreshed semantic reasoning is expected to improve downstream planning. We formulate VLM invocation as an information-gain-driven decision that explicitly balances semantic utility against computational cost. Guided by this formulation, ETM integrates short-term visual dynamics, long-term temporal context, and compact motion-aware scalar cues to estimate the necessity of VLM invocation. To train the trigger policy effectively, we further introduce a hybrid supervision strategy that combines corner-case trigger labels with information gain based pseudo-labels, enabling ETM to capture both common driving situations and rare but safety-critical events. Extensive experiments on the nuScenes benchmark demonstrate that the proposed ETM method achieves comparable planning accuracy to always-triggered VLM systems while reducing the VLM invocation rate from 100% to 32%. Meanwhile, ETM decreases inference latency by nearly 50% and lowers computational cost by approximately 64%, without sacrificing planning performance. These results show that the proposed ETM method offers an effective paradigm for VLM-assisted autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, motion planning, vision-language models, multimodal learning, efficient inference, event-triggered reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tSEpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors propose an event-triggered mechanism for planning assistance from VLMs in autonomous driving. To decide when to refresh cached semantic guidance, the Event Trigger Module (ETM) utilizes fast and slow temporal streams to capture visual changes over different time scales, combining these features with ego and cache states to estimate the need for semantic refresh. ETM learns from heuristic trigger labels and risk scores with the training objective, i.e., risk-weighted focal loss for trigger classification and the auxiliary score regression loss. During inference, the VLM is invoked when the trigger probability exceeds a threshold or the cache exceeds its limits. Otherwise, the planner reuses the cached semantic guidance. All experiments are conducted on the nuScenes benchmark. The reported results show that ETM reduces computational costs through selective VLM invocation while improving overall planning performance. However, I still have several concerns, as detailed below.

Critical strengths
  1. The paper investigates how to bring the capabilities of strong VLMs into autonomous driving at a lower computational cost, which is a timely and valuable research direction given the rapid progress of VLMs.
  2. ETM is designed to leverage VLM assistance in rare and challenging driving scenarios. It focuses on the long tail of the driving distribution, giving the approach practical potential for real-world applications.
  3. The paper is clearly written and supported by informative visualizations, making the approach easy to follow. The overall framework is also straightforward to implement.
Critical weaknesses
  1. The main concern is that experimental results and analysis are unable to support the benefits of ETM.

    (a) Trigger supervision. ETM is trained using heuristic labels derived from corner cases, changes in ego motion, and other predefined conditions. However, this paper does not verify whether invoking the VLM on frames labeled as requiring a trigger actually improves planning. The validity of these hand-crafted trigger labels therefore remains unclear.

    (b) Benefits across scenario types. Furthermore, overall performance improvements do not establish whether difficult scenarios benefit or whether easy scenarios avoid degradation. Besides, as shown in Table 2, the 3 s L2 and collision results are relatively similar. Despite the computational savings, the evidence for improved planning performance remains limited without separate analyses of various scenarios.

  2. Generalization ability of ETM. ETM relies on relatively complex and heavy hand-crafted rules and thresholds, raising concerns about its generalization across datasets. It remains unclear whether the configuration selected on nuScenes can also bring performance improvements on datasets such as OpenScene or Waymo. More evaluation is necessary to determine whether ETM is able to retain meaningful performance gains under such transfer.

Minor Issues

  1. "event trigger" should be "event-triggered" on Page 9.
Questions for authors

It is suggested that the authors provide further clarification and supporting evidence to address the weaknesses discussed above. In particular, the effectiveness of ETM (Weakness 1) and generalization ability (Weakness 2) require further validation.

Reviewer_3yXbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an Event Trigger Module (ETM) that selectively invokes a frozen VLM to refresh high-level driving guidance. A fast planner continues to run using the most recent cached meta-action, which represents speed intent, maneuver, and attention direction. ETM combines short- and longer-term visual features through GRUs and cross-attention, together with scalar cues describing visual drift, cached-action entropy, ego motion, object count, and cache age. Training uses automatic trigger labels and a heuristic risk score assembled from corner-case membership, visual change, ego displacement, timeout, and safety-related keywords.

On nuScenes, ETM invokes the VLM on approximately 32% of frames while matching the always-refresh version of the same planner at 0.96 m three-second L2 error. Reported mean latency decreases from 574.84 to 296.84 ms, and compute from 16,632.75 to 5,946.55 GFLOP/frame (Table 2). These correspond to approximately 48% lower mean latency and 64% lower compute. The evaluation measures open-loop planning and efficiency; it does not execute the policy in a closed-loop driving environment.

Critical strengths
  1. The paper addresses a practical systems question: when does expensive semantic reasoning need to be refreshed? Caching a compact meta-action provides a clear mechanism for exploiting temporal redundancy.

  2. The ETM-full versus ETM-trigger comparison is well targeted. It uses the same planning architecture and VLM, making it more informative about the trigger mechanism than comparisons with unrelated planners (Section 4.2).

  3. Evaluation includes invocation rate, measured latency, compute, and GPU memory, as well as planning metrics. Fixed-interval and random-refresh baselines show that reducing invocation frequency alone can degrade planning quality.

  4. Component ablations assess both temporal streams, scalar features, and auxiliary score regression. The strong effect of removing scalar features provides useful evidence about which inputs matter (Table 3).

  5. The appendix specifies label-generation thresholds, keyword rules, model backbones, and training settings, and explicitly identifies the trigger targets as pseudo-labels rather than human ground truth.

Critical weaknesses
  1. The information-gain interpretation is not established by the implemented target. Appendix A.1 constructs a heuristic risk score, including constants such as 1.0 for corner cases and 0.8 for keyword events. The so-called information-gain loss regresses to this score. It does not measure uncertainty reduction, a fresh-versus-cached planning improvement, or a utility difference after subtracting inference cost. The manuscript does call the score a surrogate, but the broader information-gain-driven claims need empirical validation or narrower wording.

  2. The most direct baseline is missing. Because the learned trigger imitates an explicit union of event rules, the same rules should be evaluated directly as an online refresh policy. The current fixed/random controls do not establish whether the GRUs and learned score improve upon the supervision policy itself. The uncertainty- and complexity-based gates discussed in Section 2.3 also deserve controlled comparison using the same planner and a matched invocation budget.

  3. Open-loop results do not establish safety under stale semantics. NuScenes replay cannot show how a missed refresh changes the ego vehicle's state, subsequent observations, and interactions. Trigger recall is measured against heuristic labels, so high recall does not necessarily mean that the module detects events where refreshing the VLM would prevent a driving failure. The claimed safety value requires closed-loop or counterfactual validation.

  4. Temporal data construction and cache distribution need clarification. Appendix A.2 holds out 10% of trigger labels but does not specify whether complete scenes, rather than neighboring frames, are separated. Offline labels depend on previous positive triggers, whereas deployment cache states depend on the learned policy's decisions and forced refreshes. This creates a possible training/inference mismatch. Additionally, Appendix A.1 defines visual-drift labels using consecutive-frame features, while the main text describes drift relative to the cached feature. These choices should be reconciled and evaluated.

  5. Statistical evidence and runtime characterization are limited. Appendix A.2.7 states that settings are generally run once. Changes such as 0.29% to 0.27% collision rate are therefore difficult to interpret without uncertainty or collision counts. Average latency also hides the delay on frames that actually require a VLM refresh, which are particularly relevant to the proposed safety use. GPU memory stays at 14.34 GB for every strategy; the demonstrated resource benefit concerns invocation, compute, and average latency.

  6. Some comparative wording exceeds the reported numbers. Section 4.2 describes planning accuracy as nearly identical to Senna, but Table 1 reports three-second L2 of 0.96 m for ETM versus 0.86 m for Senna, and two-second L2 of 0.64 versus 0.54 m. The clearest accuracy-preservation result is against ETM-full. Also, threshold selection uses the nuScenes validation accuracy/efficiency trade-off (Appendix A.2.7); a separate final evaluation would better establish that the selected operating point transfers.

Questions for authors
  1. Run the exact label-generation rules as a refresh baseline, including the same timeout and distance safeguards. Compare this baseline, scalar-only ETM, full ETM, and existing uncertainty/complexity gates at matched invocation rates, and plot the full planning-quality/compute trade-off.

  2. On an evaluation subset, query the VLM at every frame and measure how much fresh versus cached semantics changes trajectory error, collisions, and maneuver selection. Does ETM's predicted score rank this realized benefit? Include cases where a refresh is unnecessary or harmful.

  3. Add closed-loop tests with abrupt hazards and measure missed necessary refreshes, reaction delay, collision outcomes, and event-specific recall. Report how much safety comes from the learned trigger versus mandatory cache expiration.

  4. Use scene-disjoint trigger train/validation splits, disclose the corner-case token source and overlap checks, and test robustness when deployment cache histories differ from the histories used to generate training labels.

  5. Report repeated-run or scene-bootstrap uncertainty, absolute collision counts, and a threshold selected independently of the final evaluation. Include triggered/non-triggered latency, p95/p99 latency, hardware timing methodology, and the control frequency supported by the complete pipeline.

  6. Clarify how cached meta-action probabilities/entropy and object counts are computed without a fresh language-model call, and reconcile the consecutive-frame versus cached-frame drift definitions. Present the learned output as a refresh-utility proxy unless an information-gain interpretation is demonstrated.

Reviewer_LNGROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an Event Trigger Module (ETM) for reducing redundant vision-language model (VLM) inference in autonomous driving. ETM selectively invokes a frozen VLM to refresh high-level driving meta-actions, while a downstream planner continuously generates trajectories using either updated or cached semantic guidance. The module combines short- and long-term visual features with motion and cache-state signals. Training uses corner-case annotations and heuristic event pseudo-labels, together with asymmetric focal loss and risk-score regression to encourage safety-sensitive triggering. Experiments on nuScenes compare ETM with always-on, fixed-interval, and random invocation strategies, alongside component ablations. The reported results reduce VLM invocation to approximately 32% of frames, average inference latency by 48%, and computation by 64%, while maintaining comparable open-loop planning performance to the same architecture with always-on VLM inference.

Critical strengths
  1. Substantial efficiency gains with a controlled planning comparison. ETM reduces VLM invocation from 100% to approximately 32%, average latency by 48%, and computation by 64%, while retaining the same reported 3s L2 error of 0.96m as ETM-full. Because both variants share the planner and VLM backbone, this comparison directly supports the practical value of selective semantic refresh, which is the paper’s strongest contribution.
  2. Evidence that invocation timing matters beyond reducing call frequency. At similar invocation rates, ETM achieves a 3s L2 error of 0.96m with 32% invocation, compared with 1.07m for fixed-interval refresh at 35% and 1.08m for random refresh at 36% (Table 2). These results provide initial evidence that the learned policy allocates VLM computation more effectively than simple scheduling, supporting its potential as a useful component in VLM-assisted planning systems.
Critical weaknesses
  1. The central information-gain interpretation is insufficiently substantiated. The predicted “information gain” is supervised using a manually constructed risk score based on corner cases, visual drift, keywords, and cache expiration (Appendix A.1), rather than the downstream benefit of refreshing versus retaining cached semantics. Although the authors acknowledge this score as a surrogate, they do not validate its relationship to actual planning improvement, and the training objective contains no explicit computational cost term. This weakens the paper’s central conceptual contribution: the evidence supports learning an event-refresh heuristic more directly than an information-gain-driven utility–cost trade-off.
  2. Missing comparisons with the closest adaptive invocation methods leave the research contribution unclear. The invocation-policy experiments compare ETM against always-on, fixed-interval, and random refresh, but omit existing adaptive mechanisms. VLMPlanner [1] already adjusts inference frequency through CAI-Gate, while AdaDrive [2] learns activation using comparative planning losses and an activation penalty. Without comparisons under a shared backbone and matched invocation budget, the results establish an advantage over simple schedules but do not sufficiently demonstrate the added value of ETM’s specific architecture and supervision.

References [1] Zhipeng Tang et al. VLMPlanner: Integrating Visual Language Models with Motion Planning. 2025. [2] Ruifei Zhang et al. AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving. 2025.

Questions for authors
  1. Clarification—supervision and claimed utility. Is the predicted information-gain score trained exclusively against the heuristic risk scores in Appendix A.1, or does training also incorporate the measured planning benefit of refreshing cached semantics? What evidence shows that this score predicts when fresh VLM guidance improves planning over cached guidance?
  2. Additional validation needed for the current contribution. Under the same planner, VLM backbone, and invocation budget, how does ETM compare with existing adaptive invocation mechanisms, such as CAI-Gate or AdaDrive-style activation, and with simple rules using runtime-observable drift, motion, and cache age? Such comparisons would materially affect whether I view ETM as a substantive methodological improvement over existing scheduling approaches.
  3. Clarification—forced refresh and comparison fairness. What fraction of ETM’s reported 32% invocation rate comes from learned triggering versus mandatory cache-age or distance refresh? Are identical mandatory-refresh rules applied to the fixed-interval and random baselines, and are these calls included consistently in all reported invocation rates?
Reviewer_7ubSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Event Trigger Module (ETM) for efficient VLM-assisted autonomous driving. ETM adaptively determines when to invoke the VLM based on temporal visual features and auxiliary motion signals, while reusing cached meta-actions when no update is needed. Experiments on nuScenes show that ETM reduces VLM invocation to 32% and nearly halves inference latency while maintaining comparable planning performance.

Critical strengths

--Practical motivation. The paper addresses the computational redundancy of frequent VLM invocation in autonomous driving, which is an important problem.

--Reasonable framework design. ETM combines short-term and long-term temporal features with auxiliary motion signals. The proposed caching mechanism enables efficient reuse of high-level semantic guidance.

--Promising efficiency improvements. ETM significantly reduces VLM invocation frequency and computational cost while maintaining comparable planning accuracy.

Critical weaknesses

Weakness 1: Potentially suboptimal heuristic supervision. ETM is trained using pseudo-labels generated by handcrafted heuristic rules. However, these rules themselves may not constitute an optimal VLM triggering policy. This creates a logical gap, as learning from a potentially suboptimal policy does not guarantee an optimal triggering strategy. Why not directly use this heuristic triggering policy at inference time, instead of training an additional neural network to imitate it? Is it possible to train an adaptive policy model with RL?

Weakness 2: Poor figure readability. Figure 2 is difficult to understand due to confusing arrow directions and unclear data flow. Readers must carefully consult the text to understand the framework.

Weakness 3: Limited experimental evaluation. Experiments are conducted only on nuScenes, which is insufficient to demonstrate the method's generalizability. Additional evaluations on NAVSIM or HUGSIM, particularly with closed-loop planning metrics, would strengthen the paper. Besides, although the appendix states that thresholds of 0.8 and 0.9 were evaluated, Table 2 omits these results without justification, leaving the efficiency-performance trade-off analysis incomplete. Table 2 only compares ETM with simple triggering strategies. Comparisons with existing adaptive triggering methods under the same backbone and computational budget are needed to establish ETM's superiority.

Weakness 4: Limited validation of the plug-and-play claim. The paper claims that ETM is a plug-and-play module, but only evaluates it with a single VLM-planner configuration. Its compatibility with other architectures remains unverified.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.