- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies when an embodied agent performing continuous control should call an expensive reasoning backend such as a VLM when it can afford only a limited number of calls. The authors formulate this as event-relative opportunity prediction: from causal observations, a scorer predicts whether each time step falls within a short window before an event onset, 0.5 s by default, and a separate shared selector chooses call times under budget and cooldown constraints. The proposed Predictive Reasoning Scheduler, or PRS, encodes the last 20 steps of state history with a GRU and the current bird's-eye view with a small CNN, and uses the summed probability of the time-to-event head as the opportunity score. Evaluation is performed offline on frozen CARLA trajectories using Useful-Window Coverage, or UWC, which is the fraction of distinct event windows reached by at least one call. The paper reports that PRS covers more windows than Random-k with the same number of calls.
Critical strengths
S1. The paper clearly formulates a practical problem. For agents that combine continuous control with an expensive reasoning backend, deciding when to call, rather than how often, is an important problem. Separating the learning of opportunity scores from allocation under budget and cooldown is a clean design that allows predictive quality alone to be compared under the same allocation rule.
S2. The metric and comparison conditions are designed fairly. UWC counts multiple calls on the same event only once, so a method cannot raise its score by clustering calls around easy events. All methods are also given the same candidate times, the same number of calls, and the same cooldown, and a known-window oracle serves as an upper bound, which makes the comparison fair.
S3. Negative results are reported transparently. The authors openly report the performance loss under online allocation and the controlled experiments showing that simpler alternatives outperform the proposed reference design.
Critical weaknesses
W1. The definition of the useful window ignores the backend's response time, and there is no evidence that the calls actually help. The target window is the 0.5 s before onset, but backends such as VLMs usually take longer than this to respond, and the authors themselves report that "longer lead times enabled timely returns." In other words, UWC counts as useful the call times at which a response cannot arrive in time, and the authors acknowledge that UWC does not measure the value of the returned answer. The results therefore show better placement under a surrogate target that ignores latency, but not that PRS makes reasoning calls more useful. Evaluation with windows shifted earlier by the latency, as well as results showing actual improvements in driving, are needed.
W2. The main results rely on an offline setting that uses hindsight, and the paper does not show that the gains survive under causal allocation. The offline selector compares the scores at all time steps of a completed trajectory before choosing calls, which is not possible in deployment. Under the evaluated causal allocator, coverage retention in the primary evaluation domains, Town10HD and Town07, drops to 0–16%, partly because the allocator commits far fewer calls than the budget, placing only 32 of the 165 budgeted calls on Town10HD. The paper therefore does not yet show a deployable allocation that preserves the offline gains. In addition, each of the 63 trajectories in the primary Town10HD cohort contains exactly one event, which makes the task closer to locating a single event in a completed drive than to scheduling calls during ordinary driving. There is also no comparison with stronger baselines such as a lead-time-adjusted risk trigger or accident anticipation models.
W3. The methodological novelty is limited, and missing details limit faithful reproduction. PRS combines a small GRU and CNN classifier with greedy top-K selection. In the authors' own controlled comparisons, broad 2 s supervision significantly outperforms the proposed objective and a model using only the current state performs better on average, which leaves the history-based design without clear support. In addition, the identity of the 17-dimensional input features, the simulation tick interval, the onset definition for each event type, the definitions of the time-to-event bins and auxiliary labels, and the causal allocation algorithm are not described, which limits faithful reproduction of the method and results.
Questions for authors
Q1. Does the advantage of PRS persist when it is evaluated with windows shifted earlier than the onset by the backend's response time?
Q2. Why does the performance of the causal allocator drop so sharply on Town10HD and Town07, and why does it commit far fewer calls than the budget? Can PRS outperform Random-k and a lead-time-adjusted risk trigger under online allocation and on long trajectories with no events or multiple events?
Q3. What are the state features, the bird's-eye-view channels, the tick interval, the event onset definitions, the time-to-event bins and auxiliary labels, and the causal allocator? If a current-state-only model and broad supervision perform better, why keep the history-based GRU as the reference design?
2 comments
Thank you for the detailed response and the revisions. The new figure and section on timing make the problem much clearer, and the added details on the inputs, labels, and allocator address my reproducibility concern. The new towns with more spread-out events are also a welcome addition.
However, my main concern remains. The latency-shifted results show that PRS only wins at short lead times, and at the measured latency of your original pipeline, the TTC rule covers more windows. The closed-loop test also shows no driving benefit, so I still do not see evidence that better placement actually helps the agent. The online results are also weak. On Town10HD, the allocator uses only 32 of the 126 available calls without a clear reason, and on Town07 it falls well behind offline placement even with the same number of calls.
I appreciate that these results are reported openly, but since this gap goes to the core of the paper's motivation, I will keep my score.
Thank you for your careful review and helpful comments. We address each of your questions in turn.
Q1. Coverage under latency-shifted windows
The shifted-window analysis identifies the response lead times under which predictive scores retain a placement advantage. Section 5, Table 2, and Appendix F.2 keep calls and window width fixed while shifting windows earlier, using onset as a reference deadline. With 165 calls per method and a 0.30 s shift, PRS covers 22/63 events, versus a mean of 6.35/63 for Random-k and 8/63 for lead-time TTC. The paired gains have 95% intervals of [3.09, 29.3] and [3.44, 26.3] events, respectively. At the original driving pipeline's 1.33 s p95, Table 2 instead shows higher coverage for TTC. The advantage therefore depends on the required lead time.
New Figure 1 and Section 3.3 connect these comparisons to reasoning assistance by distinguishing event onset, evidence availability, the decision deadline, and response latency. Feasible calls require relevant evidence and enough time for a response to arrive by the deadline. UWC evaluates request placement, while these additional conditions specify when a request could support a decision.
Q2. Online allocation, budget use, and event distributions
PRS produces scores from current and past observations, while the allocator handles budget and spacing. Revised Section 6.2 clarifies that the evaluated sequential allocator knows trajectory duration but does not read future scores. On Town10HD, per-trajectory flooring reduces the nominal allowance of 165 calls to a ceiling of 126, and 32 calls are realized. Flooring explains the change in the ceiling, while the remaining 94 unused calls concern sequential commitment.
Appendix D.2 controls for call quantity by comparing offline and sequential placement at matched realized counts. Their UWC values are 0.0159 and 0 on Town10HD, and 0.2846 and 0.0488 on Town07. These comparisons separate budget utilization from placement, showing that call quantity alone does not explain the online coverage difference.
Sections 3.1 and 3.2 allocate a shared budget per trajectory, so multiple events compete for the same calls and spacing capacity. The budget does not reset for each event. Section 5, Table 3, and Appendix F.3 add offline comparisons on Town05 and Town06 with more dispersed onsets, where PRS exceeds elapsed-time and route-progress rules. These results complement the single-event cohort. The requested online comparison against Random-k and lead-time TTC on long trajectories with no events or multiple events is not included in the current evaluation.
Q3. Method definitions and the reference design
State features describe vehicle motion, control, and surrounding risks, while BEV provides spatial context. Section 4.1 explains their use. Appendix A.2 lists all 17 state inputs in stored order, their provenance, and the packed semantic BEV representation of size 3 x 64 x 64. It specifies conversion to floating point and division by 255, and distinguishes the training-manifest clock from acquisition intervals and evaluation cadence.
Sections 4.1 and 4.2 define the call score as the sum of the first two TTE probabilities, jointly covering the 0.5 s before onset. Phase, horizon, and urgency heads provide auxiliary supervision without entering the score. Appendix A.1 explains recorded event assignments, zero window labels for off-event examples, and event-wise coverage. Appendix A.2 distinguishes evaluation timing categories from training-head classes. These descriptions specify the deployed scoring interval, recorded-label usage, and evaluation conventions.
Section 4.3 and Appendix A.5 specify the shared offline selection rule. Section 6.2 and Appendix D.2 describe sequential commitment and its information conditions.
We retain the GRU and BEV encoder as a lightweight reference implementation for studying opportunity prediction under shared calling constraints. Keeping the checkpoint fixed also lets the latency analysis examine existing calls without retraining effects. Section 6.1 and Appendix C report that current-state inputs and broader supervision can match or exceed the reference results. Current observations can support future-window prediction, and broader training targets can improve ranking within the same evaluation window. These findings identify useful design choices within the framework. The contribution is event-relative opportunity prediction and controlled temporal placement, with inputs, encoders, and supervision treated as implementation choices.