- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks a simple question: when a computer’s safety checks are slow, should we keep waiting or take precautionary action? Two collision-checking programs examine the same fixed path, starting from opposite ends. Other workloads compete for processor time. A monitoring program observes elapsed time and, depending on the experiment, processor-load measurements or partial checking progress. The authors investigate three approaches:
- Account for a shared busy processor: two delayed checks are not necessarily two independent warning signs: they might share one computational bottleneck. The authors compare models that account for this dependence with models that treat the delays independently.
- Use information received before checking finishes: The monitor can observe checking progress or the arrival of processor-usage samples. It estimates danger and chooses whether further waiting is worthwhile.
- Combine verified portions of the path: If one checker verifies the beginning and another verifies the end, their combined results may cover the whole path before either checker finishes individually. The monitor can then approve based on that combined evidence.
The experiments use simulated paths and measured computer executions. They do not involve flying a physical drone.
Critical strengths
The paper some key strengths that are listed below:
-
The problem that they have chosen is relevant, particularly from a deployment of AI systems point of view. This becomes even more relevant in the context of world models and physical AI system where latencies, error checking etc. could be strong motivators (although the authors only focus on the application scenario of drones in this paper). The paper recognizes that delayed results can reflect processor congestion rather than danger. Treating every delay as an independent warning could produce misleading decisions.
-
The experiments are designed such that they isolate the importance of dependence. The authors rearrange recorded checker times while preserving each checker’s individual runtime distributions. This demonstrates that the relationship between delays can change prediction performance. They appropriately distinguish this constructed replay from a physical hardware intervention.
-
The paper tries to measure execution behavior in as detailed way as possible. Checker execution, competing workloads, message receipt, and decision opportunities are recorded. Related routes remain together in training or test splits, reducing leakage.
-
The paper tries to separate prediction accuracy from decision quality. The authors explicitly show that better probability predictions need not produce fewer hazardous approvals. They separately report waiting, intervention requests, and hazardous approvals.
-
The authors have been transparent in reporting of unsuccessful outcomes. Policies are selected before each test. The authors report the failed primary resource criterion, unsuccessful calibration results, and small median waiting benefit rather than presenting every result as a success.
Critical weaknesses
The authors should address the following weaknesses:
- It appears like the primary constrained test does not meet all the success criteria that are originally stated. The selected rule lowers hazardous approvals, but its upper confidence bound for intervention use exceeds the allowed cap. Thus, the paper does not establish the primary claim of improvement while satisfying both resource constraints. This is uncertainty about compliance—not an observed point-estimate budget violation.
- The safety improvement comes with substantially more interventions. Hazardous approvals decrease from 42.82% to 37.06%, while the overall intervention rate rises from 28.59% to 52.24% (based on my understanding of Table 1). This demonstrates a trade-off, rather than an improvement at equal resource use. Comparison against policies matched more closely on actual intervention rates would help establish the additional value of the observations.
- Based on the numbers in Sec 3.1, the shared-load model’s measured benefit appears to be small. Although the dependence argument is sound, the original measured prediction improvements over independence are only 0.21% and 0.13%, with no established improvement in decision cost over the selected contextual stopping baseline. The practical benefit of the specific model is therefore less compelling than the conceptual motivation.
- The selected processor rule uses sample arrival, not processor-usage values. All selected thresholds are zero. The successful signal is whether samples have arrived, rather than how much processor time was consumed. This limits conclusions about separating danger-related work from congestion through numerical CPU measurements. Does numerical CPU-usage information provide additional benefit beyond simply observing sample arrival?
- From a deployment standpoint, better progress-based predictions do not translate into safety improvement. On the same instrumented traces, the progress policy approves 26 hazardous routes versus 11 for the timing policy, while reducing unnecessary interventions. Its probability-calibration criterion also fails. This reinforces that the reported 18.1% prediction improvement should not be read as an 18.1% safety improvement.
- It appears like the deployment relevance remains limited. While I considered the strength of the paper arises from the deployability standpoint, that very point does not seem to have been addressed completely. The experiments concern static simulated geometry on one computer with deliberately paced CPU execution. Intervention is merely a recorded request, and the replayed waiting times exclude live policy-inference and action-publication costs. The work therefore does not establish real-flight safety or a net deployment speedup.
Questions for authors
-
Does the benefit persist at matched intervention rates? The CPU-arrival policy reduces hazardous approvals but raises interventions from 28.59% to 52.24%. Can you compare policies at matched intervention rates and comparable waiting times to isolate the value of the additional information?
-
How robust is the primary resource-budget result? The intervention-rate upper confidence bound exceeds the designated cap. What calibration margin or policy adjustment would be needed to meet the constraint reliably, and would the hazardous-approval benefit remain?
-
When does shared-load modeling materially improve decisions? Its measured prediction gains are small, and lower decision cost is not established. Under which workload or contention conditions does it provide a meaningful advantage over contextual stopping?
-
Do numerical CPU measurements add value beyond sample arrival? Since all selected thresholds are zero, can an ablation compare arrival-only information against arrival plus numerical CPU values under matched resource constraints?
-
Why does progress improve prediction while increasing hazardous approvals? Can you analyze the cases responsible for this trade-off and determine whether it arises from probability miscalibration, the decision-cost settings, or limited training support?
-
Do the benefits survive live execution overhead? Can you include monitoring, policy inference, and action-publication costs in an end-to-end experiment, rather than evaluating stopping decisions only through recorded replay?
-
How well do the results generalize? Do the policies remain effective on different hardware, unseen workloads, changing load within a query, and hazard prevalences other than the balanced 50% setting?
-
How broadly useful is complementary clearance? Given the 74.1 ms mean saving but only 1.50 ms median saving, which executions account for the large benefits, and can clearance-based approval meet useful waiting and intervention budgets?
Authors responses?
Dear authors,
Author-reviewer discussions end on November 18th. Please submit your responses to the reviewers reviews as soon as possible, so the reviewers would be able to respond to you.
Thank you.