- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In the paper the authors tried to address the problem that at any-time departure rate saturates, which mean that once essentially every predicted trajectory leaves the region at some point, the metric can no longer rank models. The contributions are an identity decomposing departure into endpoint exclusion plus return (Prop. 1), finite-sample selection bounds from Hoeffding plus a union bound, and experiments on learned dynamical systems.
Critical strengths
- I like that the method distinguishes between models that leave the safe region and return, and those that never leave it, which is important for safety evaluation.
- On the main benchmark, all 30 models have a departure rate of 1, so the metric cannot distinguish between them. Adding the endpoint helps address this without extra rollouts.
- The limitations are clearly stated and honest, which I really appreciate. They acknowledge that the endpoint metric is not necessarily better than other alternatives, that the proposed screen cannot guarantee non-collapse at the target horizon, and that the findings are limited to specific systems and time horizons.
Critical weaknesses
- It is shown in the supp material that 24/30 models have concentrated endpoints, while only five are spread and one overflows. Two of the three models with the best endpoint scores have collapsed trajectories, while the third eventually overflows. So maybe the metric may be identifying models that learned the dynamics or simply models that stay in a small region.
- The strong overall correlation (around 0.9) also seems to be driven by these cases. After removing the 20 models where both metrics reach their ceiling, it drops to 0.5 for the remaining ten models. For the five spread models, it even becomes negative. I understand that five models are too few to draw strong conclusions, but this makes the overall correlation much less convincing.
- The endpoint selection performs better than occupancy; however, these improvements depend on how ties are broken. In all three cases, the worst-case regret is 0.865, exactly the same as occupancy. This means the advantage may not be consistent and can disappear depending on which tied model is selected.
- When stricter filtering is applied, the two methods actually perform equally well. So while the endpoint metric clearly helps when the departure metric is saturated, we do not know if it offers an advantage over occupancy, which is already a simple and available alternative. This is important because the paper's main contribution depends on showing that endpoint selection is useful in practice.
Questions for authors
- Does endpoint selection still outperform occupancy when considering only models that do not collapse? Would the results change if model quality were checked at the later evaluation horizon?
- How much of the improvement depends on how ties are broken? Would the advantage remain with a fixed tie-breaking rule?
- Could the paired disagreement values needed to support the theoretical claim be obtained from another run, or is there a reason they cannot be measured?
- Could the authors clarify whether the concentration check is enough to support the reported correlation, given that many models still collapse at the later horizon?
- How well does the proposed method generalize to other model families, especially those that do not show the same saturation problem?