acceptodds
Withdrawn from ICLR 2027

When Departure Rates Saturate: Endpoint Diagnostics for Learned Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The any-time departure rate is the fraction of predicted trajectories that leave a prescribed region at least once. At long horizons, this rate can saturate at one even when models differ in their return behavior. We study the additional information obtained by recording whether the same predicted trajectories end inside or outside the region. An exact decomposition distinguishes terminal exclusion from departure followed by return. A two-state calculation shows how identical departure rates can coexist with different return behavior while endpoint and later-occupancy rankings still agree. For fixed candidate models, we give finite-sample endpoint-risk bounds that allow diagnostic eligibility screens and randomized ties, while isolating the additional condition needed for temporal transfer. In a complete 30-model mushroom-billiard cohort, every departure rate equals one, yet terminal exclusion predicts later outside occupancy on disjoint starts with Spearman correlation . The association survives a diagnostic concentration screen and appears in a separate study of both Koopman autoencoders and residual predictors. Exploratory selection analyses improve on the saturated rule, including after accuracy screening; the residual-family selection gain remains unresolved. These results support a low-cost evaluation practice: report endpoint status alongside departure and predictive fidelity, and evaluate selection performance against a separately specified regional target.

Primary area:
learning on time series and dynamical systems
Keywords:
learned dynamics, trajectory evaluation, endpoint diagnostics, model selection, long-horizon prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LvFLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the paper the authors tried to address the problem that at any-time departure rate saturates, which mean that once essentially every predicted trajectory leaves the region at some point, the metric can no longer rank models. The contributions are an identity decomposing departure into endpoint exclusion plus return (Prop. 1), finite-sample selection bounds from Hoeffding plus a union bound, and experiments on learned dynamical systems.

Critical strengths
  • I like that the method distinguishes between models that leave the safe region and return, and those that never leave it, which is important for safety evaluation.
  • On the main benchmark, all 30 models have a departure rate of 1, so the metric cannot distinguish between them. Adding the endpoint helps address this without extra rollouts.
  • The limitations are clearly stated and honest, which I really appreciate. They acknowledge that the endpoint metric is not necessarily better than other alternatives, that the proposed screen cannot guarantee non-collapse at the target horizon, and that the findings are limited to specific systems and time horizons.
Critical weaknesses
  • It is shown in the supp material that 24/30 models have concentrated endpoints, while only five are spread and one overflows. Two of the three models with the best endpoint scores have collapsed trajectories, while the third eventually overflows. So maybe the metric may be identifying models that learned the dynamics or simply models that stay in a small region.
  • The strong overall correlation (around 0.9) also seems to be driven by these cases. After removing the 20 models where both metrics reach their ceiling, it drops to 0.5 for the remaining ten models. For the five spread models, it even becomes negative. I understand that five models are too few to draw strong conclusions, but this makes the overall correlation much less convincing.
  • The endpoint selection performs better than occupancy; however, these improvements depend on how ties are broken. In all three cases, the worst-case regret is 0.865, exactly the same as occupancy. This means the advantage may not be consistent and can disappear depending on which tied model is selected.
  • When stricter filtering is applied, the two methods actually perform equally well. So while the endpoint metric clearly helps when the departure metric is saturated, we do not know if it offers an advantage over occupancy, which is already a simple and available alternative. This is important because the paper's main contribution depends on showing that endpoint selection is useful in practice.
Questions for authors
  1. Does endpoint selection still outperform occupancy when considering only models that do not collapse? Would the results change if model quality were checked at the later evaluation horizon?
  2. How much of the improvement depends on how ties are broken? Would the advantage remain with a fixed tie-breaking rule?
  3. Could the paired disagreement values needed to support the theoretical claim be obtained from another run, or is there a reason they cannot be measured?
  4. Could the authors clarify whether the concentration check is enough to support the reported correlation, given that many models still collapse at the later horizon?
  5. How well does the proposed method generalize to other model families, especially those that do not show the same saturation problem?
Reviewer_CPvtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work looks to improve on evaluating learned dynamics models by making the observation that while all learned dynamics models accumulate error and eventually leave the "prescribed" region on a long-enough horizon (and therefore all have the same final error based on that criteria), there is still discriminative signal on whether the model returns to the prescribed region at the end of the trajectory.

They test this by using a toy deterministic physical system of mushroom billiards with no actions/control inputs, train Koopman autoencoders and residual MLPs on this data, and demonstrate using whether the trajectory ended inside the prescribed region is better able to separate models by quality than just whether the trajectory ever leaves the region.

Critical strengths
  1. The main result, which is that quality of the dynamics model is better predicted from endpoint information than when departure rates saturate, is convincingly shown in Figure 2.
Critical weaknesses
  1. Endpoint information does not convincingly explain whether the learned model is actually any good (e.g. persistence control would get a perfect regional score despite being useless as a simulator).
  2. Any generality of the results would be greatly helped by studying another system where the results hold, ideally a system with more complex dynamics (like any conceptual robotics example, or any system with action data.) Standard map and smooth-flow results don't test model selection.
Questions for authors
  1. Can you explain what the interval is for the 10 non-ceiling models?

  2. Can you show how the results compare to a baseline of dropping models with endpoint rate 1 and random selection among the rest of the models?

  3. Can you explain the strongest result in the paper for model selection which could hold across other systems (potentially future work)? It seems that the method is stronger than picking a model using a completely saturated signal, which doesn't seem like a surprising result.

Reviewer_UNYvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies a practical blind spot in the evaluation of long-horizon trajectories' learned dynamics: the any-time departure rate (fraction of trajectories that ever leave a prescribed region) saturates at 1 once every rollout has exited, and therefore ceases to distinguish models that differ in return behavior. The authors propose retaining the endpoint membership label—already computed while evaluating the departure score—as a free complementary diagnostic. An exact accounting identity separates "farewells forever" from departure-and-return. A simple two-state Markov calculation shows that identical departure rates can coexist with different return rates while endpoint and later-occupancy rankings still agree. Finite-sample concentration bounds support diagnostic eligibility screens and randomized tie-breaking, while making the additional membership-disagreement term required for temporal transfer explicit. The authors study a complete 30-model mushroom-billiard cohort, where every any-time rate equals 1, yet terminal exclusion predicts later outside occupancy on disjoint starts (Spearman 0.905); the association survives a concentration screen and appears for both Koopman autoencoders and residual predictors. The paper's practical recommendation is simply to report endpoint status alongside departure rate and predictive fidelity, and to declare the regional target before selection.

Critical strengths

The problem is real and cleanly stated. Once departure saturates, a widely used regional score becomes uninformative; the paper shows this is not merely theoretical.

The accounting identity and the two-state illustration are elementary, correct, and useful for clarifying what the endpoint label does and does not recover.

The finite-sample result is careful: it isolates the extra term needed to transfer endpoint selection to a later window, rather than claiming automatic validity.

Empirical evidence is good (full crossed seed grid, disjoint evaluation starts, architecture comparison, concentration and accuracy screens), and the authors are transparent about exploratory analyses and unresolved contrasts.

The recommended practice is low-cost and immediately usable: the endpoint bit is free once the rollout has been evaluated for departure.

Critical weaknesses

The regime of practical value is narrow. When departure rates have not yet saturated, the any-time score already ranks models; when they have saturated, the endpoint is informative only if the diagnostic-horizon membership correlates with the later target of interest. The paper’s own bound shows that the transfer gap is exactly the expected membership disagreement. In settings where trap density, absorption probability, or hazard grows with distance or time (I summarize this as “there is a furnace farther away”), short-horizon returns supply little information about long-horizon no-returns, and the diagnostic collapses. The approach is most useful precisely where the temporal structure is already favorable—i.e., where the phenomenon is comparatively easy.

No comparison is offered to richer diagnostics that could exploit distance or path information. A differential estimator that models the probability of eventual return as a function of distance from the boundary (or of time spent outside) would be better matched to spatially non-uniform trap densities. The paper never tests whether endpoint status remains competitive against such alternatives, nor against other simple trajectory summaries (time-averaged occupancy, first-return time, etc.) once saturation has occurred.

Selection results remain exploratory and system-specific. The clearest gain is over the fully saturated any-time rule; superiority over whole-trajectory occupancy or one-step RMSE is not resolved, tie sets can be heterogeneous (one of three endpoint minimizers later overflows), and the residual-family contrast includes zero. Only one system supplies a true longer-horizon selection target.

Scope is limited to fixed evaluation panels, small crossed seed grids, and binary membership oracles. Concentration and overflow are acknowledged but not turned into a positive method for certifying simulator fidelity.

Questions for authors

Your endpoint diagnostic is free once the rollout has been evaluated, and it can retain ranking information after departure rates saturate. However, in many systems the probability of eventual return (or permanent exclusion) is itself a function of distance from the region or time spent outside. A differential estimator that models the return/non-return probability as a function of that distance would appear better matched to spatially non-uniform trap densities. Did you explore or ablate any such path-aware or distance-conditioned alternatives, and under what conditions do you expect plain endpoint status to remain competitive with them?

Proposition 3 makes the transfer gap explicit via the membership-disagreement term. In regimes where trap density increases with distance from the starting point, short-horizon returns may supply little information about longer-horizon outcomes, so the term grows and the endpoint signal weakens. Can you characterize (analytically or empirically) the spatial or temporal conditions under which the simple endpoint label is sufficient versus when a differential, distance-aware return model becomes necessary for reliable selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.