acceptodds
Under review as a conference paper at ICLR 2027

How Much Can Sampling-Aware Uncertainty Buy? A Measurable Ceiling for Irregularly Sampled Time Series

Abstract

A large line of work conditions predictive uncertainty on observation density: a model should be less certain where a channel is sampled too sparsely to resolve its variation. We measure what that idea is worth on four irregularly-sampled benchmarks and report three findings. Units decide the reading. The density–variance sign inversion that motivates such heads is present on PhysioNet's raw per-channel scales and absent under the z-scoring the baselines use ( to , every channel positive). And a selective-risk comparison on a per-channel standardized loss must rank by the variance in the loss's units: ranked by the raw , as our own boards were, our head appeared to beat a deep ensemble by ; aligned, the two are indistinguishable. A ceiling and a floor answer different questions. The tower property caps a variance head reading the observation set alone at . We estimate the correlation ratio of a coarsening of — channel, horizon decile, density tercile — with a split-half statistic over whole sequences: on PhysioNet is raw and z-scored. A cross-fitted -only abstention rule lowers the normalised risk–coverage area by at least on PhysioNet and on USHCN for the mean of five fitted models, and a small ceiling does not bound this floor. Both are validated, including where they fail. Against a closed-form truth, interval coverage holds in heavy tails only because two biases cancel, falls to – in light tails, and collapses when cells hold fewer than targets. On six forecasters' residuals, on three benchmark settings, the ceiling is similar while the floor depends on the forecaster. Our own density-conditioned head establishes no gain from its density input, and on PhysioNet the -only lookup outranks its variance.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
irregularly sampled time series, uncertainty quantification, predictive uncertainty, observation density, conformal prediction, selective prediction, selective classification, negative results, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_sF3wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to proffer an analysis of the relationship between predictive uncertainty and sampling density. While this is generally accepted in the community, the authors find that this is not the case. The authors claim that after pre-processing this relationship between sampling density and uncertainty does not hold. Further, the authors analyze an empirical upper limit of how much information the sampling can give about the predictive error, which is quiet low according to the authors.

Critical strengths

The biggest strength might be in going against the grain. The relationship between sampling density and prediction error is largely accepted in the community, with minor caveats. If there is little or no mutual information in the sampling schedule and the prediction error, this would raise a very important point with ramifications on active learning, design of experiments, etc.

Critical weaknesses
  1. The paper is not well written at all. I had to read it many times before understanding it and I still dont understand all of it. There are multiple levels to this a) Undefined Terms: For instance, the authors use the term NABU over 50 times in the article, but it is never defined. What is this NABU? I work in time series methods and have never heard of this. There are multiple such terms. b) Circular or self-referential writing: As an instance, on page 4: "An earlier draft called NABU “theory-grounded”; we no longer do...". An earlier version of what? The reviewer has access to this draft and this draft only. I have not seen any other versions. Why do I need to know this? Similar allusions to "earlier versions" are scattered all through the submission. Whilst I am raising a few of these issues, I feel that the draft is rife with such shortcomings.
Questions for authors

I recommend a re-write from scratch. My rationale are shared in the section of Critical Weaknesses.

Reviewer_uCW5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper attempts to quantify how much information about forecasting error can be obtained from observation patterns in irregularly sampled time series, and whether sampling-aware uncertainty estimation provides meaningful benefits.

Authors derive a correlation-based upper bound on how well uncertainty estimates using observation information alone can track squared prediction errors. They also evaluate a density-conditioned uncertainty architecture, NABU, and investigate how normalization and evaluation units affect uncertainty comparisons.

Experiments cover four benchmarks and multiple forecasting models, alongside synthetic validation of the ceiling estimator. However, the empirical findings are mixed: density-specific benefits are established only in limited settings, the proposed architecture shows no consistent improvement from its density input, and the estimator exhibits known statistical limitations. The presentation is also difficult to follow, making the paper's main conclusions and contributions challenging to identify.

Critical strengths

The authors evaluate sampling-based uncertainty across multiple datasets and forecasters, including a cross-fitted sampling-only abstention baseline that demonstrates meaningful selective-risk improvements on PhysioNet. They also openly report failed ablations, statistical limitations, and corrections to earlier analyses, which improves the transparency of the empirical study.

Critical weaknesses

1. Limited novelty and unclear significance.

The paper does not convincingly motivate the significance of its central research question. The theoretical bound follows from an established correlation-ratio formulation, while the estimated ceiling applies only to a coarse summary of sampling information rather than the richer inputs used by practical models. The observation about aligning uncertainty scores with evaluation-loss units also largely reflects established evaluation principles.

2. Weak experimental design and inconclusive results.

The experiments do not establish consistent benefits from sampling-aware uncertainty. The proposed head shows no reliable improvement from its density input, and density-specific gains are limited. Furthermore, training all models for only ten epochs, using some baselines below their intended capacity, and substantially changing standard forecasting horizons weaken the comparisons. The ceiling estimator also exhibits acknowledged statistical limitations.

3. Poor writing and organization.

The manuscript is exceptionally difficult to follow. Methodological explanations are repeatedly mixed with retrospective corrections, failed hypotheses, and experimental caveats, obscuring the central contributions. Essential details of the main estimator are also deferred to appendices, while the Method section focuses largely on an architecture that is not the paper's primary contribution. This significantly undermines the clarity and assessability of the work.

Questions for authors

Q1. What motivated training all baselines for only 10 epochs, particularly when the paper acknowledges that some models were run below their intended capacity? How confident are the authors that the conclusions would hold with adequately tuned and converged baselines?

Q2. Given that the proposed uncertainty head shows no statistically reliable benefit from its sampling-density input across the evaluated settings, what evidence supports the practical importance of the proposed sampling-aware approach beyond existing uncertainty estimation methods?

Reviewer_cLR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Many uncertainty methods for irregularly sampled time series (such as patient vitals recorded at uneven times) assume a forecast should be less certain where a variable was measured less often. This paper asks how much of a model's forecast error the measurement pattern can actually explain. It gives an upper bound on how well any uncertainty estimate built only from the measurement pattern could track the real errors, and a lower bound from a simple rule that skips the forecasts the pattern flags as risky. It also reports that the surprising result behind the paper's own method, models predicting lower uncertainty where a variable is measured less often, disappears once each variable is standardized, so it came from how the data was preprocessed. This is a classic explained-variance setup: how much of the error variance the measurement pattern explains, paired with the standard rule for when a regression model should abstain from making predictions.

Critical strengths

The idea is interesting. Asking how much of the error a signal can explain before building an uncertainty method around it is a sensible question to ask first.

Critical weaknesses
  1. The paper reads like a log of the experiments the authors tried rather than a final paper for publication. The conclusion opens with "This paper began as a method for sampling-aware uncertainty in IMTS and ends as a measurement of what that idea is worth" and counts "twenty-six ledgered claims, twenty-three retracted in print". The experiment sections are framed as questions the authors asked themselves along the way ("One percent of what?", "Testing the bound with an object it actually covers"), and "earlier version" or "earlier draft" appears more than forty times. Labels like "board" and "E2B" are never defined, seven cross-references are broken, and much of the evidence sits in the appendices of a 55-page document. As a reader this is impossible to easily parse, or appreciate independent of the quality of work being presented. One has to reconstruct what is claimed now from a record of what was tried and withdrawn, which is completely counter to the point of a research publication. The authors declare that an LLM helped develop the theory, write the proofs, design the experiments, interpret the results, and draft and revise the text, and by their own cccount this has gone through several iterations but the mistakes are left here. This is not the standard for a research paper. I feel like the manuscript needs to be substantially revised into a clear statement of its claims before it is ready even for review.

  2. The experiments don't measure what the title asks. The upper bound only holds for an uncertainty estimate that looks at nothing but the coarse grouping above. A method that looks at anything more, including the measured values themselves, is not limited by it. That covers the closest published methods, HeTVAE and LLapDiff, and the authors' own NABU. HeTVAE has public code for PhysioNet and MIMIC-III but is never run. The headline PhysioNet result also comes from models trained on unstandardized data. With the standardization the baselines use, the lower bound falls to well under half, and the measurement pattern has no contribution to the upper bound. The conclusion itself says "Where the prize is we do not know."

  3. From what i could tell, the theory is not novel as well. The ceiling in the paper is the textbook correlation-ratio bound, which the paper itself calls two lines. The floor rests on the standard optimal reject rule for regression (Denis, Hebiri and Zaoui, NeurIPS 2020), which is not cited.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.