acceptodds
Under review as a conference paper at ICLR 2027

Joint Prior-Fitted Prediction for Partially Observed Multivariate Time Series

Abstract

Multivariate time series are rarely observed as a complete history followed by a fully unknown future. In healthcare, measurements can be missing within the context, channels can stop at different times, and some covariates may be already available over the forecast horizon. Many time-series foundation models operate on fixed forecast blocks and return point predictions or marginal predictive summaries, leaving cross-time and cross-variable dependence unavailable as an explicit joint density. We introduce JointTimePFN, a mask-conditional prior-data fitted network that predicts arbitrary missing cells in a fixed-length time-by-channel grid. Its query-size-agnostic low-rank Gaussian head returns a coherent joint posterior predictive distribution and its marginals in one forward pass. The model combines mask-aware bidirectional temporal encoding, rotary embeddings, and a broad linear model of coregionalization prior. This structured prior naturally motivates posterior predictive checks against exact and MCMC-based multi-output Gaussian process references, both in-prior and under distribution shift. When the reference posterior exhibits substantial dependence across prediction targets, the joint head closes 64% of the gap left by a correlation-free baseline and mostly outperforms autoregressive chaining of its marginals in joint likelihood on these synthetic references. On MIMIC-IV, a single model checkpoint performs competitively with baselines across zero-shot forecasting, known-future conditioning, ragged horizons, and interior imputation, while improving joint likelihood over its diagonal counterpart.

Primary area:
learning on time series and dynamical systems
Keywords:
prior-data fitted networks, multivariate time series, probabilistic forecasting, missing data imputation, joint posterior predictive distributions, amortized Bayesian inference, multi-output Gaussian processes

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ph3wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JointTimePFN, a zero-shot PFN that unifies forecasting, covariate conditioning, ragged horizons, and imputation on a masked T×NT \times N time-series grid. Pretrained on a synthetic LMC prior, the model outputs a single-pass low-rank Gaussian joint distribution over queried cells, and is evaluated against exact/MCMC GP posteriors as well as on MIMIC-IV.

Critical strengths

The paper proposes JointTimePFN, a zero-shot PFN for partially observed multivariate time series. It is pretrained on a synthetic LMC prior with diverse masks, outputs a low-rank Gaussian joint distribution over queried cells in one forward pass, and is evaluated against exact/MCMC GP posteriors and on MIMIC-IV.

Critical weaknesses

On the forecasting tasks (Conditions A–C in Table 2a), marginal MSE and CRPS9CRPS_9 are only average compared to the baselines. In my view, the strongest contributions are the mask-conditional setup and the posterior-fidelity evaluation, whereas the low-rank joint head is more of an initial exploration. However, the paper frames the joint head as the central result, even though its real-data gains are relatively modest.

Questions for authors

On MIMIC-IV, why does AR extraction beat the single-pass joint head? Is this mainly due to the expressive limits of the low-rank Gaussian family, or optimization issues when training with the joint NLL?

Does the joint head show larger gains on datasets or channel subsets where empirical cross-channel correlation is stronger?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ph3w (1/2)

We thank the reviewer for the careful review and the helpful framing suggestions.

Q1: Why does autoregressive chaining of the marginals do better than the joint head on real data? Is this a problem with the low-rank Gaussian family, or with how the joint NLL behaves during training?

We would like to first clarify that comparing autoregressive chaining with our joint head is not a like-for-like comparison because the true preceding values enter AR's input at each autoregressive step.

We think one of the main reasons is the Gaussian family. The low rank itself does not seem to be the limit, as higher ranks do not improve the joint NLL (Table S20). On our synthetic LMC data the target is close to Gaussian, and there the joint head scores lower NLL than AR in conditions A–C, and approximately ties with it in D (Table 1(b)). Since this is the same model with the same training, we do not see evidence that the training of the joint NLL is the issue.

On MIMIC-IV, however, we now show that most of the gap comes from two channels: FiO$_2$ and PEEP (Section 5.2., Table S13 and Figure 4), which can remain nearly constant for long periods and then suddenly change in discrete steps. With AR extraction, as it performs KK (number of queried cells) forward passes instead of one, the preceding true values are added to the context one at a time, and entered as an input for the next forward pass. Therefore, it can quickly adapt to new changes that might not have been seen from the initial context set alone. In contrast, our model's likelihood is computed from the initial context in one forward pass, after which it follows the rule defined by its predicted covariance to compute the full joint NLL.

As we now have shown in the revised version (Table S13, Figure 4), when the two channels are not scored, the joint head is within about 0.08 nats per cell of AR, which is surprisingly good taking into account that AR can rerun the forward function every step. Finally, part of the original gap was due to a leak in our AR extraction, which is now fixed (see general response).

This overall relates to how AR computes the joint NLL: its joint density does not need to be Gaussian, since every forward pass it makes, the new mean and variance are non-linear functions of the initial context added with the already scored cells' true values, through the forward function. Against our joint model with one forward pass, this essentially gives different densities from the same network (see our response to Reviewer 8svM, second question).

Weakness: On the forecasting conditions, marginal accuracy is in the middle of the compared methods. I think the main contributions are the mask-conditional interface and the posterior-fidelity evaluation, and the joint head is a first attempt; the current framing makes the joint head look like the main result, which is where the evidence is weakest.

We agree that the interface and the fidelity evaluation are main contributions (Section 1), and also that our method is not uniformly the best in marginal forecasting accuracy. We also agree that the evidence for the joint head is strongest on synthetic data, while on real data it is mixed. The revised manuscript now states this more explicitly in the abstract and Section 5.2. We have also added analysis of where the joint head helps and where it loses on real data, including the effect of FiO$_2$ and PEEP (Table 2(b), Tables S11–S13 and Figure 4). We also added the runtime and memory comparison against AR extraction (Table S21).

AuthorsOpenReview ↗

Response to Reviewer ph3w (2/2)

Q2: Does the joint head help more on datasets (or channel subsets) where dependence between targets is empirically stronger? If so, that would give a practical rule for when to use it.

Mostly yes. We can calculate this in the following way: with r=yQ−μr = y_Q - \mu, the gain over our own diagonal is

Δ=NLLdiag−NLLjoint=1K[DKL(qθ∥qθdiag)+12(∑iri2Σii−r⊤Σ−1r)]\displaystyle \Delta = \mathrm{NLL}_{\mathrm{diag}} - \mathrm{NLL}_{\mathrm{joint}} = \frac{1}{K}\Big[ D_{\mathrm{KL}}(q_\theta \Vert q_\theta^{\mathrm{diag}}) + \tfrac12\Big(\sum_i \tfrac{r_i^2}{\Sigma_{ii}} - r^\top\Sigma^{-1}r\Big)\Big]

(Equations 10 and 11), where qθdiagq_\theta^{\mathrm{diag}} is our model with off-diagonals set to zero (Appendix B.3). The first term is the total correlation of our model's Σ\Sigma (Equation S6), and the second is the difference between the squared errors of the diagonal and the joint. For a calibrated model both squared errors have mean KK, so the second term has a mean equal to zero. Therefore, more predicted dependence should give more gain on average, unless the errors are very large.

To check this on real data, we measured the mean predicted correlation ρˉ\bar\rho between the queried cells, per dataset and condition. Then, to measure the realised correlation from the data, we calculated r^\hat r between their errors ri/Σiir_i/\sqrt{\Sigma_{ii}} (and r^trim\hat r_{\text{trim}} without the 1% most extreme cells at each tail), and the gain (Δ\Delta) (Tables 2(b) and S14). Here ρˉ\bar\rho is the mean of ρij=Σij/ΣiiΣjj\rho_{ij} = \Sigma_{ij}/\sqrt{\Sigma_{ii}\Sigma_{jj}} over all pairs of queried cells in each group, where group is one mask in one window, and r^=∑zizj/∑zi2∑zj2\hat r = \sum z_i z_j / \sqrt{\sum z_i^2 \sum z_j^2} over the same pairs, with zi=ri/Σiiz_i = r_i/\sqrt{\Sigma_{ii}}.

As can be seen from the paper's tables, on every real dataset we use, the gain is largest in condition A. We now measured that this is also where the realised correlation is strongest, while in condition D, where the gain is near zero or negative, the realised correlation is also close to zero. Since we have 4 conditions and 4 real datasets, we have 16 dataset and condition pairs, for which the Spearman correlation between r^\hat r and Δ\Delta is 0.580.58 (0.490.49 for the trimmed). With 16 pairs, however, this is only indicative. One exception is exchange rate's conditions B and C, where a few large errors make the second term clearly negative (see our response to Reviewer 8svM, second weakness, and Appendix D.2).

As a practical rule, one could calculate ρˉ\bar\rho from the same forward pass as the predictions, as it comes purely from the model's own Σ\Sigma (available even when true values are not, and without further forward passes). In our setting, ρˉ\bar\rho is a weaker predictor of the gain than r^\hat r (Spearman 0.41 over the 16 dataset/condition pairs), but it works quite well when the heavy-tailed exchange rate (in conditions B and C) is excluded (0.73 over the remaining 14 rows). Thus, the joint head tends to improve its NLL against the diagonal version when the predicted dependence is high and the errors are not heavy-tailed. Note also that when real values from the same domain is available, the gain and z2/Kz^2/K (Appendix B.3) can also be checked directly.

Overall this is in line with our limitations. The Gaussian head is sensitive to heavy-tailed errors, and future work could focus on richer output families such as Gaussian mixtures.

This analysis is based on the following table with all the real datasets we used (ρˉ\bar\rho: mean predicted correlation, r^\hat r: realised correlation, Δ\Delta: gain in nats per cell):

DatasetCondρˉ\bar\rhor^\hat rr^trim\hat r_{\text{trim}}Δ\Delta
MIMIC-IVA0.3710.3940.3130.708
B0.0770.0520.0540.113
C0.0900.0620.0560.258
D0.0030.0050.003−0.114-0.114
ETTh1A0.2640.2800.2040.632
B0.1640.1120.1180.332
C0.1280.0890.0820.365
D0.0090.0050.0050.037
Exchange rateA0.5020.4780.6941.055
B0.1450.2600.350−2.434-2.434
C0.1630.0800.238−4.044-4.044
D0.0070.0020.003−0.088-0.088
PhysioNetA0.0930.7400.4141.417
B0.0380.0400.0820.964
C0.0420.0630.0830.502
D0.0100.0050.013−0.060-0.060
Reviewer_ph3wOpenReview ↗

Reply to Author's Rebuttal

The authors clearly understand why the single-pass low-rank Gaussian joint loses to AR chaining on real data. I think this is enough to keep my score. But the novelty is limited, and the work could go deeper. This limits the paper. Whatever, we thank the hard work and patient rebuttal.

Reviewer_eDreOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method for joint probabilistic prediction over partially observed multivariate time series.

Critical strengths
  1. The paper studies an important problem by providing a unified framework for prediction and imputation on partially observed multivariate time series.
Critical weaknesses
  1. The methodological novelty is limited, as the proposed method mainly combines existing techniques.
Questions for authors

I congratulate the authors on the interesting work. I have a few minor questions.

  1. In Table S10, autoregressive extraction substantially outperforms the proposed method. Could the authors provide more discussion on this issue?
  2. The authors argue that the proposed method produces the full predictive in a single zero-shot pass, whereas the competing methods require either KK sequential passes or dataset-specific training. Could the authors provide an empirical comparison, such as inference time against autoregressive extraction, to better demonstrate the advantage of the proposed method?
  3. Section 4.4 introduces a diagonal plus low-rank parameterization of the covariance matrix. While I understand its computational advantages, the authors should further discuss the motivation and justification for this choice from a methodological perspective.
3 comments
AuthorsOpenReview ↗

Response to reviewer eDre (1/2)

We thank the reviewer for the positive assessment and the questions.

Q1: In Table S10 (now Table S13), autoregressive extraction substantially outperforms the proposed method. Could the authors provide more discussion on this issue?

As stated in the general comment, we fixed a leak in AR extraction, which reduced this difference: for example in condition B, the gap is now 1.08 nats per cell (2.45 before). The other reason is that on MIMIC-IV, two channels (FiO$_2$ and PEEP) can stay constant or nearly constant for long periods in the context (Section 5.2). Our model normalizes each channel by the variation in its observed context. Thus, when such a channel then changes in a step, our single forward pass gives a very large NLL. Meanwhile AR reruns the model after every revealed value. We measured in condition B, that on the first FiO$_2$/PEEP cell at a new level both are poor (AR 39.6, joint 45.4 nats per cell), but on the later cells of that channel AR drops to 1.2 nats, while the joint head stays at 13.3. This is because the true preceding values enter AR's input every run, and thus it is not a like-for-like competitor. TACTiS-2 likely benefits for a similar reason, as its copula also conditions each queried cell on the true preceding values (Appendix C), and it is additionally trained on the cohort. When these two channels are not scored, the gap shrinks: AR is within 0.08 nats per cell, and TACTiS-2 needs 50–75 training windows to match our model (Table S13, Figure 4).

Finally, each AR conditional is Gaussian, but its mean and variance depend non-linearly on the true values of the earlier cells, as these enter the network's input. Therefore, a density of all queried cells over repeated model calls does not need to be Gaussian (Section 4.1). In contrast, with a fixed context, the joint head's conditionals are linear in the earlier values with a fixed variance.

Q2: The authors argue that the proposed method produces the full predictive in a single zero-shot pass, whereas the competing methods require either sequential passes or dataset-specific training. Could the authors provide an empirical comparison, such as inference time against autoregressive extraction, to better demonstrate the advantage of the proposed method?

We measured the inference time and memory on the MIMIC-IV query cells (condition B, 20 windows, one NVIDIA V100, batch size 1, median over 5 repetitions). The joint head and diagonal (which is built from it) only need one forward pass for any number KK of queried cells, whereas AR needs one pass per cell. AR can also be batched when the true values are known in advance, as when computing the NLL. This gives the same density as the sequential approach but requires more memory. At prediction time, when the values are unknown, generating a sample still requires KK sequential passes. In the table, memory is the peak above the model weights. We have included this table in the revised manuscript (Table S21). For example at K=480K=480, the joint head is about 460 times faster than sequential AR and about 13 times faster than batched AR, which then needs about 1.5 GB of extra memory. For the trade-off between this and the quality of the prediction, please see our response to Reviewer 8svM (second question).

K=10K=10K=60K=60K=240K=240K=480K=480
Joint (ms)13.813.813.813.7
Diagonal (ms)13.313.113.213.1
AR, sequential (ms)13279031606320
AR, batched (ms)19.527.993.6173
Joint and diagonal, memory (MB)3333
AR, sequential, memory (MB)3333
AR, batched, memory (MB)311887491500
AuthorsOpenReview ↗

Response to Reviewer eDre (2/2)

Q3: Section 4.4 introduces a diagonal plus low-rank parameterization of the covariance matrix. While I understand its computational advantages, the authors should further discuss the motivation and justification for this choice from a methodological perspective.

We believe that the diagonal-plus-low-rank parameterization is a reasonable choice for modeling joint dependence. First, it captures correlations across cells, unlike independent marginal predictions. Second, it is particularly well suited to our evaluation framework, which assesses posterior fidelity against MOGP references whose predictive distributions are Gaussian at fixed hyperparameters. Third, every subset marginal is obtained exactly from the same joint by restricting Σ\Sigma (Section 4.4).

A similar parameterization has been used for multivariate time-series modeling in [1]. Moreover, our ablation investigated the effect of covariance rank (Appendix D.5 and Table S20 of the updated manuscript), considering R∈{4,8,16,32,64,128}R \in \lbrace 4, 8, 16, 32, 64, 128 \rbrace. The chosen rank R=32R=32 achieves the lowest joint NLL on MIMIC-IV, with no further improvement at higher ranks. This suggests that the low-rank constraint is not the main performance bottleneck in this evaluation.

Finally, as stated in the limitations of the paper, more expressive alternatives, such as Gaussian mixture models, could capture non-Gaussian dependence but introduce additional complexity. These represent relevant avenues for future work. Beyond the choice of the distribution, another possible extension stated in the limitations of the paper is to model temporal and cross-channel dependence separately, rather than through the same low-rank factors.

[1] Salinas et al. High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes, NeurIPS 2019.

Weakness 1: The methodological novelty is limited, as the proposed method mainly combines existing techniques.

The reviewer is correct that the individual components of our method, including PFNs, low-rank Gaussian parameterizations, and LMC priors, are established. Combining them sensibly is part of our methodological contribution, but we believe our contribution goes beyond that. To the best of our knowledge, existing time-series PFNs provide point predictions or marginal predictive distributions, whereas JointTimePFN offers a unified, mask-conditional formulation that returns an explicit joint distribution over arbitrary queried cells, together with its consistent marginals, in a single zero-shot forward pass. Concretely, we change the backbone to encode the full grid with the mask paired to each channel, normalize from observed cells only and use bidirectional local context (Section 4.3). Moreover, we broaden the prior with continuous kernel hyperparameters and signed loadings (Section 4.2), which we show to matter: with TimePFN's fixed kernel bank, the joint NLL rises from 1.952 to 3.298, and unsigned loadings worsen all three metrics (Table 3).

Another important aspect of our work is to provide a systematic empirical assessment of joint posterior fidelity. We compare against both an exact MOGP oracle and a Bayesian MOGP reference using MCMC, under matched and shifted prior conditions, thereby quantifying how faithfully the PFN amortizes joint Bayesian inference. Then, we further evaluate the same checkpoint across four observation regimes on real-world data, against a broad range of forecasting and probabilistic baselines, and investigate the contributions of the pretraining prior and architectural choices through ablations. We believe these findings constitute a substantive contribution beyond the novelty of the individual components.

Reviewer_eDreOpenReview ↗

Response to authors

I thank the authors for the responses and the additional experiments, which partially address my concerns. I recommend adding the motivation for the diagonal plus low-rank parameterization in the paper. I will maintain my positive assessment of the paper.

Reviewer_8svMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JointTimePFN is a mask-conditional prior-fitted network for partially observed multivariate time series. A full-grid encoder and a low-rank Gaussian head produce an explicit joint distribution and analytic subset marginals in one pass. The model combines a broadened LMC pretraining prior, signed loadings, rotary embeddings, and joint-plus-marginal likelihood training. Synthetic experiments compare posterior fidelity with exact and MCMC-based GP references; real-data experiments cover forecasting, known-future conditioning, ragged horizons, and imputation on MIMIC-IV and three additional datasets. The joint head improves over its own diagonal on MIMIC-IV, with mixed gains elsewhere.

Critical strengths
  1. The prediction interface is useful and mathematically explicit. The D + W W^T covariance supports tractable likelihoods and exact subset marginals within a fixed-context prediction. A single mask-aware model handles several observation patterns, extending the practical interface of a forecasting PFN.

  2. The controls distinguish several sources of predictive quality: exact and Bayesian GP references, the model's own diagonal, autoregressive extraction, and a cohort-trained copula model. In the synthetic in-prior forecasting condition, the head removes approximately 64% of the dependence-free KL gap and outperforms autoregressive extraction under conditions A-C. The real-data results preserve natural missingness and report cases where adding correlations hurts likelihood.

Critical weaknesses
  1. Configuration-selection independence is unresolved. Appendix D.5 evaluates rank and loss-weight variants on the full MIMIC-IV cohort, with 1,256 windows for A-C and 1,258 for D. Table S18 identifies joint NLL as the selection criterion, and the accompanying text uses those outcomes to justify retaining the composite loss. Appendix B.4 states that the main experiment scores the first 400 windows from this cohort. This raises a concrete possibility that the reported evaluation informed configuration selection; it does not by itself prove that it did. Synthetic-only gradient training and the patient-disjoint split for TACTiS-2 do not establish the independence of JointTimePFN's own selection. The chronology and data used to choose rank, loss weight, and checkpoint are necessary to assess the strength of the zero-shot transfer evidence.

  2. The practical quality-efficiency case for the joint head is incomplete. On the same 400 MIMIC-IV windows and shared subsets of at most 60 query cells, Table S10 reports joint NLLs of 1.6500, 2.9579, 2.3826, and 1.3972 for the head versus 0.8532, 0.5063, 0.5763, and 0.6273 for autoregressive extraction. The condition-B difference is 2.4516 nats per queried cell. The paper acknowledges this loss and correctly identifies the one-pass advantage, but provides no measured latency/memory comparison to show the resulting trade-off. Table S11 also shows that the head loses to its own diagonal on exchange-rate B/C and PhysioNet D. These real-data results differ from the favorable synthetic comparison. They leave unclear when the explicit Gaussian correlations offer sufficient practical value to offset their likelihood cost relative to the available alternatives.

Questions for authors
  1. Clarification: Were all 400 main evaluation windows excluded from decisions about R = 32, lambda = 0.5, and checkpoint selection? Please give the selection data and chronology. If Table S18 is post hoc, what independent rule determined the reported configuration? If those windows informed selection, are results on untouched patients available?

  2. Assessment-changing validation: On identical query subsets, what are inference time and peak memory for the direct joint, its diagonal, and autoregressive extraction? Does the one-pass quality-efficiency trade-off remain favorable as query size increases?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8svM (1/3)

We thank the reviewer for the detailed review and the concrete questions.

Weakness 1 (configuration selection): please see our answer to your first question below.

Q1: Clarification: Were all 400 main evaluation windows excluded from decisions about R = 32, lambda = 0.5, and checkpoint selection? Please give the selection data and chronology. If Table S18 (now Table S20) is post hoc, what independent rule determined the reported configuration? If those windows informed selection, are results on untouched patients available?

To clarify this, the final model was selected based on synthetic validation data and convergence statistics (and the checkpoint by the synthetic validation loss). We ran variants of RR (with λ=0.5\lambda=0.5) and of λ\lambda (with R=32R=32) (Appendix D.5), and followed how the validation loss decreases and converges. Additionally we measured the calibration (z2z^2, target 11) (Appendix B.3), both marginally and jointly, in hope to continue with a potentially calibrated model. In chronological order, we first trained the variants and compared them on the synthetic validation set, and only afterwards evaluated them on MIMIC-IV (Appendix D.5), which also informed our final choice (Appendix B.4), so the choice is not fully independent of the evaluation windows. Because of this and to answer your question, we recomputed the ablations on windows from patients that are not in the evaluation set. Note however that our final ablations reported in the paper still include these, since we used all available windows there. The 400 evaluation windows come from 393 patients. Removing all these patients from the 1258 windows (and the windows where a queried channel has no observed context) leaves 810 windows (808 in A–C) from 799 patients. On these windows, the results are the following (mean over conditions like in our Ablations in Appendix D.5):

VariantMSECRPS$_9$Marg. NLLJoint NLL
R=32R=32, λ=0.5\lambda=0.5 (our model)0.57900.36852.3162.041
R=4R=40.58600.36952.5982.220
R=8R=80.58540.36982.5472.255
R=16R=160.57850.36702.2602.110
R=64R=640.58400.36892.3522.078
R=128R=1280.59120.37092.6822.167
λ=0\lambda=00.58290.39022.1392.129
λ=0.25\lambda=0.250.58300.37112.3932.099
λ=0.75\lambda=0.750.58170.36802.4512.083
λ=1.0\lambda=1.00.58440.36792.3602.104

Our final model still is the best on joint NLL (although R=64R=64 and λ=0.75\lambda=0.75 are very close), while for other metrics it varies between other versions. Rank 16 shows the best MSE, CRPS$_9$, and marginal NLL when λ=0.5\lambda=0.5. When R=32R=32 is fixed, our model has the best MSE, while λ=1.0\lambda=1.0 has the best CRPS$_9$. Additionally, the best marginal NLL comes with λ=0\lambda=0. We did not train models with combinations excluding both λ=0.5\lambda=0.5 and R=32R=32.

AuthorsOpenReview ↗

Response to Reviewer 8svM (2/3)

Q2: Assessment-changing validation: On identical query subsets, what are inference time and peak memory for the direct joint, its diagonal, and autoregressive extraction? Does the one-pass quality-efficiency trade-off remain favorable as query size increases?

First, we can evaluate the models on identical query subsets, while increasing KK (number of queried cells). This is now also in Table S21:

K=10K=10K=60K=60K=240K=240K=480K=480
Joint (ms)13.813.813.813.7
Diagonal (ms)13.313.113.213.1
AR, sequential (ms)13279031606320
AR, batched (ms)19.527.993.6173
Joint and diagonal, memory (MB)3333
AR, sequential, memory (MB)3333
AR, batched, memory (MB)311887491500

The diagonal comes from the same forward pass as the joint, so its memory is the same, and the joint head costs approximately the same for every KK, whereas AR extraction costs roughly KK times the time (sequential). Since the true values are known in advance for NLL evaluation, the KK passes are independent and can be batched. With this, the computation increases the memory roughly linearly in KK.

To answer the second part of your question, we ran condition B on 50 MIMIC-IV windows with same set of KK = 10, 60, 240 and 480. We have now fixed the AR leak (see general response) and further evaluated the reason for the large gap, which is largely caused by the following pattern: a channel has a nearly constant context, which can then suddenly make a steep step in the future. This kind of behaviour appears in PEEP and FiO$_2$ channels on MIMIC-IV (see our response to Reviewer eDre, first question, and Section 5.2). After computing NLL per queried cell (nats) with and without the aforementioned channels scored, we get:

K=10K=10K=60K=60K=240K=240K=480K=480
All channels as targets
Joint1.8803.6793.3673.295
Diagonal1.8893.7663.9403.848
AR1.8661.5771.3090.970
Without FiO$_2$ and PEEP as targets
Joint1.2771.1621.0411.001
Diagonal1.2761.3041.3081.297
AR1.2751.1560.9570.865

With all channels, the mean is dominated by a few windows with large FiO$_2$ and PEEP errors (the median gap to AR is −0.02-0.02 at K=60K=60, 0.080.08 at K=240K=240 and 0.160.16 at K=480K=480). Without FiO$_2$ and PEEP scored, the joint head nearly matches AR up to K=60K=60, while AR achieves increasingly lower NLL at larger query sizes. At K=480K=480, AR reaches 0.865 versus 1.001 for our joint head, but requires approximately 13 times more time even when batched, together with 1.5 GB of additional memory. This is expected, because AR extraction goes through the cells one at a time, and can adapt to changes by rerunning our model's forward function KK times, with every step adding the true values of the previously evaluated cells to the context. Our model in contrast runs only one forward pass with the initial context. Therefore, AR extraction is not really a like-for-like competitor, and it is actually a bit surprising how well our model performs against it up to K=60K=60. When comparing against the diagonal, the joint head's gain remains positive throughout our test (except at K=10K=10, where they tie).

To conclude, as KK grows, AR gains in likelihood, but its cost grows roughly linearly in KK, either in time (sequential) or in memory (batched, when the true values are known), while joint head always performs only one forward pass. Thus, the efficiency advantage of the joint head grows with KK, while its likelihood gap to AR also grows. So, whether the trade-off is favorable depends on how much the KK-fold cost matters in the application.

AuthorsOpenReview ↗

Response to Reviewer 8svM (3/3)

Weakness 2: Table S11 (now Table S14) also shows that the head loses to its own diagonal on exchange-rate B/C and PhysioNet D. These real-data results differ from the favorable synthetic comparison. They leave unclear when the explicit Gaussian correlations offer sufficient practical value to offset their likelihood cost relative to the available alternatives.

To answer this, we measured the calibration z2/Kz^2/K (Appendix B.3) (which is about 1 for a calibrated model) on exchange rate. We found that a few large errors cause the strongly negative gain. z2/Kz^2/K gives 10.4 in condition B and 16.4 in C. After removing 1% of cells with the largest standardised errors at each end, z2/Kz^2/K falls to 1.5 and 1.1, respectively (Appendix D.2). These errors make the joint's squared error r⊤Σ−1rr^\top\Sigma^{-1}r much larger than the diagonal's ∑iri2/Σii\sum_i r_i^2/\Sigma_{ii}, which can happen when the errors do not match the predicted correlations. After removing them, the joint is better than its diagonal in both B and C, by 1.251.25 and 1.001.00 nats per cell, respectively.

Such errors are rare on our synthetic data, as each of our series is drawn from a Gaussian process with Gaussian noise, which likely explains the difference from the real dataset performance. Furthermore, in PhysioNet with condition D, the queried cells are nearly conditionally independent (predicted correlation 0.010, realised 0.005, or 0.013 after trimming), so a gain close to zero is also expected (gain is ≈−0.060\approx-0.060).

For when correlation helps, please see our response to Reviewer ph3w (second question), and for the cost compared with autoregressive extraction, see our answer to your second question. Overall, we see this behaviour, including inability to deal with heavy-tailed data, as a limitation of the Gaussian family. As noted in the paper, this motivates richer output families for future work, such as a mixture output, although at additional cost.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response and summary of revisions

We thank the reviewers for their careful reading and constructive feedback, which we believe has significantly strengthened our submission. In addition, before preparing our rebuttal, we revisited our method's code and evaluation pipeline and identified several issues, which we have now corrected. The main changes are:

  • Autoregressive baseline. We fixed an information leak in the AR extraction procedure, substantially reducing its performance gap relative to our joint head.
  • Joint evaluation and ablations. We corrected the diagonal comparison to use the same variance floor as the joint head, and fixed the RoPE ablation, which had inadvertently been evaluated with RoPE enabled. With the corrected diagonal comparison, the joint likelihood gain under condition D in MIMIC-IV becomes slightly negative.
  • MIMIC-IV evaluation. We excluded nine windows in which a queried channel had no observed context, leaving 391 evaluation windows (400 earlier). This mainly affected Chronos-2, whose CRPS$_9$ values were quite large before. See Appendix C for further explanation.
  • Posterior fidelity. We strengthened the MOGP reference using longer NUTS runs and convergence screening, and recomputed the corresponding KL divergences. Moreover, we removed the Q = N fidelity pool from the appendix, as the convergence screening dropped most of its configurations.
  • External baselines. We evaluated TabPFN-TS and TabImpute using their native predictive densities and expanded the TACTiS-2 training experiments.
  • Additional analyses. We investigated the joint NLL gap on MIMIC-IV, finding that it is largely driven by two channels (FiO$_2$ and PEEP). For some subjects these two features had constant or nearly constant values in the context. Excluding these channels from scoring brings our joint head close to the AR extraction of our model. In response to the reviews, we also measured inference time and memory against AR. We also added a table of the compared joint models (Table S12) and elaborated explanations of AR extraction and TACTiS-2 (Appendix C).

These corrections update several reported numerical results, but the main conclusions remain unchanged, except that on MIMIC-IV the joint head now improves on its own diagonal in three of the four conditions instead of all four. All changes have been incorporated into the updated manuscript, with the main modifications marked in red, and the tables which are rerun or new are marked accordingly in the caption. All section, figure, and table references in our responses refer to this revised version.

AuthorsOpenReview ↗

Final comment

We want to thank each reviewer once more for taking the time and effort to provide us valuable feedback during this rebuttal. We are happy to see that you have found the problem important, and the prediction interface, the mask-conditional setup, as well as the posterior fidelity experiments valuable.

Following Reviewer eDre's recommendation, we have now added a motivation for the diagonal-plus-low-rank parameterization to Section 4.4, referring to Salinas et al. (2019).

Regarding Reviewer 8svM's final comment, we would still like to clarify two points. First, we agree that the choice of rank and λ\lambda was not fully independent of the evaluation windows. However, as we showed by repeating the ablation without the evaluation patients, our final model still has the lowest joint NLL, so the evaluation windows did not change the choice. Moreover, as shown in our response (1/3), other variants would have performed slightly better on the marginal metrics, which thus indicates that the selection did not optimize the marginal results of Table 2(a).

Second, the two ventilator channels (FiO$_2$ and PEEP) explain most of the gap to AR extraction and TACTiS-2, but without them the gain over the diagonal still remains positive in conditions A–C (Table 2(b)). In condition D, as discussed during the rebuttal, the queried cells are nearly conditionally independent, so the gain there is close to zero or slightly negative in all of our experiments. Therefore, the change from four to three conditions mainly reflects condition D, where little or no gain is expected in the first place. We hope that this has made the setting clearer.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.