- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces JointTimePFN, a zero-shot PFN that unifies forecasting, covariate conditioning, ragged horizons, and imputation on a masked time-series grid. Pretrained on a synthetic LMC prior, the model outputs a single-pass low-rank Gaussian joint distribution over queried cells, and is evaluated against exact/MCMC GP posteriors as well as on MIMIC-IV.
Critical strengths
The paper proposes JointTimePFN, a zero-shot PFN for partially observed multivariate time series. It is pretrained on a synthetic LMC prior with diverse masks, outputs a low-rank Gaussian joint distribution over queried cells in one forward pass, and is evaluated against exact/MCMC GP posteriors and on MIMIC-IV.
Critical weaknesses
On the forecasting tasks (Conditions A–C in Table 2a), marginal MSE and are only average compared to the baselines. In my view, the strongest contributions are the mask-conditional setup and the posterior-fidelity evaluation, whereas the low-rank joint head is more of an initial exploration. However, the paper frames the joint head as the central result, even though its real-data gains are relatively modest.
Questions for authors
On MIMIC-IV, why does AR extraction beat the single-pass joint head? Is this mainly due to the expressive limits of the low-rank Gaussian family, or optimization issues when training with the joint NLL?
Does the joint head show larger gains on datasets or channel subsets where empirical cross-channel correlation is stronger?
3 comments
Response to Reviewer ph3w (2/2)
Q2: Does the joint head help more on datasets (or channel subsets) where dependence between targets is empirically stronger? If so, that would give a practical rule for when to use it.
Mostly yes. We can calculate this in the following way: with , the gain over our own diagonal is
(Equations 10 and 11), where is our model with off-diagonals set to zero (Appendix B.3). The first term is the total correlation of our model's (Equation S6), and the second is the difference between the squared errors of the diagonal and the joint. For a calibrated model both squared errors have mean , so the second term has a mean equal to zero. Therefore, more predicted dependence should give more gain on average, unless the errors are very large.
To check this on real data, we measured the mean predicted correlation between the queried cells, per dataset and condition. Then, to measure the realised correlation from the data, we calculated between their errors (and without the 1% most extreme cells at each tail), and the gain () (Tables 2(b) and S14). Here is the mean of over all pairs of queried cells in each group, where group is one mask in one window, and over the same pairs, with .
As can be seen from the paper's tables, on every real dataset we use, the gain is largest in condition A. We now measured that this is also where the realised correlation is strongest, while in condition D, where the gain is near zero or negative, the realised correlation is also close to zero. Since we have 4 conditions and 4 real datasets, we have 16 dataset and condition pairs, for which the Spearman correlation between and is ( for the trimmed). With 16 pairs, however, this is only indicative. One exception is exchange rate's conditions B and C, where a few large errors make the second term clearly negative (see our response to Reviewer 8svM, second weakness, and Appendix D.2).
As a practical rule, one could calculate from the same forward pass as the predictions, as it comes purely from the model's own (available even when true values are not, and without further forward passes). In our setting, is a weaker predictor of the gain than (Spearman 0.41 over the 16 dataset/condition pairs), but it works quite well when the heavy-tailed exchange rate (in conditions B and C) is excluded (0.73 over the remaining 14 rows). Thus, the joint head tends to improve its NLL against the diagonal version when the predicted dependence is high and the errors are not heavy-tailed. Note also that when real values from the same domain is available, the gain and (Appendix B.3) can also be checked directly.
Overall this is in line with our limitations. The Gaussian head is sensitive to heavy-tailed errors, and future work could focus on richer output families such as Gaussian mixtures.
This analysis is based on the following table with all the real datasets we used (: mean predicted correlation, : realised correlation, : gain in nats per cell):
| Dataset | Cond | ||||
|---|---|---|---|---|---|
| MIMIC-IV | A | 0.371 | 0.394 | 0.313 | 0.708 |
| B | 0.077 | 0.052 | 0.054 | 0.113 | |
| C | 0.090 | 0.062 | 0.056 | 0.258 | |
| D | 0.003 | 0.005 | 0.003 | ||
| ETTh1 | A | 0.264 | 0.280 | 0.204 | 0.632 |
| B | 0.164 | 0.112 | 0.118 | 0.332 | |
| C | 0.128 | 0.089 | 0.082 | 0.365 | |
| D | 0.009 | 0.005 | 0.005 | 0.037 | |
| Exchange rate | A | 0.502 | 0.478 | 0.694 | 1.055 |
| B | 0.145 | 0.260 | 0.350 | ||
| C | 0.163 | 0.080 | 0.238 | ||
| D | 0.007 | 0.002 | 0.003 | ||
| PhysioNet | A | 0.093 | 0.740 | 0.414 | 1.417 |
| B | 0.038 | 0.040 | 0.082 | 0.964 | |
| C | 0.042 | 0.063 | 0.083 | 0.502 | |
| D | 0.010 | 0.005 | 0.013 |
Reply to Author's Rebuttal
The authors clearly understand why the single-pass low-rank Gaussian joint loses to AR chaining on real data. I think this is enough to keep my score. But the novelty is limited, and the work could go deeper. This limits the paper. Whatever, we thank the hard work and patient rebuttal.
Response to Reviewer ph3w (1/2)
We thank the reviewer for the careful review and the helpful framing suggestions.
Q1: Why does autoregressive chaining of the marginals do better than the joint head on real data? Is this a problem with the low-rank Gaussian family, or with how the joint NLL behaves during training?
We would like to first clarify that comparing autoregressive chaining with our joint head is not a like-for-like comparison because the true preceding values enter AR's input at each autoregressive step.
We think one of the main reasons is the Gaussian family. The low rank itself does not seem to be the limit, as higher ranks do not improve the joint NLL (Table S20). On our synthetic LMC data the target is close to Gaussian, and there the joint head scores lower NLL than AR in conditions A–C, and approximately ties with it in D (Table 1(b)). Since this is the same model with the same training, we do not see evidence that the training of the joint NLL is the issue.
On MIMIC-IV, however, we now show that most of the gap comes from two channels: FiO$_2$ and PEEP (Section 5.2., Table S13 and Figure 4), which can remain nearly constant for long periods and then suddenly change in discrete steps. With AR extraction, as it performs K (number of queried cells) forward passes instead of one, the preceding true values are added to the context one at a time, and entered as an input for the next forward pass. Therefore, it can quickly adapt to new changes that might not have been seen from the initial context set alone. In contrast, our model's likelihood is computed from the initial context in one forward pass, after which it follows the rule defined by its predicted covariance to compute the full joint NLL.
As we now have shown in the revised version (Table S13, Figure 4), when the two channels are not scored, the joint head is within about 0.08 nats per cell of AR, which is surprisingly good taking into account that AR can rerun the forward function every step. Finally, part of the original gap was due to a leak in our AR extraction, which is now fixed (see general response).
This overall relates to how AR computes the joint NLL: its joint density does not need to be Gaussian, since every forward pass it makes, the new mean and variance are non-linear functions of the initial context added with the already scored cells' true values, through the forward function. Against our joint model with one forward pass, this essentially gives different densities from the same network (see our response to Reviewer 8svM, second question).
Weakness: On the forecasting conditions, marginal accuracy is in the middle of the compared methods. I think the main contributions are the mask-conditional interface and the posterior-fidelity evaluation, and the joint head is a first attempt; the current framing makes the joint head look like the main result, which is where the evidence is weakest.
We agree that the interface and the fidelity evaluation are main contributions (Section 1), and also that our method is not uniformly the best in marginal forecasting accuracy. We also agree that the evidence for the joint head is strongest on synthetic data, while on real data it is mixed. The revised manuscript now states this more explicitly in the abstract and Section 5.2. We have also added analysis of where the joint head helps and where it loses on real data, including the effect of FiO$_2$ and PEEP (Table 2(b), Tables S11–S13 and Figure 4). We also added the runtime and memory comparison against AR extraction (Table S21).