- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes BELIEF, which uses the observed distribution under the final result condition and Bayesian update to learn the changing path of the probability of the final result in the event stream. Experiments show that it can more accurately recover the probability path than the baseline on synthetic data and some real tasks, and maintains competitive final prediction performance. However, it does not fully outperform other methods in final prediction, and the reference path in real data is not equivalent to the true posterior.
Critical strengths
The method performed well and achieved competitive results in multiple datasets. Although the method proposed in the paper is quite simple, its performance on multiple datasets is excellent, which proves the effectiveness of the method.
Critical weaknesses
I don't see much practical value in this topic. For events that change fast in the duration, the predictions fall to overfit, since the predictions can not be made by incomplete datas. For example events sharing the same prefix that may have different results in the end.
The method relies on a known set of outcomes. The paper does not discuss how to handle new outcomes or candidate sets that change at test time.
The efficiency evaluation is incomplete. The reported timing results cover only binary synthetic data. It remains unclear how the cost grows with the number of candidates or compares with direct predictors.
The method can not be used in open world.
Questions for authors
For cases that sharing same prefix but leading to different results, did they share the same label during training? How to handle this conflict? In other words, the gt labels are made because of the full context, why should an incomplete prefix predict the same labels? This is my main concern of this paper.
How do inference time and memory usage change as the number of candidate outcomes increases, compared with PREFIX, TERMINAL, and other direct predictors?
10 comments
Response to Reviewer Tu4N (2/2)
W3 / W5 — New outcomes and open-world use
The method relies on a known set of outcomes. The paper does not discuss how to handle new outcomes or candidate sets that change at test time. The method can not be used in open world.
In general classification tasks, the candidate outcomes are defined by the task. In BELIEF, once labeled trajectories of a new outcome are available, the outcome can be added to the model. Since each candidate enters the evidence model through an outcome embedding, BELIEF only needs to train the new embedding and the shared weight branch while keeping the encoder fixed (about 6% of the parameters), whereas PREFIX and TERMINAL update all parameters. We withheld one class in the synthetic four-class settings and added it later (Appendix D.7, Table 10):
| Linear | Nonlinear | |||
|---|---|---|---|---|
| Path KL↓ | Old-class F1 change↑ | Path KL↓ | Old-class F1 change↑ | |
| BELIEF | 0.0515 | −0.0361 | 0.0394 | −0.0256 |
| PREFIX | 0.4402 | −0.1159 | 0.1447 | −0.0792 |
| TERMINAL | 0.5870 | −0.1154 | 0.5037 | −0.0804 |
After the new outcome is added, BELIEF has much lower path KL and less forgetting of old classes. For outcomes not yet labeled, BELIEF admits a natural extension: its posterior-predictive likelihood (Appendices B.4 and D.7) can flag trajectories whose evidence is poorly explained by all known outcomes, and the corresponding outcomes can then be added as new candidates once labeled, so that the candidate set grows as new outcomes appear.
W4 / Q2 — Cost versus number of candidates
The efficiency evaluation is incomplete. The reported timing results cover only binary synthetic data. It remains unclear how the cost grows with the number of candidates or compares with direct predictors. How do inference time and memory usage change as the number of candidate outcomes increases, compared with PREFIX, TERMINAL, and other direct predictors?
We have added a comprehensive efficiency evaluation (Section 4.7, Fig. 4 and Appendix D.5), measuring complete-path inference across different numbers of candidate outcomes and events against PREFIX and TERMINAL on the same backbone.
The cost does not grow linearly with the number of candidates, because all candidates are evaluated in one batched operation. The history encoding and the mixture components are shared by all candidates and computed once per event. Only the mixture weights depend on the candidate, so stacking all candidate embeddings into a matrix yields every candidate's likelihood in a single matrix operation, including on the grid of values used for continuous outcomes. With 200 events on one RTX 3090:
| Candidates | Latency (ms)↓ | Peak memory (MiB)↓ | ||
|---|---|---|---|---|
| BELIEF | PREFIX | BELIEF | PREFIX | |
| 2 | 0.810 | 0.569 | 16.42 | 15.27 |
| 16 | 0.783 | 0.572 | 16.44 | 15.28 |
| 64 | 0.784 | 0.582 | 17.65 | 15.30 |
Compared with direct predictors, the extra cost is small. TERMINAL has essentially the same cost as PREFIX.
Additional response
In addition, we respond to the following remark in your summary.
However, it does not fully outperform other methods in final prediction, and the reference path in real data is not equivalent to the true posterior.
Final prediction. Our main task is to learn belief paths, that is, how the probability of the eventual outcome should evolve as evidence arrives. The downstream experiments are therefore designed to verify that this objective does not come at the cost of final prediction. BELIEF achieves the best results or stays close to the best method on these tasks (Table 3), confirming that learning belief paths does not noticeably affect downstream performance.
Reference paths. We acknowledge that the true posterior is rarely observable in real settings, and this is what motivates the task. BELIEF provides a theoretically grounded way to recover it from evidence and outcomes alone (Section 3.5). On real data, we therefore compare with reference paths, which represent the beliefs of professional forecasters and markets about the same outcome. BELIEF is not trained on them, so its close agreement with them shows that it learns comparable beliefs directly from raw evidence.
In response to W4/Q2, could the authors please clarify the following observations:
-
As the number of candidates increases, the latency of BELIEF decreases, whereas that of PREFIX increases. This appears counterintuitive, since conventional prediction methods should be largely insensitive to the number of candidates, while BELIEF is expected to involve substantially more computation. Could the authors explain the underlying reasons for this behavior?
-
Why does the peak memory consumption of PREFIX increase as the number of candidates grows?
-
Why was TERMINAL not included in the comparison? If possible, could the authors provide the corresponding results?
Response to Reviewer Tu4N (1/2)
Thank you very much for your careful review and for your many thoughtful and insightful comments. Your main question on shared prefixes concerns the core design of BELIEF, so we first clarify this point and then respond to each comment.
Q1 — Shared prefixes and the training label
We would like to clarify that BELIEF does not train an incomplete prefix to predict the final label. Instead, it learns the distribution of observations conditioned on each candidate outcome:
Training. At every step, BELIEF takes the observation history, the time and variable of the next observation, and the realized outcome as inputs, and predicts the distribution of the next observed value. It is trained by maximizing the likelihood of the value actually observed (Eq. 3).
Inference. The outcome is unknown. For each new observation, BELIEF evaluates the learned distribution at the observed value under every candidate outcome and updates the posterior over outcomes by Bayes' rule (Eq. 4), rather than predicting the final label directly.
Hence, for two trajectories that share a prefix but end in A and B, one contributes to the observation distribution under A and the other to that under B. These are distributions under different conditions, so the two trajectories never provide conflicting targets. If the prefix observations are equally likely under A and B, their likelihood ratio is close to one and the posterior stays near the prior. Identical prefixes receive identical beliefs until later evidence separates them (Fig. 3C).
W1 — Practical value
In many applications, the probability of the eventual outcome must be updated in real time while the process is only partially observed, and this evolving probability is the belief path that BELIEF learns. In intensive care, for example, a patient's risk of in-hospital mortality is updated as vital signs and laboratory results arrive, supporting early warning (P12, MIMIC-III). In futures markets, the probability that a price reaches a given level at expiry is revised as supply and demand data are released, guiding hedging (Steel-RB).
Such evolving probabilities are in fact common in practice, although they are usually produced by mechanistic models or empirical formulas, such as precipitation probability forecasts, option-implied probabilities and outcome probabilities in sports. Our real-data reference paths (Weather, Steel-RB, Soccer) are of exactly this kind. These applications need an accurate probability at every moment, not only at the end.
W2 — Overfitting on incomplete prefixes
Training incomplete prefixes directly against the realized final label may lead to unreliable intermediate predictions when the available evidence is insufficient to distinguish outcomes. As explained in Q1, BELIEF never uses the final label as a target for an incomplete prefix: an uninformative observation has a likelihood ratio near one and leaves the belief almost unchanged, so the belief moves only as distinguishing evidence arrives.
We also evaluated BELIEF on fast-changing events, using all 1,306 qualifying binary synthetic test trajectories from the original generator whose exact posterior reverses rapidly (two opposite moves, each within six hours, Appendix D.6). BELIEF tracks these reversals much more closely than all baselines: