- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a issue that how to learn a transferable, discrete predictive representation across different stochastic processes, where the alphabets of event types are not aligned—and apply it to zero-shot or in-context next-event prediction. To address this constraint, the authors propose a world model for high-dimensional discrete event sequences organized around meta-causal predictive states, which groups equivalence classes of histories whose conditional future distributions coincide up to event relabeling. Experimental results demonstrate satisfactory performance.
Critical strengths
- The problem formulation is both significant and novel. Making predictions across processes where event alphabets are misaligned and no gradient updates in the target domain are required, which is also a core challenge for foundation models of event sequences.
- This paper draws a clear distinction between causal states within a process and meta-predictive states across processes, offering an compelling theoretical perspective. This definition provides an explanation for why two processes with entirely distinct sets of observable symbols may nonetheless share the same predictive role.
- The combination of methods is engineeringly sound. DES-JEPA employs JEPA/VQ to learn a discrete codebook, while DES-PFN utilizes PFN for context inference. The prototype-based decoding then dynamically reconstructs the distribution of event types, resulting in a self-consistent overall workflow.
Critical weaknesses
- There remains a gap between theoretical assumptions and practical implementation. Hypothesis 4.1 posits that the MCPS is finite and can be quantized using symbols. However, the paper provides neither a proof nor an upper bound on the approximation error. In practice, DES-JEPA quantizes the "nearest-neighbor codeword of the next-event teacher embedding" rather than the full future distribution . Consequently, there is currently only indirect evidence and a lack of quantitative verification to support the claim that the learned codebook truly corresponds to the MCPS.
- The definition of "admissible isomorphism" in Definition 2 is ambiguously constrained and inconsistent with the proposed methodology. The definition requires a bijection , which implies that only processes with alphabets of the same cardinality can be directly compared. However, DES-JEPA clearly utilizes shared codebooks across synthetic priors where —or even up to 1024. If different bases are permitted, the definition requires extension. If they are not, a gap emerges between theory and method.
- The term "zero-shot" is open to debate, as the decoding process relies on the original labels provided in the context. During inference, DES-PFN requires a context sequence of 128 items, and its prototype-based decoding mechanism necessitates that an event type be present within that context. If a specific event type does not appear in the context, no corresponding prototype exists, making it impossible to predict.
Questions for authors
- Are there any theoretical proofs or, at the very least, empirical bounds on the approximation error? How was determined? Is it possible to provide a quantitative error bound between and the true future distribution?
- How to quantitatively verify that the learned discrete codebooks truly correspond to Minimal Sufficient Statistics for Future Prediction (MCPS), rather than merely compressing the embeddings of the next event? For instance, could this be evaluated using the distance between predicted and actual future distributions (on held-out data), tests for predictive equivalence, or the mutual information with a causal state partition?
- In Definition 2, are arbitrary bijections permitted? If two processes have alphabets of different cardinalities, how is an admissible isomorphism defined? Does this conflict with the practice in DES-JEPA of sharing codebooks across alphabets of different cardinalities?
- This paper refers to the approach of "performing no gradient updates while providing 128 tokens of context" as "zero-shot." How does this differ from few-shot or in-context learning? Furthermore, given that it underperforms FIM-PP on low-cardinality tasks, how can the claim that it is competitive be justified?
- In the context of temporal event prediction, forecasting the timing of the next event is crucial. However, the paper only reports the accuracy of the predicted event type. How does the time-prediction head of the proposed method perform? Could the author also provide results for time prediction performance?
3 comments
Answer to R3 (2/2)
Q4: Verifying MCPS approximation on Markov chain
We tested this directly on pretraining families with known causal states. We generate Markov and burst-renewal processes from the pretraining families using during pretraining, apply a fresh random relabeling to every held-out process instance, and compare DES-JEPA codes with the true predictive-state partition. Within a single process (4 predictive states), codes recover the state with 85% (Markov) and 80% (burst) held-out accuracy, versus 50% and 56% without state information. This leaves 0.41 of 1.68 bits of state uncertainty (Markov) and reaches 93–97% of a supervised linear-probe ceiling, despite unseen labels. The code-only next-event law halves the TV error relative to no state (0.11 vs. 0.25), but is not sufficient (oracle 0.005). We therefore state the claim as: codes are relabeling-invariant, approximately state-identifying summaries within a process, not sufficient MCPS.
Q5: Time prediction
You are right, we will add time-head metrics (log-bin accuracy and MAE) in addition to the OTD (which already depends on predicted times).This will show more direct evidence that the time channel is used predictively.
I appreciate the authors' response. I have carefully read the rebuttal. The responses address my main concerns. Therefore, I decide to keep the score.
Answer to R3 1/2
We thank R3 for their complete feedback. We regrouped multiple questions that appear closely related.
W1 & Q1: Relationship between the codebook and MCPS, choice of K
We agree that the paper gives no approximation bound between the learned representation and the true MCPS. A bound for infinite families of DGPs would likely need strong assumptions (high Shannon redundancy, low-memory processes), and we think it deserves its own paper. Hypothesis 4.1 is a working assumption: predictive structure recurs across DGPs, admits a finite-vocabulary approximation, and can be learned self-supervised.
K=768 was fixed a priori, not derived. The post-hoc rate-distortion analysis (Fig. 3) is an empirical capacity check: distortion is nearly flat from 768 down to roughly 384-512 used codes (6.19 -> 6.40) and rises sharply below 256. This shows redundancy in teacher-embedding space, not that the codes equal MCPS. We will state this explicitly. We will also add a direct test on synthetic processes with known causal states (e.g. Markov chain families), measuring the mutual information between assigned codes.
W2 & Q3: Definition 2 and alphabets of different cardinalities
The reviewer is right that Definition 2 as written is too restrictive. We wrote the Definition 2 as primarily to facilitate comprehension for the reader with exemple 1. A bijection $\sigma: X_m -> X_m' $ exists only if $|X_m| = |X_m'| $. Read literally, SM would split into disjoint blocks by cardinality, we will fix it.
Only the symbols that the predictive law actually charges matter for comparing two futures. We will therefore require σ to be a bijection between effective supports:
$ A_m(h) = { x \in X_m : P_m(X_s = x $, for some $ s ≥ t | H = h) > 0 }. $
Then (m,h)∼MPS(m′,h′) iff there is a bijection σ:Am(h)→Am′(h′) with \sigma#P_m(F|h) = P_m'(F|h'). This has four consequences:
(1) The login/logout example is unchanged. (2) A 3-cycle embedded in an alphabet of 1024 symbols and a 3-cycle in an alphabet of 5 become equivalent. Unused symbols are irrelevant. (3) Processes that genuinely use different numbers of symbols (iid uniform over 3 vs. 10 symbols) remain inequivalent, which is correct since their predictive structure differs. (4) The relation stays an equivalence relation, so the quotient S_M is well defined.
This matches the implementation. Occurrence statistics are computed only from observed symbols. The prototype decoder (Eq. 10) is built only from identifiers present in the context, so it also operates on an effective support. This is consistent with the empirical results across vocabularies from 3 to 700 types (Table 1, Fig. 2). Those experiments do not prove the revised equivalence holds in general, and we will say so.
W3 & Q4: "Zero-shot"
We use "zero-shot" for no gradient update and no training on any real process, with the target process seen only through a 128-sequence context. This is the in-context regime of PFNs and FIM-PP (ICLR 2026). We agree that decoding needs raw identifiers from the context. An event type absent from the context gets no prototype and cannot be predicted. We will state this as a limitation and report how often query targets are unseen in the context for each dataset. "In-context/amortized zero-shot" may be a more precise term, and we will adopt it. On competitiveness: DES-PFN beats FIM-PP on Retweet and Amazon (next-event), and on Retweet at both rollout horizons. FIM-PP is stronger on Taobao and HDFS. We will tone down the claim to "competitive at low cardinality, and the only method that runs at 500-700 types.
Q2: Verifying that the codebook corresponds to MCPS.
Thank you for this question. Our claim should be made precise. The representation R should satisfy (i) approximate sufficiency: the future law depends on H only through R(H), and (ii) relabeling invariance. We do not claim minimality or exact recovery of MCPS, as we believe this is unrealistic for infinite DGPs across different families. Note also that DES-JEPA quantizes the teacher embedding of the next event rather than the future law, so the rate-distortion analysis (Fig. 3) shows redundancy in embedding space. We will revise the text accordingly.