acceptodds
Under review as a conference paper at ICLR 2027

Meta-Causal Predictive States: A World Model for High-Dimensional Event Sequences

Abstract

While current temporal foundation models transfer across domains through a shared numerical observation space, high-cardinality discrete event streams, such as those in cloud systems or clinical care, describe system dynamics through symbols whose identities are local to each domain. This limits direct vocabulary sharing even when different systems exhibit similar predictive structures. To address this constraint, we propose a world model for high-dimensional discrete event sequences organized around meta-causal predictive states, which group equivalence classes of histories whose conditional future distributions coincide up to event relabeling. Our main hypothesis is that transferable event dynamics can be expressed through a shared predictive vocabulary, while context binds this vocabulary to local event identities. We realize this through a two-stage framework: (i) DES-JEPA, which learns discrete event representations from causal occurrence statistics using predictive teacher-student training, and (ii) DES-PFN, which learns in-context dynamics over the frozen representations and decodes event types, enabling autoregressive forecasting without target-task parameter updates. Pretrained on a mixture of six synthetic process families and evaluated on six diverse real-world domains, our analysis demonstrates that DES-PFN is competitive on low-cardinality datasets and, to our knowledge, the first method to successfully scale zero-shot forecasting to hundreds of event types.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
foundation models, event sequence, forecasting, jepa, discrete event sequences, time series, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XuHROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a issue that how to learn a transferable, discrete predictive representation across different stochastic processes, where the alphabets of event types are not aligned—and apply it to zero-shot or in-context next-event prediction. To address this constraint, the authors propose a world model for high-dimensional discrete event sequences organized around meta-causal predictive states, which groups equivalence classes of histories whose conditional future distributions coincide up to event relabeling. Experimental results demonstrate satisfactory performance.

Critical strengths
  1. The problem formulation is both significant and novel. Making predictions across processes where event alphabets are misaligned and no gradient updates in the target domain are required, which is also a core challenge for foundation models of event sequences.
  2. This paper draws a clear distinction between causal states within a process and meta-predictive states across processes, offering an compelling theoretical perspective. This definition provides an explanation for why two processes with entirely distinct sets of observable symbols may nonetheless share the same predictive role.
  3. The combination of methods is engineeringly sound. DES-JEPA employs JEPA/VQ to learn a discrete codebook, while DES-PFN utilizes PFN for context inference. The prototype-based decoding then dynamically reconstructs the distribution of event types, resulting in a self-consistent overall workflow.
Critical weaknesses
  1. There remains a gap between theoretical assumptions and practical implementation. Hypothesis 4.1 posits that the MCPS is finite and can be quantized using KK symbols. However, the paper provides neither a proof nor an upper bound on the approximation error. In practice, DES-JEPA quantizes the "nearest-neighbor codeword of the next-event teacher embedding" rather than the full future distribution Pm(F∣H)P_m(F|H). Consequently, there is currently only indirect evidence and a lack of quantitative verification to support the claim that the learned codebook truly corresponds to the MCPS.
  2. The definition of "admissible isomorphism" in Definition 2 is ambiguously constrained and inconsistent with the proposed methodology. The definition requires a bijection σ:Xm→Xm′\sigma: X_m \to X_m', which implies that only processes with alphabets of the same cardinality can be directly compared. However, DES-JEPA clearly utilizes shared codebooks across synthetic priors where ∣Xm∣∈[3,500]|X_m| \in [3, 500]—or even up to 1024. If different bases are permitted, the definition requires extension. If they are not, a gap emerges between theory and method.
  3. The term "zero-shot" is open to debate, as the decoding process relies on the original labels provided in the context. During inference, DES-PFN requires a context sequence of 128 items, and its prototype-based decoding mechanism necessitates that an event type be present within that context. If a specific event type does not appear in the context, no corresponding prototype exists, making it impossible to predict.
Questions for authors
  1. Are there any theoretical proofs or, at the very least, empirical bounds on the approximation error? How was K=768K=768 determined? Is it possible to provide a quantitative error bound between qK∘ϵMq_K \circ \epsilon_M and the true future distribution?
  2. How to quantitatively verify that the learned discrete codebooks truly correspond to Minimal Sufficient Statistics for Future Prediction (MCPS), rather than merely compressing the embeddings of the next event? For instance, could this be evaluated using the distance between predicted and actual future distributions (on held-out data), tests for predictive equivalence, or the mutual information with a causal state partition?
  3. In Definition 2, are arbitrary bijections permitted? If two processes have alphabets of different cardinalities, how is an admissible isomorphism defined? Does this conflict with the practice in DES-JEPA of sharing codebooks across alphabets of different cardinalities?
  4. This paper refers to the approach of "performing no gradient updates while providing 128 tokens of context" as "zero-shot." How does this differ from few-shot or in-context learning? Furthermore, given that it underperforms FIM-PP on low-cardinality tasks, how can the claim that it is competitive be justified?
  5. In the context of temporal event prediction, forecasting the timing of the next event is crucial. However, the paper only reports the accuracy of the predicted event type. How does the time-prediction head of the proposed method perform? Could the author also provide results for time prediction performance?
3 comments
AuthorsOpenReview ↗

Answer to R3 1/2

We thank R3 for their complete feedback. We regrouped multiple questions that appear closely related.

W1 & Q1: Relationship between the codebook and MCPS, choice of K

We agree that the paper gives no approximation bound between the learned representation and the true MCPS. A bound for infinite families of DGPs would likely need strong assumptions (high Shannon redundancy, low-memory processes), and we think it deserves its own paper. Hypothesis 4.1 is a working assumption: predictive structure recurs across DGPs, admits a finite-vocabulary approximation, and can be learned self-supervised.

K=768 was fixed a priori, not derived. The post-hoc rate-distortion analysis (Fig. 3) is an empirical capacity check: distortion is nearly flat from 768 down to roughly 384-512 used codes (6.19 -> 6.40) and rises sharply below 256. This shows redundancy in teacher-embedding space, not that the codes equal MCPS. We will state this explicitly. We will also add a direct test on synthetic processes with known causal states (e.g. Markov chain families), measuring the mutual information between assigned codes.

W2 & Q3: Definition 2 and alphabets of different cardinalities

The reviewer is right that Definition 2 as written is too restrictive. We wrote the Definition 2 as primarily to facilitate comprehension for the reader with exemple 1. A bijection $\sigma: X_m -> X_m' $ exists only if $|X_m| = |X_m'| $. Read literally, SMS_M would split into disjoint blocks by cardinality, we will fix it.

Only the symbols that the predictive law actually charges matter for comparing two futures. We will therefore require σ\sigma to be a bijection between effective supports:

$ A_m(h) = { x \in X_m : P_m(X_s = x $, for some $ s ≥ t | H = h) > 0 }. $

Then (m,h)∼MPS(m′,h′)(m, h) ∼_MPS (m', h') iff there is a bijection σ:Am(h)→Am′(h′)\sigma: A_m(h) → A_m'(h') with \sigma#P_m(F|h) = P_m'(F|h'). This has four consequences:

(1) The login/logout example is unchanged. (2) A 3-cycle embedded in an alphabet of 1024 symbols and a 3-cycle in an alphabet of 5 become equivalent. Unused symbols are irrelevant. (3) Processes that genuinely use different numbers of symbols (iid uniform over 3 vs. 10 symbols) remain inequivalent, which is correct since their predictive structure differs. (4) The relation stays an equivalence relation, so the quotient S_M is well defined.

This matches the implementation. Occurrence statistics are computed only from observed symbols. The prototype decoder (Eq. 10) is built only from identifiers present in the context, so it also operates on an effective support. This is consistent with the empirical results across vocabularies from 3 to 700 types (Table 1, Fig. 2). Those experiments do not prove the revised equivalence holds in general, and we will say so.

W3 & Q4: "Zero-shot"

We use "zero-shot" for no gradient update and no training on any real process, with the target process seen only through a 128-sequence context. This is the in-context regime of PFNs and FIM-PP (ICLR 2026). We agree that decoding needs raw identifiers from the context. An event type absent from the context gets no prototype and cannot be predicted. We will state this as a limitation and report how often query targets are unseen in the context for each dataset. "In-context/amortized zero-shot" may be a more precise term, and we will adopt it. On competitiveness: DES-PFN beats FIM-PP on Retweet and Amazon (next-event), and on Retweet at both rollout horizons. FIM-PP is stronger on Taobao and HDFS. We will tone down the claim to "competitive at low cardinality, and the only method that runs at 500-700 types.

Q2: Verifying that the codebook corresponds to MCPS.

Thank you for this question. Our claim should be made precise. The representation R should satisfy (i) approximate sufficiency: the future law depends on H only through R(H), and (ii) relabeling invariance. We do not claim minimality or exact recovery of MCPS, as we believe this is unrealistic for infinite DGPs across different families. Note also that DES-JEPA quantizes the teacher embedding of the next event rather than the future law, so the rate-distortion analysis (Fig. 3) shows redundancy in embedding space. We will revise the text accordingly.

AuthorsOpenReview ↗

Answer to R3 (2/2)

Q4: Verifying MCPS approximation on Markov chain

We tested this directly on pretraining families with known causal states. We generate Markov and burst-renewal processes from the pretraining families using during pretraining, apply a fresh random relabeling to every held-out process instance, and compare DES-JEPA codes with the true predictive-state partition. Within a single process (4 predictive states), codes recover the state with 85% (Markov) and 80% (burst) held-out accuracy, versus 50% and 56% without state information. This leaves 0.41 of 1.68 bits of state uncertainty (Markov) and reaches 93–97% of a supervised linear-probe ceiling, despite unseen labels. The code-only next-event law halves the TV error relative to no state (0.11 vs. 0.25), but is not sufficient (oracle 0.005). We therefore state the claim as: codes are relabeling-invariant, approximately state-identifying summaries within a process, not sufficient MCPS.

Q5: Time prediction

You are right, we will add time-head metrics (log-bin accuracy and MAE) in addition to the OTD (which already depends on predicted times).This will show more direct evidence that the time channel is used predictively.

I appreciate the authors' response. I have carefully read the rebuttal. The responses address my main concerns. Therefore, I decide to keep the score.

Reviewer_tSZdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a shared discrete representation for event sequences across domains with different event vocabularies. It groups histories with equivalent future distributions up to event relabeling. DES-JEPA learns this representation, while DES-PFN uses it for in-context forecasting. Zero-shot experiments show competitive performance with small event vocabularies.

Critical strengths
  1. The paper addresses a meaningful problem of zero-shot forecasting across unaligned event vocabularies.
  2. The scheme is clear and elegant. The model is only pretrained on synthetic priors, and no parameter updates are needed during inference.
Critical weaknesses
  1. The high cardinality setting does not contain any baselines. In Table 1, on Vehicle datasets, DES-PFN is the only reported method. The scaling claim cannot be assessed.
  2. The theory is disconnected from the experiments. No experiment tests whether the learned codebook approximates MCPS (Hypothesis 4.1). This could actually be checked on synthetic processes with known causal states.
  3. Minor: There are also some typos in Table 1, such as "Raobao" and "Rmazon".
Questions for authors
  1. From my perspective, the theory part serves mainly as motivation. It does not determine the method’s design, nor is it validated by the experiments. Can the author clarify this more?
  2. What happens if the codebook is removed and DES-PFN uses the statistical embeddings directly?
3 comments
AuthorsOpenReview ↗

Answer to R2 1/2

We thank R2 for their substantial reviewing effort and address the concerns bellow.

W1 - Baselines and FIM-PP

FIM-PP supports at most 22 event types, while the fully trained neural baselines exceeded the 600-second per-task budget († in Table 1). The Vehicle results therefore demonstrate feasibility at 500–700 event types under this protocol, rather than superior accuracy against unrestricted baselines. We will qualify our claims accordingly and agree that inexpensive baselines and larger-budget comparisons would strengthen the evaluation. We can however extrapolate that using a context of 128 sequences for 500-700 event types will not suffices to estimate empirical frequencies. We will add simple baseline that use empirical frequencies for event apparition in the coming days.

W2 - Codebook Analysis

The codebook analysis on the final page provides empirical evidence consistent with this hypothesis: across sampled DGPs from several process families, the learned codebook is reused (supporting the learned predictive structures across DGPs), and post-hoc restriction to a smaller set of frequently used codes produces limited quantization distortion until the codebook is reduced substantially (supporting the number of codebook size used).

Since our DGP families got assigned random parameters during pretraining and evaluation, this directly suggests that the representation can capture recurring predictive patterns across a broad range of parameterized processes. However, these experiments sample finitely many tasks and do not prove formally that the representation quantizes the MCPS for all possible DGPs. We will revise the claims to make this distinction explicit.

Regarding the approximation of the MCPS via the learned codes, we tested this directly on pretraining families with known causal states. We generate Markov and burst-renewal processes from the pretraining families using during pretraining, apply a fresh random relabeling to every held-out process instance, and compare DES-JEPA codes with the true predictive-state partition. Within a single process (4 predictive states), codes recover the state with 85% (Markov) and 80% (burst) held-out accuracy, versus 50% and 56% without state information. This leaves 0.41 of 1.68 bits of state uncertainty (Markov) and reaches 93–97% of a supervised linear-probe ceiling, despite unseen labels. The code-only next-event law halves the TV error relative to no state (0.11 vs. 0.25), but is not sufficient (oracle 0.005). We therefore state the claim as: codes are relabeling-invariant, approximately state-identifying summaries within a process, not sufficient MCPS.

W3 - Typos

Thank you for pointing these out, we will correct them.

Q1 - Theory

That is correct, the theory as stated only motivates the paper currently. A deeper dive into the theory is planned as future work, since we think, it is more suitable as a stand alone paper rather than burying it into the Appendix. In this paper we focus on the method and the intuition behind the meta-causal states and the predictive patterns shared across stochastic processes.

Q2 - Statistical Embeddings vs DES-JEPA

This is a relevant remark. To be totally honest we couldn't run this additional ablation. However, we can argue that by replacing DES-JEPA with a statistical embedding directly (1) will increase the number of parameters to train during DES-PFN and reduce practicability as we will need to learn a representation of the DGPs for each zero-shot next-event prediction tasks. (2) It will kill the current paradigm, as we will need to retrain the embedding for each DES-PFN training and (3) also limit the interpretability of the learned statistical embedding vs the codebook, which could be used in unsupervised learning (think about sequence classification, anomaly detection) because you would need to retrain DES-PFN instead of doing a simple forward pass of DES-JEPA.

Thank you for the complete review and hope that R2 will consider increasing its scores after resolving the concerns.

I thank the authors for the detailed response and additional experiments.

  1. W1: I understand the resource constraint in the high-cardinality setting, however, the empirical-frequency baseline is still missing.
  2. W2: The newly added predictive-state results are supportive evidence of approximate state identification, not evidence of the codebook approximating MCPS.
  3. Q2: Thank the authors for the honesty, and I understand the difficulty of running ablation. The argument made by the authors makes sense, but the empirical validation is missing.

Overall, the rebuttal clarifies part of the concerns and questions, but lots of the experiments are missing. I will keep my original rating.

AuthorsOpenReview ↗

R2 Second Answer

Dear R2,

Thank you for answering.

For W1 and the empirical baselines, instead of copy pasting, we refer you to Q3/Q4 of R1 which expose three new empirical baselines that you mentioned as well as a new decoding strategy for DES-PFN which does not touch on the core method and improve significantly next-event prediction over both FIM-PP and empirical baselines. This places DES-PFN above FIM-PP on every dataset FIM-PP can run.

Critically, the remarks of R1 and R2 has helped us to improve the decoding strategy while being transparent with additional empirical baselines. We believe that now the evaluation and soundness is largely improved.

We hope that this will clarify some of your concerns, For W2 and Q2, we understand. We will add the codebook and statistical embedding ablation into the final version. Thank you !

Reviewer_cKqWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies zero-shot forecasting of discrete event sequences across domains with unrelated event vocabularies. It introduces meta-causal predictive states, which group histories whose conditional future distributions agree up to an alphabet relabeling. DES-JEPA learns a shared discrete codebook from occurrence statistics, and DES-PFN uses reference sequences and a context-built prototype decoder to predict local event labels and times. Both stages are pretrained exclusively on six synthetic process families. Evaluation covers next-event prediction and autoregressive rollouts on six real datasets with 3 to 700 event types, alongside codebook analyses.

Critical strengths
  1. The paper addresses an important transfer problem. Occurrence-based inputs and a context-built output head offer a promising way to handle changing event vocabularies without learning a new global event embedding or classifier for each dataset. This addresses a meaningful limitation of existing zero-shot event-sequence models, particularly the reported vocabulary limit of FIM-PP.

  2. Synthetic-only pretraining produces encouraging transfer results. DES-PFN outperforms FIM-PP on Retweet and Amazon in both next-event tables without target-task gradient updates, and achieves the best reported Retweet rollout results at both horizons. These findings support the use of synthetic process mixtures for transfer to real event sequences.

Critical weaknesses
  1. My main concern is how the model preserves the connection between event labels in the query and those in the context. The occurrence statistics are invariant to relabeling within a sequence, and DES-PFN receives only the resulting codes and timing information. Raw identifiers are used to construct the context prototypes in Eq. (10), but I could not find a mechanism that connects a query identifier to the same identifier in the context. As written, relabeling only the query while keeping the context fixed appears to leave the predicted distribution over raw output labels unchanged. Consider a deterministic cycle from A to B to C to A, with equal time gaps and random starting phases. Query prefixes (A, B), (B, C), and (C, A) have identical occurrence statistics and positional features, but their correct next labels are C, A, and B. The described model appears unable to distinguish them. There may be an implementation detail that resolves this, but it is essential to the method and needs to be explained and demonstrated. The codebook analyses do not answer this question because code utilization and quantization error do not establish that the representation retains the information needed to predict the correct event label.

  2. The experiments provide insufficient support for the claim of state-of-the-art performance at hundreds of event types. No competing method reports a Vehicle result, and simple comparisons such as majority-class, repeat-last, and smoothed transition predictors are missing. The Vehicle rollout results also need a reference point. Eq. (12) uses 1/N event masses, whereas Table 5 appears to report summed costs. Assuming summed costs, deleting all N predictions and inserting all N targets costs 2N times the mean insertion/deletion penalty. The penalty grid in Appendix C.1.1 is {0.05, 0.5, 1, 1.5, 2, 3, 4}, whose mean is approximately 1.7214. This gives reference costs of 17.21 at N = 5 and 68.86 at N = 20. The four Vehicle scores are only about 0.2% to 3.0% lower than these reference costs. The nearly flat per-event curve could therefore reflect metric saturation. The normalization needs clarification, and simple rollout baselines are needed to judge whether these scores represent useful forecasting. Finally, Tables 1 and 3 specify the same evaluation protocol but report different results. HDFS DES-PFN accuracy changes from 0.161 to 0.222, FIM-PP’s HDFS result disappears, and the StackOverflow point-estimate ranking reverses. Appendix A.4 describes a 10-layer model with width 512 and 64 inducing vectors, while B.2.2 describes a 5-layer model with width 384 and 16 inducing vectors. These discrepancies leave me uncertain about which results and model configuration support the paper’s conclusions.

Questions for authors
  1. Where does the implementation connect a raw query identifier to occurrences of that identifier in the context? Are occurrence statistics computed independently per sequence or jointly across the task? Please explain how the model distinguishes the random-phase cycle prefixes described above and provide a controlled test of this behavior.

  2. Which of Tables 1 and 3 is authoritative, and what explains their differences? Which architecture generated the results: A.4’s 10-layer model with width 512 or B.2.2’s 5-layer model with width 384? Were the unusually low StackOverflow few-shot RMTPP/NHP accuracies checked for label alignment and training problems?

  3. What transport constraints and normalization does the implemented OTD use? Please reconcile Eq. (12) with Table 5 and report the delete-all/insert-all reference alongside the scores. How do the Vehicle rollouts compare with repeat-last and context-marginal sampling under this metric?

  4. How does Vehicle next-event accuracy compare with majority-class and smoothed transition predictors fitted to the same context? Please report the hardware, measured baseline fitting times, DES-PFN inference time, and the specific reason for each missing baseline result.

  5. What does the “bidirectional teacher” in A.3 attend over? If it processes multiple sequence positions, how is future information excluded from query representations during evaluation? Please specify its input and attention mask, and clarify whether query tokenization is performed separately for each observed prefix.

2 comments
AuthorsOpenReview ↗

Answer to R1 - 1/2

Dear R1, thank you for the great feedback. We believe it has substantially improve the paper. We now answer your concerns bellow.

Q1/W1 Query$\leftrightarrow$context label linkage (the A$\to$B$\to$C cycle)

Occurrence statistics are computed independently per sequence, not jointly across the task. Each context row and the query are tokenized separately; at position tt the embedding of v=xtv{=}x_t uses only that sequence's own causal history (i≤ti\le t): its running count nt(v)=∑i≤t1[xi=v]n_t(v)=\sum_{i\le t}\mathbb 1[x_i{=}v], frequency nt(v)/tn_t(v)/t, local frequency, time and step distance since vv's previous occurrence, and the mean and dispersion of vv's own gaps. Raw identifiers act only as grouping keys. These features are therefore invariant to a consistent relabeling of one sequence which is the purpose of building a common vocabulary of DES-JEPA.

Taking your example, the query enters only through its codes and times; the prototype decoder (Eq.~10) uses context identifiers to pool prototypes but receives no query identifier. We verified this directly: applying a random permutation to the query's identifiers while keeping the context fixed leaves the decoded distribution bit-identical (maximum absolute difference 00) on the cycle below. The query is linked to context labels by dynamical role: its hidden state is matched to the context prototype whose statistics and timing play the same role, not to the same identifier. This exemple in particular motivated the improved decoding of DES-PFN detailed in the second part of this rebuttal answer.

Q2 Which of Tables 1/3 is authoritative; architecture discrepancy

Thank you for catching these. both are our errors in the draft, not two different systems. Table 1 (the real-world next-event table) is authoritative. The numbers that differ in the draft's second table come from an earlier, superseded run. We will delete the stale table and report a single set of numbers from one run throughout. The released architecture and evaluated checkpoint is the smaller 5-layer, width-384, 16-inducing-point model (App. B.2.2); App.~A.4's 10-layer / width-512 / 64-inducing text is stale copy from an earlier configuration and will be corrected to match the shipped model. We also re-checked the few-shot RMTPP/NHP StackOverflow numbers: they are genuinely low because those nets are refit from scratch on only the 128-sequence context (not the full split), which is insufficient for 22-way discrimination in a few epochs. We label this regime explicitly and it is the intended, honest ``gradient-trained-from-scratch in the low-data regime'' comparison we stated.

Q3 OTD normalization and a delete-all/insert-all reference

R1 is right to ask for the normalization and a saturation reference. (i) We will state the exact transport constraints and whether Table5 reports per-event (1/N1/N) or summed cost, reconciling it with Eq.(12). (ii) We now report, on the highest-cardinality dataset (Vehicle, ∣X∣=500|\mathcal{X}|{=}500), the OTD of \despfn\ against the two baselines R1 asked for repeat-last (persistence) and context-marginal sampling together with R1's trivial delete-all/insert-all saturation reference, Table 2).

MethodOTD N=5N{=}5OTD N=20N{=}20FLOPs / rolloutwall-clock
Context marginal17.2168.74<105<10^5<1<1 ms
Persistence (repeat-last)16.8667.77<105<10^5<1<1 ms
Smoothed order-116.8868.64<105<10^5<1<1 ms
DES-PFN (30.1M params)16.2768.380.73–2.34 TFLOP1–3 s
Delete/insert reference17.2168.86——

Two points. First, the metric is not saturated: context-marginal sampling merely matches the trivial saturation reference (17.2117.21 / 68.7468.74 vs 17.2117.21 / 68.8668.86), whereas \despfn\ and repeat-last both clear it. Second, we report the ranking honestly: \despfn\ attains the best short-horizon OTD (N=5N{=}5, 16.2716.27) and the second-best long-horizon OTD (N=20N{=}20, behind repeat-last). This win comes at a compute cost of 0.730.73--2.342.34 TFLOPs and 30.130.1M parameters per rollout, versus the sub-10510^5-FLOP, zero- parameter lookup baselines, which we state plainly. The takeaway is that amortized in-context prediction overtakes empirical-frequency baselines precisely in the high-cardinality regime where the latter cannot be estimated from limited context, at a cost premium that is appropriate to flag.

We answer bellow to Q4 and provide a significant decoding improvement that do not touch to our checkpoints or model configuration thanks to R1.

AuthorsOpenReview ↗

Answer to R1 - 2/2

Q4 : Why the baselines looked strong and How we improved our decoding without changing the model

Q4.1 A lot of verbatim repeated event

A small caveat to the listed empirical baseline: the real-world datasets that we used have a lot of repeated next events (A A A A A B B B B), therefore using a simple repeat last event baselines will dominate simply without exploiting any long-memory pattern useful for other DGPs.

Q4.2 Corrected decoder surpasses FIM-PP and empirical baselines on every dataset

Nevertheless, motivated by the rebuttal previous answers, we diagnosed why the baselines were competitive and found the cause in our decoder, not the model. The submitted decoder commits to the most likely code, z^=arg⁡max⁡zpϕ(z∣h,C)\hat z = \arg\max_z p_\phi(z \mid h, \mathcal{C}), then looks up a raw event. Because the code-to-event map is many-to-one, this discards the posterior. The correct operation is to marginalize,

$ \hat P(x_{t+1} = v) = \sum_z p_\phi(z \mid h, \mathcal{C}), \hat P(v \mid z, \mathcal{C}), $

interpolated with two explicit identity channels (the context's order-1 table applied to the query's last raw event, and a recency distribution over the query's own raw history) plus the context marginal, under a per-context likelihood fit to calibrate the different distributions. Importantly, this is a decode change only, on the same frozen DES-JEPA and DES-PFN checkpoint: nothing is retrained and both core contributions are untouched. The only new quantities are a few scalars fit on each context's own held-out rows, which is a negligible cost.

About MCPS intuition. If a code is a learned meta-causal predictive state, the Bayes-optimal next-event law is the mixed-state predictor of computational mechanics, P(x∣h)=∑sP(s∣h) P(x∣s)P(x \mid h) = \sum_s P(s \mid h)\, P(x \mid s), which averages over the posterior belief across causal states instead of committing to one (Crutchfield 1994; Shalizi & Crutchfield 2001), providing a plausible explanation.

Table D. Decode ablation on the same frozen checkpoint (top-1 accuracy, mean of 3 seeds, nctx=128n_\text{ctx} = 128, nquery=20n_\text{query} = 20). λcode\lambda_\text{code} = weight on the model's code channel. Controlled harness; absolute values differ slightly from Table 1 in the paper.

Dataset∣X∣\lvert\mathcal{X}\rvertsubmittedcorrected decodingMarginalPersist.Transitionλcode\lambda_\text{code}
retweet30.5770.5870.5820.5020.5540.20
stackoverflow220.4470.4650.4470.3220.4620.15
taobao170.3660.5860.4020.5760.5750.14
amazon160.2300.3090.2300.1970.2340.18
Vehicle5000.1660.2140.1660.0960.2040.14

Where the model contributes (λcode≈0.14\lambda_\text{code} \approx 0.14–0.200.20), the correction meets or beats the baselines, most clearly on taobao and amazon, we will state that plainly.

We compared this new decoding strategy with the result of FIM-PP (top-1 accuracy). FIM-PP values are the submission's Table 1; FIM-PP is capped at 22 marks.*

Dataset∣X∣\lvert\mathcal{X}\rvertFIM-PPDES-PFN
retweet30.4620.587
stackoverflow220.4410.465
taobao170.5160.586
amazon160.2070.309
HDFS140.2660.408
Vehicle500n/a0.214

With this new decoding DES-PFN surpasses FIM-PP on every dataset FIM-PP can run, at low and high cardinality, and extends beyond its 22-mark cap. Against fully-trained specialists it is competitive but not dominant (it trails NHP on retweet, 0.587 vs. 0.591, and amazon, 0.309 vs. 0.319).

Thanks to R1, we will incorporate this new decoding strategy and update the results accordingly in the main paper

Comments on the paper

1 comment

We thank the reviewers for their time, careful reading, and constructive feedback. We are encouraged that the reviewers recognized the importance of forecasting across unaligned event vocabularies (XuHR, tSZd, cKqW), the promise of the proposed meta-causal predictive state framework (XuHR), the clarity and elegance of the overall approach (tSZd), and the encouraging transfer performance achieved through synthetic-only pretraining (cKqW). We also appreciate the recognition of the engineering soundness of DES-JEPA and DES-PFN (XuHR). We address the comments below and will incorporate your feedback accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.