acceptodds
Under review as a conference paper at ICLR 2027

Predictive State Has a Scope

Abstract

A temporal model can recognize a relation without remembering its history. We study predictive state scope: where different histories are written and which states future predictions can read. Shared state pools observations but can merge histories with distinct futures, creating predictive aliasing; local states retain separate histories but receive fewer observations. We introduce Factor-Addressed Predictive State (FA-State), a jointly trained neural architecture coupling a shared prediction path with pair-addressed recurrent states. FA-State shares how history is encoded and updated across addresses while preserving where relation-specific history is written and retrieved. Bayes log-risk and random-effects analyses characterize the cost of predictive aliasing and the finite-sample trade-off between shared and local states. Across temporal graph forecasting tasks, FA-State improves prediction with multiple backbones. Holding trained weights and online bank contents fixed, deranging only the retrieval addresses systematically degrades prediction, showing that trained predictors use the correspondence between historical content and its address. Controlled streams and recurrence analyses further reveal how available history shapes the shared–local trade-off. These findings establish state scope as a design choice in temporal representation learning itself: what a model remembers depends on where it writes history and how it reads it back.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
temporal representation learning, predictive state, temporal graphs, neural memory, relational event streams

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3xeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies predictive state scope in temporal representation learning: where historical information should be stored and how it should be retrieved at prediction time. The authors propose FA-State, which maintains a separate recurrent state for each directed pair alongside a shared temporal backbone, using a shared GRU to update relational history. At prediction time, the model retrieves the historical state associated with each candidate pair and combines it with the backbone’s prediction. The paper also uses Bayes risk and random-effects analysis to discuss predictive aliasing and the trade-off between shared and local states.

Experiments cover synthetic data, temporal graph forecasting, and temporal knowledge graph prediction. FA-State improves prediction performance across multiple backbones, with particularly notable gains on tgbn-trade. Frozen retrieval experiments further show that matching historical content to the correct retrieval address matters for prediction. However, the lack of direct comparisons with the most relevant strong baselines leaves FA-State’s practical advantage over existing methods unclear.

Critical strengths
  1. The research question is meaningful. The paper examines how historical information is stored and retrieved, focusing on the predictive aliasing that shared states can introduce and the trade-off between shared and pair-specific states. Although the method itself is relatively simple, the state-scope perspective offers useful insights into temporal representation learning.

  2. The experimental results are reasonably convincing. FA-State improves performance across multiple temporal backbones and datasets, with especially large gains for TGNv2 on tgbn-trade. The authors also use synthetic data to control relational dependence and recurrence, helping clarify how the method behaves under different conditions.

  3. The ablations and analyses are fairly thorough. Beyond the standard shared-only and wrong-address controls, the authors conduct frozen retrieval experiments that change only the retrieval addresses while keeping model parameters and the memory bank fixed. These provide stronger evidence than conventional ablations that the model uses the correct correspondence between historical content and its address.

Critical weaknesses
  1. The methodological novelty is limited, and the need for persistent pair-specific memory remains unclear. FA-State essentially adds a separate GRU state for each directed pair to an existing temporal backbone and combines predictions through additive logits. Related structured or factorized memory ideas have been explored in Neural Datalog Through Time, FACTS, and other work. Transformers or cross-attention could also retrieve pair-specific history dynamically at prediction time without maintaining persistent states. Although the wrong-address and frozen retrieval experiments establish the importance of correct addressing, they do not show that FA-State’s design is better than existing approaches to history modeling.

  2. Important baseline comparisons are missing, and the evaluation protocol is not comprehensive enough. The paper mainly compares against backbones such as TGN, TGAT, and TGNv2 and their variants, without directly evaluating strong methods for the same task, such as NAVIS (ICLR 2026), even though NAVIS is discussed in Related Work. The main experiments also use occupancy-matched candidate sets, restricting prediction to previously observed relations, which may favor pair-specific memory. Although full-candidate NDCG is reported, the models are still trained with a matched objective. A fair comparison with stronger baselines under a standard full-candidate protocol is therefore needed.

  3. The theoretical analysis is only loosely connected to the actual method. Proposition 1 largely presents the standard relationship between conditional mutual information and Bayes log-risk. It applies to compressed representations in general and does not directly establish a specific advantage for FA-State. Proposition 2 uses a classical Gaussian random-effects model to discuss the shared–local trade-off, but the architecture does not implement the recurrence-dependent shrinkage emphasized by the analysis. These results help explain why retaining relational history may be useful, but do not sufficiently justify FA-State’s particular way of organizing that history.

Questions for authors
  1. Could the authors add comparisons with strong baselines such as NAVIS under a standard full-candidate protocol? Simple baselines based on pair-specific historical statistics or time-decayed aggregation would also help clarify FA-State’s practical advantage.

  2. Why is a persistent recurrent state needed for each directed pair? Could the authors compare FA-State with query-time pair-conditioned history modeling using Transformers or cross-attention? Ideally, these comparisons would use the same backbone and historical inputs and report predictive performance, memory usage, and computational cost.

  3. Proposition 2 emphasizes recurrence-dependent shrinkage, whereas FA-State mainly uses a global learned gain. Have the authors considered dynamically weighting shared and local predictions according to the amount of history available for each pair? Could this improve performance on sparse or low-recurrence relations?

Reviewer_p3L8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how the organization of historical states affects prediction in temporal relational models. The authors identify predictive aliasing, where shared states merge relation-specific histories that imply different future outcomes. To address this, they propose FA-State, which combines a shared prediction backbone with directed-pair recurrent states while sharing update parameters across pairs. The paper provides theoretical analyses of aliasing risk and the trade-off between shared and local states. Experiments on synthetic streams and temporal graph forecasting tasks show improvements across multiple backbones. Wrong-address controls and frozen-model retrieval interventions further demonstrate that correctly matching historical content to its retrieval address is important for prediction.

Critical strengths
  1. Clear motivation and intuitive design. The paper identifies predictive aliasing as a limitation of shared temporal states. FA-State addresses this by maintaining pair-specific histories while sharing update parameters, offering a straightforward way to balance shared and local information.

  2. Convincing experimental controls. The wrong-address comparisons and frozen-model retrieval interventions help isolate the importance of correct state addressing. In particular, the frozen-model experiment provides direct evidence that the trained predictor relies on the correspondence between historical content and retrieval addresses.

  3. Comprehensive evaluation and useful analysis. The method is evaluated across multiple temporal backbones and prediction tasks. The controlled experiments and recurrence-stratified results further clarify when pair-specific history is beneficial and where its advantages are limited.

Critical weaknesses
  1. Limited evidence for the necessity of learned recurrent states. The experiments show that correct addressing matters, but it is less clear whether a learned recurrent state is needed to achieve these gains. Comparisons with simpler pair-specific history summaries, such as interaction counts or recent event aggregation, would help clarify the contribution of the proposed architecture.

  2. Unclear scalability and limitations in sparse settings. Maintaining a separate state for each active pair introduces additional memory costs, but runtime and memory overhead are not reported. The gains are also less consistent for pairs with limited history, while the main node-affinity experiments rely on occupancy-matched training, leaving the method's broader applicability less clear.

  3. The figures contain excessive whitespace and disproportionately large titles and annotations; improving the layout, font-size consistency, and space allocation would enhance readability and facilitate comparison of experimental results.

Questions for authors
  1. How much of the improvement comes from recurrent state learning? Could the authors compare FA-State with simpler correctly addressed history summaries, such as time-decayed counts or recent event aggregation, under the same training and evaluation settings?

  2. What is the computational overhead of FA-State? Could the authors report the additional GPU memory usage, training time, and inference cost relative to the original backbones? How would these costs scale as the number of active pairs increases?

  3. How robust is FA-State under sparse histories and different training objectives? Would adapting the addressed contribution based on pair recurrence help in sparse settings? Do the improvements persist when training directly with the full candidate set rather than the occupancy-matched objective?

Reviewer_opwKOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies predictive state scope and proposes FA-State, which combines shared prediction with separate recurrent states for directed relationships.

Critical strengths
  • The proposed new architecture combines features learned across relationships with separately stored relationship histories.

  • The paper provides synthetic and real-data experiments, where the proposed model achieves good performance on several tasks.

Critical weaknesses
  • The writing lacks clarity. Wording requires significant improvements. For example, Sec. 2 uses "equivalence cells" and "predictively homogeneous almost surely" without sufficient explanation.

  • If I understand it correctly, Proposition 1 applies a standard log-loss and conditional mutual information identity, thereby, it lacks new theoretical results.

  • It seems FA-State introduces additional storage for relationship histories. The paper does not report measured memory or runtime overhead.

Questions for authors
  • Can the authors give a concrete example explaining states, equivalence cells, and predictive aliasing?

  • Can the authors further explain what the proposed theory establishes about FA-State's learned memories?

  • Can authors introduce the formalization of what predictive state is and its potential application examples? A background section could be benefit readers to understand the work better.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.