acceptodds
Under review as a conference paper at ICLR 2027

LLMODE: Aligning ODEs with LLMs via Gated Token Injection for Irregular Spatio-Temporal Forecasting

Abstract

Large language models (LLMs) have shown promise for spatio-temporal forecasting, but existing approaches often rely on regularly sampled token sequences and struggle with irregular observations because of temporal asynchrony, representation-space misalignment, and limited context windows. We propose LLMODE, a token-efficient framework for irregular spatio-temporal forecasting with a frozen LLM backbone. LLMODE first uses a graph-aware ODE encoder to reconstruct irregular graph observations as a continuous-time latent trajectory. A Fixed-Budget Perceiver Resampler then compresses this variable-length trajectory into a fixed number of dynamic memory tokens. In parallel, compact statistical descriptors are encoded and resampled into context memory tokens. A dual-source gated cross-attention module injects both memories into the frozen LLM, enabling controlled utilization of external spatio-temporal evidence. Experiments on three real-world urban datasets and two physical-dynamics benchmarks show competitive overall performance, with clearer advantages under sparse or dynamically complex irregular sampling. Additional evaluations on unseen urban regions further demonstrate strong zero-shot generalization without adaptation.

Primary area:
learning on time series and dynamical systems
Keywords:
ML: Time-Series & Data Streams

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Ph12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on irregular spatiotemporal forecasting on graphs, using a frozen LLM as the forecasting backbone. The paper argues that existing LLM-based forecasting methods rely on regularly sampled token sequences, making it difficult to handle asynchronous and sparse observations, while also suffering from representation misalignment and context-length limitations. To address these issues, the paper proposes LLMODE, which uses a pretrained graph-aware neural ODE to encode irregular observations, compresses the resulting latent trajectories and segment-wise statistics into a small number of memory tokens, and injects them into a frozen Vicuna-7B through gated cross-attention to produce forecasts via regression. Within the paper's experimental settings, LLMODE achieves competitive results on three NYC urban datasets and two particle dynamics benchmarks, generally outperforms the baselines in zero-shot region transfer, and substantially reduces token usage and inference latency compared with UrbanGPT.

Critical strengths

1. Using continuous-time models to enable LLMs to process asynchronous and sparse spatiotemporal observations is a meaningful and relatively underexplored direction. I therefore appreciate the paper's problem setting and motivation.

2. The experiments cover five datasets across urban and physical domains, 11 baselines, and zero-shot settings, with ablations examining token budgets, evidence selection, injection mechanisms, backbone replacement, and solvers. I therefore appreciate the breadth of the experimental evaluation.

Critical weaknesses

1. LLMODE's performance advantages cannot be attributed to the frozen LLM, leaving the paper's central claim unsupported. According to the provided code, the ODE encoder in the first stage is trained not only with the two loss terms in Eq. (14), but also through a prediction head supervised by observations within the prediction horizon. In other words, the ODE encoder itself is already trained as a predictor for the same target, yet the paper neither explains this nor reports the standalone performance of this predictor. Therefore, the reported results cannot distinguish whether the performance advantages come from the frozen LLM itself or from the ODE encoder.

2. The implementation of a key novel module does not match its description in the paper. The core design of the Resampler is to use the magnitude of the ODE vector field, ∣fθ(h)∣|f_\theta(h)|, as an attention bias (Eq. (6)) to preserve states from intervals of rapid change. However, in the provided code, this bias is actually computed as the norm of the difference between adjacent latent states after a learnable projection, and β\beta is also learnable rather than fixed at 0.1 as described in Appendix A.3. Therefore, the ablation removing the dynamic bias in Figure 5, as well as the paper's conclusions about dynamics-aware compression, does not validate the design described in the paper.

Questions for authors

1. How is the missing rate in Figure 3 defined, what missing rate does the default Springs setting in Appendix A.1 correspond to, and do the Springs results in Table 1 use this default setting?

2. Table 7 shows that removing compression barely changes latency, and only K=5K=5 and K=8K=8 outperform UrbanGPT on both metrics. How much of the reported 7.7$\times$ speedup over UrbanGPT is attributable to fixed-budget compression? How was KK selected?

3. Please clarify the following:

(a) Does the latent trajectory fed into the Resampler contain states within the prediction horizon? What is the standalone forecasting accuracy of the first-stage encoder together with its prediction head?

(b) In Figure 5, Random-5 performs several times worse than LLMODE, and even worse than the standalone GRU-ODE in Table 1, despite using states from the same pretrained ODE. Why is this the case?

(c) What exactly is the prediction target on NYC? Appendix A.1 defines it as cumulative flow since midnight of the current day, whereas Appendix A.16 defines it as the subsequent 12 regular time steps. Figure 2 instead shows irregular query times, and the ground truth in Figure 8 also does not appear to be cumulative flow that resets daily.

14 comments

We sincerely thank the reviewer for the detailed review and constructive feedback. We respond to each point below.

W1 & Q3(a): Stage-1 Training and Standalone ODE Performance

We thank the reviewer for this important observation. We clarify the Stage-1 training objective, the construction of latent trajectories, and the standalone forecasting performance of the ODE encoder.

1. Stage-1 training and latent trajectories.

The Stage-1 objective follows the pre-update and post-update loss formulation in Eq. (14). In our implementation, the post-update loss additionally incorporates forecasting supervision through an auxiliary prediction head, which predicts future latent states and is trained against observations within the prediction horizon.

During inference, the trajectory provided to the Resampler includes these model-generated future latent states, without using future ground-truth observations for hidden-state updates. We will make this implementation detail more explicit in Appendix A.2.

2. Standalone ODE forecasting performance.

To assess the forecasting capability of the first-stage encoder independently, we evaluate the ODE encoder together with its prediction head, without the Resampler or LLM. The NYCtaxi supervised results are summarized below.

ModelNYCtaxi-Sup MAEMSENYCtaxi-Zero-shot MAEMSE
ODE Encoder + Prediction Head0.03170.00810.10040.0554
LLMODE0.02940.00620.08010.0396

The complete LLMODE achieves lower prediction errors than the standalone ODE predictor, indicating that the downstream forecasting framework provides additional benefits beyond the first-stage encoder. Furthermore, our controlled comparison between pretrained and randomly initialized frozen Vicuna-7B backbones provides complementary evidence for the contribution of language pretraining.

We will clarify the Stage-1 objective and latent-trajectory construction in Appendix A.2 and include the standalone ODE comparison in the revised experimental results.

W2: Clarification of Dynamics-Aware Bias and Ablation Validity

We thank the reviewer for carefully examining our implementation. We clarify the relationship between the continuous-time formulation and its discrete implementation, the setting of the bias coefficient β\beta, and the interpretation of the existing ablation results.

1. Continuous-time formulation and discrete approximation.

Eq. (6) describes the dynamics-aware bias using the magnitude of the continuous-time ODE vector field.

In implementation, we use first-order differences between adjacent latent states as a discrete approximation of the continuous-time derivative. Since the ODE solver uses a fixed integration step, these differences provide a practical measure of temporal variations without explicitly evaluating the vector field.

The differences are computed after feature projection because the ODE latent states are mapped into the Resampler's feature space for alignment with the LLM. This allows the bias to operate on the same representations used for attention and memory compression.

For equally spaced latent states, the projected difference approximates the projected ODE derivative up to a constant time-step factor, which cancels under maximum-based normalization. Thus, the difference-based score is a discrete approximation of continuous-time dynamics in the projected space, rather than an unrelated mechanism. We will clarify this approximation and its conditions in the revised manuscript.

2. Setting of the bias coefficient β\beta.

We initially adopted a learnable parameterization to adjust the strength of the dynamics-aware bias. During training, we observed only minor changes in this coefficient, suggesting limited sensitivity to its optimization under the evaluated settings.

This motivated our subsequent adoption of a fixed coefficient of β=0.1\beta = 0.1 for implementation simplicity. We will clarify the parameterization and corresponding experimental configurations in Appendix A.3.

3. Interpretation of the dynamics-aware ablation.

Figure 5 isolates the contribution of dynamics-aware attention guidance by removing the bias while retaining the same underlying Perceiver resampling mechanism. The resulting performance degradation supports our central design motivation that incorporating temporal dynamics into attention improves forecasting performance.

Since the implemented finite differences approximate the continuous-time dynamics underlying our formulation, this experiment provides evidence for the proposed dynamics-aware design principle, although it does not independently validate the exact vector-field norm.

We will refine Eq. (6) and Section 3.2 to clarify the relationship between the theoretical formulation and its numerical implementation, and update Appendix A.3 to accurately describe the bias coefficient and experimental configuration.

Q1: Definition of Missing Rate and Default Springs Setting

We thank the reviewer for requesting this clarification. In Figure 3, the missing rate is defined as the proportion of unobserved particle–time positions within the eligible history window, i.e., rmiss=1−Nobs/Ntotalr_{\mathrm{miss}}=1-N_{\mathrm{obs}}/N_{\mathrm{total}}.

As described in Appendix A.1, the Springs dataset uses a history window of 60 sampled time steps, with approximately 24 observations retained per particle on average. This corresponds to a nominal missing rate of 60%, although the realized rate varies with the randomly sampled observation counts.

Figure 3 further evaluates missing rates of 20%, 40%, 60%, and 80% to examine robustness under different observation densities.

The Springs results in Table 1 use the same default observation-sampling protocol, corresponding to approximately 60% missingness.

We will explicitly clarify the missing-rate definition, default setting, and its relationship to Figure 3 and Table 1 in Appendix A.1.

Q2: Efficiency Gains and Selection of K

We thank the reviewer for this question. As shown in Table 7, removing compression increases inference latency, but the increase is relatively small. To understand this observation, we conducted an additional module-level latency analysis on NYCbike (batch size = 1, same GPU).

ComponentLatency (ms/sample)
ODE Solver0.6033
Context Encoder4.8565
Resampler1.1124
LLM Processing77.4478
Total84.0200

The results show that LLM processing dominates inference latency (92.2%), whereas the Resampler accounts for only 1.3%.

This helps explain the limited latency reduction from compression observed in Table 7: compression reduces the memory tokens processed by cross-attention, but does not directly shorten the LLM backbone's self-attention sequence, where much of the computational cost lies.

The 7.7× speedup over UrbanGPT therefore reflects a broader architectural advantage.

UrbanGPT incorporates spatio-temporal tokens into the LLM input sequence, increasing self-attention overhead. In contrast, LLMODE injects dynamic and context memories through gated cross-attention without extending the backbone's self-attention sequence.

Fixed-budget compression further reduces cross-attention overhead, but is not the primary explanation for the overall speedup.

Regarding K selection, our initial design goal was to achieve substantial compression to reduce inference overhead. We therefore evaluated different memory-token budgets to examine their effects on forecasting accuracy and latency, as reported in Table 7.

The results show that appropriate compression can improve forecasting accuracy while reducing computational overhead. We hypothesize that this is because consecutive ODE states contain substantial temporal redundancy, and the dynamics-aware Resampler can selectively aggregate informative temporal variations while reducing redundant information.

However, excessive compression may discard useful dynamics, whereas larger K values increase computational cost without consistent accuracy improvements. Based on this trade-off between prediction accuracy and inference efficiency, we selected K=5 as the default setting.

Reviewer_5GmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLMODE, a framework for irregular spatio-temporal forecasting with a frozen LLM backbone. A graph-aware ODE encoder reconstructs continuous-time latent trajectories, and a Fixed-Budget Perceiver Resampler compresses them into dynamic memory tokens using a dynamics-aware attention bias and explicit time labels. Statistical descriptions of the observed history are encoded into context memory tokens, and dual-source gated cross-attention injects both memories into the frozen LLM before a regression head maps query-slot hidden states to forecasts. Experiments on three NYC datasets and two physical-dynamics benchmarks report competitive forecasting performance, generalization to unseen NYC regions, and reduced token usage and inference latency relative to UrbanGPT.

Critical strengths
  • The paper addresses a meaningful forecasting problem involving irregular observation intervals, asynchronous node observations, and spatial dependencies. The combination of a graph-aware ODE encoder, compact memories, and a frozen LLM provides a coherent framework for handling these requirements.

  • The reported unseen-region results are promising. Table 2 shows the lowest errors on five of the six evaluation metrics, with improvements on both MAE and MSE for NYCbike and NYCcrime. These results support the potential usefulness of the framework for forecasting in regions that were not used for training, without additional adaptation.

  • The ablation studies examine several important design choices. The comparisons with a discrete spatio-temporal encoder, alternative compression methods, removal of the dynamics-aware bias or explicit time labels, and shuffled context memory provide useful evidence for the contributions of the proposed components.

Critical weaknesses
  • The NYC benchmark construction is not sufficiently specified to reproduce the evaluation reliably. Appendix A.1 reports observation-interval ranges and cumulative-count definitions, but the exact procedure for selecting observation timestamps and constructing the NYCcrime sliding windows remains unclear. Appendix A.16 also states that NYC evaluation uses 12 regular prediction steps, without specifying their duration for each dataset. This limits the interpretation of the experiments as evidence for forecasting at arbitrary query timestamps and leaves the relationship to the stated forecasting horizons insufficiently explained.

  • The benchmark comparisons leave important questions about directly relevant methods and information conditions unresolved. Although tPatchGNN and ISTS-PLM are included, LG-ODE from the cited work of Huang et al. (2020) is not evaluated, despite addressing irregularly sampled interacting systems and using Springs and Charged. The construction of the graphs, the availability of POI information to the baselines, and the implementation of baseline transfer to unseen regions also require clarification. These omissions limit confidence in the completeness and comparability of the reported results.

  • The methodological advance over the cited approaches needs clearer justification. The encoder uses the GRU-ODE-Bayes parameterization, while Perceiver resampling and zero-initialized gated cross-attention with a frozen language model are established components of Flamingo. Figure 5 provides evidence for the dynamics-aware bias and explicit time labels through individual ablations, which is a useful strength. However, a comparison retaining the same encoder and both memory sources while jointly removing these two additions would more directly establish the improvement over the existing resampling and injection mechanisms.

  • The contribution of language pretraining is not adequately separated from differences in architecture, capacity, and training conditions. Section 4.6 and Table 3 compare frozen Vicuna-7B with an MLP and a trainable, randomly initialized six-layer Transformer. These comparisons support the performance of the complete framework, but they do not isolate language pretraining as the reason for the better unseen-region results. This limits the strength of the interpretation that pretrained representations specifically provide the observed transfer benefit.

Questions for authors
  • What exact procedure selects the NYC observation timestamps, and how are intervals without recorded events represented? Please specify the NYCcrime sliding-window construction and the duration of each regular prediction step in Appendix A.16, including how the 12 prediction steps correspond to the stated 48-day horizon.

  • How is the graph constructed for each dataset, and what graph and POI information is provided to each baseline? Please also explain how the baseline models trained on the selected 80 regions are applied to the additional 80 unseen regions.

  • Can the authors provide a comparison with LG-ODE from Huang et al. (2020) using the same observations, graph information, prediction targets, and normalization on Springs and Charged? This comparison would materially affect the assessment of the improvement over existing continuous-time modeling of interacting systems.

  • Can the authors isolate the effect of pretraining by comparing pretrained and randomly initialized versions of the same backbone while keeping the remaining architecture, trainable modules, and training procedure unchanged? Such a comparison would help distinguish the contribution of pretraining from the other differences in Table 3.

  • Figure 5 removes the dynamics-aware bias and explicit time labels individually. What is the performance when both are removed together, while retaining the same graph-aware ODE encoder, dynamic and context memories, token budget, and dual-source gated cross-attention? This would clarify their combined contribution beyond the underlying Perceiver resampling and gated injection components.

3 comments

We thank the reviewer for the valuable comments and suggestions. Our responses are provided below.

W1 & Q1: Clarification of NYC Benchmark Construction

We thank the reviewer for highlighting the need for clearer benchmark construction details. We provide the following clarifications.

1. Observation timestamp selection and event representation.

For NYCtaxi and NYCbike, observation timestamps are independently generated for each spatial node.

Specifically, sampling intervals are drawn from an exponential distribution with a scale parameter of 60 minutes, clipped to 10–360 minutes, and rounded to integer minutes. These intervals are then accumulated to construct irregular observation timestamps.

Intervals without recorded events contribute zero to the cumulative event count.

2. Sliding windows and prediction horizons.

As described in Appendix A.1, NYCcrime uses sliding windows to construct cumulative crime counts, with a 96-day history and a 48-day forecasting horizon.

Appendix A.16 further specifies that we evaluate the NYC datasets using 12 regularly spaced prediction steps to ensure fair comparisons with baselines designed for regular temporal grids. Each step corresponds to 1 hour for NYCtaxi/NYCbike and 4 days for NYCcrime, covering 12 hours and 48 days, respectively.

Importantly, the use of regular prediction timestamps is solely an evaluation choice for fair comparison, rather than a limitation of LLMODE, which supports forecasting at arbitrary query timestamps.

In the revised manuscript, we will explicitly describe the observation timestamp sampling procedure and NYCcrime sliding-window construction in Appendix A.1, and clarify the duration of each prediction step and its correspondence to the forecasting horizon in Appendix A.16.

W2 & Q2 - Q3: Baseline Comparisons and Evaluation Protocol

We thank the reviewer for these valuable suggestions. We address the missing LG-ODE comparison and clarify the graph construction, information availability, and zero-shot evaluation protocol.

1. Comparison with LG-ODE.

We have added LG-ODE as a graph-aware continuous-time baseline on Springs and Charged, both of which were also evaluated in the original LG-ODE paper.

To verify the reliability of our implementation, we first compared our LG-ODE results with its published results on Charged, obtaining comparable MSE.

We then evaluated LG-ODE using the same observations, graph information, prediction targets, and normalization as LLMODE to ensure a fair comparison. Notably, our Springs setting uses a 5× larger downsampling interval than the original LG-ODE protocol, resulting in a longer extrapolation horizon.

MethodSprings-X MSESprings-Y MSECharged-X MSECharged-Y MSE
LG-ODE0.02230.02190.06490.0654
LLMODE0.01910.01960.04830.0493

Under matched settings, LLMODE reduces the average MSE across both spatial coordinates by 12.4% on Springs and 25.1% on Charged compared with LG-ODE. These results further support the effectiveness of LLMODE in modeling physical dynamics. We will include these results in the revised Table 1.

2. Graph construction.

For the NYC datasets, spatial graphs are constructed based on geographical adjacency, with each node representing a spatial region. Taking NYCtaxi as an example, the NYC Taxi and Limousine Commission (TLC) provides a Taxi Zone Lookup Table containing region identifiers (LocationID) and a Taxi Zone Shapefile defining the polygon boundaries of 263 taxi zones.

We use the lookup table to identify graph nodes and derive the adjacency matrix from the polygon relationships in the Shapefile. An undirected edge connects two distinct zones if their boundaries touch, including shared edges or vertices. This yields a fixed binary adjacency matrix without self-loops.

For training and zero-shot evaluation, we extract the corresponding 80 × 80 adjacency submatrices independently, without cross-partition edges. The graph topology remains fixed over time and is independent of observation timestamps.

3. Graph and POI information for baselines.

All graph-based baselines are provided with the same dataset-specific adjacency matrices and historical observations as LLMODE.

As illustrated in Appendix A.10, we use region-level POI profiles to describe the functional characteristics of each spatial region. For example, the NYCtaxi prompt describes a Bronx region using its dominant POI categories, including Cultural Facility, Residential, and Education Facility.

For baselines that support POI inputs, we provide the same region-level POI information as used by LLMODE, ensuring consistent auxiliary information across applicable methods. Baselines without POI-input mechanisms follow their original input configurations.

4. Zero-shot transfer to unseen regions.

As described in Section 4.3 and Appendix A.1, the zero-shot setting uses 80 training regions and another 80 disjoint test regions. The unseen regions are excluded from both model training and spatio-temporal encoder pretraining.

During evaluation, each model receives the historical observations and applicable graph information of the unseen regions, while its learned parameters remain fixed. No test-region labels are used for training, and no fine-tuning is performed.

For graph-based baselines, the training-region and test-region adjacency submatrices are used in their respective stages, ensuring that no edges connect the two partitions.

In the revised manuscript, we will incorporate the LG-ODE results into Table 1, clarify graph construction and the zero-shot data split in Appendix A.1, and explicitly document baseline graph/POI input conditions and transfer procedures in Appendix A.16.

W3 & Q5: Methodological Novelty and Joint Ablation

We thank the reviewer for this constructive suggestion.

Following the reviewer's recommendation, we conduct an additional joint ablation that removes both the dynamics-aware bias and explicit time labels while retaining the same ODE encoder, dual memory sources, gated cross-attention, and training configuration, allowing a more direct assessment of their contributions beyond the existing resampling and injection mechanisms.

ConfigurationNYCtaxi Supervised (MSE)NYCtaxi Zero-shot (MSE)
w/o time labels0.00680.0432
w/o dynamics bias0.00750.0475
w/o both0.00790.0502
LLMODE0.00620.0396

Jointly removing both mechanisms increases MSE by 27.4% in supervised evaluation and 26.8% in zero-shot evaluation compared with the full model.

We will include these results in Section 4.5 and further clarify our methodological contributions in Sections 3.2–3.3.

W4 & Q4: Contribution of Language Pretraining

We thank the reviewer for this valuable suggestion. The original six-layer trainable Transformer was designed to evaluate a non-pretrained alternative.

To further isolate the contribution of language pretraining, we additionally compare pretrained and randomly initialized frozen Vicuna-7B backbones, keeping the architecture, parameter count, freezing strategy, trainable modules, and training procedure identical.

BackboneNYCtaxi-Sup MAEMSENYCtaxi-Zero-shot MAEMSE
Random Vicuna-7B (Frozen)0.03540.00890.18210.1648
Pretrained Vicuna-7B (Frozen)0.02940.00620.08010.0396

As shown above, the pretrained backbone achieves substantially lower prediction errors, particularly on unseen regions.

Although the randomly initialized backbone retains the same Transformer architecture, its frozen attention and feed-forward layers lack learned feature transformations. In contrast, language pretraining can provide reusable sequence-processing capabilities beyond the original language domain (Lu et al., 2022).

In LLMODE, these pretrained capabilities may help the frozen backbone utilize ODE-derived dynamics and statistical context through gated cross-attention, potentially contributing to its stronger generalization to unseen regions.

We will include this comparison in the revised Table 3 and clarify the role of language pretraining in Section 4.6.

Reviewer_ftGLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses forecasting from asynchronous, partially observed graph time series using a frozen language model. A graph-aware GRU-ODE encoder propagates node states between observations and corrects them when measurements arrive. A Perceiver resampler compresses its latent trajectory into a fixed number of dynamic memory tokens, with a bias based on latent derivative magnitude and explicit temporal labels. A second memory represents statistical descriptions of the observed history. Gated cross-attention and feed-forward modules inject both memories into a frozen LLM, and timestamp-specific query states are mapped to numerical predictions. The evaluation covers three NYC datasets and two interacting-particle benchmarks, including held-out urban regions, missing-rate sweeps, compression and injection ablations, and alternative prediction backbones. The results support the usefulness of the complete system, particularly under sparse observations and regional transfer.

Critical strengths

1.The empirical evaluation reasonably convincing. The method works across both urban forecasting and physical systems, and its advantages are more apparent when observations are sparse or the test regions are unseen. This makes the paper more interesting to me than a method that only improves standard forecasting accuracy. The regional-transfer results are its strongest point.

  1. The authors make a useful effort to understand which parts of the system matter. The comparisons with simpler predictors, alternative compression methods, and shuffled context go beyond simply removing modules. I particularly appreciated the observation that removing the LLM has a larger effect on regional transfer than on supervised prediction. Although the explanation is not fully settled, this is a useful finding.
Critical weaknesses
  1. I am not yet convinced that language pretraining is what explains the transfer gains. The replacement Transformer is smaller and trained differently, so the comparison changes several things at once. It shows that the chosen LLM performs better than that alternative, but leaves open whether the benefit comes from pretraining, capacity, or the training setup. This is my main reservation because it affects the motivation for using an LLM in the first place.

  2. The graph setup needs a clearer explanation. I could not determine exactly how edges are constructed or what information from neighboring regions is available when evaluating an unseen region. That makes it difficult to interpret how demanding the transfer task actually is and whether all methods have access to the same information. A clear description of the training and test graphs would resolve much of this concern.

  3. Much of the interface follows an established design. The resampler and gated cross-attention are closely related to Flamingo, which the paper does cite. The interesting contribution here is how this design is adapted to irregular graph dynamics, especially through the dynamics bias and time labels. I would like the paper to be more explicit about that distinction and about what we learn from the adaptation beyond the fact that the combination works.

  4. The efficiency discussion gives compression more credit than the comparison clearly supports. The compressed and uncompressed versions of LLMODE have fairly similar inference times, while the large speedup is measured against UrbanGPT, whose interface is different in several respects. Compression appears useful for controlling memory size and preserving accuracy, but its contribution to the overall speedup should be stated more carefully.

  5. I also have a smaller concern about scaling to more irregular inputs. The model constructs the full ODE trajectory before compressing it, and the integration step depends on the smallest observation gap. A few closely spaced observations could therefore increase the work substantially even with a fixed number of memory tokens. The current experiments do not show how the method behaves in that situation.

Questions for authors
  1. Can you provide a control that changes pretraining while preserving architecture, freezing policy, memory interface, and readout? A suitably smaller matched pretrained/random pair would also be informative if repeating this at 7B scale is impractical. How much of the regional-transfer advantage remains under that comparison?

  2. How are graph edges constructed for each dataset, and which nodes, edges, histories, and region descriptors are accessible during each training stage and at test time? Are held-out nodes excluded from all encoder pretraining? Please state whether the reported setting is inductive or transductive and whether baselines receive the same spatial information.

  3. Which parts of the resampler and gated injection are inherited from the established interface, and which are new? What specific finding about irregular graph forecasting is supported by the dynamics-bias, time-label, and dual-memory experiments beyond the usefulness of combining these components?

  4. Can you separate latency for the ODE encoder, context encoder, resampling, and LLM processing, and identify which changes account for the UrbanGPT-to-LLMODE speedup? Please make clear whether all reported latency measurements include the complete inference pipeline under matched batching and hardware settings.

  5. Is the minimum observation gap used globally, per sample, or per node, and is there a lower bound or cap on the resulting integration grid? A small timing sweep over history length or minimum gap, with K fixed, would materially clarify the scope of the efficiency claim. How does total token count change with the number of forecast queries?

4 comments

We appreciate the reviewer's thoughtful feedback. We address the questions and concerns below.

W1 & Q1: Isolating the Contribution of Language Pretraining

We thank the reviewer for highlighting this important concern. The original six-layer trainable Transformer was intended to evaluate a non-pretrained alternative.

To directly isolate the effect of language pretraining, we conducted an additional controlled experiment comparing pretrained and randomly initialized frozen Vicuna-7B backbones, while keeping the architecture, model capacity, freezing policy, memory interface (including gated cross-attention), prediction readout, and training protocol identical.

BackboneNYCtaxi-Sup MAEMSENYCtaxi-Zero-shot MAEMSE
Random Vicuna-7B (Frozen)0.03540.00890.18210.1648
Pretrained Vicuna-7B (Frozen)0.02940.00620.08010.0396

As shown above, the pretrained backbone retains a substantial advantage in unseen-region forecasting under the matched setting, indicating that the transfer gains cannot be explained by model capacity or architectural differences alone.

Although the randomly initialized backbone retains the same Transformer architecture, its frozen attention and feed-forward layers lack learned feature transformations. In contrast, language pretraining establishes reusable sequence-processing capabilities that can transfer beyond the original language domain (Lu et al., 2022).

In LLMODE, these pretrained capabilities may facilitate the integration of ODE-derived dynamics and statistical context through gated cross-attention, supporting stronger generalization to unseen regions. We will incorporate this controlled comparison into the revised Table 3.

W2 & Q2: Graph Construction and Inductive Evaluation Protocol

Graph construction.

We construct spatial graphs based on geographical adjacency, where each node represents a spatial region. Taking NYCtaxi as an example, the NYC Taxi and Limousine Commission (TLC) provides a Taxi Zone Lookup Table containing unique region identifiers (LocationID) and corresponding region information, along with a Taxi Zone Shapefile defining the polygon boundaries of all 263 taxi zones.

We use the lookup table to identify graph nodes and derive the adjacency matrix from the spatial relationships between polygons in the Shapefile. Specifically, an undirected edge is established between two distinct zones if their geographical boundaries touch, including shared edges or vertices. This produces a fixed binary adjacency matrix without self-loops.

We then extract the corresponding 80 × 80 adjacency submatrices for the training and unseen-region evaluation sets, respectively. The graph topology remains fixed over time and is independent of observation timestamps.

Training and test information.

During both encoder pretraining and downstream training, only the 80 selected training regions, their internal graph edges, historical observations, and corresponding region descriptors are accessible. All held-out regions are strictly excluded from every training stage.

During zero-shot evaluation, the trained model is directly applied to 80 disjoint unseen regions without fine-tuning. The test graph contains only these unseen nodes and their internal edges, with no connections to training nodes. Each test node can access its own observed history, neighboring nodes' histories within the test graph, and its region descriptors, but not future target values.

Inductive setting and baseline fairness.

Our evaluation is inductive rather than transductive, since the unseen nodes and their observations are never involved in training.

Graph-based baselines use the same dataset-specific adjacency matrices, graph partitions, and historical observations as LLMODE. All baselines follow the same data splits and forecasting targets, while methods without textual input interfaces do not use POI descriptors. We will clarify these settings in the revised appendix.

W3 & Q3: Novelty and Insights of Irregular-Dynamics Adaptation

We thank the reviewer for this insightful comment. Our contribution lies not in the Perceiver Resampler or gated cross-attention themselves, but in adapting these mechanisms to irregular graph dynamics and identifying how forecasting-relevant information can be preserved during compression and injection.

Inherited components and our adaptations.

We inherit the fixed-budget latent-query resampling mechanism and zero-initialized gated cross-attention from Flamingo. Our adaptations are threefold:

  • (1) Dynamics-aware resampling introduces the normalized magnitude of the ODE vector field into attention logits, encouraging the selection of states with pronounced temporal changes;
  • (2) explicit time labeling computes attention-weighted representative timestamps and embeds them into the resampled tokens, preserving temporal locations after compression;
  • and (3) dual-source memory separately represents ODE-derived dynamics and historical statistical context, allowing prediction queries to retrieve complementary evidence through gated cross-attention.

Insights from the experiments.

Figures 5 and 6 reveal three important findings about irregular graph forecasting.

  • First, removing the dynamics-aware bias degrades performance, suggesting that uniformly treating ODE-reconstructed states overlooks informative transitions. Prioritizing states with pronounced changes helps preserve dynamics relevant to forecasting.

  • Second, removing explicit time labels also reduces accuracy, indicating that temporal information cannot be fully recovered from compressed state representations alone. For irregular observations, similar states occurring at different times may imply different future trajectories, making their temporal locations important.

  • Third, removing or shuffling statistical context degrades performance, showing that local temporal dynamics and global statistical characteristics provide complementary, sample-specific information.

Together, these results suggest that effective compression of irregular graph trajectories requires preserving what changes, when it changes, and the broader context in which those changes occur, rather than relying solely on content-based token compression.

We will clarify the distinction between inherited components and our task-specific adaptations, and emphasize these findings in the revised manuscript.

W4 & Q4: Latency Breakdown and Attribution of Efficiency Gains

We thank the reviewer for this important observation. To clarify the sources of efficiency gains, we conducted an additional module-level latency analysis on NYCbike.

1. Latency breakdown.

To measure per-sample inference latency, all methods were timed on the same GPU with batch size set to 1. The measured latencies of the main inference components are summarized below:

ComponentLatency (ms/sample)
ODE Solver0.6033
Context Encoder4.8565
Resampler1.1124
LLM Processing77.4478
Total (sum of components)84.0200

LLM processing accounts for approximately 92.2% of the measured module latency, while ODE solving and resampling together contribute only 2.0%, indicating relatively small computational overhead from these components. The remeasured total (84.02 ms/sample) is consistent with the previously reported inference latency of 83.81 ms/sample in Table 7, differing by approximately 0.25%.

2. Sources of the overall speedup.

The key architectural difference lies in how spatio-temporal information interacts with the LLM.

UrbanGPT directly incorporates spatio-temporal tokens into the LLM input sequence, increasing the length processed by self-attention. In contrast, LLMODE injects separate dynamic and context memories through gated cross-attention without extending the backbone's self-attention sequence.

Consequently, LLMODE's backbone self-attention is less sensitive to the number of spatio-temporal tokens, while fixed-budget resampling further bounds memory size and cross-attention cost. These differences help explain the substantial speedup over UrbanGPT, although their individual latency contributions are not isolated by this comparison.

In the revised manuscript, we will clarify in Section 4.5 that the overall speedup over UrbanGPT reflects differences in the token-processing architecture rather than compression alone.

We will also extend Appendix A.13 with the module-level latency breakdown and measurement settings, alongside the existing token-budget and end-to-end efficiency results.

W5 & Q5: Integration Step Configuration and Scalability

We thank the reviewer for raising this concern. We clarify that the ODE integration step is determined by the dataset's temporal resolution and fixed during inference, rather than dynamically recomputed from the minimum observation gap of each sample or node.

1. Integration step configuration.

For the physics-based datasets (Springs and Charged), the ODE integration step is set to the minimum sampling interval of the predefined temporal grid. The original simulated trajectories are downsampled by factors of 500 and 100, respectively, and irregular observations are subsequently selected from these discrete grids.

For the NYCtaxi and NYCbike datasets, observation intervals are independently sampled for each node from an exponential distribution, constrained to 10–360 minutes, and rounded to integer minutes. Therefore, when observations from different nodes are combined, the minimum possible positive interval between timestamps is one minute.

Accordingly, we set the ODE integration step to Δt=1\Delta t = 1 minute, matching the temporal resolution of the observations. Since this step size remains fixed during inference, closely spaced observations do not trigger a finer integration grid.

2. Computational scalability.

To further evaluate computational scalability, we conduct an additional timing sweep on NYCbike by varying the history length from 12 to 48 hours while keeping the integration step at one minute and the memory budget fixed at K = 5.

History LengthKODE Latency (ms)Resampler Latency (ms)End-to-End Latency (ms)
24 h50.60331.112484.0200
48 h51.09691.864785.4669

As shown in our module-level latency analysis in W4/Q4, ODE solving accounts for only approximately 0.7% of the total inference latency, indicating its relatively small computational overhead.

To further examine scalability, we extend the history length from 24 to 48 hours while keeping K=5 fixed. Although the latencies of ODE solving and resampling increase by 81.8% and 67.6%, respectively, end-to-end latency increases by only 1.72%.

These results demonstrate that, although longer histories increase trajectory encoding and resampling costs, their impact on overall inference latency remains limited under the evaluated settings.

3. Token scaling with forecast queries.

LLMODE maintains K dynamic memory tokens and K context memory tokens, giving a fixed memory budget of 2K tokens regardless of trajectory length.

Each forecast timestamp corresponds to one query slot in the prompt. Thus, for LqL_q forecast queries, the logical token budget is Nfixed+Lq+2KN_{\mathrm{fixed}} + L_q + 2K, where NfixedN_{\mathrm{fixed}} denotes the fixed prompt tokens.

Therefore, the memory-token budget remains constant, while the total token count grows linearly with the number of forecast queries.

In the revised manuscript, we will clarify the dataset-specific integration-step configurations in Appendix A.12, incorporate the timing sweep and its analysis in Appendix A.13, and explicitly describe query-dependent token scaling in Section 3.3.

Reviewer_ftGLOpenReview ↗

Thank you for your response.

Thank you for your response. Most of my concerns are addressed, my score is therefore updated.

Reviewer_FBPrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLMODE addresses forecasting from irregular and asynchronous observations. It uses a graph-aware ODE to encode the history, compresses the trajectory into a few time-aware tokens, and injects them alongside historical statistics into a frozen LLM through gated cross-attention.

The main contribution is adapting this pipeline to irregular spatio-temporal data through dynamics-aware compression, time labels, and two memory sources. The Perceiver resampler and gated injection already appear in Flamingo, so the novelty mainly lies in how these components are adapted and combined.

Critical strengths
  1. The design fits the problem: the ODE follows actual time gaps and updates its state when new observations arrive.

  2. Results are broadly consistent across five datasets, with the best performance on five of six unseen-region metrics and clear gains under sparse observations. (Tables 1–2, Figure 3)

  3. The ablations directly test the main design choices, including compression, time labels, dynamics bias, gating, and context information. (Figures 5–6)

Critical weaknesses
  1. The closest graph ODE comparison is missing. LG-ODE handles asynchronous, partially observed graph dynamics and evaluates on Springs and Charged, yet it is absent from Table 1. The GRU-ODE baseline lacks comparable graph modeling, leaving the advantage over existing graph ODE methods unclear.

  2. The benefit of language pretraining is not isolated. Table 3 compares a frozen 32-layer Vicuna-7B with a trainable six-layer Transformer. Model size, architecture, and training setup all change, so the unseen-region gains cannot be attributed to pretraining alone.

  3. Key details of the urban experiments are missing. The paper gives observation-interval ranges but does not explain how NYC timestamps are selected or how graph edges are built. Both choices affect the task and the information available to the model, making the results difficult to interpret and reproduce.

Questions for authors
  1. How does LLMODE compare with LG-ODE using the same observations and graph information?

  2. What happens if the same frozen backbone uses randomly initialized weights instead of pretrained weights?

  3. How exactly are NYC observation times and graph edges generated?

2 comments

We thank the reviewer for the careful review and constructive comments. We provide our responses below.

W1 & Q1: Comparison with LG-ODE

We thank the reviewer for this valuable suggestion. We have added LG-ODE as a graph-aware continuous-time baseline on Springs and Charged, both of which are also evaluated in the original LG-ODE paper.

To ensure a reliable comparison, we first checked our LG-ODE results against its published results on Charged, obtaining comparable MSE.

For fairness, we then evaluated LG-ODE under the same data, graph, and forecasting settings as LLMODE. Notably, our Springs setting uses a 5× larger downsampling interval than the original LG-ODE protocol, resulting in a longer extrapolation horizon.

MethodSprings-X MSESprings-Y MSECharged-X MSECharged-Y MSE
LG-ODE0.02230.02190.06490.0654
LLMODE0.01910.01960.04830.0493

Under matched settings, LLMODE reduces the average MSE across both spatial coordinates by 12.4% on Springs and 25.1% on Charged compared with LG-ODE, demonstrating its advantage over an existing graph-based ODE method. We will include these results in the revised Table 1.

W2 & Q2: Contribution of Language Pretraining

We thank the reviewer for this valuable suggestion. The original six-layer trainable Transformer was designed to evaluate a non-pretrained alternative.

To further isolate the contribution of language pretraining, we additionally compare pretrained and randomly initialized frozen Vicuna-7B backbones, keeping the architecture, parameter count, freezing strategy, trainable modules, and training protocol identical.

BackboneNYCtaxi-Sup MAEMSENYCtaxi-Zero-shot MAEMSE
Random Vicuna-7B (Frozen)0.03540.00890.18210.1648
Pretrained Vicuna-7B (Frozen)0.02940.00620.08010.0396

As shown above, pretraining yields substantially stronger performance, particularly on unseen regions.

Although the randomly initialized backbone retains the same Transformer architecture, its frozen attention and feed-forward layers lack learned feature transformations. In contrast, language pretraining establishes reusable sequence-processing capabilities that can transfer beyond the original language domain (Lu et al., 2022).

In LLMODE, these pretrained capabilities may facilitate the integration of ODE-derived dynamics and statistical context through gated cross-attention, helping explain the stronger generalization to unseen regions. We will include this comparison in the revised Table 3.

W3 & Q3: NYC Observation Sampling and Graph Construction

Graph construction.

We construct spatial graphs based on geographical adjacency, where each node represents a spatial region. Taking NYCtaxi as an example, the NYC Taxi and Limousine Commission (TLC) provides a Taxi Zone Lookup Table containing unique region identifiers (LocationID) and corresponding region information, along with a Taxi Zone Shapefile defining the polygon boundaries of all 263 taxi zones.

We use the lookup table to identify graph nodes and derive the adjacency matrix from the spatial relationships between polygons in the Shapefile. Specifically, an undirected edge is established between two distinct zones if their geographical boundaries touch, including shared edges or vertices. This produces a fixed binary adjacency matrix without self-loops.

We then extract the corresponding 80 × 80 adjacency submatrices for the training and unseen-region evaluation sets, respectively. The graph topology remains fixed over time and is independent of observation timestamps.

Observation timestamps.

We construct irregular observations in two steps: generating observation timestamps and aggregating event records at those timestamps.

First, for each region, we independently sample inter-observation intervals from an exponential distribution and cumulatively add them to generate non-uniform timestamps. For NYCtaxi and NYCbike, the exponential scale parameter is 60 minutes, with sampled intervals clipped to 10–360 minutes and rounded to integer minutes.

At each generated timestamp, we compute the cumulative regional inflow and outflow from 00:00 of the corresponding day, using the actual pickup and drop-off records.

For NYCcrime, the same general procedure is applied on a coarser temporal scale (1–16-day intervals, approximately 4 days on average), with cumulative crime counts calculated from the beginning of each sliding window.

Since timestamps are generated independently for different regions, the resulting observations are irregular and asynchronous across nodes. We will clarify these steps and the sampling parameters in Appendix A.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.