- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper focuses on irregular spatiotemporal forecasting on graphs, using a frozen LLM as the forecasting backbone. The paper argues that existing LLM-based forecasting methods rely on regularly sampled token sequences, making it difficult to handle asynchronous and sparse observations, while also suffering from representation misalignment and context-length limitations. To address these issues, the paper proposes LLMODE, which uses a pretrained graph-aware neural ODE to encode irregular observations, compresses the resulting latent trajectories and segment-wise statistics into a small number of memory tokens, and injects them into a frozen Vicuna-7B through gated cross-attention to produce forecasts via regression. Within the paper's experimental settings, LLMODE achieves competitive results on three NYC urban datasets and two particle dynamics benchmarks, generally outperforms the baselines in zero-shot region transfer, and substantially reduces token usage and inference latency compared with UrbanGPT.
Critical strengths
1. Using continuous-time models to enable LLMs to process asynchronous and sparse spatiotemporal observations is a meaningful and relatively underexplored direction. I therefore appreciate the paper's problem setting and motivation.
2. The experiments cover five datasets across urban and physical domains, 11 baselines, and zero-shot settings, with ablations examining token budgets, evidence selection, injection mechanisms, backbone replacement, and solvers. I therefore appreciate the breadth of the experimental evaluation.
Critical weaknesses
1. LLMODE's performance advantages cannot be attributed to the frozen LLM, leaving the paper's central claim unsupported. According to the provided code, the ODE encoder in the first stage is trained not only with the two loss terms in Eq. (14), but also through a prediction head supervised by observations within the prediction horizon. In other words, the ODE encoder itself is already trained as a predictor for the same target, yet the paper neither explains this nor reports the standalone performance of this predictor. Therefore, the reported results cannot distinguish whether the performance advantages come from the frozen LLM itself or from the ODE encoder.
2. The implementation of a key novel module does not match its description in the paper. The core design of the Resampler is to use the magnitude of the ODE vector field, , as an attention bias (Eq. (6)) to preserve states from intervals of rapid change. However, in the provided code, this bias is actually computed as the norm of the difference between adjacent latent states after a learnable projection, and is also learnable rather than fixed at 0.1 as described in Appendix A.3. Therefore, the ablation removing the dynamic bias in Figure 5, as well as the paper's conclusions about dynamics-aware compression, does not validate the design described in the paper.
Questions for authors
1. How is the missing rate in Figure 3 defined, what missing rate does the default Springs setting in Appendix A.1 correspond to, and do the Springs results in Table 1 use this default setting?
2. Table 7 shows that removing compression barely changes latency, and only and outperform UrbanGPT on both metrics. How much of the reported 7.7$\times$ speedup over UrbanGPT is attributable to fixed-budget compression? How was selected?
3. Please clarify the following:
(a) Does the latent trajectory fed into the Resampler contain states within the prediction horizon? What is the standalone forecasting accuracy of the first-stage encoder together with its prediction head?
(b) In Figure 5, Random-5 performs several times worse than LLMODE, and even worse than the standalone GRU-ODE in Table 1, despite using states from the same pretrained ODE. Why is this the case?
(c) What exactly is the prediction target on NYC? Appendix A.1 defines it as cumulative flow since midnight of the current day, whereas Appendix A.16 defines it as the subsequent 12 regular time steps. Figure 2 instead shows irregular query times, and the ground truth in Figure 8 also does not appear to be cumulative flow that resets daily.
14 comments
W2: Clarification of Dynamics-Aware Bias and Ablation Validity
We thank the reviewer for carefully examining our implementation. We clarify the relationship between the continuous-time formulation and its discrete implementation, the setting of the bias coefficient , and the interpretation of the existing ablation results.
1. Continuous-time formulation and discrete approximation.
Eq. (6) describes the dynamics-aware bias using the magnitude of the continuous-time ODE vector field.
In implementation, we use first-order differences between adjacent latent states as a discrete approximation of the continuous-time derivative. Since the ODE solver uses a fixed integration step, these differences provide a practical measure of temporal variations without explicitly evaluating the vector field.
The differences are computed after feature projection because the ODE latent states are mapped into the Resampler's feature space for alignment with the LLM. This allows the bias to operate on the same representations used for attention and memory compression.
For equally spaced latent states, the projected difference approximates the projected ODE derivative up to a constant time-step factor, which cancels under maximum-based normalization. Thus, the difference-based score is a discrete approximation of continuous-time dynamics in the projected space, rather than an unrelated mechanism. We will clarify this approximation and its conditions in the revised manuscript.
2. Setting of the bias coefficient .
We initially adopted a learnable parameterization to adjust the strength of the dynamics-aware bias. During training, we observed only minor changes in this coefficient, suggesting limited sensitivity to its optimization under the evaluated settings.
This motivated our subsequent adoption of a fixed coefficient of for implementation simplicity. We will clarify the parameterization and corresponding experimental configurations in Appendix A.3.
3. Interpretation of the dynamics-aware ablation.
Figure 5 isolates the contribution of dynamics-aware attention guidance by removing the bias while retaining the same underlying Perceiver resampling mechanism. The resulting performance degradation supports our central design motivation that incorporating temporal dynamics into attention improves forecasting performance.
Since the implemented finite differences approximate the continuous-time dynamics underlying our formulation, this experiment provides evidence for the proposed dynamics-aware design principle, although it does not independently validate the exact vector-field norm.
We will refine Eq. (6) and Section 3.2 to clarify the relationship between the theoretical formulation and its numerical implementation, and update Appendix A.3 to accurately describe the bias coefficient and experimental configuration.
Q1: Definition of Missing Rate and Default Springs Setting
We thank the reviewer for requesting this clarification. In Figure 3, the missing rate is defined as the proportion of unobserved particle–time positions within the eligible history window, i.e., .
As described in Appendix A.1, the Springs dataset uses a history window of 60 sampled time steps, with approximately 24 observations retained per particle on average. This corresponds to a nominal missing rate of 60%, although the realized rate varies with the randomly sampled observation counts.
Figure 3 further evaluates missing rates of 20%, 40%, 60%, and 80% to examine robustness under different observation densities.
The Springs results in Table 1 use the same default observation-sampling protocol, corresponding to approximately 60% missingness.
We will explicitly clarify the missing-rate definition, default setting, and its relationship to Figure 3 and Table 1 in Appendix A.1.
Q2: Efficiency Gains and Selection of K
We thank the reviewer for this question. As shown in Table 7, removing compression increases inference latency, but the increase is relatively small. To understand this observation, we conducted an additional module-level latency analysis on NYCbike (batch size = 1, same GPU).
| Component | Latency (ms/sample) |
|---|---|
| ODE Solver | 0.6033 |
| Context Encoder | 4.8565 |
| Resampler | 1.1124 |
| LLM Processing | 77.4478 |
| Total | 84.0200 |
The results show that LLM processing dominates inference latency (92.2%), whereas the Resampler accounts for only 1.3%.
This helps explain the limited latency reduction from compression observed in Table 7: compression reduces the memory tokens processed by cross-attention, but does not directly shorten the LLM backbone's self-attention sequence, where much of the computational cost lies.
The 7.7× speedup over UrbanGPT therefore reflects a broader architectural advantage.
UrbanGPT incorporates spatio-temporal tokens into the LLM input sequence, increasing self-attention overhead. In contrast, LLMODE injects dynamic and context memories through gated cross-attention without extending the backbone's self-attention sequence.
Fixed-budget compression further reduces cross-attention overhead, but is not the primary explanation for the overall speedup.
Regarding K selection, our initial design goal was to achieve substantial compression to reduce inference overhead. We therefore evaluated different memory-token budgets to examine their effects on forecasting accuracy and latency, as reported in Table 7.
The results show that appropriate compression can improve forecasting accuracy while reducing computational overhead. We hypothesize that this is because consecutive ODE states contain substantial temporal redundancy, and the dynamics-aware Resampler can selectively aggregate informative temporal variations while reducing redundant information.
However, excessive compression may discard useful dynamics, whereas larger K values increase computational cost without consistent accuracy improvements. Based on this trade-off between prediction accuracy and inference efficiency, we selected K=5 as the default setting.
We sincerely thank the reviewer for the detailed review and constructive feedback. We respond to each point below.
We thank the reviewer for this important observation. We clarify the Stage-1 training objective, the construction of latent trajectories, and the standalone forecasting performance of the ODE encoder.
1. Stage-1 training and latent trajectories.
The Stage-1 objective follows the pre-update and post-update loss formulation in Eq. (14). In our implementation, the post-update loss additionally incorporates forecasting supervision through an auxiliary prediction head, which predicts future latent states and is trained against observations within the prediction horizon.
During inference, the trajectory provided to the Resampler includes these model-generated future latent states, without using future ground-truth observations for hidden-state updates. We will make this implementation detail more explicit in Appendix A.2.
2. Standalone ODE forecasting performance.
To assess the forecasting capability of the first-stage encoder independently, we evaluate the ODE encoder together with its prediction head, without the Resampler or LLM. The NYCtaxi supervised results are summarized below.
The complete LLMODE achieves lower prediction errors than the standalone ODE predictor, indicating that the downstream forecasting framework provides additional benefits beyond the first-stage encoder. Furthermore, our controlled comparison between pretrained and randomly initialized frozen Vicuna-7B backbones provides complementary evidence for the contribution of language pretraining.
We will clarify the Stage-1 objective and latent-trajectory construction in Appendix A.2 and include the standalone ODE comparison in the revised experimental results.