- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors provide LUNA, a state-estimation framework that combines a frozen language model with classical Kalman filtering. It propagates sigma points through a learned transition model, predicts additional uncertainty using an evidential regression head, and corrects the resulting state estimate using new observations through a UKF update. Experiments on six synthetic and real-world benchmarks show promising improvements in estimation accuracy and predictive likelihood, alongside cross-system transfer results.
Critical strengths
1- they are addressing an important problem by combining learned state prediction, uncertainty propagation, and classical measurement correction. The predicted covariance directly affect how the filter balances model predictions against new observations.
2- LUNA achieves the good reported RMSE on five of six benchmarks, including a tie, and the best NLL on four.
3- The experiments examine cross-system transfer without parameter updates, using target-system context, and robustness to changing dynamics and noise which is insightful.
4- The paper evaluates multiple backbones, uncertainty components, and transition-network choices. These comparisons help identify which design choices contribute to performance and shows that the framework can also support simpler prediction models.
Critical weaknesses
I list the weaknesses as follow:
1- At each sigma point, the aleatoric and epistemic covariance terms share the same matrix structure and differ only by a scalar factor. Please explain how they enter the predictive likelihood and what allows their separate identification (to me they look identical and referring your figure 3, the epistemic vs full covariance differences is marginal). Controlled experiments would also make your interpretation stronger: with the underlying noise fixed, increasing relevant training data should generally reduce epistemic uncertainty (that I cannot see in the results), while aleatoric uncertainty should approach the underlying process-noise level (how to manipulate it as it stems from your LLM function approximator?).
-
The method retains the classical filtering (the whole storyline), while introducing a learned prediction model (by your proposed LLM). Please explicitly present the transition likelihood, observation likelihood, and recursive posterior, and identify where the learned transition, sigma-point approximation, and Gaussian approximation enter. This would clarify what is exact under the assumed model and what is approximated. If you discuss it appropriately, then the previous point regarding covariances would clarified as well.
-
Related to the previous point, please clarify whether the NIW distribution represents a learned conditional distribution over transition means and covariances, or a posterior obtained through an explicit Bayesian update (after 3 times reading, still unclear to me). This distinction matters for interpreting its components as process noise and uncertainty about the transition model. A clearer derivation would support these interpretations.
-
Although the experiments include Lorenz96 and VL20, etc., the method assumes a known, fixed-dimensional state representation. Its behavior as state dimension increases, or when the number of state variables changes, remains unclear, specially when you use strong function approximator like LLM, such extension worth a check. Please discuss this limitation and the associated computational scaling. A simple example would be inputs as images containing a varying number of moving objects that additionally require perception and state discovery (with reconsideration of your query perhaps as images rather than exact states).
-
Please report the number of independent training runs and provide means with standard deviations or confidence intervals for the main comparisons. Without these statistics, it is difficult to judge whether the reported improvements are stable, particularly when numerical differences are small (I checked your appendix, but could not find it, otherwise please point me to the correct location in appendix).
-
The statement that FM-UKF’s prediction step provides no uncertainty quantification should be revised because FM-UKF does sigma-point trajectories through its foundation model and reconstructs predictive covariance using the unscented transform. The statements regarding the baselines need to be fair and accurate.
Questions for authors
My questions are mainly embedded in the weaknesses please see them above.
Besides them, there is one more question:
You are modeling the prediction step with strong function approximator which is LLM. But you switch to conventional classic kalman formulation for update, after observing your observation. Why not apply your context aware LLM for update formulation as well?
6 comments
Summary of Paper Revision & Detailed Responses to Reviewer bp7h(2/4)
. Likelihoods, recursive posterior, and approximation steps
Probability model and approximations. LUNA learns the transition likelihood , uses the known observation likelihood , and updates the recursive posterior through Bayes' rule. Appendix A.1, Eq. 10 presents these objects together and labels their corresponding modules. The LLM–NIW model specifies the transition likelihood, sigma points approximate predictive moments over the previous recursive posterior, and Gaussian approximations represent both the prediction distribution and recursive posterior Eq\. 11. The Bayesian relations and total-covariance identity Eq\. 13 are exact under the learned model, while the learned transition and moment/distribution approximations make the implemented filter approximate.
. Meaning of the learned NIW distribution
Learned conditional distribution. NIW represents a learned conditional distribution over transition means and covariances, rather than a posterior obtained through an explicit Bayesian update. Its parameters are predicted from the input state and system context by modules trained offline. During filtering, the UKF updates the posterior over the state, without performing an explicit Bayesian update of the NIW distribution. Appendix A.1 presents the filtering recursion, and A.3 defines the input-dependent NIW distribution.
. Variable state dimensions and computational scaling
Changing state dimension. OOn four datasets provided by LLMFilter, we evaluate each trained model at its original and reduced state dimensions without retraining. Full uses the original state dimension. Half and Quarter retain the first half and first quarter of the state dimensions in their original order, respectively. For Lorenz96, these correspond to the first 36 and first 18 dimensions of the original 72-dimensional state. Table 15 in Appendix E.3 reports RMSE and joint NLL, with lower values indicating better performance at the same state dimension. LUNA maintains similar estimation accuracy across the tested dimensions, with RMSE differing from the full-state result by less than 0.9%.
Table 15 in Appendix E.3: State dimension sensitivity.
| Dataset | Full, RMSE | Full, NLL | Half, RMSE | Half, NLL | Quarter, RMSE | Quarter, NLL |
|---|---|---|---|---|---|---|
| Oscillator | 0.803 | 2.390 | 0.796 | 1.187 | N/A | N/A |
| Hopf | 0.769 | 2.315 | 0.764 | 1.151 | N/A | N/A |
| Lorenz96 | 4.017 | 202.752 | 3.987 | 101.176 | 3.983 | 50.579 |
| VL20 | 0.881 | 93.137 | 0.879 | 46.513 | 0.880 | 23.274 |
N/A indicates that the quarter-state condition was not evaluated for the two-dimensional systems.
Computational scaling. A d-dimensional state requires 2d+1 sigma-point transition evaluations and d² entries per full covariance. The separate timing study uses six backbones on 2D Oscillator and 72D Lorenz96, with synchronized complete-step timing and peak allocated memory . It shows higher latency and nearly constant backbone-dominated memory at these operating points. Because system identity also changes, this comparison is not an isolated dimension-scaling experiment.
Table 18 in Appendix E.4: Runtime and memory for six backbones.
RTX 4090.
| Backbone | Oscillator time | Oscillator memory | Lorenz96 time | Lorenz96 memory |
|---|---|---|---|---|
| MLP | 3.513 | 0.013 | 9.627 | 0.053 |
| RNN | 3.831 | 0.021 | 11.736 | 0.378 |
| Transformer | 4.054 | 0.038 | 18.443 | 0.530 |
| TimesFM-2.5 | 18.104 | 0.941 | 24.398 | 0.987 |
| TimesFM-3.0 | 20.281 | 1.338 | 27.631 | 1.374 |
| LLaMA-7B (LUNA) | 20.488 | 13.251 | 35.723 | 13.316 |
H800.
| Backbone | Oscillator time | Oscillator memory | Lorenz96 time | Lorenz96 memory |
|---|---|---|---|---|
| MLP | 5.294 | 0.038 | 13.512 | 0.078 |
| RNN | 5.638 | 0.063 | 15.868 | 0.420 |
| Transformer | 6.173 | 0.063 | 16.990 | 0.555 |
| TimesFM-2.5 | 25.840 | 0.966 | 34.566 | 1.012 |
| TimesFM-3.0 | 29.305 | 1.363 | 40.316 | 1.399 |
| LLaMA-7B (LUNA) | 22.505 | 13.276 | 30.603 | 13.341 |
Images and changing object counts. The coordinate-subset tests do not evaluate images with varying object counts or online state discovery. Those settings require perception and object identity tracking beyond the predefined state representation. Section 5 discusses this distinction, the restriction to reduced dimensions, and the need for more scalable uncertainty propagation.
Summary of Paper Revision & Detailed Responses to Reviewer bp7h(3/4)
. Stability across independent training runs
Training stability. Using the datasets provided by LLMFilter, we initialize and train each learned method's trainable modules with five seeds. Tables 13–14 of Appendix E.3 report mean ± standard deviation of RMSE and NLL, with lower means better. LUNA maintains stable estimation accuracy, with RMSE standard deviations of 0.4–2.3% of the corresponding means, and achieves the lowest mean RMSE and NLL among learned methods on all five datasets.
Table 13 in Appendix E.3: Five-seed RMSE comparison on simulated datasets.
| Method | Oscillator | Hopf | Tracking | Lorenz96 | VL20 |
|---|---|---|---|---|---|
| EKF† | 0.801 | 0.768 | 1.299 | 3.096 | 0.785 |
| UKF† | 0.798 | 0.768 | 1.299 | 3.081 | 0.785 |
| FM-UKF | 11.127 ± 2.530 | 12.984 ± 0.398 | 5.738 ± 0.363 | ||
| LLMFilter | 3.573 ± 0.053 | 2.009 ± 0.037 | 14.827 ± 0.701 | 6.792 ± 0.008 | 6.589 ± 0.014 |
| LLMFilter+UKF | 1.955 ± 0.028 | 1.133 ± 0.018 | 13.324 ± 0.780 | 6.722 ± 0.006 | 3.321 ± 0.004 |
| MOP | 0.983 ± 0.004 | 0.893 ± 0.008 | |||
| LUNA (Ours) | 0.805 ± 0.005 | 0.773 ± 0.003 | 1.947 ± 0.045 | 4.045 ± 0.036 | 0.882 ± 0.012 |
Table 13 in Appendix E.3: Five-seed NLL comparison on simulated datasets.
| Method | Oscillator | Hopf | Tracking | Lorenz96 | VL20 |
|---|---|---|---|---|---|
| EKF† | 2.39 | 2.31 | 6.35 | 180.57 | 84.73 |
| UKF† | 2.38 | 2.31 | 6.35 | 180.28 | 84.73 |
| FM-UKF | 4371.01 ± 296.79 | 1102.54 ± 135.52 | |||
| LLMFilter | N/A | N/A | N/A | N/A | N/A |
| LLMFilter+UKF | 8.68 ± 0.21 | 3.69 ± 0.08 | 1578.96 ± 376.40 | ||
| MOP | N/A | N/A | N/A | N/A | N/A |
| LUNA (Ours) | 2.40 ± 0.01 | 2.33 ± 0.01 | 8.37 ± 1.27 | 203.25 ± 0.90 | 93.23 ± 0.99 |
N/A indicates that no covariance-based NLL is reported.
Real-world variability. In Table 12, we evaluate RMSE and NLL on UTIAS and KITTI over five independent training runs. Entries are mean ± standard deviation, with lower values better for both metrics. The UTIAS results below show larger variability than the datasets provided by LLMFilter: LUNA's RMSE is 0.711 ± 0.063, with a standard deviation of approximately 8.9% of its mean, and NLL also varies more across runs. LUNA still achieves the lowest mean RMSE and NLL among learned methods.
Table 12 in Appendix E.3: RMSEs on two real datasets UTIAS and KITTI results.
| Method | UTIAS<br>RMSE | UTIAS<br>NLL | KITTI<br>RMSE | KITTI<br>NLL |
|---|---|---|---|---|
| EKF† | 0.246 | 2.140 | 2.980 | 6.920 |
| UKF† | 0.235 | 1.800 | 2.980 | 6.920 |
| FM-UKF | 0.862 ± 0.059 | 20.780 ± 10.893 | 3.260 ± 0.807 | 9.550 ± 3.099 |
| LLMFilter | 1.270 ± 0.015 | N/A | 4.310 ± 0.002 | N/A |
| LLMFilter+UKF | 1.250 ± 0.013 | 209.000 ± 3.710 | 4.050 ± 0.017 | 11.400 ± 0.082 |
| MOP | 2.110 ± 0.009 | N/A | 9.840 ± 3.340 | N/A |
| LUNA(Ours) | 0.711 ± 0.063 | 18.900 ± 8.900 | 2.790 ± 0.603 | 5.420 ± 2.770 |
. Fair characterization of FM-UKF
FM-UKF uncertainty quantification. The Introduction and Related Work now acknowledge that FM-UKF reconstructs predictive covariance from sigma-point trajectories using the unscented transform. The intended distinction is that LUNA additionally estimates input-dependent uncertainty in the learned transition through NIW, rather than suggesting that FM-UKF provides no prediction-side UQ. Appendix A.1 makes these two covariance sources explicit.
Summary of Paper Revision & Detailed Responses to Reviewer bp7h(1/4)
Summary of Paper Revision
We have revised the paper to address your concerns as follows:
For W1, we have clarified the additive covariance reconstruction in Appendix A and added fixed-noise, varying-data experiments in Section 4.3 (Table 3).
For W2 & W3, we have reorganized Appendix A.1 to present the likelihoods, recursive posterior, and approximation steps, and clarified the learned conditional NIW distribution in Appendix A.3.
For W4, we have added state-dimension experiments in Appendix E.3 (Table 15), runtime and memory measurements in Appendix E.4 (Table 18), and a discussion of dimensional limits and visual state discovery in Section 5.
For W5, we have added five independent training runs with mean and standard deviation in Appendix E.3 (Tables 12–14).
For W6, we have revised the Introduction and Related Work to distinguish FM-UKF's propagated uncertainty from uncertainty in the pretrained model's predictions.
For Q1, we have added Average fusion and Learned-gate fusion experiments in Appendix E.3 (Table 16) and discussed observation-side uncertainty in Section 5.
Detailed Responses
We thank the reviewer for the constructive feedback. We address each concern below with additional experiments, analysis, and clarifications.
W1. NIW component identification and data dependence
Predictive likelihood. LUNA incorporates the aleatoric and epistemic terms into the predictive likelihood through their summed covariance, estimated in one NIW forward pass. This additive reconstruction follows the law of total covariance Appendix A\.1, Eqs\. 13–14, with the derivation in Appendix C\.2.
Separate identification. Their shared matrix structure follows from NIW, and the predictive likelihood alone does not uniquely identify their split [1, 2]. NIW defines them as the expected transition covariance and variation in the transition mean, respectively, with $\boldsymbol{\Sigma}^{epi} = \frac{ \boldsymbol{\Sigma}^{ale}}{κ} $ AppendixA3˙. Figure 3's small gap indicates a modest additional contribution from the epistemic term in that setting.
Fixed-noise, varying-data experiment. In Table 3, we follow the work [3] and measure the test-set mean dtr(Σepi) and tr(Q)tr(Σale), where tr(Σepi) and tr(Σale) indicate the total epistemic and aleatoric variances, respectively, d is the state dimension, and tr(Q) is the total variance of the true process noise. By varying the training samples eg˙,˙5 on four datasets, we clearly find that the 100% training samples lead to dtr(Σepi) close to zero, indicating the least epistemic uncertainty, and tr(Q)tr(Σale) close to the true process noise level.
Table 3 in Section 4.3: NIW uncertainty versus training data.
Epistemic: dtr(Σepi).
Aleatoric: tr(Q)tr(Σale).
[1] Nis Meinert and Alexander Lavin. Multivariate Deep Evidential Regression. arXiv:2104.06135, 2021.
[2] Nis Meinert, Jakob Gawlikowski, and Alexander Lavin. The Unreasonable Effectiveness of Deep Evidential Regression. AAAI, 2023, pp. 9134–9142. DOI: 10.1609/aaai.v37i8.26096.
[3] Luís Marques and Dmitry Berenson. Quantifying Aleatoric and Epistemic Dynamics Uncertainty via Local Conformal Calibration. arXiv:2409.08249, 2024.