acceptodds
Under review as a conference paper at ICLR 2027

K-PRISM: Kalman-based Physics-informed Residual Innovation with Sensor Multimodality

Abstract

Classical Kalman filtering is optimal under known linear-Gaussian dynamics, but real systems exhibit partially-known dynamics, unknown observation models, and time-varying sensor reliability. Neural-augmented filters such as KalmanNet relax the need for known noise statistics, yet rely on recurrent gain estimation that adapts slowly to abrupt sensor degradation and tends to memorize training-specific spatial corrections. We propose K-PRISM (Kalman-based Physics-informed Residual Innovation with Sensor Multimodality), which (i) decomposes the state transition into a known physics prior and a learned residual, (ii) computes a translationally invariant gain from instantaneous velocity and innovation features, enabling single-timestep adaptation without spatial overfitting, and (iii) extends to heterogeneous multi-sensor fusion via cross-modal gating. The resulting update generalizes the structure of the classical multi-sensor Kalman update, with Lipschitz-bounded innovations and formal error contraction. Empirically, K-PRISM matches or exceeds KalmanNet's accuracy while using to fewer parameters and eliminating its catastrophic-failure mode. On the Lorenz attractor, it matches steady-state accuracy while recovering from sensor dropout with zero failures; in a controlled two-sensor study under alternating reliability, the cross-modal gate identifies the informative sensor online, beating independent fusion by dB. On real-world NCLT localization it converges in every run, whereas our KalmanNet reproduction diverges in , and it transfers to visual-odometry velocity filtering on EuRoC. These results position compact, translationally invariant filtering as a robust alternative to recurrent gain estimation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
state estimation, Kalman filtering, neural-augmented filtering, sensor fusion, physics-informed learning, translational invariance, multimodal fusion, parameter efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zCn7OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1(−1)
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes K-PRISM, a neural-augmented Kalman-style filter for partially known dynamics, unknown observation models, and time-varying sensor reliability. The prediction combines a known physics prior f0f_0 with a learned residual fΔf_\Delta that sees only position-independent quantities. The gain is a feedforward MLP conditioned only on velocity features and the current innovation, which the authors argue yields translational invariance and single-timestep adaptation. For multiple sensors, a joint gating network computes normalized per-sensor, per-dimension weights from the context of all sensors. The paper also gives a convex-combination interpretation of the update, a Lipschitz bound on the sensor heads, and an error-contraction result under a lower bound γmin⁡\gamma_{\min} on the total gain. Experiments on the Lorenz attractor, NCLT, and EuRoC show accuracy comparable to or better than KalmanNet with far fewer parameters and more stable training.

Critical strengths

1. K-PRISM achieves accuracy comparable to or better than KalmanNet (29K-107K parameters) with only about 0.5K-1.4K parameters. This is appealing in terms of parameter efficiency and training stability.

2. The authors openly acknowledge that K-PRISM does not surpass the published KalmanNet result, that it is inferior to KalmanNet in the Lorenz steady-state setting, and that most of the performance gain comes from the residual. They also report their KalmanNet reproduction results per seed, which increases the credibility of the comparison.

3. On the data-limited NCLT setting, the paper compares against LSTM and Transformer baselines at multiple capacities. K-PRISM substantially outperforms them even at the same parameter count, which illustrates the benefit of retaining the Kalman structure.

Critical weaknesses

1. Structural issue with the multi-sensor normalization (Eq. 12).

Since wt(i)=αt(i)∑jαt(j)+ϵw^{(i)}_t=\frac{\alpha^{(i)}_t}{\sum_{j}\alpha^{(j)}_t+\epsilon}, for small ϵ\epsilon we have ∑iwt(i)≈1\sum_{i}w^{(i)}_t\approx 1 unless all αt(i)\alpha^{(i)}_t are close to zero. The update then reduces to a weighted average of the sensor proposals, and the physics prior is effectively ignored. Figure 3, where the two weights sum to roughly 1, is consistent with this.

However, the learned gains on NCLT fusion are 0.05-0.10 (Table 13), which is difficult to reconcile with Eq. 12. Please clarify the value of ϵ\epsilon and which update equation is actually used in each experiment.

2. Translational invariance does not hold for the multi-sensor mechanism.

The gating context (Eq. 11) includes the encoder features ht(i)=ϕ(i)(zt(i))h^{(i)}_t=\phi^{(i)}(z^{(i)}_t). For a sensor that observes absolute position, such as GPS, these features depend on position. The claim that translational invariance holds for the full multi-sensor mechanism is therefore inconsistent with Definition 1.

3. Core claims are not validated on real data.

The only evidence for cross-modal gating is the synthetic two-sensor Lorenz study. In NCLT fusion, GPS is suppressed by the dataset's validity mask rather than by the learned gate. Likewise, the "unknown observation model" setting is not really tested, since all observations are low-dimensional and nearly identical to the state.

4. The contribution of the gain mechanism is unclear.

On NCLT GPS-only, the scalar gain (20.35 dB) and the fixed 0.5 gain (20.82 dB) perform similarly to K-PRISM (20.64 dB), and even the fixed gain outperforms KalmanNet (22.20 dB). On EuRoC, the gap between KalmanNet+res and K-PRISM is also small. The improvement appears to come mainly from the physics prior and the learned residual rather than from the proposed gain.

5. Missing related work.

Lee et al. (IROS 2020, "Multimodal Sensor Fusion with Differentiable Filters") address multimodal sensor fusion within differentiable filters. Recent works also approximate the Kalman gain with a learned gating network (e.g., arXiv:2602.11183). Neither line of work is discussed, which makes it difficult to assess the novelty of the proposed method.

Questions for authors

Q1. What value of ϵ\epsilon is used in Eq. 12? Could you show the distribution of ∑iwt(i)\sum_{i}w^{(i)}_t after training, to indicate how far it deviates from 1 in practice? In addition, please clarify whether the NCLT fusion experiment uses Eq. 9 or Eq. 12. If GPS is the only correcting sensor, how can the learned gain converge to 0.05-0.10?

Q2. The gating context includes encoder features from sensors that observe absolute position. In what sense is the multi-sensor mechanism translationally invariant? Do you have results with these features removed from the context?

Q3. How does K-PRISM differ from the multimodal differentiable filter of Lee et al. (IROS 2020) and from recent works that approximate the Kalman gain with a learned gating network (e.g., arXiv:2602.11183)?

2 comments
AuthorsOpenReview ↗

Response to reviewer zCn7

Thank you for highlighting the compactness of K-PRISM and the points requiring clarification. We have revised the paper to distinguish its compact physical-state estimator from the broader properties that the experiments do not establish.

Q1. Normalization, epsilon and the NCLT update

The normalized two-sensor Lorenz gate uses ε = 10⁻⁶ in Eq. (12), separately for each state coordinate. Its direct prior coefficient is

1−∑iwt,k(i)=ϵ∑iαt,k(i)+ϵ.1-\sum_i w_{t,k}^{(i)}=\frac{\epsilon}{\sum_i\alpha_{t,k}^{(i)}+\epsilon}.

Thus, when summed raw confidences are appreciable relative to epsilon, the direct prior weight is nearly zero. Section 3.3 and the discussion of Proposition 1 now state this explicitly. We have removed the claim that this normalization preserves a meaningful prior contribution or establishes rejection of all unreliable sensors. No post-training weight distribution is reported, so we do not infer the requested quantiles or below-0.9 frequency from the algebra or the plotted average traces.

NCLT fusion instead uses the unnormalized sigmoid gain of Eq. (9) with the prediction-correction update in Eq. (10), not Eq. (12). Wheel/IMU measurements drive prediction; GPS is the correcting observation, and the supplied GPS-validity mask suppresses missing updates. Table 13 reports constant per-coordinate gains from the scalar-gain fusion variant, which uses the same unnormalized, masked GPS correction. Nothing forces these gains to sum to one, so values of 0.05–0.10 are consistent with that variant (Appendix F.5). Sections 3.3 and 4.2.2 and Appendix F make this distinction explicit.

Q2. Translational invariance and encoder features

We have corrected the unconditional multi-sensor invariance claim. In Eq. (11), an encoder of absolute-position measurements can retain position information. Omitting a separate position input therefore does not establish invariance of the generic joint gate.

The narrower position-independent input construction concerns the specified NCLT/EuRoC implementations. The original synthetic position-toggle comparison keeps encoded observations in both variants and changes parameter count. Appendix C and Table 6 now describe it as the effect of an additional explicit position input, not a parameter-matched feature-removal experiment. We do not claim that removing the encoder features is cost-free; no such matched ablation is reported.

Q3. Relationship to the cited methods

Lee et al. (2020) learn crossmodal weights for combining unimodal filter estimates/covariances or particle-filter measurement contributions. K-PRISM instead corrects a common predicted physical state using sensor proposals, without propagating a posterior covariance. We have added this work and the distinction in Section 2.2; crossmodal weighting itself is not claimed as new.

NeuroKalman uses a feedforward sigmoid elementwise gain with a recurrent prior and visual-memory measurements in a latent navigation representation. K-PRISM emphasizes an explicit nominal physics prior plus a learned residual in a compact physical-state estimator. Section 2.2 now discusses this overlap and distinction. We do not claim novelty for feedforward gains alone or superiority over either cited method.

Scope of the real-data and gain-contribution claims

The contribution is the compact combination of an explicit nominal physics prior, learned residual and physical-state correction. We have narrowed the abstract, contributions, conclusion and real-data claims accordingly. The real-data tests use known observation structure, and NCLT supplies a validity mask; they do not establish autonomous joint reliability estimation. We also no longer present the existing comparisons as isolating the gain architecture’s independent advantage. The GPS-only values cited in the review are now confined to historical appendix diagnostics (Appendix F.3), not matched-accuracy evidence: the cited KalmanNet reference uses odometer-velocity observations, whereas the GPS loader used full-session ground truth before splitting (Appendix F.6). No new GPS comparison is included.

I thank the authors for their detailed response and transparent corrections. However, the response has reinforced several of my concerns.

  1. Lack of real-world validation of the multi-sensor architecture. The authors have confirmed that the NCLT fusion experiment uses the single-sensor gain (Eq. 9) rather than Eq. 12. Consequently, the joint gating mechanism (Eqs. 11–13), which is central to both the title and the proposed methodology, has been evaluated only on synthetic Lorenz experiments.

  2. Lack of controlled evidence for translational invariance. The clarification that the comparison in Appendix C was not parameter-matched removes the evidence previously used to rule out model capacity as a confounding factor in the NCLT "w/o invariance" ablation. As a result, there is no remaining controlled evidence supporting the claim that translational invariance improves generalization.

  3. Withdrawal of the NCLT GPS-only results. I consider it a serious concern that the reported KalmanNet results were obtained under a different observation setting and that the GPS data loader used ground-truth information from the entire session before splitting the data. Since these results supported a central claim in the abstract and the comparisons against LSTM and Transformer baselines, all affected results should be removed or recomputed without data leakage.

  4. Reduced scope of the contribution. After these corrections, the remaining contribution is a compact estimator combining an explicit physics prior, a learned residual, and a feedforward gain. This combination substantially overlaps with existing work, and the supporting real-world evidence is limited to NCLT fusion and EuRoC, with most of the performance improvement on EuRoC attributable to the learned residual.

I appreciate the authors' transparency throughout the rebuttal process. However, given the withdrawal of several central results and the limited novelty of the remaining contribution relative to prior work, I have decided to lower my rating.

Reviewer_LYfaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

K-PRISM combines a physics prior, learned residual dynamics, and feedforward diagonal gains, with a joint gating extension for multiple sensors. Experiments on Lorenz, NCLT, and EuRoC evaluate the authors' claims of parameter efficiency, faster adaptation to sensor degradation, and improved training stability relative to recurrent gain estimation.

Critical strengths
  1. The method retains a simple prediction–correction structure while substantially reducing the reported parameter count. This is a useful practical direction for learned filtering under resource constraints.
  2. The controlled two-sensor experiment and KalmanNet+res comparison provide useful evidence about sensor gating and the role of residual dynamics, making parts of the design easier to assess.
Critical weaknesses
  1. I am not yet convinced that the methodological contribution is sufficiently established. KalmanNet already uses difference-based gain inputs, so excluding absolute position does not by itself distinguish the proposed approach. The NCLT coordinate-conditioning ablations also change model capacity. In EuRoC, the residual accounts for much of the improvement: K-PRISM ties KalmanNet+res on MH_04 and differs by only 0.001 m/s on MH_05 (Table 5). These results leave the gain architecture's independent contribution unclear.

  2. The comparisons do not yet support the broader robustness claims. KalmanNet uses default, unregularized training, while K-PRISM receives additional training measures and a different budget (Section 4.2.2; Appendices F.2/F.6). The 47% versus 0% failure rates therefore cannot be attributed specifically to the gain architecture. Real-data validation is also narrower than the multi-sensor claim: NCLT supplies a GPS validity mask, and EuRoC filters VO velocity without IMU integration. Evidence for autonomous joint reliability gating comes mainly from the synthetic study.

Questions for authors
  1. How much improvement remains when recurrent and feedforward gains share the same residual model, input features, parameter budget, and training protocol? Variability estimates would help assess the differences.
  2. Does the NCLT stability advantage persist with comparable baseline tuning and training safeguards? Please report per-run outcomes and the failure criteria.
  3. Can the joint gate be evaluated on real sensor degradation without a supplied validity mask, against independent gating or an adaptive filtering baseline?
2 comments
AuthorsOpenReview ↗

Response to reviewer LYfa

Thank you for recognizing the practical value of a compact prediction-correction design. We have refocused the paper on that contribution and revised the comparisons to separate full-system results from claims about an individual component.

Q1. Independent contribution of the gain

The submitted experiments do not provide the requested comparison in which recurrent and feedforward gains share residuals, inputs, capacity and training. KalmanNet already uses difference-based inputs, and our coordinate-conditioning controls also vary capacity. In EuRoC, K-PRISM ties KalmanNet+res on MH_04 and differs by only 0.001 m/s on MH_05. In addition, the observation-refinement paths are not matched, and the submitted recurrent control contains an always-zero state-update feature. Its impact has not been isolated by a corrected rerun. Section 4.3 and Appendix G now present these as results of the tested full implementations, not an independent gain advantage or evidence of inherent recurrent instability. We cannot supply variability estimates for an unperformed matched control.

We have moved the original GPS-only scores to Appendix F.3 as historical diagnostics, rather than matched-accuracy evidence. The cited KalmanNet values concern odometer-velocity observations, not the GPS-position task, and the original loader fits alignment and rejects readings using full-session ground truth before splitting (Appendix F.6). Section 4.2.1 now summarizes these qualifications and directs the reader to the appendix.

We do not use the original GPS-only table or its scalar/fixed-gain rows to establish superiority over KalmanNet or isolate the gain mechanism. No new GPS performance comparison is added to this reply. The revised claim is the compact explicit-physics/residual design under the documented task-specific protocols.

Q2. NCLT fusion stability and failure accounting

The submitted 47% label combined distinct outcomes. Revised Table 8 (Appendix F.2) now labels three attempts as no checkpoint, two as nonfinite loss and two as large finite errors; the other eight have finite reported trajectory results. These categories are not pooled into a single divergence rate. We have omitted the unverified per-run Avg column, cross-run mean and dispersion, and derived best-result summary. The per-trajectory values are retained from the reported record, not newly log-verified results. We no longer claim elimination of catastrophic failure or attribute these outcomes to recurrence.

Whether a stability advantage persists under comparable tuning and safeguards is not established by the submitted comparison. No relative-accuracy or causal-stability conclusion is drawn from those local records. We explicitly acknowledge that the published KalmanNet result, 8.01 m, is more accurate than the reported K-PRISM mean of 9.43 m; the relevant contribution is the compact 536-parameter implementation and its observed accuracy-size trade-off, not superior accuracy over that published baseline.

Q3. Real-data joint reliability gating

The real-data experiments do not provide the requested no-mask joint-gating comparison. In NCLT, wheel/IMU inputs drive prediction, GPS is the correcting stream and the supplied mask suppresses missing GPS; EuRoC filters precomputed VO velocity without IMU integration. These distinctions are now explicit in Sections 3.3 and 4 and in the limitations.

The two-sensor Lorenz result remains a controlled proof of concept, not evidence that crossmodal context alone explains the 5.2 dB difference. Appendix B now states that the independent comparator uses fixed averaging, while the concatenated comparator changes the encoder/head; the implementations do not differ only in their gain. We retain this limited synthetic evidence without claiming autonomous real-sensor reliability estimation or promising an unperformed real-data experiment.

Thanks for the response from the author and parts of my concern have been addressed.

However, there is no further experiments regarding the above questions. I partially agreed with the author that since KalmanNet, the main referenced literature, doesn't provide practical experimental results in the original paper. However, I do think, experiments using real data are required to provide more persuasive analysis. Platforms with wheel encoders, IMUs and visual sensors can further expose the weakness of the proposed method, e.g., parameter tuning, etc.

Reviewer_sZmHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper developed a learning-augmented KF that learns a residual term and then uses it to assist sensor fusion.

Critical strengths

This paper works on their claimed dataset.

Critical weaknesses
  1. As a data-driven method, the proposed work is not demonstrated in a cross-dataset setting. Testing on an "in-distribution" dataset will lead to better results, without a doubt. Therefore, the result is not very informative.

  2. As a KF-type method, I did not see the formal guarantees provided by the KF framework. The uncertainty propagation is missing. The gating function also makes this even more difficult. The gating makes it even harder to deal with asynchronous sensors.

  3. The author does not test the proposed methods on more advanced Kalman-type filters for localization, such as the Invariant EKF and the unscented Kalman filter on SE(3).

Questions for authors

As in the weakness.

2 comments
AuthorsOpenReview ↗

Response to reviewer sZmH

Thank you for the review. We have narrowed the abstract, contributions and conclusion while retaining the evidence-supported case for a compact physics-augmented point estimator.

1. Cross-dataset generalization

NCLT fusion holds out complete sessions: 22 training, 2 validation and 3 test trajectories (Appendix F.1), following the released Song et al. partition. This tests unseen-session generalization within one sensing setup, rather than reusing training trajectories for testing. Task-specific within-dataset evaluation also appears in the original KalmanNet NCLT protocol (Revach et al., Section IV-E). Our Lorenz and EuRoC experiments use separately trained task-specific implementations. They demonstrate applicability of the design across tasks, not zero-shot reuse of a checkpoint. We have corrected the transfer wording accordingly. A frozen cross-dataset test would address a stronger claim, but its absence does not make the held-out-session evaluation uninformative.

2. Uncertainty, guarantees and asynchronous sensing

K-PRISM is a point estimator and does not propagate or calibrate a posterior covariance. We now distinguish this clearly from Kalman uncertainty propagation in Section 3.4 and the limitations.

The state-error bound is conditional on assumptions not guaranteed by the architecture, including a transition Lipschitz bound, a positive gain floor and bounded disturbance/proposal errors. Section 3.4 and Appendix A make these conditions explicit. The spectral-normalization result concerns the sensor head in feature space, not unconditional stability of the entire changing-gate filter.

Asynchronous and multi-rate fusion remain untested. We do not infer support for them from the gating formulation or from the synchronized experiments.

3. Invariant EKF and UKF on SE(3)

Invariant EKF and UKF on SE(3) are important baselines for broader localization claims. Our EuRoC experiment, however, is velocity filtering of precomputed stereo VO, not full visual-inertial SE(3) localization; IMU is not integrated. A meaningful comparison would need matched observations, outputs and evaluation. We have clarified this task in Section 4.3 and Appendix G rather than treating the current results as a comparison against those methods.

The revised paper claims a compact combination of explicit physics, learned residual correction and physical-state updates in the evaluated settings. It does not claim state-of-the-art general localization, calibrated uncertainty, asynchronous fusion or demonstrated zero-shot cross-dataset transfer.

Thank you!

I have an additional question: I see at least 40+ papers called PRISM. I wonder if this is suggested by AI?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.