acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Retrospective Streaming Generation of Physical Fields under Sparse Structured Sensing

Abstract

Reconstructing continuous physical fields from sparse measurements is central to scientific monitoring, inverse modeling, and digital-twin construction. Generative reconstruction has recently emerged as a promising paradigm for this task by learning data-driven physical priors that complete plausible full fields from limited observations. However, existing methods largely assume fixed, batch conditioning, whereas real sensing systems often produce structured streams: probes scan local regions, instruments observe moving fields of view, and communication constraints may leave entire frames missing. We propose TRACE, a retrospective streaming generative reconstruction framework for physical fields under structured sensing. TRACE performs approximate Bayesian inference in a learned continuous-coordinate latent space, converting sparse off-grid measurements into generative latent evidence, fusing it with a state-space temporal prior through Kalman-style filtering, and refining under-observed past frames via retrospective smoothing. Experiments on active matter, ocean sound-speed fields, and supernova simulations show that TRACE remains competitive with or surpasses frame-wise generative reconstructors, offline spatiotemporal methods, and streaming data-assimilation baselines in reconstruction quality under temporally sparse and spatially localized sensing protocols.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
generative reconstruction, physical fields, sparse sensing, state-space models, Kalman filtering, retrospective smoothing, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_PbxnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TRACE (Temporal Retrospective Accumulation for Continuous-field Estimation), a reconstruction framework for continuous physical fields designed for sparse, structured streaming data—such as sequences with temporally sparse or missing frames and spatially moving local observation windows.

  1. TRACE employs a Functional Tucker Model (FTM) to map continuous spatiotemporal physical fields into a compact, continuous-coordinate latent space, thereby enabling arbitrary off-grid spatial queries. During streaming inference, the model utilizes a pre-trained latent-space diffusion prior (EDM), combined with diffusion posterior sampling (DPS) and moment matching, to transform sparse observations from each frame into "Gaussian generative latent evidence." This evidence is then integrated with an i.i.d. Matérn-3/23/2 state-space Gaussian process (SS-GP) temporal prior via a "tempered Product of Experts" (PoE) mechanism within a Kalman filtering framework, providing causal, online estimation. As subsequent observations arrive, a Rauch-Tung-Striebel (RTS) backward smoothing pass is applied to retrospectively refine earlier frames that were poorly observed or missing.

  2. The framework was evaluated on three benchmark datasets—Active Matter (2D), ocean sound-speed fields (3D), and supernova explosions (3D)—under conditions involving intermittent temporal gaps (misses/blackouts) and spatially moving observation windows (S-curve or circular trajectories). The results demonstrate that TRACE significantly outperforms frame-by-frame reconstruction methods, offline spatiotemporal approaches (such as SDIFT and LRTFR), and streaming data assimilation baselines (DBF) in scenarios characterized by rapidly evolving dynamics or repetitive spatial scanning, while remaining highly competitive in scenarios with slower-varying dynamics.

Critical strengths
  1. By combining FTM continuous basis functions, DPS moment-matching Gaussian summaries, and Matérn-3/23/2 state-space Gaussian processes (SS-GP), TRACE elegantly transforms single-frame, non-Gaussian inverse problems into O(N)\mathcal{O}(N) Kalman filtering and RTS smoothing tasks within a linear Gaussian augmented state space. A diagonal covariance approximation (“diagonal fusion”) further reduces the single-frame fusion complexity from O(d3)\mathcal{O}(d^3) to O(d)\mathcal{O}(d), ensuring numerical stability and computational efficiency even in latent spaces with dimensions as high as d=32,768d=32,768 (as seen in the Supernova experiment).

  2. The paper provides a systematic model for common real-world sensor scenarios, such as “complete frame loss/disconnection” and “moving local scanning windows.” In the experimental section, the authors employ TRACE-Frame, TRACE-Filter, and TRACE-Smoother to clearly disentangle the respective contributions of single-frame priors, causal accumulation, and retrospective smoothing. Furthermore, the appendix demonstrates a high level of academic rigor and integrity—for instance, by proactively augmenting the MMGN baseline with latent-space linear interpolation (Appendix E.2), providing a detailed analysis of the implementation specifics and failure boundaries for DBF, EnSF, and LD-EnSF (Appendices E.4 and E.5), and offering an honest discussion of the method’s own limitations in scenarios involving long-duration, slowly-varying data gaps (Appendices F.9 and F.10).

Critical weaknesses
  1. TRACE applies a zero-mean stationary Matérn-3/23/2 prior independently to each latent space dimension, without learning cross-dimensional couplings or the true, non-stationary physical evolution operator. Consequently, during extended periods of continuous missing data (such as "Blk-10" in Table 1a), the forward filter relaxes toward the zero-mean prior, while the RTS smoother essentially functions as a bridge between the observed frames at either end of the gap. This explains why, under the "Blk-10" configuration for the Ocean and Supernova datasets, simple offline linear interpolation in the latent space (MMGN) outperforms the TRACE-Smoother (e.g., an error of 0.092 for MMGN versus 0.124 for the TRACE-Smoother on the Ocean Blk-10 case).

  2. While state-space filtering and fusion require only a few milliseconds (Appendix F.6, Table 16), obtaining reliable moment-matched Gaussian evidence necessitates running S=20S=20 parallel DPS reverse-diffusion trajectories—each with N=100N=100 steps—per frame. This process takes approximately 3.5 seconds per frame on a high-end RTX 5090 GPU. For high-frequency, real-time streaming sensing systems, this front-end evidence extraction step constitutes the primary latency bottleneck.

Questions for authors
  1. Currently, the per-frame generation evidence qngen(zn∣On)q_n^{gen}(z_n \vert{} \mathcal{O}_n) is obtained via DPS sampling from an unconditional prior, entirely independently of historical frames; it is only subsequently fused with the temporal predictive prior pn−(zn)p_n^-(z_n) as per Equation (15). When single-frame observations are extremely sparse or the viewing window is very narrow (as seen in the failure cases in Figure 15), single-frame DPS is prone to converging to incorrect local modes. Have the authors considered incorporating the temporal predictive distribution pn−(zn)=N(μn−,Σn−)p_n^-(z_n) = \mathcal{N}(\mu_n^-, \Sigma_n^-) directly as an additional Gaussian prior term within the DPS guidance gradient in Equation (12), or using μn−\mu_n^- to "warm-start" the reverse diffusion process? Such an approach could potentially improve single-frame identifiability while simultaneously reducing the number of diffusion steps NN required.

  2. In Equations (58)–(61) of Appendix A.5, the authors define a Markov transition matrix (Ai,Qi)(A_i, Q_i) over an augmented state ξ∈R2d\xi \in \mathbb{R}^{2d}. However, when deriving the bridge distribution via the method of completing the square in Equations (60)–(61), the 2d2d-dimensional matrix Q1−1A1Q_1^{-1} A_1 is multiplied by the dd-dimensional boundary-value means μk\mu_k and μk+1\mu_{k+1} (rather than the augmented smoothed means mksm_k^s and mk+1sm_{k+1}^s), with the result denoted as q(z(t∗))=N(μ∗,V∗)q(z(t^*)) = \mathcal{N}(\mu^*, V^*). Is there an inconsistent use of notation here (i.e., are equations (60)–(61) actually computed using the augmented state ξ∗\xi^* and the boundary augmented states (mks,mk+1s)(m_k^s, m_{k+1}^s), and subsequently projected onto z(t∗)z(t^*) via HH)? If so, could the authors please correct the notation and clarify whether the boundary smoothing covariances PksP_k^s and Pk+1sP_{k+1}^s are marginalized out when calculating E[z(t∗)]\mathbb{E}[z(t^*)]?

Reviewer_6RzJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE (Temporal Retrospective Accumulation for Continuous-field Estimation), a retrospective streaming generative reconstruction framework for physical fields, where sparse off-grid measurements arrive as temporally sparse frames or spatially localized moving-window observations. Fields are represented in a compact continuous-coordinate latent space via a Functional Tucker Model, converted into latent evidence by diffusion posterior sampling with a pretrained EDM prior, and fused with a Matérn-3/2 SS-GP prior through Kalman-style filtering, followed by Rauch-Tung-Striebel smoothing. Experiments on active matter, ocean sound-speed, and supernova simulations compare three variants of the method against LRTFR, MMGN, SDIFT, and DBF, showing competitive or improved field-domain RMSE.

Critical strengths

S1: The proposed method of structured sensing is practically motivated, the temporal inference is computationally efficient, and codes are provided for reproducibility.

S2: The TRACE-Frame/TRACE-Filter/TRACE-Smoother ablation, the observed-vs-missing error split (Table 12), and the regime-structured analysis (Fig. 3) make it possible to see where retrospective smoothing actually contributes.

Critical weaknesses

W1: The reported RMSE values are means over held-out test trajectories that appear to come from a single run, and no standard deviations, confidence intervals, or significance tests are provided, although App. C.3 cites a cross-trajectory std of about 0.03. This would be relevant to assessing whether small margins, such as Supernova 3-loop (0.318 vs. 0.321) and Ocean Miss (0.095 vs. 0.097), reflect genuine improvements rather than run-to-run or trajectory variance.

W2: Although code is provided, parts of it seems to not match the paper as written. For example, the DPS guidance in Model/diffusion/dps_sampler.py uses a gradient with respect to a detached denoised estimate rather than differentiating through DθD_θ as in Eq. (12), and the filter update in Model/ssgp/fusion/diag.py drops the value-derivative cross-covariance, unlike the lift described in Sec. 3.3 and App. A.3. In addition, config/exp/ocean_phase1_main.yaml uses a Tucker rank and SIREN frequency that differ from Table 2, so it is unclear which implementation produced the reported results.

Questions for authors

Q1: Could the authors report the number of seeds/trajectories and the variance for Tables 1, 5-7, 12, and 15, as well as error bars for the averaged curves in Fig. 3 and Fig. 5?

Q2: The paper (e.g. in the abstract) states that the proposed method is compared against “frame-wise generative reconstructors, offline spatiotemporal methods, and streaming data-assimilation baselines”, but the reported experiments include only one streaming baseline (DBF; EnSF and LD-EnSF are excluded after failed reproduction, App. E.5) and no external frame-wise generative reconstructor, the closest being the authors’ own TRACE-Frame/vanilla DPS (Tables 1 and 8). Could the authors clarify which methods correspond to each category, and either add such baselines or refine the wording accordingly?

Q3: Could the authors clarify which implementation produced the reported results, given the differences between provided codes and the paper as written? Also, many of the provided codes contain extensive non-English docstrings. These comments carry the implementation notes on data splits, masks, hyperparameters, etc., and could hinder readers trying to understand or reproduce the pipeline. Would the authors consider translating them into English in the released version?

Reviewer_zSHaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TRACE for reconstructing continuous physical fields from sparse measurements that arrive as structured streams, such as missing frames or a moving local observation window. Fields are represented as latent Tucker cores of a functional Tucker model, so off-grid measurements and arbitrary queries share one decoder. For each observed frame, diffusion posterior sampling with a pretrained latent diffusion prior produces samples, which are summarized as Gaussian latent evidence. This evidence is combined with a Matérn state-space GP prior through tempered Gaussian fusion and Kalman-style filtering to give causal online estimates. An RTS smoother then refines earlier, under-observed frames once later observations arrive.

Experiments on Active Matter, Ocean sound-speed, and Supernova data compare TRACE with LRTFR, MMGN, SDIFT, and DBF under temporally sparse and spatially localized sensing. The results show competitive or improved reconstruction accuracy across different structured sensing settings, particularly when retrospective smoothing can exploit later observations.

Critical strengths

The paper addresses a realistic setting that is relatively underexplored in generative field reconstruction: observations arrive over time, and individual frames can be missing or only partially observed. TRACE combines a learned generative prior with a state-space temporal prior, allowing information from earlier and later frames to help reconstruct under-observed frames. The gains are particularly clear in the moving-window setting. On Active Matter and Ocean, TRACE-Smoother outperforms all compared baselines, e.g., achieving an RMSE of 0.106 versus 0.275 for the best non-TRACE baseline on Ocean with one loop. Its reconstruction error also generally decreases as the observation window revisits the domain, while the MMGN baseline remains largely unchanged. This is practically valuable because sensing systems such as mobile probes and scanning instruments naturally produce this type of structured observation stream.

Critical weaknesses

TRACE mainly combines existing components: a functional Tucker latent representation, a latent diffusion prior with DPS, and a state-space GP with Kalman filtering and RTS smoothing. The closest baseline, SDIFT, already combines a functional Tucker representation with a diffusion prior and GP-based temporal modelling, although it operates offline. TRACE's main distinction is therefore the sequential filtering and retrospective smoothing formulation rather than a fundamentally new generative model. The temporal prior is also a stationary Matérn GP applied independently to each latent dimension, without learned dynamics.

The strongest results generally come from TRACE-Smoother, which requires future observations and is therefore not strictly causal. In temporally sparse settings, TRACE-Filter can perform worse than the offline MMGN baseline (e.g., 0.429 vs. 0.313 on Active Matter Miss-3). Even TRACE-Smoother does not consistently outperform MMGN, as shown by Ocean Blk-10 (0.124 vs. 0.092) and several Supernova settings. Furthermore, although the authors discuss the reproduction difficulties of DBF, EnSF, and LD-EnSF in Appendix E, the lack of a strong, competitive streaming baseline limits the evidence for TRACE's online reconstruction advantage.

These concerns matter because the paper's main contribution is its streaming inference framework. While the retrospective smoothing results are promising, the current evidence does not fully establish the advantages of the causal streaming component over existing approaches.

Questions for authors
  1. The strongest results generally come from TRACE-Smoother, which requires future observations. In several temporally sparse settings, TRACE-Filter performs worse than the offline MMGN baseline. Could the authors further clarify the practical advantages of TRACE's causal streaming component, particularly in scenarios where retrospective smoothing is not available?

  2. Appendix E discusses the reproduction difficulties of DBF, EnSF, and LD-EnSF, leaving limited comparisons with competitive streaming methods. Could the authors provide additional evidence or comparisons to better establish TRACE's advantage over existing causal data-assimilation approaches?

Reviewer_zvArOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

TRACE reconstructs physical fields from sparse measurements as they arrive, then uses later measurements to revise earlier estimates. It combines a per-frame generative model with temporal filtering and retrospective smoothing.

Critical strengths

The problem is practical. The experiments include challenging cases such as long observation gaps and moving observation windows, with substantial reported improvements in some settings.

Critical weaknesses
  1. Contradictions in the core equations. The claimed equivalence to the textbook Kalman update is mathematically incorrect as written. Appendix A.3 first fuses the predicted SS-GP prior with the generative evidence, then treats that already-fused distribution as another virtual observation into the same predicted state. This counts the temporal prediction twice. Consequently, the composition of Eq. 45 to Eq. 51 does not reproduce the textbook Kalman recursion claimed after Eq. 54.

  2. Contradictions of baseline results across tables. Table 12 decomposes Table 1(a), while 24 of the 27 corresponding LRTFR, SDIFT and DBF results differ, all with higher error in Table 1(a). TRACE and MMGN match across tables, thus the comparison with the strongest baseline unchanged, but the main table presents larger improvements over the other baselines.

  3. The benefit of temporal fusion is not isolated. MMGN, which simply interpolates latent codes linearly across missing frames (Appendix E.2), has lower missing-frame error than TRACE-Smoother on Ocean/Blk-10 (0.123 vs. 0.235) and Supernova/Blk-10 (0.319 vs. 0.385; Table 12), despite worse observed-frame error. Because the frame models differ, the needed control is to apply linear interpolation, a fixed-noise Kalman filter/RTS smoother, and the proposed fusion to identical TRACE frame estimates. The filtering version of this control would also test the streaming claim, since the only streaming baseline, DBF, degenerates to a near-mean field in the moving-window setting (Appendix F.9).

Questions for authors
  1. Which update is implemented? If it is the appendix version, how do the main results change once corrected?
  2. Which baseline totals are correct, and how to explain the inconsistencies?
  3. Using the same TRACE-Frame latent estimates at observed frames: (a) on Blk-10, how does linear interpolation across missing frames compare with TRACE-Smoother; and (b) on Blk-10 and the moving-window settings, how do a standard Kalman filter and RTS smoother, using the same SS-GP prior but a fixed, validation-tuned observation variance, compare with TRACE-Filter and TRACE-Smoother, respectively?
Reviewer_BKfYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces TRACE, a framework to assimilate observations with filtering (estimating a state from past observations) and smoothing (correcting past state estimates with new observations). TRACE targets a challenging setting where entire frames might be missing, with structured temporal observations, such as a moving sensor, which causes observations to be very important to propagate both forward and backward in time when assimilating them. The method works by representing a physical field with a smaller vector that is directly optimized at training (in an auto-decoder manner) using a functional Tucker model. A diffusion model is trained as a static (non-temporal) prior over the fitted latent coefficients of the training states. This model is then used at inference by performing posterior sampling (with DPS) in latent space, and fitting a normal distribution over an ensemble of samples. Kalman-style filtering is performed by fusing this distribution with Matérn state-space Gaussian process, followed by Rauch–Tung–Striebel smoothing. They evaluate their method on three physical datasets: Active matter, Ocean, and Supernova, with temporally sparse observations and moving-window sensors. The results display competitive accuracy with existing offline reconstruction methods, with specific advantages in fast-moving dynamics and temporally complementary observations.

Critical strengths

The paper targets a practically relevant reconstruction problem, where observations are sparse and continuous, as well as temporally intermittent. The problem is not new, but is often targeted in a restricted manner, such as using fixed discrete grids. Here, the authors develop and showcase a modular method with less constraints. The combination of the continuous spatial decoder, the generative prior, and the temporal inference is practical and well-motivated.

The method obtains good results on moving-window experiments, and provides ablation for individual-frame/filtering/smoothing to isolate the improvement.

Critical weaknesses

W1: In Eq (15), q_gen(z | O) already contains the prior p(z), so the authors are double counting it, which they acknowledge as a non-proper Bayesian update, as written in the paper. However, unlike claimed, tempering does not generally mitigate this completely, as q_gen should be divided by p(z) for this equation to be correct. This might cause uncertainty to be underestimated, which could explain the undercoverage the authors report. The effect is probably largest with sparse observations, which is the setting the authors claim advantages in.

W2: The experiments do not sufficiently isolate the benefits of the proposed components. Despite the combination being new and motivated, it lacks baselines and clear isolation of the components. Filtering and smoothing are ablated, but a baseline using direct Kalman filtering with the FTM with the linear observator would provide a meaningful comparison. It also lacks comparison with gridded representations with interpolation, and only uses gridded states in the experiments despite mentioning a strong practical target with off-grid measurements. For example, I think a comparison with a model like Appa (Andry et al., 2025) could strengthen the results and the motivation for this architecture. Indeed, smoothing seems like an important component, which makes a comparison with offline data assimilation methods fair. Andry et al. evaluate at a large scale observations that are mapped to the nearest neighbor on a grid, which is based on SDA that is cited. This could certainly be applied, especially on the given gridded experiments, as an additional baseline. I am not asking for an additional experiment with a large model like Appa, but a smaller SDA experiment would certainly improve the soundness.

Some other minor limitations are not discussed clearly enough in my opinion. A dedicated limitations paragraph in the main text would be appreciated. Limitations I identify include notably the same-field assumption for observations. How could the method be used for moving-window nonlinear observations of satellite radiance for reconstructing atmospheric data? Would adapting DPS with a supplementary observation process suffice? Scalability is also a concern of mine, see below.

Questions for authors
  • Do the authors expect the method to be scalable for high-dimensional systems with more physical fields? Considering there are already 32,768 coefficients for the Supernova dataset, can this method be used at larger scales, such as atmospheric data cited above? My main concern is the expressivity of the FTM for more complex data.
  • Have the authors experimented with more recent posterior sampling algorithms, such as MMPS (Rozet et al. 2024)? Likewise, have the authors considered more advanced observation models such as nonlinear observations by backpropagating through the entire decoding process?
  • How are the off-grid coordinates genuinely generated and evaluated? Are they interpolated between gridded values?
  • Could the authors justify the update in Eq. 15 or compare it with the corrected update using q_gen/p(z)?

Addressing the concerns, especially those with the baselines, including a direct Kalman filter and a (small) experiment with SDA could lead me to raise my score. Right now, these concerns are too strong for me to consider a higher score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.