acceptodds
Under review as a conference paper at ICLR 2027

Temporal Causal Observability: Separating Persistent Causal State from Perceptual Access

Abstract

Temporal causal representation learning (CRL) can recover latent causal factors and their dynamics, yet still fail when perceptual access to those factors changes over time: a change in observation need not imply a change in the underlying causal state. We formalize this distinction as Temporal Causal Observability, separating causal-state evolution from time-varying factor-wise perceptual access and studying controlled interventions that alter access while preserving the causal trajectory. For a frozen temporal CRL model, we show that synchronized reference evidence can evaluate the expected benefit of a fixed measurement correction without observing the true state, under a conditional orthogonality assumption. We then establish an identifiability boundary: evaluating the benefit of a fixed correction does not in general identify when that correction should be applied, because the same observation change may arise from either state dynamics or changing perceptual access. This motivates Regime-Marginalized Elimination (RME), which marginalizes latent access regimes instead of making hard observability decisions. Across multiple temporal CRL systems, reduced perceptual access causes substantial degradation despite strong full-observation recovery. On frozen CITRIS representations, recursive RME initialized from a single clean encoding reduces hidden-state error by 22–74% relative to robust median fusion and by 7–34% relative to recursive hard regime decisions under the same evidence and uncertainty-propagation procedure, across all eight representations. A matched structural ablation further shows that recursive maintenance benefits from aligning access regimes with the causal-factor partition learned by the CRL model. In one-step updates, RME also outperforms hard regime decisions on BISCUIT. These results identify temporal causal observability as a distinct post-identification problem: learning a causal state and its dynamics does not determine whether current perceptual evidence should update that state.

Primary area:
causal reasoning
Keywords:
causal representation learning, temporal causal representation learning, causal identifiability, partial observability, latent state inference, causal dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gtDAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to address the difficulty of maintaining causal states when perceptual access changes. It introduces Temporal Causal Observability and analyzes when observations provide reliable evidence for state updates. A method named RME combines predictions and observations by considering different possible access conditions. Experiments on frozen causal representations demonstrate reduced state estimation errors when some observations are replaced.

Critical strengths
  1. The paper distinguishes changes in causal states from changes in perceptual access. The controlled replacement experiments make it possible to study observation changes without altering the underlying causal trajectory.
  2. The paper compares different update rules under the same observation conditions and includes ablation experiments on scale calibration and uncertainty propagation.
Critical weaknesses

-- The experimental assumptions limit the applicability of the proposed method. The method accumulates evidence within observation blocks whose boundaries are known in advance, and assumes that different views lose access independently. These assumptions make it easier to determine which observations should be trusted. However, access may change unpredictably or be lost across several views at the same time, making the available evidence less informative. The experiments mainly examine partial replacement while valid observations remain available, so they do not adequately establish performance under sustained loss of access. Further evaluation under less restrictive observation conditions is necessary to support the applicability of the method.

-- The reliability of the uncertainty estimation is not sufficiently demonstrated. The proposed method uses uncertainty to determine how much the predicted state and the current observations should influence each update. Although uncertainty propagation improves the experimental results, the estimated variance does not adequately reflect the growth of prediction error over time. This raises concerns about whether the method can maintain an appropriate balance as estimation errors accumulate. The assumption of independent measurement errors is also inconsistent with the strong correlations between the views used in the experiments, which may lead to excessive confidence in repeated evidence. The influence of these limitations on continuous state estimation needs further analysis.

-- The comparison with existing filtering methods does not fully support the advantage in continuous state estimation. The paper compares the proposed method with filtering baselines when updating from a clean previous observation. However, its main objective is to maintain the state through repeated updates, where earlier estimation errors can affect later predictions. Under this setting, the comparison mainly focuses on whether to average over possible access conditions or select a single condition. This explains the benefit of the proposed decision rule, but does not show whether its advantage over the filtering baselines persists as errors accumulate. Comparing these methods under the same recursive setting is therefore necessary for a more complete evaluation.

-- The contribution of causal information is not sufficiently distinguished from the design of the observation replacements. The grouping experiment shows that the learned causal-factor groups perform better than rearranged groups. However, the observation replacements are also organized by causal factor, so coordinates in the same group share the same access condition by construction. Rearranging the groups breaks this correspondence and makes the shared-access assumption less appropriate. Consequently, the performance difference may reflect the match between the groups and the replacement mechanism, without establishing an advantage specific to causal information. A comparison with groups learned from observational dependence would help clarify whether causal grouping provides benefits beyond capturing shared access conditions.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gtDA — Part 1/2

Thank you for four specific requests. We ran each of them. The filtering baselines, the boundary-free test and the grouping controls support the method. The tests with shared replacement and with no valid view and the calibration analysis confirm your concerns, and the paper now states these limits.

Your pointWhat we didOutcome
W3. Filtering baselines under recursionχ2\chi^2 gate and Student-t, with the same recursion and draws as RMERME has 17–78% lower error, and lower in every representation
W1. Known boundariesFixed hazard 0.25 insteadAdvantage over hard validity kept (3/3 Pong and 5/5 Voronoi representations, both donors)
W1. Views lost together; sustained lossCorrelated masks, 4–16-frame blocks, separate or shared donorsKept with separate donors; can fail with shared donors, and fails with no valid view; now stated
W2. Reliability of the uncertaintyCommon-error channel; calibration by sequence ageOver-counting reduced, ordering kept; long-horizon calibration remains a limit
W4. GroupingGroupings learned from observational dependence (MI, MI on increments); one shared regime; 8 shufflesLearned causal grouping lowest except Pong-IND (MI 1% lower); the others are worse everywhere

W3. Filtering baselines under the same recursion. The χ2\chi^2 gate (99%) and the Student-t update (ν\nu = 4) now run recursively. Each feeds back its own mean and variance, with the same 64 Monte Carlo draws as RME.

SettingRMEχ2\chi^2 gateStudent-t
Pong, independent donors0.2900.4330.349
Pong, coherent donors0.2970.4260.379
Voronoi, independent donors0.0420.1920.148
Voronoi, coherent donors0.0510.2210.184

RME has lower EHE_H in every representation. Recursion widens the gap: on Pong with independent donors, the χ2\chi^2 gap grows from 0.034 in one-step updates to 0.142. The reason is lock-out. Once its prediction drifts, the χ2\chi^2 gate keeps rejecting valid views. In the correlated-loss test (8-frame blocks), in the first two frames after access returns, it rejects a valid view in 31% of frames on Pong and 8% on Voronoi, against 6% and 0.3% for RME. Student-t has slightly lower all-valid error on Pong (0.244 against 0.253), a trade-off we report; on Voronoi, RME is lower on both measures.

W1. Known boundaries. They are not needed for the advantage over hard validity. With a fixed hazard of 0.25 instead, RME beats hard validity in all 3 Pong and all 5 Voronoi representations, for both donor types (on Pong, hard-minus-RME EHE_H is +0.021 [0.016, 0.026] and +0.021 [0.011, 0.031]). On both benchmarks RME stays 19–67% below median fusion (reply to fHSU, Q1).

W1. Views losing access together, and sustained loss. You were right that this reduces the evidence. We added masks in which views share one access state with probability 0.5 or 1, with blocks of 4, 8 and 16 frames, and with separate or shared donors (Appendix F.7).

  • With separate donors, RME beats hard validity throughout. With 16-frame blocks, for example, its EHE_H is 0.075 against 0.094 (5/5).
  • With shared donors, the hidden views show the same wrong value, and their agreement misleads RME. On Voronoi a χ2\chi^2 gate is then better: 0.313 against 0.341 when views share access with probability 0.5, and 0.908 against 1.332 when they always do. On Pong with fully shared masks and donor, the gate is also best, and both hard rules match or beat RME.
  • When no view is valid, updating is worse than keeping RME's own prediction: MSE rises by 0.084 and 0.061 on Pong and by 0.143 and 0.162 on Voronoi (independent and coherent donors). The independent access prior gives this case only (1−0.8)3(1-0.8)^3 = 0.8% prior mass, and on these frames RME's posterior weight on it stays low: median 0.08 and 0.01 on Pong, 0.16 and 0.09 on Voronoi, below 0.5 in every representation.
  • When access returns, RME recovers fast: on Voronoi its error over the first two frames is 0.012, against 0.298 for the χ2\chi^2 gate.
AuthorsOpenReview ↗

Response to Reviewer gtDA — Part 2/2

To check the role of the prior, we reused the runs with assumed p from 0.6 to 0.95 (reply to fHSU, W1). As the prior mass on the all-invalid case falls from 6.4% to 0.0125%, the harm grows steadily in all four settings:

Settingp = 0.6 (6.4% prior)p = 0.8 (0.8%)p = 0.95 (0.0125%)
Pong, independent donors+0.0662+0.0840+0.1090
Pong, coherent donors+0.0381+0.0607+0.0892
Voronoi, independent donors+0.1034+0.1431+0.2065
Voronoi, coherent donors+0.1248+0.1620+0.2048

Values are the MSE increase from updating on frames with no valid view; Appendix F.11 gives all five values of p with intervals. So the rare all-invalid prior contributes to the failure, but a larger prior does not remove it. We keep p = 0.8 for the reported method. A prior that allows correlated loss of access, and an explicit regime for shared replacements, are natural next steps.

W2. Reliability of the uncertainty. We agree. The propagated variance does not track the slow growth of error over long horizons, and the paper now says plainly that better state estimates are not calibrated long-horizon uncertainty. We made two additions. First, a common-error channel reduces the over-counting of repeated evidence that you describe, since three agreeing views no longer count as three independent measurements. The ordering holds under it (Voronoi: 0.0427 against 0.0576 for hard validity, 5/5). Second, Appendix F.8 reports coverage, NLL and dispersion by sequence age: on Pong, squared error reaches 3–4 times the predicted variance after 512 frames.

W4. Causal grouping versus the replacement design. We ran the comparison you suggested. We learned two alternative groupings from observational dependence on training encodings, without access labels: mutual information (MI) between coordinates, and MI between their increments. We also include one regime shared by all coordinates and the eight size-matched shuffles of the original ablation. Mean EHE_H, lower is better:

SettingLearned causalMIMI on incrementsOne shared regimeShuffles (mean of 8)
Pong, independent donors0.29010.28680.30230.42940.3004
Pong, coherent donors0.29720.29870.31820.44160.3110
Voronoi, independent donors0.04240.04540.05040.17320.0609
Voronoi, coherent donors0.05140.05590.05930.18590.0728

The learned causal grouping has the lowest error in every setting but one. MI comes closest: on Pong the two are within about 1% (MI slightly lower with independent donors); on Voronoi the learned grouping's error is 6–8% lower. Against the other alternatives the learned grouping is lower in every setting: by 4–16% against MI on increments, 32–76% against one shared regime, and 3–4% on Pong and about 30% on Voronoi against the shuffles. So observational dependence can come close to the causal grouping, but none of the alternatives improves on it across settings. Section 5.3 now reports these controls (Appendix F.9).

In short: under the original masks, RME has lower hidden-state error than both recursive filtering baselines in every representation; its advantage over hard validity holds without known boundaries on both benchmarks; and the learned causal grouping holds up against the data-driven groupings you suggested. The limits you pointed to (shared evidence, no valid view, long-horizon calibration) are now measured and stated in the paper. We hope this fuller comparison addresses your concerns, and we would be grateful if you would reconsider your assessment in light of these results.

AuthorsOpenReview ↗

Short follow-up before the discussion closes

Thank you again for your review. Since the discussion closes tomorrow, here is the short answer to your four points.

W3: under the same recursion, RME's mean hidden-state error is 30–78% lower than a χ² gate's and 17–72% lower than a Student-t update's, and it is lower in every representation.

W4: grouping by mutual information on the encodings comes within about 1% on Pong, and the causal grouping's error is 6–8% lower on Voronoi; MI on increments, one shared regime and size-matched shuffles are worse in every setting.

W1: without known boundaries, RME keeps its advantage over hard validity in every representation, and with correlated loss and separate donors it also holds. Shared erroneous evidence and frames with no valid view are now stated as limits in Section 6.

W2: a common-error channel reduces the over-counting of agreeing views, and the ordering holds under it; long-horizon calibration remains a stated limit.

We would be grateful to know whether this addresses your concerns, or which concern remains.

Reviewer_coY2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Temporal Causal Observability: given a causal representation, deciding whether evidence at frame t should update the current state belief. They show (via an R2R^2 probe and two propositions) that, when a donor is drawn from P(Ck∣C⁻k)P(C^k|C^{⁻k}) and rendered by the same mechanism, access and no-access are statistically indistinguishable from the current frame — so no per-frame detector beats chance. Their method, Regime-Marginalized Elimination (RME), therefore keeps all 2V2^V access patterns alive, weights them by prior + integrated likelihood, and averages the regime-conditioned state updates. Experiments with frozen CITRIS/BISCUIT and boundary cases (CaRiNG, DMS) on Pong and Voronoi, withholding 20% of four-frame blocks per factor across three synchronized views, show RME cuts any-hidden error 22–74% versus a memoryless median and 7–34% versus hard regime decisions, across all eight representations.

Critical strengths
  • Clear problem formulation and motivation.
  • Table 1B asks about the representation at fault with the right control: retraining on partial access leaves recovery at .548, essentially the frozen encoder's .557. The state-supervised GRU (.691) is then honestly labelled a diagnostic, not a matched baseline.
  • Tables 2 and 3 hold the evidence fixed and vary only the decision rule. Tables 4A and 4B then separate the calibrated scale from the propagated uncertainty. That is a clean decomposition, and Table 4A shows the missing scale of the median as a dash in its own column.
  • The manipulation check works. Table 1A drops from .985 to .557 and .595 against a predicted .591 on the two Voronoi systems, from .599 to .363 against .359 on DMS, and from .802 to -.802 on CaRiNG where the target is hidden so effective p = 0. The encoders really do pass the donor through.
Critical weaknesses
  • The theory and the method do not meet. Prop. 2 tells you when a synchronized reference can score a fixed correction, and Sec. 4 then says directly that "the reference score in Section 3.2 does not determine the regime weights in Eq. 12".
  • Proposition 1 assumes a donor drawn from P(Ck∣C−k)P(C^k | C^{-k}), but the experiments do not draw donors that way. Sec. 5 says IND resamples the donor value at each frame from the factor's empirical pool. Sampling from the pool is not sampling from the conditional law given the other factors, so the joint law of (donor, other factors) need not match the real one.
  • Fig. 1a motivates the problem with a real occlusion in nuScenes, but no experiment uses real imagery or a carrier that is physically degraded. The donors replace evidence, and the Discussion notes that real occlusion also removes and distorts it.
Questions for authors
  • Can you implement the donor that Prop. 1 assumes and report the Eq. 4 deviation under it? If the degradation is partly detectable, how much of the "the frame is uninformative" argument survives, and should the retraining result still be described as consistent with Prop. 1?
4 comments
AuthorsOpenReview ↗

Response to Reviewer coY2 — Part 1/2

Thank you for checking our numbers against the predictions and for a precise question. It led to our most informative new experiment. You were right that our original donors do not follow the conditional law, and a single frame carries a weak but detectable cue about them. With donors approximately matched to that law, the tested detectors fall to near chance and RME's advantage over hard validity grows. Section 4 now also links Proposition 2 and RME explicitly.

DonorLogisticGradient boostingCNNRME EHE_HHard validity EHE_H
Original empirical-pool donor0.5480.6200.5950.04240.0577
Approximately matched to the conditional law0.4990.5020.4940.04730.0716

The first three columns are single-frame detection AUROC; the last two are recursive hidden-state error. Voronoi, independent donors, original masks. RME wins in 5/5 representations in both rows.

The conditional donor. For each replaced factor, we draw the donor from training states whose other factors are close to the current ones (nearest neighbours, K = 10). K was chosen by a training-only balance test, which all five representations passed. This approximates the conditional law; it is not an exact match. Trajectories, masks and models are unchanged. With these donors, RME's hidden-error reduction relative to hard validity is 34%, against 26% with the original donors. Replacing one factor at a time gives the same picture: EHE_H 0.0477 for RME, 0.0617 for hard validity and 0.0634 for MAP (5/5 representations). So RME's advantage remains when the tested detectors can no longer find a single-frame cue.

Eq. 4 under conditional donors. Eq. 4 assumes that the donor is independent of the true value. A conditional donor is correlated with it through the other factors, so Eq. 4 does not apply directly. With conditional donors the observed R2R^2 is 0.643–0.669, above Eq. 4 (0.589–0.593) and above a first-order generalization that accounts for this correlation (0.593–0.631). The exact split puts the difference in the cross-moment term: on replaced frames the probe output stays more correlated with the true value than the estimated predictability of the factor from the others implies. This concerns how much state information a replaced frame keeps, not whether the replacement can be detected; the detectors above stay near chance (Appendix F.6; per-factor values in F.11).

How much of the argument survives. Proposition 1 is an exact statement under conditional matching, not a claim about our empirical-pool donors. With those donors, the tested single-frame detectors reach AUROC up to 0.62; with approximately conditional donors, they are near chance. The retraining result holds either way. Partial-access retraining reaches R2R^2 0.548 and 0.585. Eq. 4 gives 0.591 for an encoder that simply encodes the donor, while a per-frame model that detected every replacement, and predicted the factor from the others on those frames, could reach at least p⋅Rfull2p\cdot R^2_{\mathrm{full}} ≈ 0.79. Retraining stays below the first value, so it does not close the gap, even though the original donors carry a weak cue. "Consistent with Proposition 1" is removed; Section 5.1 now reads: "Retraining the encoder on partially observed data leaves recovery at 0.548, below the 0.591 of Eq. 4 and far below the p⋅Rfull2p\cdot R^2_{\mathrm{full}} ≈ 0.79 or more that a per-frame model detecting every replacement could reach, although these donors are weakly detectable from one frame (AUROC up to 0.62; Appendix F.6)."

AuthorsOpenReview ↗

Response to Reviewer coY2 — Part 2/2

W1. Theory and method. We agree that the link was not explicit. Section 4 now reads:

Proposition 2 characterizes when synchronized reference evidence can evaluate the expected benefit Δt\Delta_t of a correction fixed before scoring; it does not determine the latent access regimes required for adaptive updates. RME instead specifies a measurement model over these regimes and marginalizes their corresponding state estimates. The two connect through the objective: since ∥Dt−Zt⋆∥Q2\lVert D_t - Z_t^{\star}\rVert_Q^2 does not depend on the correction, minimizing conditional quadratic risk is the same as maximizing the expected benefit E[Δt∣evidence]\mathbb{E}[\Delta_t \mid \mathrm{evidence}]. Under the model, the posterior mean does this for a given prediction (Appendix A.5), whereas the recursive implementation remains an approximate inference procedure. Two further results shape the rule: under its matched-replacement assumption, Proposition 1 rules out single-frame validity detection, so RME draws on synchronized views and the recent past; and reference scores do not generally retain unbiasedness when used to choose a correction per sample (Appendix A.4), so RME weighs regimes by likelihood instead of gating on Γt\Gamma_t. The reference analysis thus identifies an information boundary, and RME is a constructive update rule under additional observation assumptions.

One more way to see the link: with Q = Σ−1\Sigma^{-1}, the reference score is twice a Gaussian log-likelihood ratio, Γt=2[log⁡N(Wt;Ut,Σ)−log⁡N(Wt;Dt,Σ)]\Gamma_t = 2[\log \mathcal{N}(W_t;U_t,\Sigma)-\log \mathcal{N}(W_t;D_t,\Sigma)]. RME weighs access regimes with the same kind of likelihood ratio.

W3. Real imagery and physical degradation. We agree this is a limit, and Section 6 states it. We did not add such a test for two reasons. First, the tests need the same causal trajectory under two access conditions, which natural video cannot provide; nuScenes serves only as motivation. Second, in these benchmarks pixel-level occlusion does not act on a single factor: Voronoi colours mix the factors through the flow, and the ball factors in Pong share pixels. Occluding pixels would change several factors' access at once and break the controlled intervention. As controlled alternatives, we added conditional donors and independent pixel noise per view; the ordering held in both.

In short: we now approximate the donor law of Proposition 1 with a training-only conditional sampler; the tested single-frame detectors fall to near chance, and RME's advantage over hard validity grows. Section 4 now links the theory and the method through the expected benefit Δ\Delta. We hope this addresses the mismatch you identified, and we would be grateful if you would consider updating your assessment.

I would like to thank the authors for addressing my questions. Paper limitations are now explicitly stated in the Limitations subsection. I will increase the score to 3.

We sincerely thank the reviewer for their thoughtful feedback and for reconsidering the assessment. We appreciate your time and constructive engagement throughout the review process.

Reviewer_fHSUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a post-identification problem the authors call Temporal Causal Observability (TCO): after a temporal causal representation learning (CRL) model has recovered causal factors and their dynamics, when should a new observation update the maintained causal state? The key observation is that a change in the observation need not reflect a change in the underlying state — e.g., a cyclist passing behind a car keeps moving while only perceptual access drops. The authors formalize this by separating the persistent causal state C_t from factor-wise perceptual access O_t, and by intervening only on access: for one factor at a time, the image content carrying it is replaced with a plausible donor value (independent resampling, IND, or a coherent segment from another trajectory, COH) while the causal trajectory and transition inputs stay fixed. For a frozen temporal CRL model they show: (i) a closed-form manipulation check (Eq. 4) plus an impossibility result (Proposition 1) showing that under distribution-matched replacement no single-frame detector can identify access (ROC AUC 1/2), and that access-matched retraining does not help (verified empirically); (ii) a positive boundary result (Proposition 2) that a synchronized reference can identify the expected benefit of a fixed correction only under a conditional orthogonality condition that replaced references violate; and (iii) Regime-Marginalized Elimination (RME), which treats factor-wise access as a latent regime over 2³ binary validity patterns across three synchronized views and, instead of committing to the most probable regime, marginalizes regime-conditioned Gaussian state updates over the posterior on regimes, using a representation-calibrated noise scale and recursively propagated uncertainty through the frozen transition. Experiments on frozen CITRIS (interventional Pong, Voronoi), BISCUIT (robotic-arm Voronoi), CaRiNG, and DMS show that reduced access degrades recovery despite near-perfect full-access recovery, and that recursive RME reduces hidden-state error by 22–74% relative to median fusion and 7–34% relative to hard regime decisions across all eight CITRIS representations. Ablations show that reliable maintenance additionally requires representation-specific scale calibration, propagated state uncertainty, and alignment of latent access regimes with the learned causal-factor partition.

Critical strengths
  1. A well-posed post-identification problem with a clean impossibility result. Proposition 1, combined with the closed-form manipulation check of Eq. 4 (verified across four backbones, Table 1), explains why retraining or a better encoder cannot resolve access changes: the encoder is faithful and the replacement is distribution-matched. This is consequential because it establishes TCO as a distinct problem rather than a failure of factor recovery, and it justifies why evidence beyond the current frame (synchronized views, temporal history) is needed.
  2. Disciplined experimental design. All compared methods receive identical frozen encodings, transitions, masks, donors, and Monte Carlo propagation draws, and differ only in the decision rule; the unit of replication is the representation with paired bootstrap intervals. Ablations isolate each requirement (Table 4: calibrated scale, uncertainty propagation, causal-factor partition), and negative results (single-view DMS worse than the dynamics prior; fixed-scale recursion losing to a memoryless median) are reported rather than hidden.
Critical weaknesses
  1. The solution depends on information that is rarely available in practice. RME requires synchronized multi-view renderings of the same instant, the nominal access probability p, and known block boundaries. The authors acknowledge this (Section 6), but it materially limits significance: the main empirical gains are obtained under privileged inputs, while the realistic single-view setting (DMS) is precisely where RME underperforms the dynamics prior. Inferring boundaries online or operating with a single view is left open.
  2. The channel model is misspecified and its uncertainty is uncalibrated. Views are replicated renderings (correlation 0.65–0.76) yet modeled as independent Gaussians, so the regime likelihood acts as a scoring rule rather than a calibrated posterior; the propagated variance stays nearly flat while prediction error grows with time since the clean start. Consequently, the recursive protocol's "uncertainty" does not reflect true belief uncertainty, and the degree to which RME's advantage over hard decisions reflects principled marginalization versus scale tuning is under-determined.
Questions for authors
  1. (Clarification) Regime weights require known block boundaries and the nominal access probability p. What happens to hidden-state error when boundaries are misaligned or unknown, and is there a tolerable boundary error?
  2. (Clarification) How sensitive are the regime weights and the RME-vs-hard-decision ordering to the view correlation (0.65–0.76)? Would modeling the actual correlation structure, or using truly independent renderings, change the results?
  3. (Clarification) The recursive protocol starts from one clean encoding at t=0. How does performance behave with a noisy or partially hidden initialization?
  4. (Clarification) The update resembles an interacting-multiple-models / probabilistic-data-association filter on a frozen learned encoder and transition — effectively an action-free RSSM-style filter. How would RME compare to a posterior that learns its uncertainty jointly with the dynamics (as in RSSM or Recurrent Kalman Networks) rather than calibrating Gaussian scales post hoc?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fHSU — Part 1/3

Thank you for the careful summary and for asking for specific numbers. We tested the information requirements you raised (Appendix F). RME's advantage over hard validity survives without known boundaries, when view correlation is modeled or independent pixel noise is added, with true noise scales in simulation, and at every tested value of the access prior. With a single view, RME still improves overall recovery; hidden frames then carry no valid view, so the prediction stays best there.

Your concernNew testResult
Known block boundaries (Q1)Fixed hazard 0.25; shifted boundariesRME beats hard validity in 3/3 Pong and 5/5 Voronoi representations, both donors; any shift, even of one frame, is worse than no boundaries
Nominal access probability (W1)Assumed p from 0.6 to 0.95, true p = 0.8Mean EHE_H changes by at most 7.5%; RME beats hard validity in every representation at every p
View correlation (Q2)Common-error channel; fixed correlation sweep; independent pixel noiseHard rules have higher error than RME in every setting tested
Scale tuning (W2)Simulation with the true transition and noise scalesRME 0.318 against 0.348 for hard validity
Initialization (Q3)Restarts from the prior meanNo difference after 16 frames
Single view (W1)One view on CITRISOverall R2R^2 up in every representation; hidden frames then carry no valid view, so the prediction stays best there

Q1. Known block boundaries. We replaced the known boundaries with a fixed hazard h = 0.25: at each frame, the access regime is redrawn from its prior with probability 0.25, and no realized boundary or access label is given to inference. A redraw can keep the same regime, so this is not a switch every four frames. The model still assumes that access persists for a while, and it keeps the nominal prior.

SettingKnown boundariesNo boundary inputShifted by 2 framesMedian fusion
Pong, independent donors0.29010.29390.30920.3770
Pong, coherent donors0.29720.30720.32600.3797
Voronoi, independent donors0.04240.05360.07490.1615
Voronoi, coherent donors0.05140.06210.08530.1540

Without boundaries, EHE_H rises by 1–3% on Pong and 21–26% on Voronoi, and it stays 19–67% below median fusion. RME still beats hard validity in every representation: hard-minus-RME EHE_H is +0.021 [0.016, 0.026] and +0.021 [0.011, 0.031] on Pong (independent and coherent donors, 3/3 each), and +0.017 [0.014, 0.021] and +0.027 [0.019, 0.036] on Voronoi (5/5 each). On the tolerable boundary error: every shift we tested, even by one frame, gives higher error than giving no boundaries, in all four settings (Appendix F.2). So unless boundaries are exact, it is better to give none and use the fixed hazard.

W1. Nominal access probability. We ran every assumed p in {0.6, 0.7, 0.8, 0.9, 0.95} while the true p stayed at 0.8, with known boundaries and with the fixed hazard. Relative to p = 0.8, mean RME EHE_H changes by at most 1.6% on Pong and 7.5% on Voronoi; the largest change is Voronoi with coherent donors, no boundary input and p = 0.95 (0.0621 to 0.0667). RME beats hard validity in every representation at every p, in both boundary conditions. The reported method still uses p = 0.8; we did not use the sweep to pick a new value. Appendix F.11 lists all values, with EVE_V and paired comparisons.

AuthorsOpenReview ↗

Response to Reviewer fHSU — Part 2/3

W1. Synchronized views. The strongest results use three synchronized views, but RME also runs with one, weighing that view against the prediction from the recent past. With one view, a replaced frame carries no information about the state, so the prediction is the best estimate available there; the ideal rule would keep the prediction on hidden frames and use the view on valid ones. We tested how close RME comes on CITRIS, with one-step updates from the previous clean encoding; each value averages the three single-view runs within a representation.

SettingDynamics R2R^2One-view RME R2R^2Dynamics EHE_HOne-view RME EHE_H
Pong, independent donors0.64630.71370.34900.3819
Pong, coherent donors0.64630.71250.34900.3883
Voronoi, independent donors0.73580.90530.22160.2982
Voronoi, coherent donors0.73580.90520.22160.2992

One view raises aggregate R2R^2 above the dynamics prediction in every representation, for both donor types (paired gains +0.067 [0.064, 0.072] and +0.066 [0.063, 0.070] on Pong, 3/3; +0.170 [0.162, 0.176] and +0.169 [0.161, 0.177] on Voronoi, 5/5). As expected, its error on hidden frames stays above that of the prediction: the rule cannot see which frames are hidden, and with a prediction and one measurement no rule can accept every real change and reject every donor (Appendix A.4). With three views, 78–79% of hidden frames keep two valid views, and one-step RME reaches EHE_H 0.257 on Pong and 0.033 on Voronoi (Table 3), far below the prediction (about 0.35 and 0.22). This is what repeated views add.

The DMS result you mention follows the same pattern. In Table 16 of the revision (Appendix C.9), one-view RME has slightly higher aggregate R2R^2 than the dynamics prior (0.582 against 0.572), close to the update given the true access labels (0.589). Its hidden-frame error is 0.793 against 0.747 for the prior, while using the view directly raises it to 2.615. "Worse than the dynamics prior" in Section 5 referred to this hidden-frame gap; the revision now says so. Several cameras that observe the same scene are one setting with repeated evidence; we have not tested real sensors.

W2. Scale tuning versus marginalization. In Table 2, RME, Block MAP and hard validity share the same calibrated scale, prior, evidence and Monte Carlo draws; only the decision rule differs. So the ordering does not come from scale tuning. To check this further, we ran a simulation with the true transition and the true noise scales, where nothing is tuned. Recursive RME has EHE_H 0.318, against 0.348 for hard validity, 0.359 for MAP and 0.376 for Student-t. With coherent donors, the values are 0.341, 0.375, 0.391 and 0.384.

Q2. View correlation. We tested it in two ways and added a robustness check (Appendix F.3).

  1. A common-error channel, in which valid views share one error term whose size is estimated on training renders only. On Voronoi, RME has EHE_H 0.0427 against 0.0607 for MAP and 0.0576 for hard validity (5/5 representations); with coherent donors, 0.0501 against 0.0724 and 0.0677. On Pong the training estimate did not pass our pre-registered eligibility rule, so we report only the fixed sweep there.
  2. A fixed sweep of the error correlation (0, 0.25, 0.5, 0.75, 0.9). Both hard rules have higher EHE_H than RME at every value, in every representation and donor type, on both benchmarks.
  3. Independent pixel noise per view, with the level chosen on training images. Pong: 0.326 against 0.373 (MAP) and 0.382 (hard). Voronoi: 0.046 against 0.070 and 0.062; coherent donors are similar. One Voronoi representation selected zero noise, which we report. It lowers the cross-view residual correlation only slightly (Pong 0.76 to 0.69; Voronoi unchanged), so we treat it as a robustness check.

W2. Calibration. We agree that the propagated variance tracks short-term error but not its slow growth over long horizons. Section 6 now says plainly that better state estimates are not calibrated long-horizon uncertainty, and Appendix F.8 reports coverage, NLL and dispersion by sequence age: on Pong, squared error reaches 3–4 times the predicted variance after 512 frames, while on Voronoi the ratio stays near or below 1. The likelihood still ranks access well (validity AUROC 0.91–0.996).

Q3. Initialization. We restarted the recursion every 100 frames from the prior mean, with no clean encoding. After 16 frames, EHE_H differs from clean starts by at most 0.0002 on Pong and less than 0.00001 on Voronoi, and all intervals include zero. We claim this for restarts within an episode, not for arbitrary initial states.

AuthorsOpenReview ↗

Response to Reviewer fHSU — Part 3/3

Q4. RSSM and Recurrent Kalman Networks. We did not add a jointly learned baseline, and we do not claim to beat these model families. Training one changes the representation, the dynamics and the uncertainty together, so it tests a broader question than updates of a fixed causal state. Two points still bear on your question. An RKN-style update takes its observation uncertainty from the current frame; under matched replacement, that uncertainty cannot flag replaced frames, and the Gaussian update then averages the donor with the prediction. An RSSM posterior also sees the history and could, in principle, learn to discount inconsistent frames; testing this needs training with access changes, which we leave to future work. Our comparison instead isolates the decision rule: RME, MAP and hard validity share the same scales and draws, and the simulation above uses the true scales.

In short: RME's advantage over hard validity survives without known boundaries, when view correlation is modeled or independent pixel noise is added, with true scales in simulation, and at every tested value of the access prior. With one view, it still improves overall recovery; on hidden frames, which then carry no valid view, the prediction stays best, and the paper now says so. We hope these results answer your questions, and we would be grateful if you would consider updating your assessment.

AuthorsOpenReview ↗

Short follow-up before the discussion closes

Thank you again for your review. Since the discussion closes tomorrow, here is the short answer to your two main concerns.

Privileged information: without any boundary input, RME still beats hard validity in every representation on both benchmarks. Over assumed access probabilities from 0.6 to 0.95, its mean hidden-state error changes by at most 7.5%, and it stays ahead of hard validity at every value.

Scale tuning: all decision rules in Table 2 share the same calibrated scales. In a simulation with the true transition and noise scales, where nothing is tuned, RME still beats hard validity (0.318 against 0.348). The ordering also holds when view correlation is modeled, up to a correlation of 0.9.

With a single view, RME raises overall R² in every representation; on hidden frames, a replaced view carries no state information, so the prediction stays best there.

We would be grateful to know whether this addresses your concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all reviewers and the Area Chair for their constructive feedback. We added pre-specified experiments, report their settings, and mark revisions in blue.

Summary

  1. Robustness beyond the submitted setting. RME beats hard validity in all five Voronoi representations without known boundaries, with estimated common error, or with independent pixel noise (Table R1). It also wins in all three Pong representations without boundaries, for both donor types.
  2. With approximately conditional donors, single-frame detection falls to near chance, and RME's advantage grows. The tested detectors reach AUROC 0.49–0.50, against 0.55–0.62 for the original donors, and RME's reduction in hidden-state error relative to hard validity grows from 26% to 34%.
  3. Robust-filter comparisons. Under the same recursion and original masks, RME reduces mean hidden-state error by 30–78% relative to a χ2\chi^2 gate and by 17–72% relative to a Student-tt update, with lower error in every representation.
  4. Causal grouping versus observational dependence. Grouping by mutual information on the encodings comes within about 1% on Pong, and the causal grouping's error is 6–8% lower on Voronoi; MI on increments, a single shared regime and size-matched shuffles are worse in every setting.
  5. Failure modes. Updating is harmful when no view is valid; in the one-view setting this holds on hidden frames despite improved overall recovery. If multiple views simultaneously show the same incorrect value, χ² gating can outperform RME. These limits are now explicit.

Table R1 (new Table 5)

Hidden-state error EHE_H on Voronoi, independent donors, five representations. Lower is better.

ConditionRMEHard validityHard − RME [95% CI]RME wins
Submitted setting (known boundaries)0.04240.0577+0.0153 [0.0109, 0.0197]5/5
No boundary input (fixed hazard 0.25)0.05360.0710+0.0174 [0.0143, 0.0206]5/5
Common-error channel, estimated on training data0.04270.0576+0.0148 [0.0118, 0.0182]5/5
Independent pixel noise per view0.04640.0617+0.0153 [0.0104, 0.0203]5/5
Donors approximately matched to the conditional law0.04730.0716+0.0243 [0.0189, 0.0306]5/5
Views lose access together, 8-frame blocks0.10160.1274+0.0259 [0.0173, 0.0343]5/5
Views lose access together, 16-frame blocks0.07470.0945+0.0198 [0.0149, 0.0241]5/5

For the final two rows, views share an access state with probability 0.5; their absolute errors are not directly comparable to the other rows. One representation selected zero noise on its training images. Pong results appear in Appendix F.

Experimental protocol. All new tests were specified before execution. Frozen models and, except when explicitly varied, encodings, masks, donors, and Monte Carlo draws were retained. Representations are the statistical units (five Voronoi, three Pong); 95% paired percentile CIs use 20,000 resamples. Appendix F reports full results.

Interpreting the limits. Causal representations and dynamics remain frozen; only access and measurement evidence change.

• Shared donors: Repeated identical donors are effectively one measurement. Appendix A.4 formalizes the worst-case ambiguity between a true state change and loss of access given a prediction and one measurement. χ² gating can reject shared erroneous values, but can also reject real recovery: in the first two frames after access returns in the correlated-loss test, it rejects valid Pong views on 31% of frames versus 6% for RME. With one view, RME improves valid-frame accuracy and overall R² but harms hidden-frame accuracy.

• No valid view. Independent access priors assign only 0.8% mass to this case, leaving its posterior weight too small. Raising the prior reduces, but does not eliminate, the harm (see reply to gtDA).

With the causal model fixed, whether an update helps depends on the available evidence and how access changes. This is precisely the distinction TCO formalizes. These cases motivate explicit models of correlated loss and shared replacements.

Changes in revised PDF. Abstract: new controls, three-view scope, and limitations. Section 4: explicit connection from Proposition 2 to RME via expected correction benefit (reply to coY2). Section 5: Table 5, recursive robust-filter and grouping controls, and empirical detectability results instead of the earlier “consistent with Proposition 1” phrasing. Section 6: revised limitations. New Appendix F: full experimental settings, results, and paired intervals.

The central contribution is the post-identification problem of temporal causal observability and its information boundary. Our experiments establish when regime-marginalized updates can maintain an identified causal state under changing perceptual access, and where the available evidence is insufficient.

AuthorsOpenReview ↗

Summary for the Area Chair

As the discussion closes, we summarize our contribution and the evidence for it.

Our main contribution is the problem itself: we formalize it and show that it arises in current temporal CRL models. We then give a first solution that follows from its information boundary, and test where it works and where it fails.

The problem. Changing access alone lowers factor recovery to nearly the value of the closed-form prediction in Eq. 4, across four temporal CRL backbones, even where full-access recovery is near perfect (R² 0.985 on CITRIS–Voronoi). Retraining the encoder on partially observed data does not recover it, and with donors approximately matched to the conditional law, the tested single-frame detectors are near chance.

The boundary. Maintenance therefore needs repeated evidence. With a prediction and one measurement, no rule can both accept every real change and reject every donor (Appendix A.4); our one-view experiments confirm this.

A first solution. RME uses exactly this repeated evidence: it weighs latent access regimes across synchronized views and time instead of deciding validity frame by frame. With three views, it has lower hidden-state error than matched hard decisions in every tested representation, also without known boundaries, with a common-error channel, and with conditional donors. It fails when views share the same erroneous evidence, which then acts as a single measurement, and when no view is valid. These cases are where better solutions are most needed, and our protocol makes them testable.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.