acceptodds
Under review as a conference paper at ICLR 2027

QUASAR: STRUCTURE-PRESERVING DENSITY-STATE DYNAMICS FOR RECURSIVE VIDEO PREDICTION

Abstract

Visual world models increasingly rely on structured latent dynamics, yet they are still evaluated mainly through decoded frames. In recursive prediction, latent states are fed back into later forecasts, so a model can produce accurate frames while its states leave the space its representation declares valid, a failure that standard forecasting metrics cannot detect. We introduce QUASAR, a video predictor whose local latent states are density matrices: symmetric, positive semidefinite, and unit trace. Motion-conditioned orthogonal transport preserves each state’s spectrum, and learned Kraus channels, made complete by thin QR factorization, redistribute spectral mass. Every recurrent update is therefore completely positive and trace preserving (CPTP), so states remain admissible by construction rather than by post-hoc correction. Experiments show that forecast accuracy and state validity are distinct properties. On Human3.6M, an unconstrained transition matches QUASAR’s four-frame mean squared error (MSE), yet its minimum eigenvalue reaches−1.36×10−3, versus−8.79×10−9 for CPTP. Among matched transition controls under 16-frame recursion, CPTP gives the lowest mean MSE and highest mean structural similarity (SSIM) on both evaluation cohorts, and the lowest mean MSE at all three tested temporal rates. As a predictor, QUASAR reduces Human3.6M MSE by 17.95% relative to the strongest reproduced baseline and attains the highest SSIM on TaxiBJ and KITTI/Caltech, though not the lowest MSE on either. Added to PredFormer, the density-dynamics module reduces MSE by 14.69% with about 1.5% more parameters, although its gains vary across metrics. Density entropy is moderately associated with an occlusion proxy but only weakly with prediction error. These results indicate that admissible latent dynamics can be enforced without sacrificing forecast quality, and that latent states merit auditing alongside decoded frames.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video prediction, long-horizon forecasting, structured latent dynamics, density-state representation, world models, CPTP dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qZSuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes QUASAR, a video prediction framework that represents latent states as density matrices and evolves them through completely positive and trace-preserving (CPTP) transitions. The method combines motion-conditioned orthogonal transport, Kraus operators, and density-guided feature modulation to preserve the mathematical validity of latent states during recursive prediction. Experiments on several video prediction benchmarks show competitive forecasting performance and improved state validity.

Critical strengths
  1. The paper explores an interesting direction by introducing explicit structural constraints into recurrent video prediction. The CPTP formulation provides a mathematical guarantee that the latent states remain within the defined density-matrix space.

  2. The experiments cover multiple datasets and include controlled comparisons with unconstrained transitions, Gram repair, and eigenvalue projection. The results on Human3.6M and the PredFormer backbone demonstrate the potential usefulness of the proposed approach.

Critical weaknesses
  1. The motivation for density-state constraints is not sufficiently justified. While the proposed formulation guarantees that latent states remain positive semidefinite with unit trace, it is unclear why these particular constraints are necessary for video prediction. Violating the density-matrix definition does not necessarily imply physically invalid or inaccurate visual dynamics.

  2. The empirical evidence does not fully establish the necessity of CPTP transitions. In Table 6, eigenvalue projection slightly outperforms CPTP in short-term forecasting. Although CPTP performs better under 16-frame recursion, the improvements over other structure-preserving alternatives are relatively modest. Moreover, the main recursive evaluation re-encodes the latest four frames, while the persistent-state results in Appendix B are nearly identical. These observations make it difficult to determine whether the gains primarily come from the proposed state-preserving dynamics.

Questions for authors
  1. What properties of real video dynamics motivate the use of positive-semidefinite, unit-trace latent states? Can the authors provide evidence that these constraints capture meaningful visual or physical structure beyond mathematical admissibility?

  2. How much of the forecasting improvement can be specifically attributed to CPTP transitions rather than the additional network components or generic structural regularization? In particular, how should the results of eigenvalue projection and persistent-state propagation be interpreted?

  3. Could the authors provide more architectural details for the motion-guided transport and temporal refiner, together with inference-time and memory comparisons against similarly sized baselines?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qZSu

Thank you for asking what the density constraint contributes and how much of QUASAR’s performance and cost can be attributed to the transition. We added a three-seed fixed-spectrum control and a same-session latency comparison. At the four-frame training horizon, the full model achieved better mean results on all four rollout metrics, with 3.3% higher median latency than the control.

Q1. Why require positive semidefiniteness and unit trace?

These constraints preserve the interpretation of the local state throughout a rollout. Eqs. (1–4) initialize each state ρ as a weighted mixture of outer products of local feature directions. Positive semidefiniteness and unit trace ensure that, after each recurrent update, its eigenvalues remain nonnegative and sum to one. They describe how the state distributes weight across latent modes. Entropy and purity measure the concentration of those weights. The same constraints bound the Frobenius norm of ρ by one at every forecast step.

This interpretation has an empirical counterpart. Density entropy correlates with an independently computed occlusion proxy (Spearman correlation 0.5347, p = 0.010), while its correlation with prediction error is 0.06 (Table 8C). The visibility association gives a concrete visual interpretation to the concentration of the learned spectrum.

Q2. What does the CPTP transition contribute?

The new control allows the state to move while holding its spectrum fixed. Both variants retain motion-conditioned orthogonal transport and share the allocated parameter count, objective, training schedule, and seeds (43–45). Orthogonal conjugation can rotate the eigenbasis while preserving every eigenvalue. The full model applies the Kraus channel after transport, allowing it to reweight the modes while preserving admissibility.

Over 64 persistent steps, the control exhibited nonzero state displacement in every run while entropy and purity stayed constant within 10⁻⁵. With the Kraus channel, both quantities changed in every run. Entropy rose in two seeds and fell in the third, showing that the model uses the channel both to spread and to concentrate spectral mass.

Forecast accuracy depends on the horizon. At four frames, full CPTP achieved lower mean MSE on full validation (1.025 vs. 1.038, both × 10⁻³). On the rollout cohort of 264 windows from 22 validation videos, it achieved lower mean MSE and MAE and higher mean SSIM and PSNR. At 16 frames, all four metrics differed by at most 0.3%. At 64 frames, the fixed-spectrum control achieved lower mean MSE (5.817 vs. 6.381, both × 10⁻³).

Projection and Gram repair test a different transition choice. Both restore admissibility after an unconstrained update. The Kraus transition preserves admissibility by construction. Projection achieved the lowest four-frame MSE in Table 6 (0.9693 vs. 0.9935 for CPTP, both × 10⁻³). Under 16-frame recursion, CPTP achieved the best mean MSE and SSIM in both cohorts among the controls in Table 4, and the lowest MSE at every temporal rate in Table 7. These controls share the visual architecture, objective, optimization schedule, and training horizon. In PredFormer, holding the backbone and FiLM location fixed, CPTP reduced MSE by 14.69%, compared with 12.08% for Gram repair and 6.13% for generic recurrence (Table 5).

Persistent propagation tests whether re-encoding explains the result. Carrying the state across forecast blocks gave essentially the same accuracy as blockwise re-encoding in Table 8A. In the new paired comparison, the two rollout modes also agreed closely at every tested horizon for both variants.

Q3. What are the architecture and inference costs?

QUASAR warps observed features at three scales and applies local ρ-FiLM before decoding. Each coarse forecast conditions the next state transition. A video corrector and a joint refiner then produce the final frames, with the pooled density trajectory conditioning the joint refiner.

We timed all four models in one session on the same RTX 3090 at batch size 1 and 256 × 256 resolution, using FP32 with TF32 disabled. Values are medians of 20 timed runs after five warm-ups.

ModelParameters4 frames16 frames
QUASAR (full CPTP)46.87M216.2 ms873.9 ms
Orthogonal-only46.87M209.3 ms847.7 ms
SimVP+IncepU41.16M51.0 ms204.4 ms
PredFormer12.73M21.1 ms80.7 ms

Enabling the Kraus channel added 6.9 ms at four frames (3.3%) and 26.2 ms at 16 frames (3.1%). The orthogonal-only row measures the shared QUASAR pipeline with the channel bypassed. Table 5 gives a complementary deployment result: attaching the density module to PredFormer increased its parameter count by 1.52%. The measurements reported here cover latency and parameter count.

We respectfully ask you to reconsider your assessment in light of the matched control, the spectral measurements, and the measured cost of enabling the channel.

Reviewer_tMMnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes QUASAR, a video prediction model that models the latent explicitly as density matrices, and a principled update mechanism that ensures state feasibility at transition by construction. Empirical results demonstrate that QUASAR's constrained transition mechanism allows to predict video frames on a decent level while keeping the state admissible.

Critical strengths
  1. The idea to model the latent states of a video prediction model as density matrices is generally interesting and the structured way of updating the state to preserve its feasibility makes sense.
  2. The paper signals that explicit state modeling in video prediction can be achieved without loss in prediction quality, which in theory can be significant if evaluated on more complex data, longer horizons and larger model capacities.
Critical weaknesses
  1. Motivation is rather weak. The paper claims that video prediction models are typically evaluated on pixel quality which can hide the problem of latent state admissibility, i.e. when a model predicts correct pixels while its latent state drift away from valid representation regions. This claim is unsupported in the paper, unconstrained transition yields both larger MSE and weaker state admissibility. Moreover, intuitively when the state drifts away at inference due to recurrent prediction and exposure bias the quality of the predicted frames degrades, and this is a known issue in video prediction. Related to this, despite motivating the method with the problem of state admissibility on longer horizons, the comparisons are done on horizons up to 16 frames, with most of the comparisons on horizons from 1 to 4 frames. Comparisons and ablations on longer horizons, more complex data and larger model capacities are essential for a video prediction paper.
  2. The clarity of the paper can be improved. While most of the presentation of the method is clear if read carefully, some of the parts look unmotivated and overcomplicated with missing explanations. The paper does not do a great job in providing theoretical background and backing the claims with formal proofs, e.g. why the admissibility is preserved at every step. The paper lacks implementation details for several components, such as context encoder, motion encoder, temporal refiner, etc.
Questions for authors

Could the authors clarify the following?

  • It appears that the paper never defined L that is used in equation 11.
  • Equation 15 introduces Eh\cal E_h, but equation 19 uses Ehn\cal E_h^n.
  • Section 3.4 uses several variables that were not defined above, such as eTe_T and qhq_h.
2 comments
AuthorsOpenReview ↗

Response to Reviewer tMMn

Thank you for pressing us on the motivation and soundness of the state update. Table 6 shows why admissibility needs its own test. At the four-frame training horizon, the unconstrained and CPTP transitions achieve nearly identical pixel MSE, yet the unconstrained transition already produces an invalid density state.

Four-frame result (Table 6)UnconstrainedCPTP
Forecast MSE0.00099360.0009935
Minimum eigenvalue−1.36 × 10⁻³−8.79 × 10⁻⁹

W1. What does admissibility add beyond forecast accuracy?

The MSE difference is only 10⁻⁷, while the negative eigenvalue magnitudes differ by more than five orders of magnitude. For a density state, a negative eigenvalue violates the interpretation of its spectrum as weights over latent modes. CPTP preserves that interpretation in exact arithmetic, with a residual at floating-point scale in Table 6. Under 16-frame recursion, the unconstrained transition also has the highest mean MSE among the Table 4 controls. The contribution is a recurrent state whose validity can be proved and audited even when decoded frames give the two transitions essentially the same score.

Longer horizons. We added 64-frame rollouts and trained the evolving, fixed-spectrum control described in the General Response. Both variants use the same motion-conditioned orthogonal transport, allocated parameter count, objective, and training recipe across three paired seeds. The control’s state moves while its entropy and purity remain fixed within 10⁻⁵. The full model changes both quantities in every seed. At four frames, it achieves lower mean validation MSE (1.025 vs. 1.038, both × 10⁻³) and better mean results on all four rollout metrics. At 16 frames, all four metrics differ by at most 0.3%. At 64 frames, the fixed-spectrum control has lower mean MSE (5.817 vs. 6.381 × 10⁻³, with both values on the same scale). The admissibility guarantee holds at every horizon. The measured forecast advantage depends on the horizon and comparator.

Data and capacity. Table 3 evaluates five JVET sequences at resolutions up to 3840 × 2160. QUASAR outperforms both SimVP+IncepU and persistence on every sequence and every reported metric. Table 5 tests the transition in a second backbone while fixing the PredFormer architecture, FiLM location, and training recipe across variants. CPTP reduces MSE by 14.69%, compared with 12.08% for Gram repair and 6.13% for generic recurrence. Adding the density module increases PredFormer’s parameter count by 1.52%.

W2. Why does every recurrent state remain admissible?

Proposition. For fixed conditioning, each update maps a positive semidefinite, unit-trace state to another positive semidefinite, unit-trace state.

  1. Initialization. For ε > 0, FFᵀ + εI is positive definite. Dividing by its trace gives unit trace. Orthogonal conjugation preserves eigenvalues, and the convex mixture in Eq. (6) preserves admissibility. All reported runs use the fixed gate α = 1.
  2. Kraus update. At a fixed step, the identity block gives the stacked matrix full column rank. Thin QR yields VᵀV = I, so its row blocks satisfy ∑ⱼ KⱼᵀKⱼ = I. The map E(ρ) = ∑ⱼ KⱼρKⱼᵀ is completely positive by its Kraus form. By cyclicity, Tr(E(ρ)) = Tr(ρ ∑ⱼ KⱼᵀKⱼ) = Tr(ρ).
  3. Recurrence. Every composition of E is completely positive and trace-preserving (CPTP), including the identity at order zero. The weights in Eq. (19) are nonnegative and sum to one because the last weight collects the remaining Poisson mass. Their mixture is therefore CPTP. Applying this argument at each step proves the proposition by induction.

Implementation and notation

  • L in Eq. (11) counts the motion-conditioned operator pairs from Eqs. (9–10). Including the identity gives J = 2L + 1. We use L = 8, hence J = 17.
  • Eqs. (15) and (19). Eq. (15) defines one channel application. Eq. (19) averages its n-fold compositions applied to the state after orthogonal transport. At n = 0, the state passes through unchanged.
  • Section 3.4. At scale s, transport uses the last three observed encoder features to form velocity and acceleration. The step embedding is added to the motion vector and shared across scales. The transport network predicts a flow and feature residual. Local ρ-FiLM then supplies the scale Γ and shift β before the decoder predicts its bounded residual and mask. The response to f6a7 gives the layer dimensions, two refinement stages, loss weights, and supervision targets.

We respectfully ask you to reconsider the soundness assessment in light of the stepwise proof and the matched control. Together with the state audit, they establish what the transition guarantees, show that the model uses its spectral flexibility, and quantify how its forecast accuracy changes with horizon.

Reviewer_tMMnOpenReview ↗

Response to the rebuttal

Thank you for the additional clarifications. Unfortunately, the rebuttal was published quite late and there was not much time left towards the end of the discussion period to carefully check the new results. I will therefore keep my original rating and reconsider my assessment later only after checking the rebuttal in detail.

Reviewer_f6a7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QUASAR uses local density-matrix states for recursive video prediction. Orthogonal transport preserves each state’s spectrum. Conditional Kraus channels allow the spectrum to change while keeping the state valid. Thin QR factorization enforces channel completeness. The states modulate visual features through FiLM. A temporal refiner produces the final frames.The paper evaluates forecast quality and state validity. Experiments include several update controls, sixteen-frame recursion, persistent-state propagation, different temporal rates, high-resolution sequences, and PredFormer transfer. CPTP gives the best observed sixteen-frame means in both cohorts.

Critical strengths

1.A useful design for local video states. QUASAR combines local density-matrix states with motion-conditioned updates and visual feature modulation. Orthogonal transport changes state directions, while Kraus channels can change their spectral weights. FiLM connects these states to frame prediction. This provides a concrete way to combine structured state dynamics with a video predictor.
2. Competitive prediction performance and useful transfer results. QUASAR performs well across several datasets. Controlled tests show gains under extended recursion. Its module also improves another prediction backbone with a small parameter increase. The state updates preserve validity by construction, with support from numerical checks.

Critical weaknesses

1.The benefit of changing the spectrum is not directly tested. Gram repair and eigenvalue projection are useful controls, but both can change the spectrum. Identity keeps the state fixed. None directly tests an evolving state with a fixed spectrum. CPTP improves sixteen-frame MSE over Gram repair by about 1.6–2.1%, while projection gives better four-frame MSE. These gains are useful, but the role of spectral redistribution remains unclear. This matters because it is a main difference from spectrum-preserving dynamics.
2.The full prediction pipeline needs a clearer description. Equation 22 does not explain how visual transport is computed. The refiner’s inputs are listed, but its structure is not given in enough detail. It is also unclear which losses apply to the coarse and final predictions. These parts directly affect forecast quality. The missing details make the full method and its training hard to assess.

Questions for authors

1.What does allowing the state spectrum to change add to prediction quality, compared with spectrum-preserving evolution?
2.Please describe the visual transport and temporal refiner, including their main computations and losses. For the full-model comparisons, do the baselines use the same loss functions and training budgets?

2 comments
AuthorsOpenReview ↗

Response to Reviewer f6a7 (1/2)

W1/Q1. What does changing the spectrum add when the state can already evolve?

Thank you for identifying the missing comparison. We trained the evolving, fixed-spectrum control you proposed. The control changes its state at every tested seed while preserving its spectrum. The full model uses the additional freedom to reweight its latent modes in all three seeds and achieves better mean results on all four rollout metrics at the four-frame training horizon.

What the control isolates. Both models retain the motion-conditioned transport in Eqs. (7–8). The orthogonal-only model updates each local state by ρ ↦ UρUᵀ, where U = exp(−Δt A) and Aᵀ = −A. This permits the eigenbasis to rotate while preserving every eigenvalue. The full model applies the learned Kraus channel after the same transport (Eq. 20), allowing the eigenvalues to change while preserving admissibility.

Both models have 46,867,574 allocated parameters. For each seed (43, 44, 45), we trained both variants with the Table 2 objective and recipe for 50 epochs, accumulated gradients over four minibatches, evaluated with EMA weights, and selected checkpoints by validation SSIM.

Does each model use the dynamics available to it? Over 64 persistent steps, the orthogonal-only control has nonzero state displacement in every run while entropy and purity remain constant within 10⁻⁵. With the Kraus channel, both quantities change in every run. Entropy rises and purity falls for seeds 43 and 44. Seed 45 shows the reverse. The learned channel can therefore both spread and concentrate spectral mass.

SeedModelEntropy (nats), step 1 → 64Purity, step 1 → 64Mean displacement
43Orth-only0.7635 → 0.76350.5559 → 0.55590.213
43Full CPTP0.4851 → 0.99400.7732 → 0.58290.096
44Orth-only0.4404 → 0.44040.7809 → 0.78090.136
44Full CPTP0.7507 → 0.78810.6205 → 0.61080.040
45Orth-only1.0535 → 1.05350.4251 → 0.42510.147
45Full CPTP1.4732 → 1.10320.2758 → 0.40180.017

Displacement is the Frobenius distance between consecutive local states, averaged over grid locations and steps.

When does spectral flexibility help prediction? At four frames, the full model achieves lower mean MSE on both the full validation set and the rollout cohort (264 windows from 22 validation videos). On that cohort it also achieves lower mean MAE and higher mean SSIM and PSNR.

Horizon and evaluation setOrth-onlyFull CPTP
4 frames, full validation1.038 ± 0.0621.025 ± 0.040
4 frames, rollout cohort1.008 ± 0.0521.001 ± 0.054
16 frames, persistent2.818 ± 0.1002.823 ± 0.161
64 frames, persistent5.817 ± 0.6586.381 ± 1.905

Values are MSE × 10³ (mean ± sample SD across seeds 43–45). At 16 frames, all four metrics differ by at most 0.3%. At 64 frames, the orthogonal-only model has lower mean MSE. Table 2 reports separate full-model runs (seeds 0–2, mean 0.9935 on the same scale). The new full-validation mean of 1.025 also remains below the strongest Table 2 baseline (1.211).

How this relates to Gram repair and projection. As you noted, projection achieves 2.4% lower four-frame MSE than CPTP in Table 6 (0.9693 vs. 0.9935 on the scale above). Gram repair and projection restore admissibility after an unconstrained update. The Kraus transition preserves it by construction. Under 16-frame recursion, CPTP achieves lower MSE than both repairs in both cohorts (Table 4). The PredFormer transfer provides a separate test with the backbone and FiLM location held fixed. There, the MSE reductions are 14.69% for CPTP, 12.08% for Gram repair, and 6.13% for generic recurrence (Table 5).

The new control answers Q1 at both levels: the spectral measurements establish what changes in the state, and the paired forecasts show where that flexibility is useful. Its clearest measured benefit is at the horizon used for training. We will include the control, spectral diagnostics, and full 4–64-frame curves in the appendix.

AuthorsOpenReview ↗

Response to Reviewer f6a7 (2/2)

W2/Q2. How do transport, refinement, and supervision contribute, and which comparisons isolate the transition?

The transition controls share the visual architecture, objective, optimization schedule, and four-frame training horizon (Tables 4, 6, 7). Thus the decoder, refiner, and loss assignment described below are common to those comparisons. Table 2 addresses the separate question of full-model performance under shared data and evaluation protocols.

1. Motion-guided visual transport (Eq. 22). For Human3.6M (4 observed → 4 predicted frames at 256 × 256), two 3 × 3 convolutions (384 → 192 → 64, group normalization and GELU) and global average pooling encode the last two bottleneck features into a motion vector m. At step h, we add a learned 64-dimensional step embedding and broadcast the sum at 256², 128², and 64², with C = 48, 96, and 192 channels, respectively.

At each scale, the local-motion network reads the last two observed features, their first difference v = e_T − e_{T−1}, and their second difference a = e_T − 2e_{T−1} + e_{T−2}. Two 3 × 3 convolutions (4C → 64 → 64) encode these inputs. Adding the broadcast motion and step embedding yields u. Separate flow and residual heads both read [u, e_T, e_{T−1}, v] and predict a two-channel flow f and a C-channel residual r. With τ = h/P and feature-map height H, the update is z = Ref(Warp(e_T, 0.3 H τ f)) + τ r. Warp uses bilinear sampling with border padding. Ref applies a 3 × 3 convolution, group normalization, and GELU.

2. Local density conditioning and feedback (Eqs. 23, 24, 16). At each of the 32² grid locations, LayerNorm and a 64 → 192 → 192 MLP with GELU encode the current density matrix. Each scale head uses a 1 × 1 convolution followed by one, two, or three stride-2 transposed convolutions to reach 64², 128², or 256². Its 2C outputs split into Γ and β, giving the pointwise update (1 + Γ) ⊙ z + β. The decoder predicts a bounded residual and mask to form each coarse frame. That frame conditions the next state transition through the feedback head.

3. Temporal refinement (Fig. 1). Both stages receive the observed frames directly. First, a nine-block U-Net (widths 96/192/384) corrects the coarse forecasts using observations, predictions, temporal differences, the last observation, and observed velocity. Second, the joint refiner stacks the four observed and four corrected RGB frames into 24 channels. A 5 × 5 stride-2 convolution maps them to 896 channels at 128², followed by six residual blocks with dilations 1, 2, 4, 1, 2, 4.

The joint refiner uses the full predicted density trajectory. We detach it at this branch, average each field over its 1,024 locations, and flatten each result into a 64-dimensional token. LayerNorm, four-head temporal attention, and temporal averaging produce a descriptor. A 64 → 1792 projection yields a and b for F ← F ⊙ [1 + 0.1 tanh(a)] + 0.1b. A zero-initialized head predicts a residual and gate. After bilinear upsampling, we add 0.5 tanh(r) ⊙ sigmoid(g) to the corrected frames and clip to [0, 1].

4. Loss assignment (Eq. 25).

Supervised outputLoss terms and weights
Final predicted framesMSE 0.30, SSIM 1.00, VGG perceptual 0.05, LPIPS 0.01, temporal 0.05
Density statesEntropy (Eq. 26) 0.10, trace error 0.005

The temporal loss is the mean absolute second difference of the predicted frames. The coarse pathway receives its reconstruction gradients through both refinement stages. The displayed Eq. (25) omitted the trace-error term used in training. We will restore it in Eq. (25) and Table 1. For CPTP, the maximum observed trace error was 2.38 × 10⁻⁷ (Table 6).

5. Baseline budgets and attribution. We trained all Table 2 baselines using official or OpenSTL implementations and method-specific objectives, including MSE for PredFormer and L1 with auxiliary losses for SVQ. Each dataset used the same split, resolution, and evaluation protocol across methods.

DatasetEpochsResolutionForecast protocol
Human3.6M50256²4→4, source-video-grouped validation (22 videos, 2,026 windows)
TaxiBJ5032²4→4
KITTI/Caltech60128²10→1, common 75-window evaluation

For transition attribution, Tables 4, 6, and 7 hold the visual pipeline and training recipe fixed. The PredFormer study also fixes the backbone, FiLM location, split, and seed pair (Table 5). Table 8 changes only the rollout mode of the same trained model.

We will add the architecture table, pseudocode, and complete loss assignment to the revision. In light of the new control addressing W1 and the training and comparison details addressing W2, we would appreciate your reconsideration of the assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

3. Why is every recurrent state admissible?

Proposition. For fixed conditioning, the update at each forecast step and grid location maps a symmetric positive semidefinite matrix of unit trace to another such matrix. Consequently, every recurrent state is admissible in exact arithmetic.

(i) Initialization. For ε > 0, FFᵀ + εI is symmetric positive definite. Dividing by its trace gives an admissible initial density (Eq. 4).

(ii) Orthogonal update. Orthogonal conjugation preserves symmetry, positive semidefiniteness, trace, and spectrum. Since the admissible set is convex, the mixture in Eq. (6) remains admissible for every gate value in [0, 1].

(iii) Kraus channel. Fix a forecast step and omit its index. The identity block gives B full column rank, so its thin QR factorization yields VᵀV = I. The row blocks of V therefore satisfy ∑ⱼ KⱼᵀKⱼ = I (Eq. 14). The map E(ρ) = ∑ⱼ KⱼρKⱼᵀ is completely positive by its Kraus form and preserves symmetry. Cyclicity of the trace gives Tr(E(ρ)) = Tr(ρ ∑ⱼ KⱼᵀKⱼ) = Tr(ρ). Thus E is completely positive and trace-preserving (CPTP).

(iv) Poisson mixture. Every iterate of E is CPTP, including the identity at order zero. The weights in Eq. (19) are nonnegative and sum to one because the final weight collects the remaining Poisson mass. Their convex combination is therefore CPTP.

(v) Induction. Initialization gives an admissible state at every location. At each subsequent step, both orthogonal transport and the channel mixture preserve admissibility. Induction over the forecast steps gives Eq. (21).

In FP32, the implementation uses only symmetrization and scalar trace normalization to correct numerical drift. Tables 6 and 8B report the residuals. The maximum trace error in Table 6 is 2.38 × 10⁻⁷.

4. Architecture, supervision, and experimental protocol

Context and initialization. The per-frame U-Net uses group normalization and GELU. Its skip features have 48, 96, and 192 channels at 256², 128², and 64². The 32 × 32 × 192 bottleneck defines the density grid. Two heads, each using a depthwise 3 × 3 convolution followed by a 1 × 1 convolution, produce four 8-dimensional vectors and four mixture logits per location (Eqs. 1–4). An eight-head temporal attention layer processes the 64-dimensional frame-level density tokens with a learned density-similarity bias. An MLP maps the final token to an 8 × 8 matrix R, giving A_c = 0.05(R − Rᵀ). All reported runs fix the initialization gate at 1, so Eq. (6) orthogonally conjugates the final observed density at each location.

Forecast pathway. Two 3 × 3 convolutions and global average pooling map the last two bottleneck features to a 64-dimensional motion vector (384 → 192 → 64 channels). At each scale, transport uses observed features, velocity, and acceleration to predict a flow and a residual. Local ρ-FiLM modulates the transported features before decoding. Each coarse forecast conditions the next state transition. A video corrector and a joint refiner produce the final frames, with the pooled density trajectory conditioning the joint refiner. The response to f6a7 gives the layer dimensions and update equations.

Losses and training. The final frames receive MSE (0.30), SSIM (1.00), VGG perceptual (0.05), LPIPS (0.01), and temporal (0.05) losses. Reconstruction gradients reach the coarse pathway through both refiners. The density states receive entropy (0.10) and trace-error (0.005) terms. The trace-error term was used in training and omitted from the displayed Eq. (25). We accumulated gradients over four minibatches (effective batch size 4), used mixed precision with density and channel computations in FP32, evaluated with EMA weights (decay 0.9995), and selected checkpoints by validation SSIM.

Seed provenance. The QUASAR entry in Table 2 averages seeds 0, 1, and 2. Table 6 reports forecast means and a seed-0 state audit. The new orthogonal-only and full-CPTP comparison uses paired seeds 43, 44, and 45.

Inference cost. On the same RTX 3090 at batch size 1 and 256 × 256 resolution, enabling the Kraus channel increased median latency from 209.3 to 216.2 ms at four frames (+3.3%) and from 847.7 to 873.9 ms at 16 frames (+3.1%). Both variants have 46.87M allocated parameters.

We respectfully ask the reviewers to reconsider their assessments in light of the admissibility proof, the paired fixed-spectrum experiment, and the measured forecasting and computational trade-offs.

AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank the reviewers for their careful reading of QUASAR. Our response focuses on two questions raised across the reviews. What does a valid density state tell us about a forecast, and when does allowing its spectrum to change improve prediction? Table 6 shows why the distinction matters while two transitions achieve nearly identical four-frame MSE, yet the unconstrained transition produces an invalid density state. To test the role of spectral change directly, we trained a fixed-spectrum control across three paired seeds. At the four-frame training horizon, the full model achieved lower mean validation MSE and better mean results on all four rollout metrics. Below we report how that comparison changes with forecast length, then give the admissibility proof and implementation details.

1. What does the Kraus channel add to an evolving state?

We trained the orthogonal-only control suggested in the reviews. Both variants retain motion-conditioned transport (Eqs. 7–8), have 46.87M allocated parameters, and use the Table 2 objective and training recipe. We trained both on the same three seeds (43, 44, 45). The control updates each local state by ρ ↦ UρUᵀ. Orthogonal conjugation can rotate the eigenbasis while preserving every eigenvalue. The full model then applies the Kraus channel, which allows the eigenvalues to change.

The control’s state moves, and the full model uses its spectral flexibility. Over 64 persistent steps, the control had nonzero mean state displacement in all three runs while entropy and purity stayed constant within 10⁻⁵. With the Kraus channel, entropy changed from 0.49 to 0.99, 0.75 to 0.79, and 1.47 to 1.10 nats across the three seeds. The channel can therefore both spread and concentrate spectral mass during a rollout.

At the training horizon, both evaluations favor the full model. On full validation, it achieved lower mean MSE and MAE. On the rollout cohort (264 windows from 22 validation videos), it achieved lower mean MSE and MAE and higher mean SSIM and PSNR.

EvaluationMetric (× 10³)Orthogonal-onlyFull CPTP
4 frames, full validationMSE1.0381.025
4 frames, full validationMAE8.898.73
4 frames, rollout cohortMSE1.0081.001
16 frames, persistentMSE2.8182.823
64 frames, persistentMSE5.8176.381

At 16 frames, all four metrics differed by at most 0.3% between the two variants. At 64 frames, 16 times the training horizon, the orthogonal-only control achieved lower mean MSE. Among the original transition controls in Table 4, CPTP achieved the best mean MSE and SSIM in both 16-frame cohorts. It also achieved the lowest MSE at every temporal rate tested in Table 7.

2. Why audit state validity separately from forecast accuracy?

At four frames, the forecast MSE gap is only 10⁻⁷, while the magnitudes of the minimum eigenvalues differ by more than five orders of magnitude.

Four-frame result (Table 6)UnconstrainedFull CPTP
Forecast MSE0.00099360.0009935
Minimum eigenvalue−1.36 × 10⁻³−8.79 × 10⁻⁹

The unconstrained transition assigns negative weight to a latent mode, violating the density-state interpretation. Trace normalization alone also permits a negative eigenvalue (−2.85 × 10⁻³). For CPTP, the residual is at floating-point scale. The state audit thus detects a violation at the training horizon even when forecast MSE is nearly identical. At 16 frames, the unconstrained transition also has the highest mean MSE in both Table 4 cohorts.

Full-model and transfer results. QUASAR achieved the best MSE, MAE, SSIM, and PSNR among the reproduced Human3.6M baselines, including 17.95% lower MSE than the strongest baseline (Table 2). It also outperformed both comparators on every reported metric across all five JVET sequences, at resolutions up to 3840 × 2160 (Table 3). In PredFormer, adding the density module reduced MSE by 14.69% with 1.52% additional parameters (Table 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.