acceptodds
Under review as a conference paper at ICLR 2027

FL4N: Feed-forward Latent Model for 4D Novel View Synthesis

Abstract

4D Novel View Synthesis (NVS) from monocular videos aims to render dynamic scenes across viewpoints and timestamps, requiring jointly inferring 3D structure and temporal dynamics. Existing feed-forward methods rely on explicit 3D Gaussian representations, which are prone to temporal inconsistency and severe rendering artifacts. We present FL4N, the first feed-forward latent 4D NVS model. We bypass explicit 3D primitives by weaving a geometrically grounded scene prior into a unified latent representation across space and time. To overcome dynamic training data scarcity, we learn a temporal residual on top of a frozen, pretrained static latent backbone. A lightweight dynamic encoder learns to isolate true scene motion from camera parallax. Trained purely on image synthesis with binary mask guidance, FL4N achieves state-of-the-art rendering quality for 4D NVS in realtime. Our proposed latent representation successfully learns a strong 4D prior, generalizes across novel views and timestamps, and cleanly decouples view and time in the latent space. Codes will be publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
4D, Latent-based NVS

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f9b3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FL4N addresses feed-forward 4D novel view synthesis from monocular videos. It augments a frozen pretrained static latent view-synthesis model with a lightweight dynamic encoder and a target-time adaptation module. The dynamic encoder cross-attends to static features to estimate temporal residuals. These residuals are conditioned on learned time tokens, added to the static latent, and decoded by the frozen camera-conditioned renderer. Intermediate timestamps are handled through time-token interpolation. Training uses image reconstruction losses and binary dynamic-mask pseudo-labels without dense geometry or 3D-motion supervision.

Critical strengths
  1. Effective reuse of a static latent prior. The residual formulation offers a practical route from pretrained static synthesis to dynamic synthesis while preserving the original renderer.

  2. Strong cross-time and novel-time results. These tasks directly test the proposed temporal control. Pose-free FL4N outperforms the evaluated Gaussian baselines across all reported metrics in Tables 2 and 3.

  3. Fast reported inference. FL4N achieves 40 FPS at 224×224224\times224, making the approach practically relevant for interactive synthesis.

Critical weaknesses
  1. The data-efficiency claim is insufficiently quantified. Reusing a static prior is compelling, but no training-data scaling experiment establishes how much dynamic data is saved.

  2. The contribution of the residual formulation is not completely isolated. A capacity-matched, explicitly time-conditioned fine-tuning baseline would help distinguish the proposed design from alternative adaptation strategies.

  3. Some conclusions exceed the evidence. The results demonstrate effective view–time conditioning, but “clean disentanglement” is stronger than what the visualizations and probes establish. Pose-free superiority is also not universal in Table 1.

  4. Generalization remains dependent on the static backbone. The acknowledged failures under viewpoint extrapolation and substantial camera errors constrain claims of arbitrary-view synthesis.

Questions for authors
  1. How does performance change when training with progressively smaller fractions of Kubric4D and CamxTime? Can this substantiate the claimed dynamic-data efficiency?

  2. How does FL4N compare with a parameter-matched adaptation baseline using explicit temporal conditioning, such as adapters or LoRA in the static backbone?

  3. Does the 40 FPS measurement include context encoding, and how does runtime scale with the number of context frames?

  4. How are the 768-dimensional backbone features and the reported 1024-dimensional time tokens made compatible?

  5. Since the time tokens begin as copies of a shared learned token, how reliably does interpolation handle irregular sampling and visually ambiguous or periodic motion?

5 comments

W1, Q1: Training-data scaling

We thank the reviewer for this valuable suggestion.

To isolate the effect of the residual formulation, we construct a single-encoder baseline that adapts the static backbone directly rather than adding a separate residual branch. It consists of a single 24-layer encoder initialized from the pretrained static encoder and fully fine-tuned on dynamic data, with 967M trainable parameters, compared with 594M for FL4N. We train FL4N and a single-encoder baseline using 50%, 75%, and 100% of the training data and report PSNR averaged across five zero-shot real-world datasets: NVIDIA, N3DV, DyCheck, ADT, and TUM.

FL4N consistently outperforms the single-encoder baseline across all data fractions and continues to improve with more training data. Notably, FL4N trained on only 50% of the data already outperforms the baseline trained on the full dataset, achieving 17.536 dB compared to 17.467 dB. These results demonstrate that our residual formulation makes more effective use of dynamic training data and achieves better performance with less data.

We further compare the training data scales of existing methods. FL4N uses fewer than 3K scenes from two datasets, substantially less than 4DGT (11K clips), MoVieS (170K clips), NeoVerse (36K clips), and C4G (12K clips). Despite its relatively limited training data, FL4N achieves superior performance on zero-shot benchmarks, further supporting the effectiveness of our model design.

We have added the data-scaling results, including PSNR, SSIM, and LPIPS across all data fractions, in Figure F, along with the corresponding discussion in Appendix D.7.

50%75%100%
Single Encoder17.37017.46317.467
Residual Encoder (FL4N)17.53617.63117.687

 

MethodTraining datasets
4DGTEgoExo4D, Nymeria, Hot3D, AEA, Epic-Fields, Cop3D, ARKitTrack (11K clips)
MoVieSRealEstate10K, TartanAir, MatrixCity, PointOdyssey, DynamicReplica, Spring, VKITTI2, Stereo4D (170K clips)
NeoVersePointOdyssey, DynamicReplica, Kubric, Spring, VKITTI2, Waymo, TartanAir, BEDLAM, MVS-Synth, GFIE, HOI4D, CoP3D, DL3DV, ScanNet++, ARKitScenes, HyperSim, MapFree (36K clips)
C4GSpring, Kubric, RealEstate10K (12K clips)
FL4NKubric4D, CamxTime (3K clips)

  

W2, Q2: Capacity-matched fine-tuning baseline

We thank the reviewer for the suggestion. We construct a capacity-matched baseline by fine-tuning the static backbone with LoRA (rank 456, 596M trainable parameters, matching FL4N’s 594M). FL4N outperforms the capacity-matched LoRA baseline by 0.712 dB PSNR on NVIDIA and 2.430 dB on N3DV, showing that the gains come from the residual formulation rather than the additional capacity.

The two approaches differ in where the residual is applied. LoRA adds a residual in weight space. It modifies the static encoder itself, so the static latent also changes and can drift away from the pretrained prior. FL4N instead adds a residual in representation space. It keeps the static latent intact and adds a time-conditioned residual on top of it. In addition, a LoRA-adapted model must learn camera-induced changes and temporal variations jointly within a single encoder. FL4N separates the two into dedicated branches. The frozen static encoder captures scene structure and camera viewpoint, while the residual branch learns only the dynamic part. Figures 4 and 5 support this separation.

VariantNVIDIAN3DV
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
LoRA18.3910.42550.484621.4460.73560.3870
FL4N (Ours)19.1030.44620.456223.8760.78970.3404

W3: View–time disentanglement and pose-free results

Following the reviewer’s suggestion, we have toned down the claim of “clean disentanglement” in the revised paper.

Regarding pose-free performance in Table 1, FL4N achieves the best PSNR and comparable SSIM and LPIPS even without test-time pose optimization (TTO), which requires access to the target frames. We agree that FL4N does not outperform the baselines on every benchmark. NVIDIA is especially challenging because each camera in its rig has individually calibrated intrinsics. Within a single scene, the principal points of the input cameras differ by up to 15.7% of the image width and may lie away from the image center, and the focal lengths differ by up to 10.7%. In contrast, the other benchmarks use a single shared intrinsic model per scene. Without context poses or test-time pose alignment, this out-of-distribution camera geometry causes a global misalignment of the rendered frame. We expect that per-view intrinsic augmentation during training, such as random off-center crops and focal-length perturbation, could address this, and we leave it to future work. We further report FL4N results with TTO, which improves camera alignment and achieves superior performance.

We have added these results and the corresponding discussion in Table I and Appendix F.

poseTTONVIDIAN3DV
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
NeoVerse✗yes15.0500.41350.462612.9530.48060.5697
C4G✗yes16.5720.41820.620318.4440.70500.5246
FL4N (Ours)✗yes19.9510.46150.449121.4940.73470.4153

  

W4: Dependency on the static backbone

We thank the reviewer for pointing out this limitation. To avoid overstating the scope of our method, we revised the term "arbitrary-view synthesis" to "novel view synthesis" throughout the paper.

We agree that our approach inherits an upper bound from the pretrained static NVS backbone, as mentioned in Appendix G, particularly when it produces inaccurate scene representations or camera poses. Our residual formulation is designed to model temporal variations on top of the static prior, rather than reconstruct the entire scene representation from scratch. Conversely, as stronger static NVS backbones with broader scene coverage and more accurate viewpoint estimation become available, FL4N can directly benefit from them. In addition, a generative decoder with learned priors for view extrapolation could be combined with our residual formulation, which we leave for future work.

  

Q3: Runtime across different numbers of context frames

We clarify that the reported runtime includes context frame encoding. The table below reports FPS for different numbers of context frames, with 8 target frames. FL4N achieves the highest FPS across all context sizes. Moreover, the pose-free baselines rely on test-time pose alignment (†) for their reported accuracy, which reduces their throughput to below 0.1 FPS, whereas FL4N requires no such optimization.

We have added these results in Appendix D.6.

Number of context frames2481632
4DGT0.750.750.760.750.74
MoVieS20.5420.2118.4713.748.11
NeoVerse53.5946.5238.7521.5110.78
NeoVerse†0.090.080.070.070.06
C4G46.0740.8527.4713.475.27
C4G†0.040.040.040.040.04
FL4N (Ours)54.6549.9240.7221.9210.88

  

Q4: Clarification on feature dimension

We thank the reviewer for pointing out the typo. The backbone uses 1024-dimensional features, which are projected to 768 dimensions through a single linear layer before entering the time module. We have updated the implementation details in Appendix A.1.

Q5: Discussion on time-token interpolation

We thank the reviewer for raising this interesting question. Although the time tokens are initialized from a shared learned token, they become frame-specific through the dynamic encoder. Each token attends to its corresponding frame content as well as other context frames, allowing it to encode the dynamic state at that timestamp.

Irregular sampling. Our interpolation uses the two context tokens temporally adjacent to the target timestamp, with interpolation weights computed from the actual timestamps rather than frame indices. Thus, it naturally accounts for irregular temporal intervals.

Visual ambiguity and periodic motion. To further examine what the time tokens encode, we added a visualization in Figure 6 comparing token similarities for periodic (top) and monotonic (bottom) motion. In the periodic example, frames that return to the same dynamic state have highly similar time tokens even when they are far apart in time. For example, (τ0\tau_0) and (τ5\tau_5) are nearly identical, as are (τ3\tau_3) and (τ4\tau_4). This suggests that the tokens capture the underlying dynamic state rather than simply encoding elapsed time. In the monotonic example, although the frames look visually similar, their time tokens represent distinct temporal states, with similarity decreasing as the temporal gap increases. These observations suggest that the learned tokens capture temporal changes while recognizing recurring dynamic states.

Limitations of interpolation. We acknowledge that interpolating between two adjacent tokens may be insufficient when motion between context observations is strongly non-linear or visually ambiguous. In such cases, the observed frames alone may not provide sufficient information to determine the motion trajectory between them. Recovering the correct intermediate states would require additional motion information about the unobserved time intervals, which is unavailable in our current formulation. This is a general limitation of any method that reconstructs 4D scenes from sparse image observations alone, including Gaussian-based methods that rely on linear or velocity-based motion interpolation. Incorporating additional temporal context or explicit motion cues to provide such information is an interesting direction for future work. We clarified this limitation in Appendix G.

  

[1] Lin et al. MoVieS: Motion-aware 4D dynamic view synthesis in one second. CVPR 2026.

[2] Yang et al. NeoVerse: Enhancing 4D world model with in-the-wild monocular videos. CVPR 2026.

Reviewer_8DJPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FL4N, a feed-forward latent model for 4D novel view synthesis from monocular videos. It freezes a pretrained static latent NVS model (encoder and renderer) and adds a lightweight dynamic encoder, which uses learnable time tokens and cross-attention to static features to predict a time-conditioned residual on top of the static latent. The model is trained on synchronized multi-view videos with image losses and flow-based pseudo dynamic masks, and is evaluated under three protocols (novel view, cross-time view, and novel time synthesis) on eight datasets against Gaussian-based feed-forward baselines.

Critical strengths
  1. The idea of formulating 4D NVS as a time-conditioned residual on top of a frozen pretrained static latent NVS model is simple and reasonable. It reuses a strong static prior, which allows training with limited multi-view dynamic data and without dense depth or motion supervision.

  2. The evaluation splits 4D NVS into novel view, cross-time view, and novel time synthesis, which is a useful way to examine view and time generalization separately. The paper also includes analyses of the learned latent (PCA, gate activation, and probing) that help illustrate what the residual branch captures.

Critical weaknesses

1. Unreconciled baseline discrepancies undermine the main performance claims. The NTS evaluation on ADT/TUM claims to follow C4G, yet the reported baseline results are substantially lower than those in the C4G paper. On ADT, C4G drops from 22.35 to 18.90 PSNR, NeoVerse from 21.94 to 16.18, MoVieS from 20.35 to 15.43, and 4DGT from 19.22 to 16.71. On TUM, C4G drops from 19.52 to 17.51. Under the published numbers, pose-free FL4N (19.69 / 17.85) would no longer be the best pose-free method on either dataset. These discrepancies directly affect the paper's central SOTA claim and require reconciliation of evaluation protocols, checkpoints, resolutions, and pose alignment procedures.

2. The experimental evidence is insufficient to establish the effectiveness and generalization of the proposed 4D representation. The largest gains occur on Kubric4D and CamxTime, which are also FL4N's training datasets, albeit with held-out test scenes. The improvements on real monocular NTS benchmarks are considerably smaller. Moreover, the main ablation is limited to a single in-domain experiment trained for 10k steps, with only 0.16–0.37 dB improvements in dynamic-region PSNR. Crucial temporal components, including the time module, token interpolation, and residual gate, lack individual ablations. Consequently, it remains unclear how much of the reported improvement comes from the proposed temporal modeling rather than the pretrained static backbone and the favorable training/evaluation setup.

Questions for authors
  1. Baseline discrepancies. Why are the baseline results on ADT/TUM substantially lower than those reported in C4G under apparently similar evaluation protocols? For example, on ADT, C4G drops from 22.35 to 18.90 PSNR, NeoVerse from 21.94 to 16.18, and MoVieS from 20.35 to 15.43. Please clarify the differences in evaluation frames, resolution, checkpoints, preprocessing, and test-time pose alignment. Can the authors reproduce or reconcile the published results? This is critical because the discrepancies directly affect whether FL4N achieves SOTA among pose-free methods.

  2. Evaluation fairness and generalization. How do the authors ensure a fair comparison given that Kubric4D and CamxTime are part of FL4N's training distribution, while the baselines may have different training distributions? Are all methods evaluated using identical input and target view–time pairs, and how closely do these configurations match their respective training setups?

  3. Pose estimation and target information. For the real-world evaluations using VGGT-estimated camera parameters, are the target RGB frames included in VGGT's inputs? How are target camera poses obtained and specified in the pose-free setting? This clarification is important to rule out target-information leakage and establish whether the methods receive comparable information.

  4. Evidence for the proposed temporal modeling. Can the authors provide quantitative comparisons against the frozen static backbone without the dynamic branch, and isolate the contributions of the time module, time-token interpolation, and residual gate? The current ablation is limited to 10k steps on Kubric4D, with relatively small dynamic-region improvements (0.16–0.37 dB). These results make it difficult to determine whether the proposed temporal components are responsible for the reported performance gains.

5 comments

W1, Q1: Baseline discrepancy and evaluation fairness

We thank the reviewer for pointing out these discrepancies. We identified an error in the pose normalization procedure for 4DGT and MoVieS, which led to lower performance. We have corrected the normalization for each baseline and updated the results in the revised manuscript. FL4N achieves state-of-the-art PSNR and LPIPS among posed methods, improving PSNR by 2.098 dB in ADT and 3.113 dB on dynamic regions of the Charge dataset, while being superior or comparable in SSIM.

The discrepancy also stems from differences in evaluation resolution. While following the same protocol as C4G for selecting input and target frames, we evaluate at the full GT resolution rather than downsampled resolution (224 ×\times 224) after center-cropping, as done in C4G. This lower-resolution setting can inflate evaluation metrics by suppressing high-frequency details and reducing sensitivity to fine texture differences, geometric misalignments, and rendering artifacts such as ghosting. Center-cropping also excludes peripheral regions where geometric errors may be more pronounced.

For completeness, we additionally provide results under the C4G evaluation protocol, using center-cropping and resizing to (224 ×\times 224), in the table below. Under this setting, our reproduced baseline results closely match those reported by C4G, and FL4N maintains its superior performance, improving the best baseline by +4.867 dB PSNR in pose-free comparison and +10.383 dB in posed setting in ADT. On TUM, it reduces LPIPS by 0.0904 in the pose-free setting and 0.2112 in the posed setting. FL4N shows slightly lower PSNR and SSIM than C4G on TUM in the pose-free setting. This is because PSNR tends to favor blurry predictions, and both PSNR and SSIM heavily penalize slight pixel misalignments. Such misalignments occur in pose-free FL4N, which, unlike the other pose-free baselines, does not use test-time pose alignment. Figure G shows a visual comparison on TUM at the C4G resolution, with per-image PSNR. FL4N produces much cleaner renderings, yet C4G obtains higher PSNR with blurrier predictions, which accounts for the gap. Qualitative comparison and detailed discussion is provided in Figure H and Appendix D.9.

poseADTTUM
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
4DGT✓19.2090.71680.256617.2090.54980.3509
MoVieS✓20.6870.58780.357915.4900.42320.3975
NeoVerse✗20.6520.56730.347015.2600.37570.3363
C4G✗19.3110.55410.460519.7090.60120.3162
FL4N (Ours)✗25.5190.77460.129318.6930.55280.2258
FL4N (Ours)✓31.0700.91460.058422.4810.76340.1397

W2, Q4: Effectiveness and generalization of the temporal modeling

Generalization beyond the training datasets. We note that NVIDIA, N3DV, Charge, ADT, DyCheck, and TUM are never seen during training. Results on these benchmarks are therefore zero-shot evaluations, covering five real-world datasets and one synthetic dataset. On the monocular NTS benchmarks, FL4N still improves over the strongest baseline by 1.926 dB PSNR in DyCheck, 2.485 dB in ADT, 2.736 dB in TUM dataset. The largest gains do appear on the held-out scenes of Kubric4D and CamxTime, but evaluating on held-out scenes of the training datasets follows standard practice in recent feed-forward 4D methods. MoVieS [1] and NeoVerse [2] also report results on held-out scenes of their own training data.

Scale and evaluation of ablation study. Due to limited computational resources, we trained the ablation variants at a smaller scale than the main model, as has also been done in prior works [3, 4]. All variants are trained for the same number of steps, so the comparison reflects performance under an identical training budget. We agree that the resulting gaps in dynamic-region PSNR are small. At this scale, we observe that the weaker variants have not yet clearly separated dynamic regions from the static background, which narrows these gaps. To address this, we trained every variant in the additional ablations of this response for 20k steps and evaluated them on real-world out-of-domain benchmarks.

Ablation on temporal components. We provide additional ablations of the time conditioning in the time module, the time-token interpolation, and the residual gate in the table below, together with the frozen static backbone without the dynamic branch. To isolate the contribution of temporal modeling beyond the in-domain setting, we evaluate all variants on the real monocular NTS benchmarks, ADT and TUM, with a temporal stride of 4. For the time module, we replace AdaLN with addition, token-wise concatenation, and cross-attention. For time-token interpolation, we replace our learnable time-token interpolation with a continuous sinusoidal time embedding. We also ablate the residual gate by removing it. Figure 3 of the main paper also provides a qualitative comparison between the static backbone and FL4N, where the static backbone fails to render moving regions at the target time while FL4N recovers them. We have added this result in Appendix D.8, at Table E and Figure G.

We also note that our time module follows a design similar to the time-conditioned decoder in V-DPM [5]. Both use learnable time tokens with AdaLN conditioning. Our module is lighter, however. It uses only two global attention layers, while V-DPM uses eight alternating attention layers. We have added this discussion in Section 3.2.

VariantMeanADTTUM
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
Static backbone14.2970.46450.524712.7140.44520.611815.8790.48380.4376
Time module conditioning: Addition15.1900.48950.516914.4610.49210.611015.9190.48680.4228
Time module conditioning: Token-wise concatenation15.1030.48610.514514.3000.48590.607115.9060.48630.4220
Time module conditioning: Cross-attention15.1200.48870.513614.2070.48470.608016.0330.49260.4191
Sinusoidal time token15.2220.49160.511414.4230.48690.603916.0220.49640.4190
No residual gate15.2400.48930.512114.4770.48540.604416.0030.49310.4198
FL4N (ablation)15.4140.49400.509614.7530.49530.601416.0760.49260.4178

Q2: Evaluation fairness and generalization

First, we clarify that all models use the identical input and target view–time pairs for all evaluation.

Following recent feed-forward 4D methods, we include held-out scenes of our training datasets in the evaluation. MoVieS [1] likewise holds out 10% of the scenes of its training datasets as a test set, and NeoVerse [2] evaluates on the ScanNet++ test split while also training on ScanNet++.

NVIDIA, N3DV, Charge, ADT, DyCheck, and TUM are never seen during FL4N's training. Results on these benchmarks are therefore fair zero-shot evaluations, covering five real-world datasets and one synthetic dataset. With or without context poses, FL4N achieves the best PSNR while achieving the best or second-best SSIM and LPIPS. Qualitatively, FL4N renders geometrically consistent frames without the temporal ghosting, holes, or aliasing that appear in explicit Gaussian-based methods.

Regarding training setups, 4DGT is trained only on real-world monocular videos, which corresponds to our NTS setting. C4G, like FL4N, is supervised with target views rendered from cameras that are not given as input, which corresponds to our NVS and CTVS settings. As for supervision, all Gaussian-based models rely on geometry and motion supervision such as depth, normal, and tracking losses to learn strong 3D/4D priors, whereas FL4N is trained with only an image reconstruction loss and a binary dynamic mask loss.

Finally, the baselines are trained on substantially larger data than FL4N, as summarized below. FL4N is trained on under 3K scenes from two datasets, considerably fewer than 4DGT (11K clips), MoVieS (170K clips) and NeoVerse (36K clips). 4DGT is trained only on monocular videos, whereas MoVieS, NeoVerse, C4G, and FL4N all include multi-view data in training. Despite using the least training data among these methods, FL4N shows superior performance on the zero-shot benchmarks, which indicates that its improvements come from the model design rather than the scale or domain of its training data.

MethodTraining datasets
4DGTEgoExo4D, Nymeria, Hot3D, AEA, Epic-Fields, Cop3D, ARKitTrack (11K clips)
MoVieSRealEstate10K, TartanAir, MatrixCity, PointOdyssey, DynamicReplica, Spring, VKITTI2, Stereo4D (170K clips)
NeoVersePointOdyssey, DynamicReplica, Kubric, Spring, VKITTI2, Waymo, TartanAir, BEDLAM, MVS-Synth, GFIE, HOI4D, CoP3D, DL3DV, ScanNet++, ARKitScenes, HyperSim, MapFree (36K clips)
C4GSpring, Kubric, RealEstate10K (12K clips)
FL4NKubric4D, CamxTime (3K clips)

  

Q3: Target pose usage and information leakage

Following LagerNVS [3] (which is our static encoder), target frames are used as VGGT input to estimate target pose. However, only the resulting 6D camera parameters are used to construct Plücker embeddings, and no other features are used. This is standard protocol used in pose-free baselines [2, 6], and thus does not introduce any target information leakage.

Moreover, pose-free baselines (NeoVerse and C4G) perform an additional 2,000-step test-time pose alignment with the target RGB images, relying on target information more strongly than ours.

  

[1] Lin et al. MoVieS: Motion-aware 4D dynamic view synthesis in one second. CVPR 2026.

[2] Yang et al. NeoVerse: Enhancing 4D world model with in-the-wild monocular videos. CVPR 2026.

[3] Szymanowicz et al. LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis. CVPR 2026.

[4] Jin et al. LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias. ICLR 2025.

[5] Sucar et al. V-DPM: 4D Video Reconstruction with Dynamic Point Maps. CVPR 2026.

[6] Kim et al. Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction. arXiv preprint 2026.

Reviewer_ozQmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FL4N extends a pretrained static novel-view synthesis model to dynamic scenes. It freezes the static encoder and renderer and learns a time-conditioned latent residual. The method is evaluated on novel-view, cross-time view, and novel-time synthesis across eight datasets.

Critical strengths

The method offers a simple way to reuse a strong static model for dynamic view synthesis. Evaluation covers several datasets and complementary protocols, including cross-dataset tests and dynamic-region metrics. These results provide useful evidence that the complete system works.

Critical weaknesses
  1. The paper does not discuss FSM-LVSM from Fast Spatial Memory with Elastic Test-Time Training, which also synthesizes images at queried views and times without explicit Gaussian rendering. Although its architecture differs from FL4N, it is relevant to the paper’s positioning. A comparison would be useful; at minimum, the authors should discuss the differences and clarify what FL4N adds beyond this existing latent-based approach.
Questions for authors
  1. How does FL4N differ from FSM-LVSM in Fast Spatial Memory with Elastic Test-Time Training, and what new contribution does it offer beyond that approach?
  2. LagerNVS inherits geometry-aware features from VGGT pretraining. How can the gains attributed to latent-based rendering be separated from those provided by this pretrained geometry prior? The current comparisons do not isolate these two factors.
4 comments

W1, Q1: Comparison with FSM-LVSM

We thank the reviewer for bringing FSM-LVSM [1] to our attention. We have added it to the related work of the revised paper and provide a detailed comparison below.

Different problem settings. FSM-LVSM and FL4N both synthesize novel views from latent view–time queries, but they address different problems. FSM-LVSM targets long-context spatial memory. It requires posed input images, compresses them into fast weights updated by elastic test-time training, and is pretrained on large-scale 3D and 4D data, including RealEstate10K, DL3DV, PointOdyssey, Spring, Multi-Cam Video, DynamicReplica, and Stereo4D.

Contribution of FL4N. FL4N models dynamics as a residual on top of a frozen static backbone, which decomposes the 4D problem into two parts. The frozen static encoder captures scene structure and camera geometry, while the residual encoder captures only what changes over time. Recovering dynamic objects from a moving monocular camera is highly challenging as camera motion and object motion are entangled. The problem becomes even harder in the pose-free setting, which FL4N supports but FSM-LVSM does not, since no camera poses are available to anchor the camera motion. Since the majority of the scene stays static, FL4N decouples where the time and camera conditions are applied. A time module transforms a residual latent into a time-specific residual, which is then added to the static latent to yield the dynamic latent, and the dynamic latent is then decoded into an image conditioned on the target camera. This separation is built into the formulation rather than learned from data. Since the residual only needs to model temporal variation, which typically occupies a small part of the scene, FL4N requires far less dynamic training data than FSM-LVSM and is trained on only two multi-view datasets without depth or motion supervision.

We view the two directions as orthogonal and complementary. The long-context memory of FSM-LVSM could potentially be combined with our residual formulation.

Quantitative comparison. For completeness, we compare against FSM-LVSM in the table below. In the NVS protocol, FL4N shows the best PSNR on all benchmarks with comparable SSIM and LPIPS in full-frame evaluation, and it outperforms FSM-LVSM on all metrics in dynamic-region evaluation.

Qualitative comparison. Figure 7 shows a qualitative comparison. FL4N shows a clearly better view-time disentanglement, which we attribute to our architectural design. FSM-LVSM often renders the state of a nearby context frame instead of the target time, whereas FL4N consistently renders the target time across different viewpoints. We attribute this difference to how each model separates time from viewpoint. FSM-LVSM encodes the time of each view as a per-pixel timestamp concatenated with the RGB image and Plücker rays, and it stores all context views in a single set of fast weights. Its architecture therefore has no explicit mechanism to separate time from viewpoint, and this separation must be learned from data. In contrast, FL4N applies the time and camera conditions in separate branch, so object motion is modeled independently of camera motion.

Full-frameposeNVIDIAN3DVKubric4DCamxTimeCharge
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
FSM-LVSM✓19.3340.48720.383922.6640.77370.383520.6890.65360.344621.4500.75330.304721.4790.81500.4735
FL4N✓20.4350.48350.386722.8960.77160.356022.2090.67370.335025.7300.82700.168122.9590.83240.4616
Dyn-regionposeKubric4DCharge
PSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
FSM-LVSM✓13.0630.27790.026719.5530.65210.1054
FL4N✓14.8000.34850.023619.9930.67110.1029

[1] Ma et al. Fast Spatial Memory with Elastic Test-Time Training. ECCV 2026.

Q2: Role of the VGGT geometry prior

We thank the reviewer for raising this important point. We clarify that VGGT initialization is not exclusive to LagerNVS [2], but the baselines including MoVieS, NeoVerse, and C4G also initialize from VGGT. Therefore, geometry-aware features are shared across most baselines, and the performance gains of FL4N cannot be attributed solely to the VGGT prior. Rather, they demonstrate the effectiveness of FL4N's latent-based residual formulation.

In addition, LagerNVS shows that latent NVS can be trained without the VGGT prior, although it converges more slowly. We expect FL4N to behave similarly with longer training, and we leave this direction to future work.

[2] Szymanowicz et al. LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis. CVPR 2026.

Thank you for the detailed response, including the added discussion and comparison with FSM-LVSM. I appreciate the clarifications and will keep my positive rating.

Thank you for your thoughtful feedback and for maintaining your positive rating. We truly appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.