- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FL4N addresses feed-forward 4D novel view synthesis from monocular videos. It augments a frozen pretrained static latent view-synthesis model with a lightweight dynamic encoder and a target-time adaptation module. The dynamic encoder cross-attends to static features to estimate temporal residuals. These residuals are conditioned on learned time tokens, added to the static latent, and decoded by the frozen camera-conditioned renderer. Intermediate timestamps are handled through time-token interpolation. Training uses image reconstruction losses and binary dynamic-mask pseudo-labels without dense geometry or 3D-motion supervision.
Critical strengths
-
Effective reuse of a static latent prior. The residual formulation offers a practical route from pretrained static synthesis to dynamic synthesis while preserving the original renderer.
-
Strong cross-time and novel-time results. These tasks directly test the proposed temporal control. Pose-free FL4N outperforms the evaluated Gaussian baselines across all reported metrics in Tables 2 and 3.
-
Fast reported inference. FL4N achieves 40 FPS at , making the approach practically relevant for interactive synthesis.
Critical weaknesses
-
The data-efficiency claim is insufficiently quantified. Reusing a static prior is compelling, but no training-data scaling experiment establishes how much dynamic data is saved.
-
The contribution of the residual formulation is not completely isolated. A capacity-matched, explicitly time-conditioned fine-tuning baseline would help distinguish the proposed design from alternative adaptation strategies.
-
Some conclusions exceed the evidence. The results demonstrate effective view–time conditioning, but “clean disentanglement” is stronger than what the visualizations and probes establish. Pose-free superiority is also not universal in Table 1.
-
Generalization remains dependent on the static backbone. The acknowledged failures under viewpoint extrapolation and substantial camera errors constrain claims of arbitrary-view synthesis.
Questions for authors
-
How does performance change when training with progressively smaller fractions of Kubric4D and CamxTime? Can this substantiate the claimed dynamic-data efficiency?
-
How does FL4N compare with a parameter-matched adaptation baseline using explicit temporal conditioning, such as adapters or LoRA in the static backbone?
-
Does the 40 FPS measurement include context encoding, and how does runtime scale with the number of context frames?
-
How are the 768-dimensional backbone features and the reported 1024-dimensional time tokens made compatible?
-
Since the time tokens begin as copies of a shared learned token, how reliably does interpolation handle irregular sampling and visually ambiguous or periodic motion?
5 comments
W3: View–time disentanglement and pose-free results
Following the reviewer’s suggestion, we have toned down the claim of “clean disentanglement” in the revised paper.
Regarding pose-free performance in Table 1, FL4N achieves the best PSNR and comparable SSIM and LPIPS even without test-time pose optimization (TTO), which requires access to the target frames. We agree that FL4N does not outperform the baselines on every benchmark. NVIDIA is especially challenging because each camera in its rig has individually calibrated intrinsics. Within a single scene, the principal points of the input cameras differ by up to 15.7% of the image width and may lie away from the image center, and the focal lengths differ by up to 10.7%. In contrast, the other benchmarks use a single shared intrinsic model per scene. Without context poses or test-time pose alignment, this out-of-distribution camera geometry causes a global misalignment of the rendered frame. We expect that per-view intrinsic augmentation during training, such as random off-center crops and focal-length perturbation, could address this, and we leave it to future work. We further report FL4N results with TTO, which improves camera alignment and achieves superior performance.
We have added these results and the corresponding discussion in Table I and Appendix F.
| pose | TTO | NVIDIA | N3DV | |||||
|---|---|---|---|---|---|---|---|---|
| PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | |||
| NeoVerse | ✗ | yes | 15.050 | 0.4135 | 0.4626 | 12.953 | 0.4806 | 0.5697 |
| C4G | ✗ | yes | 16.572 | 0.4182 | 0.6203 | 18.444 | 0.7050 | 0.5246 |
| FL4N (Ours) | ✗ | yes | 19.951 | 0.4615 | 0.4491 | 21.494 | 0.7347 | 0.4153 |
W4: Dependency on the static backbone
We thank the reviewer for pointing out this limitation. To avoid overstating the scope of our method, we revised the term "arbitrary-view synthesis" to "novel view synthesis" throughout the paper.
We agree that our approach inherits an upper bound from the pretrained static NVS backbone, as mentioned in Appendix G, particularly when it produces inaccurate scene representations or camera poses. Our residual formulation is designed to model temporal variations on top of the static prior, rather than reconstruct the entire scene representation from scratch. Conversely, as stronger static NVS backbones with broader scene coverage and more accurate viewpoint estimation become available, FL4N can directly benefit from them. In addition, a generative decoder with learned priors for view extrapolation could be combined with our residual formulation, which we leave for future work.
Q3: Runtime across different numbers of context frames
We clarify that the reported runtime includes context frame encoding. The table below reports FPS for different numbers of context frames, with 8 target frames. FL4N achieves the highest FPS across all context sizes. Moreover, the pose-free baselines rely on test-time pose alignment (†) for their reported accuracy, which reduces their throughput to below 0.1 FPS, whereas FL4N requires no such optimization.
We have added these results in Appendix D.6.
| Number of context frames | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|
| 4DGT | 0.75 | 0.75 | 0.76 | 0.75 | 0.74 |
| MoVieS | 20.54 | 20.21 | 18.47 | 13.74 | 8.11 |
| NeoVerse | 53.59 | 46.52 | 38.75 | 21.51 | 10.78 |
| NeoVerse† | 0.09 | 0.08 | 0.07 | 0.07 | 0.06 |
| C4G | 46.07 | 40.85 | 27.47 | 13.47 | 5.27 |
| C4G† | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 |
| FL4N (Ours) | 54.65 | 49.92 | 40.72 | 21.92 | 10.88 |
Q4: Clarification on feature dimension
We thank the reviewer for pointing out the typo. The backbone uses 1024-dimensional features, which are projected to 768 dimensions through a single linear layer before entering the time module. We have updated the implementation details in Appendix A.1.
Q5: Discussion on time-token interpolation
We thank the reviewer for raising this interesting question. Although the time tokens are initialized from a shared learned token, they become frame-specific through the dynamic encoder. Each token attends to its corresponding frame content as well as other context frames, allowing it to encode the dynamic state at that timestamp.
Irregular sampling. Our interpolation uses the two context tokens temporally adjacent to the target timestamp, with interpolation weights computed from the actual timestamps rather than frame indices. Thus, it naturally accounts for irregular temporal intervals.
Visual ambiguity and periodic motion. To further examine what the time tokens encode, we added a visualization in Figure 6 comparing token similarities for periodic (top) and monotonic (bottom) motion. In the periodic example, frames that return to the same dynamic state have highly similar time tokens even when they are far apart in time. For example, () and () are nearly identical, as are () and (). This suggests that the tokens capture the underlying dynamic state rather than simply encoding elapsed time. In the monotonic example, although the frames look visually similar, their time tokens represent distinct temporal states, with similarity decreasing as the temporal gap increases. These observations suggest that the learned tokens capture temporal changes while recognizing recurring dynamic states.
Limitations of interpolation. We acknowledge that interpolating between two adjacent tokens may be insufficient when motion between context observations is strongly non-linear or visually ambiguous. In such cases, the observed frames alone may not provide sufficient information to determine the motion trajectory between them. Recovering the correct intermediate states would require additional motion information about the unobserved time intervals, which is unavailable in our current formulation. This is a general limitation of any method that reconstructs 4D scenes from sparse image observations alone, including Gaussian-based methods that rely on linear or velocity-based motion interpolation. Incorporating additional temporal context or explicit motion cues to provide such information is an interesting direction for future work. We clarified this limitation in Appendix G.
[1] Lin et al. MoVieS: Motion-aware 4D dynamic view synthesis in one second. CVPR 2026.
[2] Yang et al. NeoVerse: Enhancing 4D world model with in-the-wild monocular videos. CVPR 2026.
W1, Q1: Training-data scaling
We thank the reviewer for this valuable suggestion.
To isolate the effect of the residual formulation, we construct a single-encoder baseline that adapts the static backbone directly rather than adding a separate residual branch. It consists of a single 24-layer encoder initialized from the pretrained static encoder and fully fine-tuned on dynamic data, with 967M trainable parameters, compared with 594M for FL4N. We train FL4N and a single-encoder baseline using 50%, 75%, and 100% of the training data and report PSNR averaged across five zero-shot real-world datasets: NVIDIA, N3DV, DyCheck, ADT, and TUM.
FL4N consistently outperforms the single-encoder baseline across all data fractions and continues to improve with more training data. Notably, FL4N trained on only 50% of the data already outperforms the baseline trained on the full dataset, achieving 17.536 dB compared to 17.467 dB. These results demonstrate that our residual formulation makes more effective use of dynamic training data and achieves better performance with less data.
We further compare the training data scales of existing methods. FL4N uses fewer than 3K scenes from two datasets, substantially less than 4DGT (11K clips), MoVieS (170K clips), NeoVerse (36K clips), and C4G (12K clips). Despite its relatively limited training data, FL4N achieves superior performance on zero-shot benchmarks, further supporting the effectiveness of our model design.
We have added the data-scaling results, including PSNR, SSIM, and LPIPS across all data fractions, in Figure F, along with the corresponding discussion in Appendix D.7.
W2, Q2: Capacity-matched fine-tuning baseline
We thank the reviewer for the suggestion. We construct a capacity-matched baseline by fine-tuning the static backbone with LoRA (rank 456, 596M trainable parameters, matching FL4N’s 594M). FL4N outperforms the capacity-matched LoRA baseline by 0.712 dB PSNR on NVIDIA and 2.430 dB on N3DV, showing that the gains come from the residual formulation rather than the additional capacity.
The two approaches differ in where the residual is applied. LoRA adds a residual in weight space. It modifies the static encoder itself, so the static latent also changes and can drift away from the pretrained prior. FL4N instead adds a residual in representation space. It keeps the static latent intact and adds a time-conditioned residual on top of it. In addition, a LoRA-adapted model must learn camera-induced changes and temporal variations jointly within a single encoder. FL4N separates the two into dedicated branches. The frozen static encoder captures scene structure and camera viewpoint, while the residual branch learns only the dynamic part. Figures 4 and 5 support this separation.