acceptodds
Under review as a conference paper at ICLR 2027

AnySpaceTime: Feed-Forward Spacetime Novel View Synthesis from Arbitrary Observations

Abstract

Novel view synthesis (NVS) has progressed significantly, but existing settings are typically tailored to specific observation patterns, such as static multi-view scenes, dynamic videos, or synchronized multi-camera captures. This paper introduces Space-Time NVS, a unified task that takes timestamped and posed images distributed across time and viewpoint as input and synthesizes images at a target viewpoint and a target time within the observed temporal interval, subsuming the above settings as special cases. We propose AnySpaceTime, an end-to-end feed-forward framework for this task that supports independently specified time and view queries in a single forward pass. Our key idea is to decouple spatial representation from temporal adaptation: we freeze a pretrained NVS encoder to preserve strong spatial priors and introduce a lightweight motion branch that predicts time-dependent updates in the latent space. The target time determines the latent update, while the target camera pose determines rendering, with static scenes handled as the shared-state special case. This design handles static multi-view inputs, dynamic videos, and asynchronous multi-camera observations without requiring a fixed camera layout, synchronized or contiguous frames, or supervision from depth, optical flow, or 3D motion. It generalizes to unseen scenes and irregular observations, and experiments across multiple datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis、Spatiotemporal Rendering、Dynamic Scene Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H8izOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnySpaceTime addresses space–time novel view synthesis from images with known camera parameters and timestamps. It extends a pretrained spatial NVS model through a motion encoder, a shared low-dimensional motion basis, target-time-conditioned feature updates, and lightweight renderer adaptation. One model is trained across static multiview, dynamic monocular, synchronized multiview, and asynchronous multiview observation patterns. Experiments demonstrate strong dynamic reconstruction results and largely preserved static-scene capability.

Critical strengths
  1. The architecture is coherent, and a single model supports all 4 input patterns without setting-specific fine-tuning is practically useful.

  2. Table 1 reports substantial improvements across three dynamic benchmarks, including relevant monocular baselines.

  3. Component ablations provide supporting evidence for the motion basis and renderer adaptation.

Critical weaknesses
  1. The comparisons do not cleanly separate architectural gains from training exposure. In Table 1, the monocular eval is tested with 8 reference image, which is exactly how anyspacetime is trained, while other models might not be exposed to such train inference matching. For example, 4DGT’s documented training uses 16 and subsequently 64 images, while NoPo4D targets synchronized multiview capture rather than the evaluated monocular setting. In Table 2, AnySpaceTime uniquely reports explicit training on both evaluated multiview patterns among the compared methods, while all other do not report equivalent training exposure to both patterns, so making the comparison kind of mixed. If I view this as a method paper, I would like to see training data and exposure controlled to determine how much of the improvement comes from the proposed framework versus better alignment between training and evaluation. However, that said, if it is standard practice in this field to compare methods at a system level only, then I am open to removing this weakness.

  2. Positioning against closely related predecessors is incomplete. BTimer is cited but not compared, despite being a relevant feed-forward predecessor using similar posed, timestamped observations and supporting target-time reconstruction with target-view rendering. This does not establish that it already matches AnySpaceTime across all four settings, but it might be more suitable as a direct comparable candidate than the baselines author used. As some of them are more narrow-task-specific methods and take in different inputs. Additionally, CAT4D and 4DiM should also be discussed as alternative predecessors as they also share similar input interface with anyspacetime. I understand they are diffusion based models, but I would not ignore their existance. As the problem they are targeting is similar to anyspacetime and have been around for about 2 years. Therefore, the novelty should be distinguished more precisely from the broader space–time synthesis objective.

  3. The asynchronous experiment uses a regular four-camera offset pattern, with target times represented among the reference observations. It does not fully establish performance for irregular camera availability combined with unseen target times. Separately, Table 3 includes only two dedicated static baselines, so it supports retention of static capability more strongly than broad competitiveness with specialized static methods.

Questions for authors
  1. Can you provide a focused comparison with a compatible predecessor, such as 4DGT or BTimer? may be using matched training data and observation sampling to better isolate the architectural contribution. If its not possible, please explain the constraints.

  2. How sensitive is the Table 1 gap to reference count? may be try more out of distribution reference count for anyspacetime and other baseline methods.

  3. Were Table 2’s baselines adapted or trained for synchronized and asynchronous inputs? may be specify checkpoints and method-specific input adaptations if available.

  4. What distinguishes the formulation and method from BTimer? and why was BTimer omitted from comparison?

Reviewer_REQtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AnySpaceTime, which is a feedforward approach for dynamic Novel View Synthesis (NVS). The key idea and insight of the paper stem from the observation that pretrained static NVS models provide a sufficient spatial prior, which can be adapted with a motion branch to encode a dynamic representation. Further, the approach utilises a LoRA adapter to provide target camera conditioning to the model. The proposed approach uses either multiview or monocular dynamic capture as an input and renders both novel views and time (interpolated). The experimental sections show substantive improvements over the state-of-the-art competitors.

Critical strengths

One of the strengths of the paper is its performance on the evaluated datasets. Notably, the approach unifies both monocular and multiview scenarios, performing well in both. Performing evaluation with a single shared model is a strong point.

The concept and empirical motivation for the usefulness of static models' spatial priors are compelling. The ablation proves the relevance of the design choices. Notably, reusing the existing prior is a good idea and certainly leads to easier training. The motion description is sound, and its application at the feature level seems to be a nice idea. Additionally, decoupling time from the camera pose by means of motion branch modulation is conceptually convincing.

Critical weaknesses

Despite showing a convincing performance on 3 datasets, I believe the evaluation is missing, especially in the monocular scenario. Subsampling the multiview input to create a monocular path does not best reflect monocular camera motion [1]. To this end, it would be great to see comparisons on a dedicated mono dataset, e.g. DyCheck [1] (one could include Nvidia [2], Nerfies [3], HyperNeRF [4], often present in monocular papers). Similarly, Technicolor [5] would be a nice multiview addition, as there are some elements in the scene that are appearing and disappearing, which could give some insight into feature propagation in time. On the note of mono evaluation, it would be good to see a table focused on observed target time. Even though it is not suitable, one could use LagerNVS in the dynamic tables to get a glimpse of the impact of the underlying spatial model on the overall performance. Finally, adding per-scene optimisation competitors would be beneficial to gauge the gap, if any, from such approaches.

[1] H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa, "Monocular Dynamic View Synthesis: A Reality Check". NeurIPS. 2022. [2] J. S. Yoon, K. Kim, O. Gallo, H. S. Park, and J. Kautz, "Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera". CVPR. 2020. [3] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B Goldman, S. M. Seitz, and R. Martin-Brualla, "Nerfies: Deformable Neural Radiance Fields". ICCV. 2021. [4] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B Goldman, R. Martin-Brualla, and S. M. Seitz, "HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields". SIGGRAPH Asia. 2021. [5] N. Sabater, G. Boisson, B. Vandame, P. Kerbiriou, F. Babon, M. Hog, R. Gendrot, T. Langlois, O. Bureller, A. Schubert, and V. Allie, "Dataset and Pipeline for Multi-View Light-Field Video". CVPRW. 2017.

Questions for authors
  • Could the method work with a different static backbone? Have the authors tried?
  • Is there a specific design decision on choosing the direct renderer as a backbone as opposed to using a 3D representation?
  • What is the ray embedding used?
  • Is the backbone decoder frozen as well, with LoRA trained, or is the decoder trained as well?
  • What are the details on camera selection in the asynchronous setup?
  • Could authors explain in more detail the Unified Reference Set opposed to prior representation?
  • In training, are you always predicting forward in time, or is bidirectional prediction viable?
  • How does the method deal with noisy camera poses?
  • How does the method handle dynamic objects leaving and re-entering the camera view?
Reviewer_xm69OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new novel view synthesis technique for dynamic scenes.

Critical strengths
  • Good quantitative results, although I want to see what other reviewers think as I might be missing some recent works.
  • Relatively clear writing, I have not realised any major problems.
Critical weaknesses
  • Very limited visual results, a few short clips only, makes me wonder about the reported tables as well.
  • The fundamental insight as stated in the paper: "This leads to our key insight: a pretrained static NVS representation already encodes each source-time state in a renderable form, anchored to the features of its source reference, so dynamic NVS can be cast as adapting these features to the target time before rendering, rather than learning" is not new and does not tell me much.
  • The technique itself has lots of design choices and no thorough justifications.
Questions for authors

Have you seen success across videos, meaning you get good reconstructions at any time stamp? The video is not very useful to illustrate this.

Can you elaborate on why you made the design choices you did? The paper reads like a procedural explanation of what is done and not why.

Comments on the paper

1 comment
Area_Chair_MYMiOpenReview ↗

Please Engage in the Author–Reviewer Discussion

Dear Reviewers,

As the author–reviewer discussion period is approaching its deadline (November 18, AoE), I would like to kindly remind you to review the authors' rebuttals and actively engage in the discussion as soon as possible. Please address any outstanding questions or concerns and update your assessments and ratings if appropriate. Thank you very much for your time and efforts!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.