acceptodds
Under review as a conference paper at ICLR 2027

AttenDiffCrafter: Attention-Crafted Consistent Novel Views from a Single Image

Abstract

Generating consistent novel views from a single image is challenging under large camera motions, where unseen regions must be synthesized while preserving scene structure across frames. Existing approaches face different limitations: autoregressive Transformers tend to accumulate errors over long sequences, whereas diffusion models built on U-Net backbones have limited global interaction and may produce color shifts or implausible content in regions outside the input view. We propose AttenDiffCrafter, a diffusion framework for novel-view synthesis built on attentioncentric Transformer architecture. Instead of relying on convolution-dominated feature extraction, the model uses self-attention throughout as the primary mechanisms for spatial and cross-view feature interaction. To better relate different viewpoints, we introduce a learnable query-key cross-attention module that learns view-dependent correspondences between source and target features. Camera motion is represented by relative poses and Plücker coordinates, and temporal attention is used to model the relations along the camera trajectory. Experiments on Matterport3D and RealEstate10K show that AttenDiffCrafter achieved a good overall performance in terms of visual quality, geometric consistency, and temporal coherence.

Primary area:
generative models
Keywords:
Novel View Synthesis、Single Image Generation、Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q3ozOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AttenDiffCrafter, a diffusion framework for generating novel-view sequences from a single image and a specified camera trajectory. It uses a two-stream, multiscale Transformer architecture to process the clean source image and noisy target image. The proposed learnable query-key cross-attention combines image features, camera embeddings, and learned query embeddings to model source–target correspondences. Camera conditioning incorporates relative poses, Plücker embeddings, and temporal attention in the pose encoder. Experiments on Matterport3D and RealEstate10K report improvements in reconstruction metrics and short-term FVD.

Critical strengths
  • Experiments demonstrate the paper achieves better performance on the task of novel view synthesis compared to baseline models.
  • The paper is well-written and easy to understand.
Critical weaknesses
  • Adopting transformer for novel view synthesis is very common and can not be seen as a contribution. The proposed method largely based on the structure of Pixel-Art and adds plucker ray to achieve camera control, this is not novel and all existing technique. So to me, the tech novelity and contribution of the paper is very limited.
  • The paper emphasizes that the method does not require a pretrained video diffusion model, but the impact of pretraining is not sufficiently evaluated. I would recommend that the authors compare training from scratch with pretrained initialization under comparable settings, reporting both generation quality and training cost.
Questions for authors

N.A.

Reviewer_YMNbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AttenDiffCrafter, a diffusion model for generating a sequence of novel views given a camera trajectory from a single image. The model is a two-stream, weight-shared, U-shaped transformer denoiser in latent space (AutoencoderKL, 256×256). To keep attention affordable, it downsamples tokens the same way U-DiT does. Camera information enters through Plücker embeddings of the relative pose between the two views. The main new component, LQCA, is a cross-attention layer between source and target features with an added learnable query. The model is trained from scratch, separately on RealEstate10K and Matterport3D.

For validation they report pixel metrics on 20-frame sequences, in various length settings. They also report CLIP frame consistency and camera-pose error estimated with DUSt3R. FID and FVD are reported only for the short clips. GeoGPT, LookOut and PC-NVS are compared on both datasets, while the newer methods (GenWarp, CamI2V-Epi, ViewCrafter, SceneSplatter) appear only on RealEstate10K. They also provide ablations to compare LQCA with epipolar attention and check the effect of the Plücker encoding and the pose attention.

Critical strengths

I see the main strength as the competitive results that presented with respect to baselines used (though the re-implementation of the baselines are concerning)

On Table 1, reports best PSNR, LPIPS values on RE10K wrt methods compared to also for rotation distance. On Table 2 lowest in many of the metrics for the baselines considered.

Ablations on the proposed LQCA mechanism vs epipolar-masked attention variants show the methodolical contribution is leading to a boost with respect to alternatives that are commonly used.

Critical weaknesses

My main concern is how the system and the baselines are trained is not made clear by authors and some of the umbers are confusing. I detail below:

The authors mention that methods are trained separately for separate datasets. The reason for this is not discussed. Many of these methods use pretrained video models. These only appear on one of the datasets and not the rest, no clear justification given.

Some older methods give better results than recent models, this makes me believe something is off, and metrics used are different to those provided in the papers, for instance CamI2V-Epi, they do report results on RE10K, but the different metrics and reimplementations that are not clearly defined make the comparisons much harder. For instance, rotation error, FVD already reported in some of the baselines and these are not matching the paper. Without clear descriptions of reimplementations it is very hard to understand and rely on these numbers.

Secondly, the paper closely follows U-DiT(2024), and though some transformer-based camera controlled approaches are cited in the discussion however not compared to the method described. Attention-centric denoiser fixes the colour shifts and "unknown regions" of U-Net NVS diffusion, this is used as a description but not evidenced clearly.

The LPIPS values in Tables 1 and 5 (around 1.8 to 3.6) are well outside the range the standard metric produces, which is usually below 1. This points to either an unstated scaling or an error in the evaluation code, and since the same pipeline was used for all baselines, it makes me less confident in the other numbers too.

Questions for authors

Can the authors clarify baseline computations, why are there clear discrepancies in terms of metrics to other papers? Is there a specific reason the authors go for reimplementation of baselines (these are not discussed at all)?

There are many curious points that are not clarified in the paper. FVD is computed even at times on " 3 frame" clips, how can this be compared to existing approaches? Why would LPIPS be differing so much between ablations and actual results, and why is the nominal value of LPIPS not [0,1].

Why MP3D only considers 2021-2023 baselines, and some recent baselines score significantly worse on RE10K contradictory to their own reporting?

Did the authors consider comparing to U-net denoiser, or provide a justiufication about colour-shift mentioned from abstract on as a key argument of the paper?

Reviewer_bDAaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new image DiT architecture for novel-view synthesis (NVS). Specifically, it introduces a learnable query–key cross-attention (LQCA) module that combines VAE latent features with source and target camera Plücker embeddings to improve cross-view correspondences. Based on experiments on RealEstate10K and Matterport3D, the paper reports improved visual quality and content coherence in the generated novel views.

Critical strengths
  • The paper provides detailed explanations of the model architecture and implementation, which help readers understand and reproduce the method.
Critical weaknesses
  • The evaluation setup for some baselines is unclear, and the reported results appear inconsistent with those in their original papers. For example, the paper reports severe color shifts in ViewCrafter. Such artifacts are understandable when multiple rollouts are used to generate long sequences. However, the evaluation involves only 20 novel views, which ViewCrafter should be able to generate jointly without such severe artifacts, judging from the RealEstate10K examples in its original paper. In addition, more recent DiT-based NVS methods, such as TrajectoryCrafter, are not included in either the related work or the evaluation.
  • Although LQCA is the main novelty, its motivation and importance remain unclear, and the ablation study does not adequately establish its benefits. plucker embeddings are already widely used for camera control in diffusion-based NVS. What is the intuitive advantage of incorporating them through LQCA? The ablation should compare against commonly used conditioning strategies, such as concatenating plucker embeddings with image features before self-attention, injecting them through AdaLN, and using vanilla cross-attention.
Questions for authors

Please see the Critical Weaknesses section above. Additionally:

  • In Figure 4, I do not see a clear visual-quality advantage of AttenDiffCrafter over the other methods. For example, ViewCrafter appears to generate a better-quality chair. Could the authors clarify the advantage illustrated in this figure?
  • Given the substantial training time of 20–25 days, why train AttenDiffCrafter’s DiT from scratch rather than incorporate LQCA and the other proposed components into an existing image or video DiT-based NVS model and finetune it? This could reduce training effort and provide a fairer, more convincing evaluation of the proposed module.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.