acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Central Ray: Ray Differential Attention for Novel View Synthesis

Abstract

Generalizable novel-view synthesis reconstructs unseen views from posed images using transformer in a single forward pass. Transformers tackle this task by aggregating features across views, where performance critically depends on how attention encodes camera geometry. Existing methods predominantly represent each token as an infinitesimal central ray, discarding the patch's local angular scale, anisotropy, and shear across the viewing sphere. To address this limitation, we introduce Ray Differential Attention (RDA). First, we represent each token by a ray differential—a unit ray along with its spatial image derivatives capturing the local patch footprint without requiring scene depth. Second, RDA incorporates this geometry into both attention routing and feature aggregation. For routing, dual-primal query–key transforms yield relative, world-frame-invariant attention scores. For aggregation, an invertible value transform enables consistent cross-view content transport via an exact query-side inverse. This establishes a broader principle that camera geometry should actively govern both token interaction and feature expression, rather than serving as passive conditioning. In a controlled LVSM host, RDA outperforms five camera baselines across PSNR, SSIM, and LPIPS on RealEstate10K and DL3DV. Rigorous ablations confirm that both the full ray differential footprint and the dual read/write pathways are essential to these gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
generalizable novel view synthesis, ray differential attention, multi-view transformers, geometry-aware attention, camera geometry, relative attention, depth-free rendering, neural rendering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZHeCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Ray Differential Attention for novel view synthesis without explicit depth estimation. The method uses a central ray and its derivatives along the two image axes to capture the local angular scale, anisotropy, and shear of image patches. These geometric cues are incorporated into both attention scoring and feature aggregation. Specifically, the Q/K transformations model relative geometric relationships, while the V/O transformations enable cross-view feature transfer through invertible mappings. Under a unified LVSM architecture, the authors compare their method against five camera encoding approaches and achieve the best average performance across three metrics on RealEstate10K and DL3DV.

Critical strengths
  1. The local geometric representation is well motivated. Section 3.2 uses ray differentials to preserve local metric information that is lost through normalization in orthogonal ray frames, and incorporates it into invertible token transformations. The method requires no explicit depth estimation and does not increase the attention channel dimensionality, making it a geometrically well-grounded design.
  2. The main experiments demonstrate consistent improvements under controlled settings. Table 1 compares RDA against five external baselines across three random seeds. RDA improves PSNR by 0.446 dB over UCPE, while also achieving better SSIM and LPIPS scores.
Critical weaknesses
  1. Limited novelty. GTA (GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers) already incorporates geometry-aware feature comparison and value aggregation in §3.1 and Eqs. (4)–(6), and maps the aggregated features back to the query coordinate frame. PRoPE (Cameras as Relative Positional Encoding) aso introduces projective transformations involving both camera intrinsics and extrinsics, combined with 2D RoPE. Therefore, the claim in Section 1 that this work establishes a new principle of using geometry to jointly govern token interactions and feature transformations appears to overstate its actual novelty. UCPE(Unified Camera Positional Encoding for Controlled Video Generation) also constructs token-level local ray transformations. Compared with these prior works, the main contribution of this paper lies in preserving local metric information through ray differentials and adopting distinct transformation designs for attention scoring and feature aggregation. Although the authors cite these works, they do not sufficiently explain why these specific differences constitute a substantial advance. In particular, PRoPE already incorporates camera intrinsics, so the absence of explicit ray derivatives should not be equated with the absence of camera calibration information.
  2. Insufficient experimental evidence. Table 4 provides the most direct evidence for the benefit of full ray differentials, reports results only for seed 0. Compared with the orthogonal frame baseline, the improvement is limited to 0.115 dB in PSNR and 0.0088 in LPIPS. While these results are encouraging, the lack of repeated experiments makes it difficult to assess the robustness of this key contribution.
Questions for authors
  1. As mentioned I'm concerned about this paper's novelty. Beyond replacing the geometric representation and adjusting channel allocation, what technical contributions distinguish RDA from GTA , PRoPE, and UCPE’s local ray transformations? Should the claim of establishing a principle that geometry should govern both attention scoring and feature aggregation be narrowed?
  2. Could you provide a control that retains the content pathway’s 2D RoPE while replacing only its geometric blocks with identity mappings?
  3. Could you compare the full ray differential against the orthonormal frame across additional random seeds and under the same crop/resize interventions? Please also clarify how the Unit ray configuration in Table 4 is constructed to yield an invertible transform.
Reviewer_oAEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes Ray Differential Attention (RDA), which incorporates central rays and their image-space derivatives into attention to represent local patch geometry. With this geometric information, RDA outperforms existing camera-aware methods, including GTA and PRoPE, on novel view synthesis benchmarks.

Critical strengths

1. Focused geometric contribution: A concrete attempt to improve camera encoding through local ray geometry, without increasing attention width or requiring depth prediction. RDA preserves the host model’s 144-dimensional attention heads.

2. Principled formulation: Explicit first-order ray variation, world-frame-invariant attention scores, and analytically invertible value transformations. The paper provides algebraic proofs for the relative-score construction and the inverse content transform.

Critical weaknesses

W1. Insufficiently convincing motivation.

  • It is unclear whether central-ray-based encodings actually lack the information that RDA aims to introduce. Local footprints may already be recoverable from neighboring rays and their image-grid positions, making the distinction between missing information and more explicit encoding important (see Q1).

  • The “key observation” in L49 states a geometric property, but its practical significance is not convincingly established. Figures 1–2 illustrate differences between ray representations without demonstrating a concrete failure of existing methods attributable to footprint ambiguity.

W2. Limited performance improvements

  • The overall performance gains are modest, particularly on DL3DV, where RDA improves PSNR over UCPE by only 0.011 dB (Table 2).

W3. Concerns about the evaluation setup.

  • The overall PSNR in Table 2 is too low (14.5–14.7 dB). It raises questions about the validity of evaluation setting. Clarification of whether the view configuration is excessively challenging and providing additional evaluation setup would help demonstrating the method’s practical benefits.
Questions for authors

Q1. Could existing central-ray-based attention methods also recover local footprint information, such as a token’s angular size, from neighboring central rays and their positions on the image grid?

Reviewer_VdpNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper improves generalizable novel-view synthesis without depth prediction or per-scene optimization. Central-ray representations omit patch extent, anisotropy, and shear, limiting cross-view geometric reasoning. Ray Differential Attention (RDA) captures these properties through calibrated ray derivatives and incorporates geometry into both attention routing and feature transport. Relative query–key transformations ensure world-frame-invariant matching, while invertible value transformations align aggregated features with the query frame. RDA outperforms five camera-encoding baselines on RealEstate10K and DL3DV, with small PSNR gains on the latter. Ablations support both the full ray-differential representation and the two geometric pathways.

Critical strengths
  1. Precise representational motivation. Ray differentials address geometric information omitted by central rays and orthonormal frames, retaining angular scale, anisotropy, and shear without depth estimation.
  2. Rigorous mathematical correspondence. Relative transformations of queries and keys implement coordinate invariance, while invertible value transport ensures consistent feature aggregation.
  3. Well-controlled experiments. A shared backbone, matched training conditions, five baselines, three seeds, component ablations, and efficiency profiling support credible comparisons.
Critical weaknesses
  1. Concentrated methodological novelty. The main novelty lies in the ray differential representation, while relative attention and geometric feature transport follow existing constructions, and the individual necessity of scale, anisotropy, and shear remains unverified.
  2. Partial mechanistic isolation. Principal pathway and representation ablations use one seed, while crop and resize tests jointly alter image sampling and intrinsics, preventing clean attribution to calibration sensitivity.
  3. Restricted experimental scope. Evaluation covers one backbone, two context views, calibrated static scenes, and 256×256 rendering, with only a 0.011 dB PSNR advantage on DL3DV.
Questions for authors

Here are some suggestions for rebuttal:

  1. Novelty and necessity. Isolate or analysis the contributions of scale, anisotropy, and shear, and compare against standard attention receiving the same ray differential information.
  2. Mechanistic validation. Introduce calibration controls using identical images to separate geometric effects from changes in image sampling.
  3. Experimental scope. Evaluate more additional backbones and input-view settings, and report paired scene-level confidence intervals on DL3DV to contextualize the small PSNR gain.

Comments on the paper

1 comment

Thank the new exp and response. The explicit construction of the unit-ray baseline resolves our invertibility question. The three seed comparison shows consistent gains in every seed, which strengthens the evidence for the core contribution. I also appreciate the more precise positioning relative to GTA, PRoPE, and UCPE.

Some concerns remain. Exact invertible transport and fixed head width are also properties of GTA and PRoPE. For pinhole cameras, the ray differential is determined by K, R, and the pixel coordinate, which PRoPE already uses, so it is unclear what this parameterization adds. The Q2 interventions are run at inference time and do not show that the geometric blocks help beyond 2D RoPE. A model trained with the geometric blocks set to identity is still needed. The zoom result (+0.185 dB) is measured at a single scale without multiple seeds, and it is smaller than the average gain. Given these remaining concerns, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.