acceptodds
Under review as a conference paper at ICLR 2027

DVSM: The Simplest Decoder-Only View Synthesis Model Performs Best

Abstract

Regression-based view-synthesis models have explored encoder-decoder, decoder-only, and foundation-model designs, but it remains unclear which composition is most effective. We revisit this design space through controlled experiments and find that the simplest choice works best: one decoder should both reconstruct a reusable scene state and render novel views. DVSM prefills a per-layer KV cache from the context views, then renders each target through unidirectional cross-attention using the same weights. Every tested form of weight decoupling reduces quality despite increasing the parameter count. Our feature-space analysis provides a consistent explanation: for the same viewpoints, complete weight sharing keeps reconstruction and camera-only rendering features aligned and enables the renderer to recover the corresponding oracle features. Foundation features can be injected during context prefill as an orthogonal enhancement. DVSM establishes state-of-the-art results on DL3DV and zero-shot out-of-distribution benchmarks, outperforming both geometry-free and geometry-based baselines without explicit 3D inductive biases. On most benchmarks, our best-performing variant is also the smallest model, showing that the gains come from our design rather than model scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel view synthesis, Large view synthesis model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GPuYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits some of the design choices in LVSMs, specifically how "scene reconstruction" and "scene rendering" interact with each other. The resulting architecture is a decoder-only transformer with unidirectional cross-attention from rendering tokens to scene tokens. Several minor changes, including the use of DINO-v3 feature, alternating attention blocks, further improve performance.

Critical strengths
  1. The paper explored several possible combinations inside the design space of a regression-based view-synthesis model (a.k.a. LVSM). The resulting weight-sharing model is supported by solid experiments.
  2. The evaluation covers sparse and dense inputs, higher resolutions, and multiple zero-shot datasets.
  3. Multiple components further improve the overall quality (with DINO-v3 features and alternative attention) and rendering speed (with KV-Cache).
Critical weaknesses

The main weakness is the generalizability of the paper's conclusion. The architectural conclusion may not hold under different data and model scales. The overall contributions are also somewhat incremental, as the final architecture is basically a decoder-only LVSM (with target->seen cross-attn.) plus VGGT's alternative attention. While the empirical improvements are notable, the architectural novelty and the extent to which these conclusions generalize remain limited.

Questions for authors

none

2 comments

We thank the reviewer for recognizing the empirical improvements and the solid experimental support for our weight-sharing model.

W1. Conclusion scope

We agree that our main conclusion about the benefit of sharing decoder weights between reconstruction and rendering may not hold at other model or training-data scales. We have only tested a common LVSM model size and training recipe, and now state this limitation. Our ps8 model outperforms the larger LaCT and LagerNVS models on the OOD benchmarks (Table 4), despite LagerNVS benefiting from VGGT pretraining and broader training data. These comparisons support the generalization of our final model, but do not establish that the weight-sharing benefit holds at other scales.

Thanks for the rebuttal. I will retain my score.

Reviewer_MgnnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper present DVSM, a decoder-only model for novel view synthesis with shares weights between scene encoding and target-view rendering. Given calibrated input images, the model constructs a reusable per-layer KV cache. Target-camera tokens then query this fixed cache through unidirectional cross-attention to directly predict novel-view images.

The authors find that complete weight sharing improves rendering quality over separating the two stages. Experiments report strong performance on DL3DV and zero-shot out-of-distribution benchmarks.

Critical strengths

Simple architecture with a reusable scene representation. A single decoder performs both context encoding and target-view rendering, using a fixed per-layer KV cache to reuse scene information across target views. This provides a clear architectural design with fewer parameters than separate reconstruction and rendering networks.

Clear architectural ablations. The paper isolates design choices under matched data, training schedules, and evaluation protocols. The feature-space analysis provides additional empirical insight into the benefits of sharing.

Critical weaknesses

I did not identify major technical weaknesses. My remaining concerns are relatively minor.

  1. Scope of the weight-sharing ablation. The controlled experiments support the benefit of weight sharing under the evaluated setting. However, it remains unclear whether the same benefit holds with very few input views, where geometric information is more limited. Question: Could the authors include a shared-versus-decoupled comparison in the two-view setting to assess whether the benefit remains consistent?
  2. Extension to settings with unknown cameras. The supplementary experiment using π3-estimated poses is encouraging, although it relies on an external pose estimator and ground-truth camera intrinsics. Question: How sensitive is DVSM to errors in the estimated camera parameters? Could the framework be extended to jointly estimate cameras and synthesize novel views when both poses and intrinsics are unknown?
Questions for authors

See Weakness

2 comments

Thank you for the positive assessment and the two focused suggestions.

W1. Weight sharing with two input views

Following our ablation setup, we trained and evaluated the shared and decoupled models in the two-view DL3DV setting:

SetupPSNR (dB) ↑
Decoupled19.21
Shared21.41

Sharing improves PSNR by >2 dB, showing that its benefit also holds in the sparse two-view setting, consistent with the dense-view result.

W2a. Sensitivity to pose errors

We analyze DVSM ps8 with π³-estimated poses in the 16-context-view DL3DV setup (supplementary Figure 5). Both pose-error metrics are normalized by the root-mean-square distance of the ground-truth context camera centers from their centroid. Their Pearson correlations with each scene's mean novel-view MSE are:

Pose-error metricPearson r with novel-view MSE
Normalized RTE0.460
Normalized ATE0.335

Scenes with larger pose errors generally have higher novel-view MSE. The substantial scatter shows that pose error alone does not determine rendering quality. Scene content can affect pose estimation and rendering differently: for example, textureless regions may be relatively easy to render but provide few visual cues for accurate pose estimation.

W2b. Joint camera estimation

Jointly estimating camera poses and intrinsics within DVSM remains future work. One option is to predict reconstruction-stage Plücker ray maps instead of using them as input conditions, inspired by DA3. Another is to prepend a VGGT frontend, following its use in LagerNVS, and jointly finetune it with DVSM. This would let us test whether joint training improves both camera estimation and novel-view synthesis.

The rebuttal resolve my concern, I decide to maintain my score

Reviewer_17o1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an analysis on different transformer based architectures for feed-forward NVS networks. The paper proposes an LVSM style network that is composed of a single decoder only transformer, with alternating local and global self-attention between the input frames, whose weights are shared for encoding the context frames and decoding the target views.A KV cache is generated from the context views, which is later cross-attended to from the target views.

Critical strengths
  • In principle the weight sharing idea is appealing and is backed up by the ablations provided in table 5.
  • The idea of KV-cache as a scene representation is powerful and offers a more expressive representation compared to the final latent tokens only.
Critical weaknesses
  • Your main argument for your improved performance seems to be hinging on weight sharing, which does have some merit, However, you have a KV-cache per layer which is more expressive than a final latent. Couldn't that be attributed to the performance improvement? The method lacks analysis on that.
  • What does the feature space analysis prove exactly? Why should we expect the NVS and recon features to be similar? There is no reason that both NVS and recon should have similar features, their features serve different purposes so it's fine if they differ.
  • The method claims that its smaller model outperforms other bigger models. While this is true, the overall memory requirements for DVSM esp, its best performing variant ps 8, needs a very large KV cache, so it's not clear if the memory savings by the smaller model size are relevant.
  • The patch size for the baselines is not mentioned which makes it hard to compare the results, since most of the wins seem to be when DVSM uses a smaller patch size.
Questions for authors
  • Something in the timing measurements looks weird. In table 3, (256×448) resolution, for your method, the 4 input views ctx time is lower than the 2 input views. Also the 6 vs 2 input timings also don’t add up, while 4 and 6 have roughly similar timings. The rendering time also is constant across the 3 input view dimensions which should scale with the kv cache size?
  • Can you please report the run-time memory analysis of your method, how much memory does your KV cache end up using as a function of the number of input frames and with differ patch sizes.
  • Can you please include LagerNVS to the comparison in the zeroshot experiments?

Minor comments:

  • The highlighting colour scheme is confusing, some columns are fully highlighted in more or less the same shade, so highlighting loses its meaning at this point.
  • Things like ps, need to be defined in the main paper not in the appendix.
  • Figure 1 typos, intral-view and alternative?
  • Mixing of units, s and ms, in the tables is confusing.
4 comments

Thank you for identifying where our attribution and presentation were unclear.

W1. Performance improvement from a fixed set of scene tokens to a KV cache

We agree that dense, input-scaled context features can contribute to the quality improvement over a fixed set of learnable scene tokens. The LVSM paper already showed that decoder-only LVSM, whose scene representation scales with the context-token count, outperforms encoder–decoder LVSM with its fixed scene-token bottleneck. Our Table 3 likewise shows that encoder–decoder LVSM's quality nearly saturates in the high-resolution, dense-view setting, with little improvement from 16 to 64 context views. Training decoder-only LVSM is prohibitively costly in this setting. We therefore examine the Efficient LVSM design in our controlled experiments (Table 5). Its key differences from DVSM are context encoding and weight sharing (Table 1). Configurations (a), (f), (g), and (h) of Table 5 show that global context encoding and weight sharing substantially improve quality while preserving the same dense KV-cache dimensions.

W2. Meaning of the feature space analysis

Our aim is to uncover behavioral differences between shared and decoupled weights. In Figure 2(a), we compare the cosine similarity of retrieved scene features f(Q)=softmax(QKᵀ)V for RGB+camera reconstruction and camera-only rendering at the same viewpoint, holding K,V fixed within each model and layer. Our intuition is that higher similarity suggests better recovery of the appearance represented by the corresponding oracle RGB-conditioned tokens. To further validate this interpretation, we add color readouts from the output feature of each of the 12 blocks by freezing each backbone and fitting new single-layer MLP heads (Figure 2b). The shared variant produces clearer structures and sharper details from the middle layers onward, while the decoupled variant remains blurrier and fails to recover parts of the scene even at the final layer. These results support our interpretation that weight sharing facilitates appearance recovery for camera-only rendering queries. We have clarified this discussion in Section 3.3 and expanded Figure 2 accordingly.

W3. Memory savings from the smaller model

We agree that trainable parameter count and runtime GPU memory are different quantities. DVSM ps8 achieves higher PSNR with 163M trainable parameters than LaCT (379M) and LagerNVS (1052M), demonstrating parameter efficiency. Fewer parameters reduce weight storage, but cache and working memory can outweigh those savings during runtime. As shown below, ps8 outperforms LaCT and LagerNVS in PSNR across all six settings, both when it uses less peak memory (2/4/6 views) and when it uses more (16/32/64 views). Its smaller parameter count therefore does not imply lower overall memory in the dense-view setting. We have added the runtime memory usage to supplementary Section D (Tables 12–15). PSNR and parameter counts are copied from Table 3; memory values are measured on an H100 GPU and reported in GiB.

DL3DV: 2 context views, 256×448.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37917.642.8382.701
LagerNVS105220.976.5856.321
DVSM ps16+dino45421.722.8162.800
DVSM ps816322.651.3061.254

DL3DV: 4 context views, 256×448.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37923.413.0112.674
LagerNVS105222.896.9136.345
DVSM ps16+dino45427.032.8812.839
DVSM ps816328.321.5951.447

DL3DV: 6 context views, 256×448.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37927.293.2462.699
LagerNVS105223.517.2656.400
DVSM ps16+dino45428.902.9492.880
DVSM ps816330.701.8801.631

DL3DV: 16 context views, 540×960.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37924.9510.1793.015
LagerNVS105221.738.9256.757
DVSM ps16+dino45426.225.5074.541
DVSM ps816326.8611.3868.045

DL3DV: 32 context views, 540×960.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37927.2217.7493.112
LagerNVS105222.1711.5677.146
DVSM ps16+dino45428.218.3056.312
DVSM ps816329.1321.75414.868

DL3DV: 64 context views, 540×960.

MethodParams (M)PSNR (dB) ↑Recon. peak (GiB) ↓Render peak (GiB) ↓
LaCT37928.6632.8703.292
LagerNVS105221.9116.9097.940
DVSM ps16+dino45429.8313.9059.857
DVSM ps816331.1142.48428.497

W4. Patch-size fairness

LVSM, decoder-only LVSM, Efficient LVSM, SVSM, and LaCT all use patch size 8 in the compared configurations, matching our highest-quality ps8 setup. LagerNVS also uses patch size 8 in its novel-view decoder; its VGGT encoder uses patch size 14. Our ps16 configuration is a faster variant that remains competitive with the baselines. We have clarified these patch sizes in the revised paper.

Q1. Timing

We have corrected Table 3 with the following H100 measurements at 256×448 resolution. All values are in milliseconds:

Model/stage (ms)2 views4 views6 views
ps16+dino reconstruction38.76437.29042.741
ps16+dino rendering9.5859.6499.602
ps8 reconstruction19.25927.20141.504
ps8 rendering9.4769.26610.115

At this low-resolution, few-view setup, the H100 can process many token operations concurrently. This is consistent with additional token and KV-cache work being absorbed by GPU parallelism, so latency need not grow proportionally with context count. Small run-to-run timing variations can also make these nearly equal measurements nonmonotonic as the number of views increases. Rendering remains near 9–10 ms and ps16+dino reconstruction varies modestly, while ps8 reconstruction still increases with the number of views.

Q2. Run-time memory and KV cache versus input frames

The table below reports retained KV-cache storage and separate reconstruction/rendering peaks for ps16+dino and ps8 at 540×960 (supplementary Table 12). We use an H100 80GB, FP32 weights with BF16 autocast, one scene, and one target per render. All memory values are in GiB; stage peaks include the cache.

Context viewsps16+dino cacheps16+dino recon.ps16+dino renderps8 cacheps8 recon.ps8 render
20.23.13.00.82.32.1
40.43.43.21.73.62.9
80.84.13.73.46.24.6
161.75.54.56.711.48.0
323.48.36.313.421.814.9
646.713.99.926.942.528.5

Cache storage grows linearly with input-view count. At fixed resolution, ps8 uses four times the cache of ps16+dino because it has four times as many context tokens. The reconstruction and rendering peaks also increase with context count.

Q3. LagerNVS zero-shot comparison

We have added the official LagerNVS general checkpoint to Table 4. DVSM ps8 retains the best PSNR, SSIM, and LPIPS in all six dataset/view-count settings. For convenient comparison, the updated PSNR results (dB) are:

MethodMip360 16vMip360 64vFree 16vFree 64vT&T 16vT&T 64v
LagerNVS21.3022.2219.8119.7219.4420.78
DVSM ps822.2926.0622.9827.5520.3424.35

We also compare at a 512-pixel long side, matching LagerNVS's preferred resolution setting. The new supplementary OOD table gives the following PSNR results (dB):

MethodMip360 16vMip360 64vFree 16vFree 64vT&T 16vT&T 64v
LagerNVS22.0223.0720.2920.2619.8721.40
DVSM ps823.3327.3123.5728.2120.6224.89

DVSM ps8 again leads in all six settings, with the same advantage in SSIM and LPIPS.

Minor comments

Following LVSM, we have updated the comparison tables to highlight the top three values in pink, orange, and yellow, with ties sharing a color. We have defined ps in the main text. We have corrected Figure 1 to “Intra-view” and “Alternating.” We retain column-wise latency units: ms for low-latency columns and s for longer reconstruction times, with a consistent unit within each column and benchmark panel. Choosing the unit by scale avoids long millisecond integers or unnecessary leading zeros in seconds.

Thanks for all the clarifications. I appreciate the decoder weight sharing idea + the investigation/experiments setup. I find the additional RGB readout in Fig. 2 to provide a convincing visual proof for the effects on the feature space. Given that and the additional results and timing fixes, I will be raising my score.

Comments on the paper

1 comment

We thank the reviewers for the constructive feedback. We appreciate the recognition of DVSM’s simple reusable scene representation, controlled ablations, and strong empirical results.

Our main finding is that simply sharing the decoder between reconstruction and rendering substantially improves novel-view synthesis quality. In our controlled experiment (Table 5), complete sharing yields a 1.41 dB PSNR gain over the fully decoupled model, and every tested form of weight decoupling reduces quality. DVSM improves PSNR over the strongest baseline in each setting by 1.1–3.3 dB on DL3DV and 0.6–1.8 dB on zero-shot out-of-distribution benchmarks.

We have strengthened the evaluation in response to the reviews:

  • Expanded Figure 2 with layerwise color readouts. Shared weights produce clearer structures and sharper details from the middle layers onward, supporting our interpretation that weight sharing facilitates appearance recovery.
  • Added LagerNVS to the zero-shot comparison (Table 4) and added comparisons at a 512-pixel long side (supplementary Tables 10–11). We outperform LagerNVS across all three quality metrics in these comparisons.
  • Corrected the sparse-view runtime measurements and reported DVSM’s cache storage and separate reconstruction/rendering memory peaks for DVSM and the baselines (supplementary Tables 12–15). These results make the quality–memory tradeoff explicit.
  • Added a 16-context-view DL3DV analysis with π³-estimated poses (supplementary Figure 5), showing that larger pose errors generally accompany higher novel-view MSE.

The revision also states that the architectural conclusion remains limited to the tested model and training-data scales. The responses below address each reviewer point with the corresponding evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.