acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Large View Synthesis Models: Capacity and Scene Memory

Abstract

Large view synthesis models (LVSMs) employ diverse architectures and scene representations, but how to allocate capacity and computation for effective scaling remains unclear. We organize these systems through scene memory constructed from context views and queried for novel view synthesis. We systematically study network organization, spatial granularity, model capacity, and the allocation of re- sources between scene memory and decoding. Within the evaluated full-attention configurations, decoder-only models favor reconstruction quality, whereas sepa- rate decoders favor rendering efficiency. Spatial granularity is a key scaling di- mension alongside model capacity, and the benefits of larger scene memory de- pend on its decoder. These findings lead us to select and train a ViT-L/16 back- bone, yielding highly competitive reconstruction quality with shorter reported re- construction times than most baselines on DL3DV. Using this backbone, we ana- lyze sparse attention for novel view synthesis. Training-free block-sparse attention preserves spatial tokens and scene-memory capacity, accelerating scene construc- tion by 1.51–2.22× over dense inference at fixed inputs with only a small quality loss. Further analysis shows how layer sensitivity and routing overhead govern the quality and speed gains. We hope our study helps the community scale up LVSMs more effectively and explore better model designs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Reconstruction Models, Novel View Synthesis, Scene Representation, Large View Synthesis Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AbqcOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper organizes large view synthesis models (LVSMs) through a common functional decomposition — context images construct a scene memory that target cameras query to synthesize novel views — and systematically studies network organization (decoder-only vs. encoder–decoder), spatial granularity (patch size), model capacity, scene representations (KV caches, Gaussians, fast weights), and the allocation of capacity between scene memory and its decoder. Within the evaluated full-attention configurations: decoder-only models favor reconstruction quality while separate decoders favor rendering throughput (Finding 1); spatial granularity is a key scaling dimension alongside model capacity (Finding 2); the KV reference decoder offers a favorable quality/speed balance while FeatureGS uses less memory (Finding 3); and scene-memory size alone does not predict quality or cost (Finding 4). The study selects a decoder-only ViT-L/16 backbone, trains it at 544×960 on DL3DV (BVSM), and analyzes training-free block-sparse cross-view attention, accelerating scene construction by 1.51–2.22× with a small quality loss (Findings 5–6), with layer-sensitivity and routing-overhead analyses.

Critical strengths
  1. A useful framing and clean resource accounting. Decomposing LVSMs into construction and query and separating parameters, scene-memory payload, training TFLOPs, and rendering FPS disentangles factors usually conflated in end-to-end comparisons; the study asks the right scaling questions.

  2. Careful, honest sparse-attention analysis. Layer-sensitivity diagnostics, routing quality vs. overhead trade-offs, and the comparison against token merging are performed at fixed weights and inputs with paired latency/quality measurements, and the paper clearly distinguishes reducing interactions from reducing storage or query cost.

Critical weaknesses

The study never compares against methods doing the same thing. Its baselines are full-attention or generic heuristics. The sparse-construction claim is evaluated only against the dense/full-attention variant and generic token-reduction heuristics (token merging, attention merging, cache pruning); learned and training-free sparse-attention methods designed for reconstruction/view synthesis (e.g., SeerAttention, DeepSeek Sparse Attention, MoBA, and the block-sparse global attention applied to VGGT/Pi3/MapAnything) are listed in related work but never benchmarked. Likewise, the model-level comparison (Table 2) is against full-attention or architecturally different systems (LVSM, Long-LRM, LaCT, tttLRM, …), not same-approach models. It is therefore unclear whether block-sparse scene construction is actually better than, or even competitive with, the same-approach literature, which undermines the paper's central practical claim.

The study's conclusions are too hedged to support acceptance. The paper repeatedly qualifies that comparisons are equal-step rather than compute-matched, and that cross-method runtimes retain heterogeneous source measurement conditions — precisely the quantities its findings depend on. Consequently the headline findings (e.g., "decoder-only favors quality", "spatial granularity is a key scaling dimension", "memory size alone is insufficient") are stated as qualitative trends within hand-picked configurations without controlled budgets, and several are likely unsurprising to experts in the area.

Questions for authors

The sparse-construction claim is benchmarked only against dense/full-attention and generic heuristics. How does BVSM-Sparse compare with other sparse-attention approaches for reconstruction (e.g., SeerAttention, DeepSeek Sparse Attention, MoBA, block-sparse global attention applied to VGGT/Pi3) under matched budgets?

Would the headline findings (especially Findings 1 and 2) survive compute-matched budgets and controlled comparisons? The paper states equal steps ≠ equal compute — can the authors provide at least one compute-matched comparison to anchor the claims? Would block-sparse patterns transfer to training (e.g., distilling dense attention into sparse patterns), and by how much could quality improve?

The layer-sensitivity diagnostic uses six scenes, how stable are the "early layers are more sensitive" conclusions across more scenes and context sizes?

4 comments
AuthorsOpenReview ↗

To Reviewer Abqc 1/2

We thank the reviewer for the detailed feedback and constructive suggestions for strengthening our experimental evaluation.

1. Same-approach comparisons and baseline selection

We agree that comparisons on the same backbone are important for evaluating sparse-attention strategies. We have therefore adapted SpargeAttention and BGSA to BVSM and conducted direct comparisons on 32 scenes at 544×960:

Method / sparse setting32-view PSNR32-view build (s)64-view PSNR64-view build (s)
Dense27.19301.49928.8955.385
SpargeAttention (CDF 0.98)27.14101.59128.8105.536
BGSA, 6.25%26.81400.77628.5852.264
Ours, 6.25%27.03650.79328.7642.358
BGSA, 12.5%26.83000.82328.7342.451
Ours, 12.5%27.09600.84928.8212.572

At the 6.25% setting, our method achieves 0.223/0.179 dB higher PSNR than BGSA at 32/64 views, with 0.017/0.094 s additional construction time. At 64 views, our 6.25% configuration also outperforms BGSA at 12.5% in both PSNR (28.764 vs. 28.734 dB) and construction time (2.358 vs. 2.451 s). SpargeAttention (sim=0.6, CDF=0.98) achieves quality close to Dense but does not reduce construction time in this experiment.

We adapt BGSA's official method to BVSM without VGGT-specific sink tokens and use its native CUDA path for 64-view evaluation. SpargeAttention dynamically selects sparse blocks. For our method, the 6.25%/12.5% retention rate applies to the final eight layers; layers 1–2 remain dense and layers 3–4 retain 50%. These results use a paired Dense baseline from the same 32-scene evaluation, separate from the main 133-scene results. Layer indices 1–12 correspond to indices 0–11 in the paper.

The baselines in Table 2 span different construction and representation approaches, including attention-based LVSM, Mamba2–Transformer models with 3DGS/FeatureGS, and fast-weight models LaCT/tttLRM. Table 2 reports complete-system results under each method's training recipe. In contrast, Table 1 aligns the input settings, global batch size, and number of training steps to study representation–decoder trade-offs. The additional sparse-attention experiments further isolate the effect of sparse routing by using the same BVSM backbone.

2. Matched compute and measured training time

Thank you for raising the question of compute-matched training. We have added comparisons between P8 and P16 under matched estimated FLOPs and matched measured training time. Both models use width 768 and 12 layers (approximately 171–173M parameters), with 144×256 images, 32 context views, and 32 target views. Training uses eight A100 GPUs, a per-GPU batch size of four, and a global batch size of 32.

Following the paper's accounting, each multiply–add counts as two FLOPs, and training FLOPs are estimated as three times forward FLOPs. Total training compute is calculated by multiplying per-scene training FLOPs by the global batch size and the number of training steps.

ConfigurationStepsTotal training EFLOPsTest PSNR ↑Test SSIM ↑Test LPIPS ↓Train Loss ↓
P16 reference50,00015.152023.79000.768000.127000.0457
P8, matched estimated FLOPs5,03615.149922.86140.745350.135090.0433
P8, matched measured training time11,60734.917624.15100.789000.116000.01542

The time-matched comparison uses measured cumulative training time. P8 takes approximately 8 h 20 min for 5,036 steps and 18 h 17 min for 11,607 steps; the latter matches the training time of the P16 reference.

At matched estimated FLOPs, P8 achieves 0.929 dB lower PSNR than P16, despite having a slightly lower training loss. At matched measured training time, P8 achieves 0.361 dB higher PSNR, alongside better SSIM and LPIPS, while using approximately 2.30× the estimated FLOPs. These results demonstrate that spatial granularity plays an important role in the quality–efficiency trade-off, with different outcomes under FLOPs-matched and time-matched training budgets. The complete training-loss curves for these experiments are shown in Fig. 9.

For Finding 1, the original study also includes configurations with nearly identical estimated model compute: FSA decoder-only requires 94.01 TFLOPs per sample and achieves 26.44 dB, while ColorGS requires 94.00 TFLOPs and achieves 21.13 dB. These estimates exclude Gaussian splatting, which we will specify in the paper.

AuthorsOpenReview ↗

To Reviewer Abqc 2/2

3. Runtime comparison protocol

Regarding runtime measurement protocols, Table 2 reports reference runtimes under the conditions used by the respective methods. By contrast, the 1.51–2.22× acceleration reported in our paper is based on paired Dense/Sparse measurements in Table 6, using the same checkpoint, input data, and A100 GPU. The new sparse-attention comparisons also measure construction time after adapting the methods to the same BVSM model. We will state these measurement protocols more clearly in the table captions and discussion.

4. Layer sensitivity across more scenes and context sizes

We have expanded the layer-sensitivity evaluation from six to 32 scenes, covering 16, 32, and 64 context views. In each experiment, only one layer is sparsified to 6.25% block retention; all other layers remain dense. Representative results are shown below using one-based layer indices:

Only sparsified layer16-view PSNR32-view PSNR64-view PSNR
Dense25.19227.19328.895
114.69815.09715.185
222.10122.57822.749
324.68126.52727.908
424.97927.03028.764
525.11327.13028.848
825.17127.17128.879
1225.19227.19328.895

Across all three context sizes, sparsifying the first two layers causes the largest quality degradation, while the later layers are much less sensitive. Sparsifying layer 12 leaves the result unchanged because its final update is not used in subsequent cache-based rendering.

We further compare three sparsity schedules at 32 views, each with a nominal average block retention of 29.17%, alongside a Dense reference:

ConfigurationPer-layer block retentionPSNR (dB)Build (s)
Dense100% throughout27.19301.499
Early-layer protection (ours)L1–2: 100%; L3–4: 50%; L5–12: 6.25%27.03650.793
Uniform29.17% throughout15.85300.831
ReversedL1–8: 6.25%; L9–10: 50%; L11–12: 100%15.24800.794

Our early-layer protection schedule achieves substantially higher PSNR than uniform and reversed sparsity allocation at similar construction time. Together with the single-layer experiments, these results directly support our layer-wise sparsity design.

An improvement of 0.2 dB over BGSA is marginal at best; it is difficult to determine whether this is truly due to a superior algorithm or simply because more computation is preserved in the earlier layers.

Furthermore, regarding the image quality advantage of the decoder-only approach—does this stem from the architecture itself, or from the fact that the two methods employ different scene?

Thank you for pointing out that our experimental settings were not sufficiently clear. BGSA and our method use exactly the same per-layer retention schedule: layers 1–2 remain dense, layers 3–4 retain 50%, and the final eight layers retain 6.25% or 12.5%. Both methods use the same BVSM checkpoint and are evaluated on the same 32 scenes. Therefore, the 0.223/0.179 dB improvements at 32/64 views under the 6.25% setting are achieved with identical layer-wise retention budgets, rather than by preserving more computation in the early layers. We will clarify this in the table caption.

Regarding the quality advantage of decoder-only models, the controlled experiments in Table 1 use the same DL3DV training dataset and follow the Long-LRM evaluation protocol, with identical test scenes and evaluation settings. We also align the input settings, global batch size, and number of training steps. Since the compared configurations differ in both architecture and scene representation, the observed quality differences reflect their combined effects under controlled conditions, rather than differences in training data or evaluation settings.

Reviewer_E8qiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies architectural choices and capacity allocation in large view synthesis models. It analyzes network organization, spatial granularity, scene memory, and decoder capacity. Based on these findings, the authors develop BVSM and apply training-free block-sparse attention to accelerate scene construction. Experiments demonstrate competitive reconstruction quality and up to 2.22× faster scene construction.

Critical strengths
  1. The paper provides extensive experiments on different LVSM architectures and capacity configurations. The analysis offers useful engineering insights.

  2. BVSM achieves competitive reconstruction quality on DL3DV. The sparse implementation provides meaningful acceleration with a relatively small quality reduction.

Critical weaknesses
  1. Limited novelty of the empirical findings. Several central findings substantially overlap with prior work. Finding 1 largely repeats DVSM's controlled analysis of decoder-only architectures, while SVSM has also examined architecture choices under matched training budgets. Finding 2 revisits the spatial granularity versus computational cost trade-off, which DVSM already investigated through patch-size and model-capacity experiments. Although the empirical study is extensive, these results provide limited new insights beyond existing research.

  2. Limited novelty of the sparse-attention method. Training-free block-sparse attention has already been studied in SpargeAttention (ICML 2025) and Block-Sparse Global Attention (CVPR 2026). The latter also uses pooled query-key similarities and block selection for efficient multi-view reconstruction, making it particularly relevant. BVSM mainly introduces sub-block log-sum-exp routing and layer-wise sparsity allocation while reusing the SpargeAttention backend. However, Table 3 shows only modest gains from the new routing strategy, and its individual components are not fully ablated. Direct comparisons with existing block-sparse methods under matched computational settings are needed to establish the independent contribution of BVSM.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

To Reviewer E8qi 1/2

We thank the reviewer for the thoughtful comments and constructive suggestions, particularly regarding the positioning of our study and the sparse-attention comparisons.

1. Relationship to prior work and contribution of the empirical study

DVSM investigates architectural design, weight sharing, and spatial granularity, while SVSM studies scaling across architectures and training strategies. Our work addresses a complementary challenge: existing NVS systems differ in their architectures, scene representations, and training procedures, making it difficult to isolate the effects of individual design choices. We therefore study representation–decoder combinations under controlled training conditions to better understand their quality, speed, and storage trade-offs.

In Table 1, we keep the input resolution, numbers of context and target views, global batch size, and 50K training steps consistent across configurations, while reporting computation separately. Selected results are shown below:

Representation and decoderScene memory (MiB)PSNR (dB)Render FPS
KV cache, shared backbone (decoder-only)648.026.4429
KV, separate decoder (r=4)216.023.25325
FeatureGS, F64 / D768168.822.24154
FeatureGS, F64 / D1024168.822.88103

These results show why representation and decoding should be considered jointly. Among these configurations, the shared KV backbone achieves the highest reconstruction quality, the separate KV decoder offers the fastest rendering, and FeatureGS requires less scene storage. Moreover, the two FeatureGS configurations use the same 168.8 MiB of memory, yet the wider decoder increases PSNR from 22.24 to 22.88 dB, with rendering speed changing from 154 to 103 FPS.

Table 2 provides a complementary comparison with representative NVS systems spanning attention, Mamba2–Transformer, and fast-weight construction, together with direct RGB prediction, 3DGS, and FeatureGS rendering. The following results are from the 32-input-view evaluation:

MethodConstruction mechanismRepresentation and renderingPSNR (dB)
LVSMAttentionAttention-based RGB prediction21.73
Long-LRMMamba2 + Transformer3DGS → RGB rasterization24.10
Long-LRM++Mamba2 + TransformerFeatureGS → feature rasterization → decoder26.43
tttLRMLaCT / fast weightsDecode to 3DGS → RGB rasterization25.07
LaCTLaCT / fast weightsQuery fast weights → RGB26.90
BVSM-DenseAttentionKV cache → shared-backbone RGB prediction27.42

Table 2 compares complete systems using their respective training recipes, whereas Table 1 controls the key training conditions to examine the effects of different representations and decoders. We will clarify this distinction and better explain how the two evaluations complement each other.

2. Direct comparisons with sparse-attention methods

To address the requested comparisons, we have adapted SpargeAttention and BGSA (Block-Sparse Global Attention) to the same BVSM model and evaluated them on 32 scenes at 544×960. These additional experiments use a paired Dense baseline and are reported separately from the main 133-scene evaluation. Layer indices 1–12 below denote the cross-view layers (indices 0–11 in the paper).

Method / sparse setting32-view PSNR32-view build (s)64-view PSNR64-view build (s)
Dense27.19301.49928.8955.385
SpargeAttention (CDF 0.98)27.14101.59128.8105.536
BGSA, 6.25%26.81400.77628.5852.264
Ours, 6.25%27.03650.79328.7642.358
BGSA, 12.5%26.83000.82328.7342.451
Ours, 12.5%27.09600.84928.8212.572

At the 6.25% setting, our method achieves 0.223/0.179 dB higher PSNR than BGSA at 32/64 views, with only 0.017/0.094 s additional construction time. At 64 views, our 6.25% configuration also achieves higher PSNR (28.764 vs. 28.734 dB) and faster construction (2.358 vs. 2.451 s) than BGSA at 12.5%. SpargeAttention (CDF 0.98) maintains quality close to Dense, but does not improve construction speed in this setting.

For implementation details, we adapt the official BGSA method to BVSM without introducing VGGT-specific sink tokens; its 64-view results use the native CUDA path. SpargeAttention uses its fused routing implementation with sim=0.6 and CDF=0.98, which dynamically selects blocks. For our method, 6.25%/12.5% denotes the retention rate in the final eight cross-view layers; layers 1–2 remain dense, and layers 3–4 retain 50% of blocks. We also acknowledge SpargeAttention's execution backend, which our method uses.

AuthorsOpenReview ↗

To Reviewer E8qi 2/2

3. Routing components and layer-wise allocation

We additionally ablate the routing components. At 32 input views, the model, evaluation scenes, execution backend, and per-layer sparsity budgets remain fixed. All variants have the same actual average block retention of 29.18%.

Routing configurationPSNR (dB)
Block mean, unnormalized26.9181
Block mean, L2-normalized27.0100
Sub-block mean, L2-normalized27.0090
Sub-block LSE, L2-normalized (ours)27.0365

L2 normalization improves the Block mean baseline by 0.0919 dB. Our Sub-block LSE routing further improves PSNR by 0.0275 dB over normalized Sub-block mean and 0.0265 dB over normalized Block mean. Casting the unnormalized Block mean descriptors to FP32 leaves the result unchanged at 26.9181 dB. These results distinguish the gains from normalization and Sub-block LSE routing.

We also compare three layer-wise sparsity schedules with the same nominal average block retention of 29.17%; Dense is included as a reference:

ConfigurationPer-layer block retentionPSNR (dB)Build (s)
Dense100% throughout27.19301.499
Early-layer protection (ours)L1–2: 100%; L3–4: 50%; L5–12: 6.25%27.03650.793
Uniform29.17% throughout15.85300.831
ReversedL1–8: 6.25%; L9–10: 50%; L11–12: 100%15.24800.794

Protecting the early layers preserves substantially higher reconstruction quality at comparable construction time. This demonstrates the importance of allocating the sparsity budget according to layer sensitivity, rather than applying a uniform sparsity ratio.

Reviewer_qkCkOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides an in-depth analysis on the factors that affect view synthesis transformers' memory capacity and scalability. Key findings are:

  1. Decoder-only architecture is better for quality when not considering the rendering speed. Encoder-decoder architecture is faster in terms of rendering.
  2. Patch granularity is as important as, or more important than, model size, especially for the decoder.
  3. Different scene representations (transformer KV cache vs Gsplat representation) have different trade-offs between rendering quality and speed.
  4. Scene memory capacity and decoder capacity are both important.

Based on these findings, the authors propose a novel sparse-attention architecture named BVSM, which achieves state-of-the-art performance on several benchmarks.

Critical strengths

The paper identifies several previously underexplored scaling factors, providing valuable insights and practical guidelines that could inform future architecture design. In particular, geometry-grounded (e.g., gsplat, pointcloud) vs geometry-free (e.g., trandformer kv) scene representation is a long-standing debate in nvs, but to my knowledge, no work has disentangled this comparison into encoder and decoder part. It was previously unclear whether the advantage of geometry-free approach primarily comes from only the decoder part, or both.

The proposed sparse attention design, which builds on these insights, is novel. The claims are well-supported by strong empirical evidence.

Critical weaknesses

I do not have any outstanding concerns. The experiments are scientifically rigorous and well-designed. Please find the minor concern in the questions section below.

Questions for authors

The paper seems to treat scene memory capacity and scene representation capacity as an equivalent concept. Is there a justification behind this assumption?

1 comment

We thank the reviewer for the careful reading and constructive feedback on our terminology and analysis.

Scene-memory size versus representational capacity

We agree that scene-memory size should be distinguished from representational capacity. In our resource analysis, scene-memory size measures the persistent storage budget (Sec. 2; App. A.3, Eq. (5)), while reconstruction quality depends on both the representation and its decoder. Table 1C provides a direct example with two FeatureGS configurations using the same feature width (F64):

Decoder widthScene memory (MiB)PSNR (dB)
768168.822.24
1024168.822.88

With identical scene-memory sizes, increasing the decoder width improves PSNR by 0.64 dB. This supports Finding 4: scene-memory size alone does not determine reconstruction quality. We will use “scene-memory size” or “storage budget” consistently when referring to measured storage and distinguish it from representational capacity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary and Response to all Reviewers

We thank all reviewers for their thoughtful feedback and constructive suggestions.

The main motivation of our work is to better understand the design choices and quality–efficiency trade-offs in large view synthesis models. Training and evaluating these models is computationally expensive, while existing NVS methods differ considerably in their architectures, scene representations, and training procedures. These differences make it challenging to isolate the effects of individual design choices. To address this, we conduct a systematic empirical study of representation–decoder combinations under controlled training conditions, examining reconstruction quality, computational cost, rendering efficiency, and scene-memory requirements. We also compare representative NVS systems to provide a broader perspective on their practical performance.

During the rebuttal, we conducted several additional experiments to address the reviewers' concerns:

  1. Comparisons with sparse-attention methods. We adapted SpargeAttention and BGSA to the same BVSM backbone. At the 6.25% setting, our method achieves 0.223/0.179 dB higher PSNR than BGSA at 32/64 input views, with comparable construction time.

  2. Additional sparse-routing ablations. We evaluated the individual routing components and different layer-wise sparsity schedules. We also expanded the layer-sensitivity experiments to 32 scenes with 16, 32, and 64 input views. The results demonstrate the effectiveness of our routing strategy and the importance of preserving dense attention in early layers.

  3. Compute- and time-matched experiments. We compared different spatial granularities under matched estimated FLOPs and measured training time. P8 achieves 0.361 dB higher PSNR than P16 at matched training time, while P16 achieves better results at matched estimated FLOPs. These experiments provide additional insights into the relationship between spatial granularity, reconstruction quality, and computational efficiency.

  4. Clarifications on methodology and evaluation. We clarified the distinction between scene-memory size and representational capacity, the different purposes of controlled experiments and complete-system comparisons, and the runtime evaluation protocols.

Overall, these additional experiments strengthen our analysis of the key factors affecting NVS reconstruction quality and efficiency, while providing further support for our sparse routing and layer-wise allocation strategies. We thank the reviewers again for their valuable feedback, which has helped us improve the clarity and experimental evaluation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.