- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper organizes large view synthesis models (LVSMs) through a common functional decomposition — context images construct a scene memory that target cameras query to synthesize novel views — and systematically studies network organization (decoder-only vs. encoder–decoder), spatial granularity (patch size), model capacity, scene representations (KV caches, Gaussians, fast weights), and the allocation of capacity between scene memory and its decoder. Within the evaluated full-attention configurations: decoder-only models favor reconstruction quality while separate decoders favor rendering throughput (Finding 1); spatial granularity is a key scaling dimension alongside model capacity (Finding 2); the KV reference decoder offers a favorable quality/speed balance while FeatureGS uses less memory (Finding 3); and scene-memory size alone does not predict quality or cost (Finding 4). The study selects a decoder-only ViT-L/16 backbone, trains it at 544×960 on DL3DV (BVSM), and analyzes training-free block-sparse cross-view attention, accelerating scene construction by 1.51–2.22× with a small quality loss (Findings 5–6), with layer-sensitivity and routing-overhead analyses.
Critical strengths
-
A useful framing and clean resource accounting. Decomposing LVSMs into construction and query and separating parameters, scene-memory payload, training TFLOPs, and rendering FPS disentangles factors usually conflated in end-to-end comparisons; the study asks the right scaling questions.
-
Careful, honest sparse-attention analysis. Layer-sensitivity diagnostics, routing quality vs. overhead trade-offs, and the comparison against token merging are performed at fixed weights and inputs with paired latency/quality measurements, and the paper clearly distinguishes reducing interactions from reducing storage or query cost.
Critical weaknesses
The study never compares against methods doing the same thing. Its baselines are full-attention or generic heuristics. The sparse-construction claim is evaluated only against the dense/full-attention variant and generic token-reduction heuristics (token merging, attention merging, cache pruning); learned and training-free sparse-attention methods designed for reconstruction/view synthesis (e.g., SeerAttention, DeepSeek Sparse Attention, MoBA, and the block-sparse global attention applied to VGGT/Pi3/MapAnything) are listed in related work but never benchmarked. Likewise, the model-level comparison (Table 2) is against full-attention or architecturally different systems (LVSM, Long-LRM, LaCT, tttLRM, …), not same-approach models. It is therefore unclear whether block-sparse scene construction is actually better than, or even competitive with, the same-approach literature, which undermines the paper's central practical claim.
The study's conclusions are too hedged to support acceptance. The paper repeatedly qualifies that comparisons are equal-step rather than compute-matched, and that cross-method runtimes retain heterogeneous source measurement conditions — precisely the quantities its findings depend on. Consequently the headline findings (e.g., "decoder-only favors quality", "spatial granularity is a key scaling dimension", "memory size alone is insufficient") are stated as qualitative trends within hand-picked configurations without controlled budgets, and several are likely unsurprising to experts in the area.
Questions for authors
The sparse-construction claim is benchmarked only against dense/full-attention and generic heuristics. How does BVSM-Sparse compare with other sparse-attention approaches for reconstruction (e.g., SeerAttention, DeepSeek Sparse Attention, MoBA, block-sparse global attention applied to VGGT/Pi3) under matched budgets?
Would the headline findings (especially Findings 1 and 2) survive compute-matched budgets and controlled comparisons? The paper states equal steps ≠ equal compute — can the authors provide at least one compute-matched comparison to anchor the claims? Would block-sparse patterns transfer to training (e.g., distilling dense attention into sparse patterns), and by how much could quality improve?
The layer-sensitivity diagnostic uses six scenes, how stable are the "early layers are more sensitive" conclusions across more scenes and context sizes?
4 comments
To Reviewer Abqc 2/2
3. Runtime comparison protocol
Regarding runtime measurement protocols, Table 2 reports reference runtimes under the conditions used by the respective methods. By contrast, the 1.51–2.22× acceleration reported in our paper is based on paired Dense/Sparse measurements in Table 6, using the same checkpoint, input data, and A100 GPU. The new sparse-attention comparisons also measure construction time after adapting the methods to the same BVSM model. We will state these measurement protocols more clearly in the table captions and discussion.
4. Layer sensitivity across more scenes and context sizes
We have expanded the layer-sensitivity evaluation from six to 32 scenes, covering 16, 32, and 64 context views. In each experiment, only one layer is sparsified to 6.25% block retention; all other layers remain dense. Representative results are shown below using one-based layer indices:
| Only sparsified layer | 16-view PSNR | 32-view PSNR | 64-view PSNR |
|---|---|---|---|
| Dense | 25.192 | 27.193 | 28.895 |
| 1 | 14.698 | 15.097 | 15.185 |
| 2 | 22.101 | 22.578 | 22.749 |
| 3 | 24.681 | 26.527 | 27.908 |
| 4 | 24.979 | 27.030 | 28.764 |
| 5 | 25.113 | 27.130 | 28.848 |
| 8 | 25.171 | 27.171 | 28.879 |
| 12 | 25.192 | 27.193 | 28.895 |
Across all three context sizes, sparsifying the first two layers causes the largest quality degradation, while the later layers are much less sensitive. Sparsifying layer 12 leaves the result unchanged because its final update is not used in subsequent cache-based rendering.
We further compare three sparsity schedules at 32 views, each with a nominal average block retention of 29.17%, alongside a Dense reference:
| Configuration | Per-layer block retention | PSNR (dB) | Build (s) |
|---|---|---|---|
| Dense | 100% throughout | 27.1930 | 1.499 |
| Early-layer protection (ours) | L1–2: 100%; L3–4: 50%; L5–12: 6.25% | 27.0365 | 0.793 |
| Uniform | 29.17% throughout | 15.8530 | 0.831 |
| Reversed | L1–8: 6.25%; L9–10: 50%; L11–12: 100% | 15.2480 | 0.794 |
Our early-layer protection schedule achieves substantially higher PSNR than uniform and reversed sparsity allocation at similar construction time. Together with the single-layer experiments, these results directly support our layer-wise sparsity design.
An improvement of 0.2 dB over BGSA is marginal at best; it is difficult to determine whether this is truly due to a superior algorithm or simply because more computation is preserved in the earlier layers.
Furthermore, regarding the image quality advantage of the decoder-only approach—does this stem from the architecture itself, or from the fact that the two methods employ different scene?
Thank you for pointing out that our experimental settings were not sufficiently clear. BGSA and our method use exactly the same per-layer retention schedule: layers 1–2 remain dense, layers 3–4 retain 50%, and the final eight layers retain 6.25% or 12.5%. Both methods use the same BVSM checkpoint and are evaluated on the same 32 scenes. Therefore, the 0.223/0.179 dB improvements at 32/64 views under the 6.25% setting are achieved with identical layer-wise retention budgets, rather than by preserving more computation in the early layers. We will clarify this in the table caption.
Regarding the quality advantage of decoder-only models, the controlled experiments in Table 1 use the same DL3DV training dataset and follow the Long-LRM evaluation protocol, with identical test scenes and evaluation settings. We also align the input settings, global batch size, and number of training steps. Since the compared configurations differ in both architecture and scene representation, the observed quality differences reflect their combined effects under controlled conditions, rather than differences in training data or evaluation settings.
To Reviewer Abqc 1/2
We thank the reviewer for the detailed feedback and constructive suggestions for strengthening our experimental evaluation.
1. Same-approach comparisons and baseline selection
We agree that comparisons on the same backbone are important for evaluating sparse-attention strategies. We have therefore adapted SpargeAttention and BGSA to BVSM and conducted direct comparisons on 32 scenes at 544×960:
At the 6.25% setting, our method achieves 0.223/0.179 dB higher PSNR than BGSA at 32/64 views, with 0.017/0.094 s additional construction time. At 64 views, our 6.25% configuration also outperforms BGSA at 12.5% in both PSNR (28.764 vs. 28.734 dB) and construction time (2.358 vs. 2.451 s). SpargeAttention (sim=0.6, CDF=0.98) achieves quality close to Dense but does not reduce construction time in this experiment.
We adapt BGSA's official method to BVSM without VGGT-specific sink tokens and use its native CUDA path for 64-view evaluation. SpargeAttention dynamically selects sparse blocks. For our method, the 6.25%/12.5% retention rate applies to the final eight layers; layers 1–2 remain dense and layers 3–4 retain 50%. These results use a paired Dense baseline from the same 32-scene evaluation, separate from the main 133-scene results. Layer indices 1–12 correspond to indices 0–11 in the paper.
The baselines in Table 2 span different construction and representation approaches, including attention-based LVSM, Mamba2–Transformer models with 3DGS/FeatureGS, and fast-weight models LaCT/tttLRM. Table 2 reports complete-system results under each method's training recipe. In contrast, Table 1 aligns the input settings, global batch size, and number of training steps to study representation–decoder trade-offs. The additional sparse-attention experiments further isolate the effect of sparse routing by using the same BVSM backbone.
2. Matched compute and measured training time
Thank you for raising the question of compute-matched training. We have added comparisons between P8 and P16 under matched estimated FLOPs and matched measured training time. Both models use width 768 and 12 layers (approximately 171–173M parameters), with 144×256 images, 32 context views, and 32 target views. Training uses eight A100 GPUs, a per-GPU batch size of four, and a global batch size of 32.
Following the paper's accounting, each multiply–add counts as two FLOPs, and training FLOPs are estimated as three times forward FLOPs. Total training compute is calculated by multiplying per-scene training FLOPs by the global batch size and the number of training steps.
The time-matched comparison uses measured cumulative training time. P8 takes approximately 8 h 20 min for 5,036 steps and 18 h 17 min for 11,607 steps; the latter matches the training time of the P16 reference.
At matched estimated FLOPs, P8 achieves 0.929 dB lower PSNR than P16, despite having a slightly lower training loss. At matched measured training time, P8 achieves 0.361 dB higher PSNR, alongside better SSIM and LPIPS, while using approximately 2.30× the estimated FLOPs. These results demonstrate that spatial granularity plays an important role in the quality–efficiency trade-off, with different outcomes under FLOPs-matched and time-matched training budgets. The complete training-loss curves for these experiments are shown in Fig. 9.
For Finding 1, the original study also includes configurations with nearly identical estimated model compute: FSA decoder-only requires 94.01 TFLOPs per sample and achieves 26.44 dB, while ColorGS requires 94.00 TFLOPs and achieves 21.13 dB. These estimates exclude Gaussian splatting, which we will specify in the paper.