acceptodds
Under review as a conference paper at ICLR 2027

FlashLVSM: A Constant-Time View-Synthesis Transformer

Abstract

Latent scene representations have emerged as a promising alternative for novel view synthesis, encoding multi-view observations into learned features without relying on predefined geometric primitives or rendering operations. Despite their promise, realizing them with modern neural architectures, such as transformers, often incurs substantial rendering cost. As the number and resolution of source views increase, every target token repeatedly attends to a growing set of source tokens through a deep neural renderer, causing the rendering cost to grow rapidly. We present FlashLVSM a constant-time view-synthesis model designed for scalable, highly efficient rendering from latent scene representations, which is carefully co-designed around four complementary principles, each addressing a distinct source of rendering cost. A richer, more 3D-expressive latent representation enables us to use a lightweight decoder without sacrificing rendering quality. Target-aware view selection restricts computation to source views geometrically relevant to the target, while target-aware token pruning further retrieves only the informative tokens within them, making rendering cost independent of the total number of input views. Finally, mixed-precision inference accelerates the remaining decoder computation. Together, FlashLVSM renders at 142 FPS with 32 input views on a single consumer-grade GPU, and maintains roughly 140 FPS as the number of views increases to 256 while preserving rendering quality. This corresponds to speedups of nearly 12x and 95x, respectively, with 204x fewer FLOPs and 25x lower peak memory at 256 views. Despite this reduction in computation, FlashLVSM remains competitive in rendering quality, matching or exceeding the baseline from 64 input views onward. These results establish efficient latent scene representations as a compelling alternative to conventional explicit representations for efficient novel view synthesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel view synthesis, Neural Rendering, Token Pruning, Quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rVB5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FlashLVSM accelerates novel view synthesis by combining a DA3-based encoder, a shallow renderer, geometry-guided view selection (based on camera poses), target-aware token pruning, and mixed-precision inference. After jointly encoding and caching all source views, it renders each target using a fixed source-token budget. The reported rendering throughput remains around 140 FPS across 32–256 input views.

Critical strengths

The rendering speedups are substantial and supported by FPS, FLOPs, and memory measurements. At 64 views, the acceleration components increase throughput from 13 to 143 FPS, with a 0.60 dB PSNR reduction relative to the same base model. Component ablations, random-pruning controls, and quantization sensitivity analyses provide useful evidence for the design choices.

Critical weaknesses
  1. The quality and efficiency advantage over competing efficient methods remains unclear. The main experiments focus on dense LagerNVS, while LaCT (already cited in the paper) also supports rendering costs independent of the input-view count after scene encoding. Given that FlashLVSM primarily combines established components, a comparison with this alternative would better establish its practical contribution and clarify the trade-off between retaining source tokens and compressing them into fast weights.

  2. Another weakness is that FlashLVSM primarily combines established components. This may make the contribution feel incremental.

Questions for authors

See weakness 1. Could you compare FlashLVSM with LaCT under matched evaluation settings, reporting rendering quality, encoding cost, scene-representation memory, and rendering throughput?

3 comments

1. Comparison with LaCT

Grouped encoding. As explained in General Response C, we observe that the many-view degradation in the paper comes from encoding 32–256 views jointly, far outside the 2–6-view training setting. We therefore partition the source cameras into equal-sized groups of 4 or 6 views by clustering on camera position (with 6-view groups, a few views are repeated to fill the last group), normalize each group exactly as a training sample, and encode it independently; the cache is the same size as with joint encoding for 4-view groups and 0.8–12.5% larger for 6-view groups. Each target reads the group nearest to it — a group-level counterpart of our target-aware view selection — followed by the paper's pruning and FP8 (PQ), i.e., 5,184 or 7,776 source tokens (2,592 or 3,888 after pruning) instead of the paper's 10,368 (5,184).

Table 1. DL3DV 256$\times$256, feed-forward (PSNR↑ / SSIM↑ / LPIPS↓).

ViewsLaCTOurs, 4-view groupsOurs, 4-view + PQOurs, 6-view groupsOurs, 6-view + PQ
3227.74 / 0.866 / 0.11627.04 / 0.863 / 0.10426.58 / 0.853 / 0.11127.60 / 0.874 / 0.09527.21 / 0.866 / 0.100
6428.91 / 0.884 / 0.10528.74 / 0.896 / 0.07928.11 / 0.885 / 0.08729.07 / 0.902 / 0.07328.59 / 0.894 / 0.079
12829.08 / 0.881 / 0.10730.18 / 0.918 / 0.06329.36 / 0.906 / 0.07130.40 / 0.921 / 0.05929.78 / 0.913 / 0.065
25629.21 / 0.882 / 0.10631.10 / 0.931 / 0.05330.11 / 0.919 / 0.06231.19 / 0.933 / 0.05130.42 / 0.923 / 0.058

LaCT is strongest near its training view count of 32: at 32 views it is 0.5 dB higher in PSNR than ours with 6-view groups, and at 64 views the gap narrows to 0.3 dB. From 128 views on, ours is higher in all three metrics: by 0.7/1.2 dB in PSNR at 128/256 views with PQ, and 1.3/2.0 dB without. LPIPS is lower for ours at every view count, and SSIM from 64 views on. As the number of views grows, LaCT saturates (28.91 → 29.21 dB from 64 to 256 views), consistent with a fixed-size memory having to hold more views; ours keeps improving (28.59 → 30.42 dB with PQ), because every token is retained and each target reads its nearest group at the same cost.

Table 2. Cost at 256$\times$256 (LaCT / ours, 4-view groups / ours, 6-view groups). Ours is measured with PQ and includes group selection cost (0.4 ms); peak memory includes model weights (1.3–1.5 GB LaCT, 5.0 GB ours).

ViewsEncoding (s)Peak encoding memory (GB)Rendering (ms/frame)Cached scene state (MB, FP32)
320.30 / 0.89 / 0.963.4 / 5.9 / 6.29.6 / 5.6 / 6.8324 / 122 / 137
640.58 / 1.78 / 1.774.9 / 6.2 / 6.49.7 / 5.6 / 6.8324 / 243 / 251
1281.14 / 3.64 / 3.577.9 / 6.5 / 6.99.7 / 5.6 / 6.8324 / 486 / 501
2562.27 / 7.20 / 6.9913.8 / 7.7 / 7.99.7 / 5.6 / 6.8324 / 972 / 980

Encoding the scene is about 3$\times$ slower than LaCT's but linear in NN, and its peak memory is lower than LaCT's from 128 views on, since LaCT's fast-weight update processes all context tokens at once. Rendering is 1.4–1.7$\times$ faster than LaCT and independent of NN (5.6 ms, about 179 FPS, with 4-view groups, against 9.7 ms, about 103 FPS for LaCT); the cache grows with NN and exceeds LaCT's fixed 324 MB beyond about 85 views.

Summary. LaCT writes the context into fixed-size fast weights and renders from a static network; we keep the context as tokens and let each target read a fixed budget of them. The two therefore differ in three trade-offs:

  • What is stored. LaCT keeps a lossy summary whose size is independent of the number of views (324 MB). We keep every encoded token, lossless with respect to the encoder but growing with NN.
  • What scene preparation costs. Both are linear in NN, and LaCT is about 3$\times$ faster. However, LaCT's fast-weight update processes all context tokens at once, so excluding model weights, its working memory grows with NN (2.1 → 12.3 GB from 32 to 256 views), while ours stays at that of one group (0.9–2.9 GB).
  • What rendering costs and delivers. LaCT runs 24 blocks over the target tokens; we run 6 blocks that cross-attend to a fixed set of source tokens selected for each target, 1.4–1.7$\times$ faster. In quality, LaCT is strongest near its training view count, whereas ours keeps improving as views are added.

Neither design dominates; they are complementary. Combining LaCT-style fast weights for the global scene with retained tokens for target-dependent detail is a promising direction, and we thank the reviewer for suggesting the comparison with LaCT.

2. Contribution

We agree that view selection, token pruning, and FP8 inference are established techniques, and claim no novelty for them individually. Our contribution is in making them work together. First, the combination is not free: applied to LagerNVS, the same combination costs 4.00/3.69/2.98/2.31 dB at 32/64/128/256 views (Table 2 of the paper, +PQ), versus 0.78/0.60/0.47/0.42 dB for ours. Second, what makes it work is a property of the representation: a target can be rendered from the tokens of a few selected views. When 64 views are encoded and view selection passes only 8 to the renderer, the VGGT-based LagerNVS drops 3.55 dB while ours drops 0.24 dB (Table 14). Third, target-aware retrieval from an uncompressed cache is a distinct design point from compressive representations (scene tokens, CLiFT, LaCT fast weights), which reduce the context to a fixed summary before any target is known, and from query-agnostic token reduction (ToMe, FastVGGT), which reduces once for all targets; under a matched token budget, query-agnostic strategies are 0.9–2.7 dB below ours (Table R1-1 in our response to Reviewer aREE). We will revise the contribution statement to present these findings rather than list the techniques.

Thank you for the clarification and additional experiments. My concerns have been addressed, and I will maintain my rating of 3.

Reviewer_UqDzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes a series of designs to improve the efficiency of novel view synthesis based on latent scene representations. Its key design combines geometry-guided view selection and target-conditioned token pruning over jointly encoded, cached source features, together with a shallow decoder and mixed-precision execution. The framework is evaluated through comparative experiments and component ablations, examining its rendering efficiency and reconstruction quality.

Critical strengths
  • The overall technical designs are well motivated and described, along with clear ablation studies to verify their effectiveness.

  • The decoding acceleration is significant, suggesting certain practical value.

Critical weaknesses
  • To my understanding, scoring all source cameras still depends on the number of views, the cached representation grows with that number, and joint encoding remains input-dependent, which does not establish constant end-to-end latency or storage. The claim of "Constant-time" should consistently refer to the bounded decoding stage.

  • The comparison with LagerNVS is a bit mixed. The method uses DA3 + 6-layer decoder while the baseline uses VGGT + 12-layer decoder. The improvement could also attribute to the difference of encoder.

Questions for authors

See the weakness part above.

2 comments

1. Constant-time claim

We agree with the reviewer that view selection, the cached representation, and joint encoding all depend on NN, so our original claim does not establish constant end-to-end latency or storage. We will use "constant-time" only for the per-frame rendering stage after one-time scene encoding. Rendering time stays at 7.1 ms from 32 to 256 views while encoding takes 1.44–43.90 s and the cache 122–972 MB; view scoring, the only per-frame operation that depends on NN, stays at 0.64–0.66 ms up to 16,384 candidate views (General Response A, Table C-1). With the grouped encoding of General Response C, scene preparation becomes linear in NN.

2. Encoder contribution

Part of the improvement does come from the encoder, which is our intended design choice: we claim no novelty for DA3 itself, and the claim of Sec. 3.2 is that a richer one-time scene representation lets the per-frame renderer be lighter. To separate the encoder from renderer depth, we trained both encoders with the same six-block renderer and a matched 65k-iteration schedule: DA3-giant improves over VGGT-1B by 2.02/3.07/3.49 dB at 2/4/6 views (General Response B, Table C-2). The matched 12-block variants are being trained to quantify the depth side, and the full VGGT/DA3 ×\times 6/12-block grid will be added in the revised paper. The stronger encoder also appears to be what makes the acceleration possible: when 64 views are encoded and only 8 are passed to the renderer, our DA3-based model loses 0.24 dB whereas the VGGT-based LagerNVS loses 3.55 dB (Table 14); the same pruning and quantization cost LagerNVS up to 4.0 dB and ours up to 0.8 dB (Table 2). Our contribution is identifying this representation–renderer trade-off and the pipeline that exploits it; the encoder is what makes this pipeline possible, and we will explicitly state it in the revised paper.

I thank the authors for the rebuttal, where my concerns have been properly addressed. Yet, I would suggest reserving “constant-time” for decoding with a fixed budget of selected views and tokens.

Overall, I would like to keep my original score.

Reviewer_aREEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FlashLVSM, an efficient transformer-based framework for novel view synthesis from latent scene representations. To address the high rendering cost associated with increasing numbers of source views, the authors combine a stronger 3D foundation-model encoder with a shallower renderer, geometry-guided view selection, target-aware token pruning, and mixed-precision FP8 inference.

The proposed framework achieves approximately 140 FPS across 32–256 source views, with substantial improvements in decoder throughput, FLOPs, and memory consumption compared with LagerNVS. The results demonstrate an effective approach to accelerating neural rendering under a fixed source-token budget.

Critical strengths
  1. Significant efficiency improvements.

The reported speedups are impressive. At 256 source views, FlashLVSM achieves approximately 141 FPS compared with 1.5 FPS for LagerNVS, while maintaining competitive reconstruction quality. The reported reductions in renderer FLOPs and peak memory are also substantial.

  1. Complementary and effective optimization strategies.

The combination of view selection, token pruning, decoder simplification, and mixed-precision inference is well motivated. The component-wise ablations provide useful evidence of their contributions to rendering efficiency.

  1. Useful analysis of implementation choices.

The experiments investigating pruning locations, token retention ratios, and FP8 quantization sensitivity provide valuable practical insights into efficient neural renderer design.

Critical weaknesses
  1. The constant-time claim requires a more precise definition.

The fixed source-token budget makes the neural decoder's computational cost approximately independent of the total number of source views. However, the complete rendering pipeline still involves operations that depend on the input-view count.

In particular, target-dependent view selection evaluates candidate source cameras, while the scene-level normalization statistic involves pairwise camera distances. Joint source encoding and feature caching also scale with the number of input views.

Therefore, the current evidence supports approximately constant decoder cost under fixed rendering conditions, rather than strictly constant end-to-end view-synthesis complexity.

The distinction is important because the headline speedups are based primarily on renderer-only measurements.

  1. The effect of the stronger encoder is not sufficiently disentangled from the proposed architectural improvements.

The primary comparison changes both the encoder and the decoder depth: LagerNVS uses VGGT with a 12-layer renderer, whereas FlashLVSM uses DA3 with a 6-layer renderer.

Consequently, it is difficult to determine how much of the observed quality improvement comes from the stronger DA3 representation rather than the proposed architectural design.

A controlled comparison using VGGT and DA3 with both 6-layer and 12-layer decoders would provide stronger evidence for the claimed representation–renderer trade-off.

  1. The experimental comparison with existing efficient rendering methods remains incomplete.

The main efficiency comparison focuses on LagerNVS. Although the appendix includes comparisons with several 3D Gaussian Splatting methods, their rendering throughput and memory costs are not reported under a unified protocol.

Moreover, the paper lacks direct quantitative comparisons of the quality–efficiency trade-off against other efficient neural rendering approaches, such as CLiFT and Efficient-LVSM.

These comparisons are important for supporting the broader claim that latent scene representations can compete with explicit representations in rendering efficiency.

  1. Limited validation of the proposed token selection strategy and its robustness.

The paper demonstrates that attention-based token selection outperforms random pruning, but does not sufficiently compare it against alternative relevance estimation or geometry-based selection strategies under matched token budgets.

Furthermore, the fixed eight-view selection budget assumes adequate coverage from nearby source cameras. The experiments do not thoroughly establish robustness under sparse coverage, wide baselines, or large viewpoint changes.

The evaluation also relies predominantly on DL3DV, leaving cross-dataset generalization less clear.

Finally, the reconstruction-quality trade-off deserves more discussion. At 512×512 resolution, the proposed method produces worse LPIPS than LagerNVS across the reported view-count settings, despite its strong throughput improvements.

Questions for authors
  1. Can the authors clarify the precise computational complexity of the complete pipeline, including source encoding, view ranking, token selection, and decoding? How do end-to-end latency and peak memory scale from 32 to 256 source views?

  2. Can the authors provide controlled encoder–decoder ablations, particularly VGGT with a 6-layer renderer and DA3 with a 12-layer renderer, to isolate the benefits of the stronger representation and reduced decoder depth?

  3. How does the proposed target-aware token pruning compare with alternative selection strategies, such as geometry-only selection or feature-similarity-based ranking, under identical source-token budgets?

  4. Can the authors provide a more comprehensive quality–speed comparison against existing efficient neural rendering approaches, including explicit-representation methods where comparable evaluation settings are available?

3 comments

1. Constant-time scope (W1, Q1)

We agree that the claim needs a precise definition and will scope the claim to per-frame rendering after one-time scene encoding: rendering time stays at 7.1 ms from 32 to 256 views while encoding (1.44–43.90 s), peak encoding memory (9.8–40.6 GB), and the token cache (122–972 MB) grow with NN; view selection alone stays at 0.64–0.66 ms up to 16,384 candidates (General Response A, Table C-1).

2. Encoder contribution (W2, Q2)

Part of the improvement does come from the encoder, which is our intended design choice: we claim no novelty for DA3 itself, and the claim of Sec. 3.2 is that a richer one-time representation lets the per-frame renderer be lighter. The experiment separates the encoder from renderer depth: with the same six-block renderer and a matched 65k-iteration schedule, DA3-giant improves over VGGT-1B by 2.02/3.07/3.49 dB at 2/4/6 views (General Response B, Table C-2). The 12-block variants are being trained under the same schedule and the full VGGT/DA3 ×\times 6/12-block grid will be in the revised paper.

3. Token pruning alternatives (W4, Q3)

We compared token-reduction strategies with everything else fixed. All methods use BF16 without FP8 and are evaluated on DL3DV dataset at 256$\times$256; each target receives the same eight selected views, and all strategies produce 5,184 tokens, 50% of the original 10,368. The geometry baseline unprojects tokens with DA3 depth and keeps those whose 3D points fall in the target frustum, ranked by distance to the target camera; ToMe-style merging is a merging rather than pruning baseline.

Table R1-1. Token-reduction strategies under a matched budget (PSNR).

Token-reduction strategy32 views64 views128 views
No pruning (10,368 tokens)26.2326.4925.68
Random23.9724.4724.30
Spatial uniform (per-view grid)24.6525.0224.62
Feature diversity (K-means, nearest token)24.1624.6324.43
Geometry (DA3 depth)25.6025.7124.78
ToMe-style merging23.3623.6323.68
Ours26.0726.3325.54

Target-agnostic strategies lose 1.1–2.9 dB against the unpruned model, target-aware geometry-based selection 0.6–0.9 dB, and ours lose 0.14–0.16 dB. Selecting tokens for each target recovers part of the loss; selecting them by the renderer's own attention, rather than by geometry, recovers a further 0.5–0.8 dB. We will add this to the revised paper.

4. Robustness to sparse coverage and large viewpoint changes (W4)

To test the fixed eight-view budget under sparser capture, we subsampled the paper's 128-view DL3DV source indices with stride 2/4/8/16 to obtain 64/32/16/8 source views, keeping the targets fixed, and compare the full model (PQ) with the same model rendering from all source views (w/o PQ), so the difference isolates the cost of the budget. Viewpoint change is measured per target as the smallest angle between its optical axis and that of any source view (nearest-view angle). Table R1-2 averages this angle over all targets for each source-view count, and Table R1-3 groups individual targets by their angle.

Table R1-2. Eight-view selection under sparser coverage.

Source viewsNearest-view angle (mean over targets)w/o PQOurs (PQ)Δ\Delta
810.5°23.3923.20−0.19
166.2°25.4624.94−0.52
323.7°26.8526.04−0.81
642.4°26.9226.33−0.59

Table R1-3. PQ cost binned by the nearest-view angle of each target (24,768 targets, 8–64 views pooled).

Nearest-view angle0–2°2–5°5–10°10–20°20–40°> 40°
Targets8,3597,2855,0252,9781,012109
w/o PQ → Ours, PSNR27.73 → 26.9926.08 → 25.5624.32 → 23.9322.41 → 22.1220.05 → 19.8918.80 → 18.72
Δ\Delta−0.73−0.52−0.39−0.29−0.16−0.08

Absolute quality drops for both models as coverage thins, but the fixed budget does not amplify the drop: the PQ cost is largest in the densest setting (−0.81 dB at 32 views, −0.73 dB for targets within 2° of a source) and smallest in the sparsest (−0.19 dB at 8 views, −0.08 dB beyond 40°), and the same trend holds at each fixed view count. The subsampled 64- and 32-view sets give the same PQ cost as the paper's original indices, so the subsampling only adds the 16- and 8-view settings without changing the result. The view-selection term is the largest component (−0.23/−0.49/−0.23 dB at 64/32/16 views); pruning and FP8 cost ≤0.16 and ≤0.22 dB regardless of density. Sparse and wide-baseline scenes remain harder for the dense model as well as for ours, but the budget is not what degrades them. We will add this analysis to the revised paper.

5. Comparison with efficient rendering methods (W3, Q4)

(a) Efficient neural renderers. On DL3DV at 256$\times$256 (times in this paragraph are measured on an RTX 5090), CLiFT renders faster from its compressed tokens (3.19/4.25/6.36 ms at 32/64/128 views) but reaches only 17.50/17.65/17.82 dB, against our 25.90/26.13/25.35 dB at 7.6 ms. Efficient-LVSM is reported for runtime only, as its released checkpoint is not trained on DL3DV; its rendering time grows with the view count (10.56/17.92/30.63 ms) and is above ours at every view count.

(b) Explicit representations. For the 3DGS baselines of Table 13 (DL3DV, YoNoSplat protocol) we measured encoding, rendering, peak memory, and representation size; quality is from Table 13.

Table R1-4. Feed-forward 3DGS vs. ours, DL3DV 224$\times$224. For ours, the scene representation is all cached tokens.

MethodViewsEncoding (ms)Rendering (ms/frame)Peak enc. / render (GB)Scene representationQuality
DepthSplat6583.872.3 / 0.760 MB (393k Gaussians)23.42 / 0.797 / 0.136
YoNoSplat6563.305.8 / 3.826 MB (301k)24.72 / 0.817 / 0.139
Ours61575.26.0 / 5.423 MB (7,776 tokens)26.93 / 0.862 / 0.081
DepthSplat241589.637.5 / 1.3240 MB (1.5M)20.09 / 0.690 / 0.240
YoNoSplat241767.176.9 / 4.1106 MB (1.2M)22.66 / 0.758 / 0.192
Ours249436.58.5 / 5.691 MB (31,104 tokens)26.19 / 0.839 / 0.089

Rasterization is faster at 6 views (3.3–3.9 ms vs. our 5.2 ms); at 24 views the Gaussian count has grown with the views while our budget has not (7.2–9.6 ms vs. 6.5 ms). Our advantage is quality, 2–6 dB higher in PSNR with roughly half the LPIPS; our costs are 2.7–6$\times$ slower encoding and about 5 GB of model weights in peak memory. We will add these comparisons to the revised paper.

6. Cross-dataset generalization (W4)

We evaluated the DL3DV-trained 256$\times$256 models zero-shot on Mip-NeRF 360 (9 scenes; 7 at 128 views) and Tanks&Temples (2 scenes), with every 8th frame as target.

Table R1-5. Zero-shot results on Mip-NeRF 360 and Tanks&Temples (PSNR↑ / SSIM↑ / LPIPS↓; submitted pipeline).

DatasetViewsLagerNVS (VGGT + 12-block)Ours
Mip-NeRF 3603221.24 / 0.597 / 0.21422.05 / 0.608 / 0.219
Mip-NeRF 3606421.77 / 0.607 / 0.19822.77 / 0.621 / 0.200
Mip-NeRF 36012822.34 / 0.618 / 0.17823.13 / 0.644 / 0.173
Tanks&Temples3220.27 / 0.679 / 0.18920.29 / 0.689 / 0.195
Tanks&Temples6420.58 / 0.686 / 0.17121.12 / 0.713 / 0.167
Tanks&Temples12820.54 / 0.674 / 0.17521.45 / 0.717 / 0.158

Ours is ahead of LagerNVS in PSNR and SSIM in every setting (+0.81/+1.00/+0.79 dB on Mip-NeRF 360, +0.02/+0.54/+0.91 dB on Tanks&Temples), with LPIPS within 0.006 at 32–64 views and lower at 128, while rendering at about 7 ms regardless of the view count.

7. LPIPS at 512$\times$512 (W4)

We swept the keep ratio pp and removed pruning and FP8 one at a time (Table R1-6). The LPIPS gap to LagerNVS is 0.024/0.014/0.005 at 32/64/128 views and narrows with more views, since from 64 views on our eight-view renderer is ahead of LagerNVS before pruning and FP8. At 64 views, removing pruning closes the gap (0.202 vs. 0.202); at 128 views, pp = 0.75 already puts ours below LagerNVS (0.221 vs. 0.229). At 32 views the gap remains even without pruning and FP8 (all-view model 0.007, view selection 0.003, FP8 0.005, pruning 0.009), in exchange for a 12$\times$ lower rendering time (7.1 vs. 87.0 ms).

Table R1-6. Keep-ratio sweep. Rendering time is ms per frame at 32 / 64 / 128 views.

Model32 views64 views128 viewsRendering time (ms)
LagerNVS (all views)24.30 / 0.755 / 0.19523.59 / 0.720 / 0.20222.51 / 0.673 / 0.22987.0 / 176.6 / 353.9
Ours w/o PQ (all views, BF16)24.70 / 0.764 / 0.20224.76 / 0.755 / 0.20023.98 / 0.717 / 0.22039.3 / 80.2 / 158.2
Ours, VS-8, BF16 (no pruning, no FP8)24.33 / 0.757 / 0.20524.50 / 0.755 / 0.19723.76 / 0.722 / 0.21010.6
Ours, VS-8 + FP8, pp = 124.20 / 0.751 / 0.21024.36 / 0.749 / 0.20223.65 / 0.717 / 0.2158.7
Ours, pp = 0.7524.12 / 0.748 / 0.21324.21 / 0.743 / 0.20723.49 / 0.710 / 0.2218.2
Ours, pp = 0.5 (paper)23.92 / 0.741 / 0.21923.93 / 0.732 / 0.21623.17 / 0.699 / 0.2347.1

Since pp is an inference-time parameter, pp = 0.75 recovers 0.006/0.009/0.013 LPIPS for 1.1 ms per frame, and removing pruning recovers 0.009/0.014/0.019 for 1.6 ms, still 10$\times$ faster than LagerNVS at 32 views.

Thanks for the additional result and analysis. I have no more questions and will keep my rating.

Comments on the paper

1 comment

General Response

We thank the reviewers for their constructive feedback. Two points are shared by Reviewers aREE and UqDz — the scope of the "constant-time" claim and the entanglement of encoder choice with renderer depth — and one finding made while preparing the LaCT comparison for Reviewer rVB5 changes how we prepare scenes; all three are collected here. Unless noted otherwise, all runtimes were re-measured on a single RTX PRO 6000 (Blackwell).

A. Scope of the constant-time claim (aREE W1/Q1, UqDz W1)

We agree that the claim needs a precise definition, and the reviewers' argument is correct: the fixed token budget makes the per-frame rendering stage independent of the number of source views NN, whereas encoding, the token cache, and view selection are input-dependent, and the headline 95$\times$/204$\times$/25$\times$ figures are renderer-stage speedups. We will revise the paper so that "constant-time" refers to per-frame rendering cost after one-time scene encoding and report the input-dependent costs in the paper. Of the stages, encoding is one DA3 pass over all NN views, quadratic in NN; the scene-level normalization σ\sigma in Eq. 2 is O(N2)O(N^2) but computed once per scene; per-frame view scoring is O(N)O(N) with a small constant; pruning and decoding run on fixed budgets of 10,368 and 5,184 tokens.

Table C-1. End-to-end cost with joint encoding, 256$\times$256.

32 views64 views128 views256 views
Encoding (s) ↓1.444.0512.6843.90
Peak encoding memory (GB, incl. weights) ↓9.814.223.240.6
Cached source tokens (MB, FP32) ↓122243486972
View selection (ms/frame) ↓0.660.640.640.64
Token pruning (ms/frame) ↓0.520.520.520.52
Rendering time per frame (ms) ↓7.17.27.17.1
Peak rendering memory (GB, incl. weights) ↓5.75.96.37.1

Encoding and storage scale with NN, whereas per-frame rendering time does not grow over the measured range. View selection is the only per-frame operation that depends on NN, and it stays at 0.64–0.66 ms from 32 to 16,384 candidate views. Encoding is a one-time cost per scene: for KK rendered frames at 64 views the end-to-end time is 4.29 s + 7.0 ms ⋅\cdot KK for ours and 5.08 s + 177 ms ⋅\cdot KK for LagerNVS (Appendix B.6). The grouped encoding described in Sec. C below replaces the first two rows with costs linear in NN (7.0 s and 7.9 GB at 256 views).

B. Encoder choice vs. renderer depth (aREE W2/Q2, UqDz W2)

We trained VGGT and DA3 with the same six-block renderer and a matched training schedule (65k iterations, batch size 16, 256$\times$256), changing only the encoder:

Table C-2. Encoder choice at a fixed six-block renderer (PSNR).

Encoder + renderer2 views4 views6 views
VGGT-1B + 6-block17.1319.0719.76
DA3-giant + 6-block19.1522.1423.25

DA3 improves PSNR by 2.02/3.07/3.49 dB at fixed renderer depth; these reduced-budget runs are compared only with each other, and their absolute values are below the paper's because of the shorter schedule. To isolate the renderer-depth trade-off we are training the 12-block variants under the same schedule and will report the full VGGT/DA3 ×\times 6/12-block grid in the revised paper.

C. A correction to scene encoding (all reviewers)

While preparing the LaCT comparison (rVB5) we found the cause of our many-view degradation. Our encoder and renderer are trained with 2–6 context views, each sample normalized to its own camera extent. Encoding 32–256 views jointly, as in the submitted paper, gives the encoder and renderer far more views than they saw in training and a scene scale normalized over the whole trajectory. Encoding the sources in groups of 4 or 6 views — clustered by camera position and each normalized exactly as a training sample, with the cache the same size as before for 4-view groups and 0.8–12.5% larger for 6-view groups — puts the encoder and renderer back in their training regime and removes the degradation without any test-time optimization: with 6-view groups and the paper's pruning and FP8, 27.21/28.59/29.78/30.42 dB at 32/64/128/256 views, above the submitted pipeline even after test-time optimization (26.85/28.10/28.98/29.78 dB, Table 6). Test-time optimization is therefore no longer needed to handle large contexts. Scene encoding becomes linear in NN (7.0 s and 7.9 GB at 256 views, against 43.9 s and 40.6 GB). The renderer and its weights are unchanged; since each target now reads one group of 4 or 6 views instead of 8 selected views, rendering with PQ takes 5.6/6.8 ms per frame instead of 7.1 ms. We will add grouped encoding to the revised paper alongside the submitted joint encoding, report both, and revise the explanation in Sec. 3.3 accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.