- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The "encode once, allocate on demand" idea is genuinely good and the execution is clean — consistent gains across three frozen backbones and two datasets convince me the mechanism works, and I like that it needs no retraining of the backbones. My remaining doubts are minor and mostly framing: the headline speedups are Effective FPS and the end-to-end advantage only appears with many target views, and robustness to pose noise is untested. None of this undermines the core idea, which I find effective for dense-context NVS. Paper Summary TACC is a target-aware compression module for feed-forward latent novel view synthesis under dense source-view contexts. Each source view is organized once into a progressive per-view Carrier hierarchy (a binary spatial refinement tree with learned regional Carriers). At target time, a training-free allocation stage scores source views by source–target geometric relevance, grants the top-ranked view a full quota, and distributes the remaining budget across support views using an entropy-adaptive shared floor plus relevance-weighted largest-remainder rounding; the corresponding frontier readout yields a compact, target-conditioned token set that feeds the unchanged (frozen) rendering pathway. Only the Carrier module is trained, via teacher–student distillation. Across LagerNVS, LVSM Decoder-Only, and CLiFT on DL3DV and RealEstate10K, TACC outperforms matched-budget baselines (ToMe, DivPrune, ZPressor) in reconstruction quality while maintaining near-constant target-time throughput (up to 9.23×/64× Effective FPS over Full Context in dense settings), and cuts peak GPU memory for LVSM at 120 views from 12.77 to 4.22 GiB (−66.96%).
Critical strengths
An elegant, reusable design. The progressive Carrier hierarchy decouples one-time source encoding from target-time cost, supports exact integer per-view quota allocation across views, and is backbone-agnostic, demonstrated by consistent gains across three frozen backbones with a single Carrier checkpoint. The frontier/refinement-order construction (area, then marginal scatter gain) and the capacity-aware largest-remainder rounding are clean, well-specified technical choices.
Consistently positive, well-controlled empirical results. TACC beats matched-budget compression baselines by a large margin (e.g., +1.2–2.3 dB over ZPressor on LagerNVS) and even exceeds the frozen Full-Context checkpoint's quality in dense settings, while keeping Effective FPS nearly constant as the source count grows. The ablations (w/o Carrier, Uniform Alloc., Anchor-only Carrier, Top-A Full Views) support the contribution of both learned regional Carriers and target-conditioned allocation, and the results hold across two datasets including cross-dataset generalization without fine-tuning.
Critical weaknesses
The headline speedup numbers are Effective FPS, which exclude one-time source encoding and Carrier construction. The amortized E2E comparison (Fig. 4c) shows TACC is slower with 10 targets and only overtakes Full Context at ~40+ targets on LagerNVS.
The allocation stage is training-free and driven by simple geometric descriptors (ray direction / camera center / Plücker moments); robustness to noisy or inaccurate poses and to failure of the geometric-relevance assumption is not analyzed. These are fixable concerns and do not detract from the core contribution.
How about compare other accelerated methods with TACC in Fig 4?
Questions for authors
What is the break-even number of target views (and how does it scale with source-view count) for end-to-end wall-clock advantage including Carrier construction? A break-even plot over (source views × target views) would substantially improve the paper's practical impact.
How sensitive is the allocation to pose noise or outliers? Are there regimes (e.g., targets with similar relevance to many sources) where relevance-based allocation underperforms uniform allocation?
Does TACC's quality edge over Full Context mainly reflect the frozen backbone degrading on dense contexts? Would the advantage persist with a backbone fine-tuned for dense inputs, or is the contribution better framed as "restoring/improving dense-context quality at low cost"?
5 comments
Response to Reviewer 3 (Part 2/3): Full Context Quality and Dense-Context Adaptation
Q3. Full Context Quality and Dense-Context Adaptation
We agree that the original Full Context backbone's limited adaptation to dense source views may partly explain TACC's large reconstruction-quality advantage. While dense-context fine-tuning substantially improves the Full Context backbone, TACC still achieves higher reconstruction quality and substantially greater target-time rendering throughput across the evaluated settings. We therefore consider preserving or improving dense-context reconstruction quality at lower rendering cost a more precise characterization of our contribution.
To examine this effect, we fully fine-tuned LVSM Decoder-Only on RE10K for dense source-view inputs, using approximately 170.8M trainable parameters and strictly disjoint training and evaluation scenes. In contrast, TACC retains the original frozen backbone and trains only 302,785 Carrier parameters (approximately 0.18% of the backbone).
| Source Views | Fine-tuned LVSM PSNR ↑ | Fine-tuned LVSM Eff. FPS ↑ | TACC PSNR ↑ | TACC Eff. FPS ↑ |
|---|---|---|---|---|
| 80 | 29.5903 | 0.3377 | 30.2851 | 10.28 |
| 100 | 29.4271 | 0.2211 | 30.1978 | 10.15 |
| 120 | 29.2294 | 0.1547 | 30.1441 | 10.03 |
Effective FPS measures target-time rendering throughput and excludes one-time source encoding and Carrier construction.
Dense-context fine-tuning narrows the quality gap observed with the original pretrained LVSM checkpoint, confirming that backbone adaptation affects the Full Context comparison. Nevertheless, TACC achieves 0.69–0.91 dB higher PSNR than the fine-tuned backbone across 80–120 source views, while substantially reducing target-time rendering costs without retraining the original backbone.
These findings show that TACC's benefits are not solely attributable to the original Full Context checkpoint's degradation. We have clarified the interpretation of the Full Context comparisons in Section 4.2 and added the dense-context fine-tuning results to Appendix C.1 (Table 9).
Manuscript location: Section 4.2, p. 8, lines 422–428; Appendix C.1, pp. 21–22, lines 1131–1164 (Table 9).
Response to Reviewer 3 (Part 3/3): Reported Speedups and Acceleration Methods
W1. Scope of Reported Speedups
We agree that Effective FPS should be distinguished from end-to-end (E2E) acceleration. While Effective FPS excludes one-time source encoding and Carrier construction, TACC follows an encode-once, render-many design: a single source encoding and Carrier construction can support arbitrarily many subsequent target-view renderings without repeating these operations. Effective FPS therefore provides a meaningful measure of sustained rendering throughput, whereas E2E FPS accounts for one-time costs when rendering a finite number of targets.
Importantly, our additional measurements show that TACC approaches E2E parity at 16 target views and achieves a 1.0732× speedup at 20 targets, substantially earlier than the approximately 40 targets suggested by the original figure. To quantify this behavior, we conducted additional E2E experiments using LagerNVS on the official 140-scene DL3DV evaluation set, with 80 source views and . All target counts use prefixes of the same ordered target-view sequence. E2E timing includes source encoding, Carrier construction, target-conditioned allocation and readout, and rendering, excluding disk loading and image decoding.
| Target Views | Full Context E2E FPS ↑ | TACC E2E FPS ↑ | E2E Speedup |
|---|---|---|---|
| 10 | 0.8862 | 0.7927 | 0.8946× |
| 16 | 1.2514 | 1.2523 | 1.0008× |
| 20 | 1.4137 | 1.5172 | 1.0732× |
| 32 | 1.9741 | 2.4155 | 1.2236× |
| 40 | 2.1798 | 2.9512 | 1.3539× |
| 60 | 2.6666 | 4.2215 | 1.5831× |
| 80 | 2.9619 | 5.2206 | 1.7626× |
| 100 | 3.1750 | 6.2883 | 1.9806× |
TACC remains slower at 10 targets due to the one-time Carrier construction overhead, but its E2E speedup increases to 1.2236× at 32 targets and 1.9806× at 100 targets. For the same 80-source-view setting, TACC achieves 27.95 Effective FPS versus 4.40 for Full Context, demonstrating its sustained target-time efficiency. We now report both metrics and clarify their measurement scopes in Sections 4.1–4.2 and Figure 5(c).
Manuscript location: Section 4.1, p. 6, lines 296–300; Section 4.2, p. 8, Figure 5(c) (lines 398–412), and p. 9, lines 432–440.
W3. Comparison with Other Acceleration Methods
We have added ToMe, DivPrune, and ZPressor to the efficiency comparisons in Figures 5(a) and 5(b), allowing direct comparisons of peak GPU memory and per-target rendering latency under the same decoder-visible token budget. We thank the reviewer for this suggestion.
Specifically, all methods are evaluated using LVSM Decoder-Only on RealEstate10K, with source-view counts ranging from 15 to 120 and a fixed budget. Together with the reconstruction-quality results in Table 2, these measurements provide a more complete comparison of reconstruction quality, memory consumption, and rendering efficiency across the compression methods.
Manuscript location: Section 4.2, p. 8, lines 398–412 (Figures 5(a)–(b)).
Response to Reviewer 3 (Part 1/3): E2E Break-even and Allocation Robustness
The number of target views required to approach E2E parity generally decreases as the source-view count increases, from approximately 48 targets at 15 source views to 16 targets at 80–120 source views. This trend arises because Full Context incurs increasing per-target rendering costs with larger source contexts, while TACC maintains a fixed decoder-visible token budget.
To quantify this behavior, we conducted additional E2E measurements using LagerNVS on the official 140-scene DL3DV evaluation set, with A=10 on a single NVIDIA A800 GPU. The timing includes source encoding, one-time Carrier construction, target-conditioned allocation and readout, and rendering of all target views.
As source-view counts increase, the growing per-target cost of Full Context allows TACC's rendering savings to amortize the one-time Carrier construction overhead over fewer target views. We thank the reviewer for suggesting this additional E2E analysis. We have added the measured near-parity analysis across 15–120 source views to Figure 6(b), with the detailed runtime protocol and results in Appendix C.5 (Tables 17–18).
Manuscript location: Section 4.2, p. 9, Figure 6(b); Appendix C.5, p. 28, lines 1481–1511 (Tables 17–18).
TACC shows limited sensitivity to small pose perturbations and sparse outliers in the tested allocation settings, although larger perturbations cause progressively greater quality degradation. Moreover, similar geometric relevance scores do not systematically favor uniform allocation, despite individual targets where uniform allocation performs better.
To investigate both effects, we conducted additional experiments using LagerNVS on the official 140-scene DL3DV evaluation set, with 80 source views, 20 target views per scene, a fixed A=10 budget, and the same trained Carrier checkpoint.
Pose noise and outliers. We perturb only the source poses used to compute relevance scores, keeping encoded features, Carrier hierarchies, and rendering inputs unchanged. We evaluate three perturbation magnitudes applied to all 80 cameras and a sparse-outlier setting with four randomly selected cameras.
R denotes the RMS distance of source-camera centers from their centroid. Rotation axes and translation directions are sampled independently. ΔPSNR reports the mean ± standard deviation across three random seeds, each evaluated on all 2,800 targets.
The PSNR drop increases from 0.079 dB to 0.886 dB as perturbations across all cameras become stronger, while sparse outliers cause only a 0.037 dB reduction. These results characterize the sensitivity of the allocation stage to pose errors without introducing perturbations into source encoding or rendering.
Similar relevance scores. We additionally compare TACC with inference-time Uniform Allocation using identical source and target views, the same trained Carrier checkpoint, and the same token budget. For each target, we measure the relevance-score gap between the highest- and tenth-ranked source views and divide targets into two groups.
Although uniform allocation outperforms TACC on some individual targets, its win rates are comparable across the two groups (3.57% versus 3.43%). TACC also retains a 4.99 dB mean PSNR advantage on targets with the smallest relevance-score gaps, indicating that similar geometric relevance does not systematically weaken the proposed allocation relative to uniform allocation. We have added both analyses to Appendix C.6 (Tables 20–21).
Manuscript location: Appendix C.6, p. 30, lines 1570–1595 (Table 20) and lines 1600–1619 (Table 21).