acceptodds
Under review as a conference paper at ICLR 2027

TACC: Target-Aware Carrier Compression for Dense-Context Novel View Synthesis

Abstract

Recent feed-forward approaches mark a paradigm shift in novel view synthesis by eliminating the reliance on per-scene optimization, directly inferring 3D representations from input views. However, their reliance on global scene-token context incurs substantial memory and latency overhead, constraining their ability to handle increasingly dense source-view contexts. Compressing source representations offers an intuitive way to limit this token budget. Prior general-purpose compression methods are not tailored for novel view synthesis, as their fixed, target-agnostic nature fails to adapt evidence allocation to individual target views. To overcome this, we introduce TACC, a target-aware compression module tailored for feed-forward latent NVS. Specifically, TACC organizes each source view into a progressive carrier hierarchy. Driven by source-target geometric relevance, it adaptively retrieves tokens to dynamically allocate exact computational quotas. This selective retrieval directly yields a compact, target-conditioned representation. Extensive experiments on RE10K demonstrate that, under dense-input settings, TACC accelerates inference by 9.23× over LagerNVS and 64× over LVSM.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
novel view synthesis, feed-forward novel view synthesis, neural rendering, multi-view representation compression, target-conditioned token allocation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SPXnOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The "encode once, allocate on demand" idea is genuinely good and the execution is clean — consistent gains across three frozen backbones and two datasets convince me the mechanism works, and I like that it needs no retraining of the backbones. My remaining doubts are minor and mostly framing: the headline speedups are Effective FPS and the end-to-end advantage only appears with many target views, and robustness to pose noise is untested. None of this undermines the core idea, which I find effective for dense-context NVS. Paper Summary TACC is a target-aware compression module for feed-forward latent novel view synthesis under dense source-view contexts. Each source view is organized once into a progressive per-view Carrier hierarchy (a binary spatial refinement tree with learned regional Carriers). At target time, a training-free allocation stage scores source views by source–target geometric relevance, grants the top-ranked view a full quota, and distributes the remaining budget across support views using an entropy-adaptive shared floor plus relevance-weighted largest-remainder rounding; the corresponding frontier readout yields a compact, target-conditioned token set that feeds the unchanged (frozen) rendering pathway. Only the Carrier module is trained, via teacher–student distillation. Across LagerNVS, LVSM Decoder-Only, and CLiFT on DL3DV and RealEstate10K, TACC outperforms matched-budget baselines (ToMe, DivPrune, ZPressor) in reconstruction quality while maintaining near-constant target-time throughput (up to 9.23×/64× Effective FPS over Full Context in dense settings), and cuts peak GPU memory for LVSM at 120 views from 12.77 to 4.22 GiB (−66.96%).

Critical strengths

An elegant, reusable design. The progressive Carrier hierarchy decouples one-time source encoding from target-time cost, supports exact integer per-view quota allocation across views, and is backbone-agnostic, demonstrated by consistent gains across three frozen backbones with a single Carrier checkpoint. The frontier/refinement-order construction (area, then marginal scatter gain) and the capacity-aware largest-remainder rounding are clean, well-specified technical choices.

Consistently positive, well-controlled empirical results. TACC beats matched-budget compression baselines by a large margin (e.g., +1.2–2.3 dB over ZPressor on LagerNVS) and even exceeds the frozen Full-Context checkpoint's quality in dense settings, while keeping Effective FPS nearly constant as the source count grows. The ablations (w/o Carrier, Uniform Alloc., Anchor-only Carrier, Top-A Full Views) support the contribution of both learned regional Carriers and target-conditioned allocation, and the results hold across two datasets including cross-dataset generalization without fine-tuning.

Critical weaknesses

The headline speedup numbers are Effective FPS, which exclude one-time source encoding and Carrier construction. The amortized E2E comparison (Fig. 4c) shows TACC is slower with 10 targets and only overtakes Full Context at ~40+ targets on LagerNVS.

The allocation stage is training-free and driven by simple geometric descriptors (ray direction / camera center / Plücker moments); robustness to noisy or inaccurate poses and to failure of the geometric-relevance assumption is not analyzed. These are fixable concerns and do not detract from the core contribution.

How about compare other accelerated methods with TACC in Fig 4?

Questions for authors

What is the break-even number of target views (and how does it scale with source-view count) for end-to-end wall-clock advantage including Carrier construction? A break-even plot over (source views × target views) would substantially improve the paper's practical impact.

How sensitive is the allocation to pose noise or outliers? Are there regimes (e.g., targets with similar relevance to many sources) where relevance-based allocation underperforms uniform allocation?

Does TACC's quality edge over Full Context mainly reflect the frozen backbone degrading on dense contexts? Would the advantage persist with a backbone fine-tuned for dense inputs, or is the contribution better framed as "restoring/improving dense-context quality at low cost"?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (Part 1/3): E2E Break-even and Allocation Robustness

Q1. End-to-End Break-even Analysis

The number of target views required to approach E2E parity generally decreases as the source-view count increases, from approximately 48 targets at 15 source views to 16 targets at 80–120 source views. This trend arises because Full Context incurs increasing per-target rendering costs with larger source contexts, while TACC maintains a fixed decoder-visible token budget.

To quantify this behavior, we conducted additional E2E measurements using LagerNVS on the official 140-scene DL3DV evaluation set, with A=10A=10 on a single NVIDIA A800 GPU. The timing includes source encoding, one-time Carrier construction, target-conditioned allocation and readout, and rendering of all target views.

Source ViewsTarget ViewsFull Context E2E FPS ↑TACC E2E FPS ↑
154816.418116.1030
203210.935910.9403
40203.81713.8998
60171.98601.9939
80161.25141.2523
100160.89280.8839
120160.67220.6636

As source-view counts increase, the growing per-target cost of Full Context allows TACC's rendering savings to amortize the one-time Carrier construction overhead over fewer target views. We thank the reviewer for suggesting this additional E2E analysis. We have added the measured near-parity analysis across 15–120 source views to Figure 6(b), with the detailed runtime protocol and results in Appendix C.5 (Tables 17–18).

Manuscript location: Section 4.2, p. 9, Figure 6(b); Appendix C.5, p. 28, lines 1481–1511 (Tables 17–18).

Q2 & W2. Robustness of Geometric Allocation

TACC shows limited sensitivity to small pose perturbations and sparse outliers in the tested allocation settings, although larger perturbations cause progressively greater quality degradation. Moreover, similar geometric relevance scores do not systematically favor uniform allocation, despite individual targets where uniform allocation performs better.

To investigate both effects, we conducted additional experiments using LagerNVS on the official 140-scene DL3DV evaluation set, with 80 source views, 20 target views per scene, a fixed A=10A=10 budget, and the same trained Carrier checkpoint.

Pose noise and outliers. We perturb only the source poses used to compute relevance scores, keeping encoded features, Carrier hierarchies, and rendering inputs unchanged. We evaluate three perturbation magnitudes applied to all 80 cameras and a sparse-outlier setting with four randomly selected cameras.

SettingCameras PerturbedRotation / TranslationPSNR (dB)ΔPSNR (dB)
Original poses0/80—27.3415—
Random perturbations80/801° / 1% of RR27.2625−0.0790 ± 0.0113
Random perturbations80/803° / 3% of RR26.9075−0.4340 ± 0.0256
Random perturbations80/805° / 5% of RR26.4560−0.8855 ± 0.0246
Sparse outliers4/8010° / 10% of RR27.3048−0.0367 ± 0.0065

RR denotes the RMS distance of source-camera centers from their centroid. Rotation axes and translation directions are sampled independently. ΔPSNR reports the mean ± standard deviation across three random seeds, each evaluated on all 2,800 targets.

The PSNR drop increases from 0.079 dB to 0.886 dB as perturbations across all cameras become stronger, while sparse outliers cause only a 0.037 dB reduction. These results characterize the sensitivity of the allocation stage to pose errors without introducing perturbations into source encoding or rendering.

Similar relevance scores. We additionally compare TACC with inference-time Uniform Allocation using identical source and target views, the same trained Carrier checkpoint, and the same token budget. For each target, we measure the relevance-score gap between the highest- and tenth-ranked source views and divide targets into two groups.

Target GroupTargetsMean TACC − Uniform PSNR (dB)Uniform Wins
Smallest 25% of score gaps700+4.992825 (3.57%)
Remaining 75%2,100+4.113172 (3.43%)

Although uniform allocation outperforms TACC on some individual targets, its win rates are comparable across the two groups (3.57% versus 3.43%). TACC also retains a 4.99 dB mean PSNR advantage on targets with the smallest relevance-score gaps, indicating that similar geometric relevance does not systematically weaken the proposed allocation relative to uniform allocation. We have added both analyses to Appendix C.6 (Tables 20–21).

Manuscript location: Appendix C.6, p. 30, lines 1570–1595 (Table 20) and lines 1600–1619 (Table 21).

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 2/3): Full Context Quality and Dense-Context Adaptation

Q3. Full Context Quality and Dense-Context Adaptation

We agree that the original Full Context backbone's limited adaptation to dense source views may partly explain TACC's large reconstruction-quality advantage. While dense-context fine-tuning substantially improves the Full Context backbone, TACC still achieves higher reconstruction quality and substantially greater target-time rendering throughput across the evaluated settings. We therefore consider preserving or improving dense-context reconstruction quality at lower rendering cost a more precise characterization of our contribution.

To examine this effect, we fully fine-tuned LVSM Decoder-Only on RE10K for dense source-view inputs, using approximately 170.8M trainable parameters and strictly disjoint training and evaluation scenes. In contrast, TACC retains the original frozen backbone and trains only 302,785 Carrier parameters (approximately 0.18% of the backbone).

Source ViewsFine-tuned LVSM PSNR ↑Fine-tuned LVSM Eff. FPS ↑TACC PSNR ↑TACC Eff. FPS ↑
8029.59030.337730.285110.28
10029.42710.221130.197810.15
12029.22940.154730.144110.03

Effective FPS measures target-time rendering throughput and excludes one-time source encoding and Carrier construction.

Dense-context fine-tuning narrows the quality gap observed with the original pretrained LVSM checkpoint, confirming that backbone adaptation affects the Full Context comparison. Nevertheless, TACC achieves 0.69–0.91 dB higher PSNR than the fine-tuned backbone across 80–120 source views, while substantially reducing target-time rendering costs without retraining the original backbone.

These findings show that TACC's benefits are not solely attributable to the original Full Context checkpoint's degradation. We have clarified the interpretation of the Full Context comparisons in Section 4.2 and added the dense-context fine-tuning results to Appendix C.1 (Table 9).

Manuscript location: Section 4.2, p. 8, lines 422–428; Appendix C.1, pp. 21–22, lines 1131–1164 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer 3 (Part 3/3): Reported Speedups and Acceleration Methods

W1. Scope of Reported Speedups

We agree that Effective FPS should be distinguished from end-to-end (E2E) acceleration. While Effective FPS excludes one-time source encoding and Carrier construction, TACC follows an encode-once, render-many design: a single source encoding and Carrier construction can support arbitrarily many subsequent target-view renderings without repeating these operations. Effective FPS therefore provides a meaningful measure of sustained rendering throughput, whereas E2E FPS accounts for one-time costs when rendering a finite number of targets.

Importantly, our additional measurements show that TACC approaches E2E parity at 16 target views and achieves a 1.0732× speedup at 20 targets, substantially earlier than the approximately 40 targets suggested by the original figure. To quantify this behavior, we conducted additional E2E experiments using LagerNVS on the official 140-scene DL3DV evaluation set, with 80 source views and A=10A=10. All target counts use prefixes of the same ordered target-view sequence. E2E timing includes source encoding, Carrier construction, target-conditioned allocation and readout, and rendering, excluding disk loading and image decoding.

Target ViewsFull Context E2E FPS ↑TACC E2E FPS ↑E2E Speedup
100.88620.79270.8946×
161.25141.25231.0008×
201.41371.51721.0732×
321.97412.41551.2236×
402.17982.95121.3539×
602.66664.22151.5831×
802.96195.22061.7626×
1003.17506.28831.9806×

TACC remains slower at 10 targets due to the one-time Carrier construction overhead, but its E2E speedup increases to 1.2236× at 32 targets and 1.9806× at 100 targets. For the same 80-source-view setting, TACC achieves 27.95 Effective FPS versus 4.40 for Full Context, demonstrating its sustained target-time efficiency. We now report both metrics and clarify their measurement scopes in Sections 4.1–4.2 and Figure 5(c).

Manuscript location: Section 4.1, p. 6, lines 296–300; Section 4.2, p. 8, Figure 5(c) (lines 398–412), and p. 9, lines 432–440.

W3. Comparison with Other Acceleration Methods

We have added ToMe, DivPrune, and ZPressor to the efficiency comparisons in Figures 5(a) and 5(b), allowing direct comparisons of peak GPU memory and per-target rendering latency under the same decoder-visible token budget. We thank the reviewer for this suggestion.

Specifically, all methods are evaluated using LVSM Decoder-Only on RealEstate10K, with source-view counts ranging from 15 to 120 and a fixed A=10A=10 budget. Together with the reconstruction-quality results in Table 2, these measurements provide a more complete comparison of reconstruction quality, memory consumption, and rendering efficiency across the compression methods.

Manuscript location: Section 4.2, p. 8, lines 398–412 (Figures 5(a)–(b)).

Reviewer_5vdYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to reduce the source tokens for dense-context feed-forward novel view synthesis. It builds a binary hierarchy of regional Carriers for each source view. Given a target camera, the method allocates a token quota to each source view based on geometry, and selects a corresponding frontier from the hierarchy. Only the Carrier module is trained while the pretrained backbone is frozen. The paper applies this method to LagerNVS, LVSM Decoder-Only, and CLiFT, and evaluates on DL3DV and RealEstate10K. It provides comparisons at the same token budgets, ablation studies, and both target-time and end-to-end efficiency results.

Critical strengths
  1. The hierarchy design. The proposed hierarchy makes sense to me. Each positive integer quota gives a frontier that covers the full image, and the full quota recovers the original source tokens. Thus the method can change the granularity for each target without re-encoding all the source images. I appreciate that the rendering backbone is kept frozen and the paper explains where the Carrier is inserted for each backbone.

  2. Detailed experimental comparisons. The paper evaluates three backbones on two datasets, and retrains the native CLiFT compressor with a matched protocol. The Top-A and residual-PMA comparisons are also useful. For example, Table 6 shows 30.1441 dB for TACC and 26.7087 dB for Top-A with the same LVSM token budget. The paper also separates Effective FPS from end-to-end FPS and reports the one-time cost. These details help understand what the reported speedup actually means.

Critical weaknesses
  1. The Full Context comparison needs some clarification. According to Appendix C.1, the frozen LVSM model is trained with two source views, but it is evaluated with up to 120 views. On the other side, the proposed compressor is trained with dense context. To my understanding, part of the large improvement over Full Context could come from this difference. The setting is useful for adapting a released model, but I am not sure whether it shows the same advantage over a renderer adapted to dense context with similar data and computation. The comparisons between compressors at the same token budget are more convincing to me. It would be better to make this distinction clear around the main speed and quality claims.

  2. The contribution of each allocation choice is not clear enough. Table 4 shows that uniform allocation is worse. However, the proposed allocation has several changes together, including fully keeping the top view, an entropy-dependent coverage floor, and relevance-weighted allocation of the remaining tokens. The appendix studies the relevance score, but I did not find results separating the other choices. In Table 9, residual PMA with the same allocation already gives 26.9088 dB, compared with 27.3415 dB of TACC. Thus I would expect more analysis on which allocation choice contributes to the improvement.

Questions for authors
  1. Is there a comparison with an LVSM or LagerNVS model adapted to dense context using similar data and computation? If not, please make clear that the Full Context result is from the frozen released checkpoint, and discuss it separately from the comparisons at matched token budgets.

  2. With the same learned Carrier, what happens if the entropy floor is removed, or if the top view is not required to keep full resolution? How much does each choice contribute over uniform allocation?

  3. It would be helpful to put the one-time Carrier construction time, the full cache size, and the number of target views needed to obtain an end-to-end speedup next to the main results. Appendix C.5 and Fig. 4 already discuss amortization, but having these numbers together would make the applicable setting easier to understand.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5vdY (Part 1/2): Full Context Fairness and Allocation Ablations

Q1 & W1. Full Context Fairness and Dense-Context Adaptation

We agree that the large quality gap over the original Full Context backbone may partly reflect its limited adaptation to dense source views. While the officially released LVSM Decoder-Only checkpoint was trained with two source views, TACC retains this frozen backbone and uses a separately trained Carrier to compress dense source contexts. We therefore distinguish the quality gains over the original Full Context checkpoint from those demonstrated by matched-budget compression comparisons.

To further assess this concern, we fully fine-tuned LVSM Decoder-Only on RE10K for dense source-view inputs, using approximately 170.8M trainable parameters and strictly disjoint training and evaluation scenes. In comparison, TACC retains the original frozen LVSM backbone and trains only 302,785 Carrier parameters.

Source ViewsFine-tuned LVSM PSNR ↑Fine-tuned LVSM Eff. FPS ↑TACC PSNR ↑TACC Eff. FPS ↑
8029.59030.337730.285110.28
10029.42710.221130.197810.15
12029.22940.154730.144110.03

Effective FPS measures target-time rendering throughput, excluding one-time source encoding and Carrier construction.

Dense-context fine-tuning substantially improves LVSM's reconstruction quality, confirming that backbone adaptation contributes to the original Full Context performance gap. Nevertheless, TACC maintains higher PSNR and substantially greater target-time throughput across all three source-view settings without fine-tuning the backbone.

Importantly, the two comparisons serve different purposes: Full Context evaluates the behavior of released or adapted renderers under dense inputs, whereas the matched-budget comparisons with ToMe, DivPrune, and ZPressor directly assess compression effectiveness under identical decoder-visible token budgets. We do not attribute the entire improvement over the original Full Context checkpoint to compression alone. We have clarified this distinction in Section 4.1 and added the fine-tuning comparison in Appendix C.1 (Table 9).

Manuscript location: Section 4.1, p. 6, lines 288–300; Appendix C.1, pp. 21–22, lines 1131–1164 (Table 9).

Q2 & W2. Contribution of Allocation Components

Both Top-1 full-view preservation and the entropy-adaptive coverage floor contribute substantially to reconstruction quality. With the same trained Carrier, removing these components independently reduces PSNR by 4.20 dB and 2.57 dB, respectively. We agree that the original ablations did not sufficiently isolate their individual contributions.

To quantify these effects, we conducted inference-time ablations on the official 140-scene DL3DV evaluation set using LagerNVS, with 80 source views, 20 target views per scene, and a fixed A=10A=10 budget (13,690 tokens). All variants share the same trained Carrier, 2,800 evaluation targets, source views, and camera poses; only the allocation rules are modified, without retraining.

Allocation StrategyPSNR ↑ (dB)SSIM ↑LPIPS ↓
Full TACC27.34150.86490.0791
w/o Entropy Coverage Floor24.77030.78040.1060
w/o Top-1 Preservation23.14020.72020.1894

Removing either component also decreases SSIM and increases LPIPS, showing that full-resolution preservation of the most relevant view and coverage across supporting views are both important to the allocation strategy.

The Uniform Allocation and Residual PMA comparisons address complementary aspects. Uniform Allocation achieves 23.0085 dB versus 27.3415 dB for TACC, but uses a separately retrained Carrier; its gap therefore reflects the overall allocation strategy rather than the isolated effect of either component. Conversely, Residual PMA achieves 26.9088 dB under the same allocation strategy as TACC but with a separately trained representation module, primarily reflecting differences in representation construction. We thank the reviewer for suggesting these controlled allocation ablations. We have clarified the training distinctions and added the corresponding results to Section 4.3 and Appendix C.4 (Table 14).

Manuscript location: Section 4.3, p. 10, lines 521–525; Appendix C.4, p. 25, lines 1297–1335 (Table 14).

AuthorsOpenReview ↗

Response to Reviewer 5vdY (Part 2/2): Carrier Costs and End-to-End Efficiency

Q3. One-Time Costs, Cache Size, and End-to-End Efficiency

TACC approaches end-to-end (E2E) runtime parity with Full Context at approximately 16 target views for 80–120 source views, with measured acceleration at 20, 18, and 18 targets, respectively. Importantly, a single source encoding and Carrier construction can support arbitrarily many subsequent target-view renderings without repeating these operations. Effective FPS therefore provides a meaningful measure of sustained rendering efficiency, while E2E FPS captures the impact of one-time costs when rendering a finite number of targets. We thank the reviewer for suggesting that these costs and their amortization be presented together.

To quantify these trade-offs, we report Carrier construction time and complete source-cache sizes in Section 4.2 (Table 3), alongside E2E near-parity and acceleration measurements in Appendix C.5 (Tables 17–18). We evaluate LagerNVS on the official 140-scene DL3DV benchmark at 256 × 256 resolution, with A=10A=10 and a target batch size of 4. Construction times are averaged per scene.

Source ViewsConstruction (s)TACC Cache (GiB)Full Context Cache (GiB)Near-Parity TargetsAcceleration Targets
803.13231.68590.31331620
1003.94742.10740.39171618
1204.70762.52880.47001618

At 16 target views, E2E runtimes differ by −0.08%, +1.01%, and +1.30% for 80, 100, and 120 source views, respectively.

E2E timing includes source encoding, Carrier construction, target-conditioned allocation and readout, and rendering. Carrier construction includes hierarchy building, frontier-table precomputation, and Carrier network precomputation. Cache sizes account for all reusable GPU-resident source representations, counting shared storage only once and excluding model weights and temporary target-side tensors.

While TACC requires a larger source cache than Full Context in LagerNVS, its fixed decoder-visible budget reduces per-target rendering costs. Cache storage and peak GPU memory are distinct metrics: our LVSM measurements further show that TACC reduces the overall peak from 12.77 GiB to 4.22 GiB at 120 source views (Figure 5(a)). These results distinguish the one-time preprocessing and cache overhead from the sustained rendering efficiency measured by Effective FPS.

Manuscript location: Section 4.1, p. 6, lines 296–300; Section 4.2, p. 8, Figure 5(a) and Table 3 (lines 398–420); Appendix C.5, pp. 28–29, lines 1481–1521 (Tables 17–18).

Reviewer_uWTuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TACC, an approach that enables novel view synthesis with any number of input views. TACC ranks each source view according to their relevance in creating the target view and extracts compreseed features which is processed through a decoder to produce the target view. TACC follows the encoder-decoder path of LVSM where the encoder is capable of compressing any number of views taking away the number of source view bottleneck from LVSM. The authors further portray rendering costs to show that the approach adds minimal computaional and memory overheads with additional views. Results show better scaling until N=40 views compared to existing approaches

Critical strengths
  1. The idea of memory efficeint approach for compressing the input views is interesting and useful and removes the existing bottleneck of transformer based feedforward novel view synthesis models
  2. The results on Re10K dataset shows the performance improves with numebr of views until N=40 which is promsiing
  3. Qualitiative results show that the approach removes existing problem of feedforward NVS specifically on the ringing artifacts near the edges as well as disoriented lines.
Critical weaknesses
  1. I'm curious to learn, why the memory does not increase substantially with increase in number of views. In the apporach, each view provides a set of tokens form which the target decoder can extract features from. However the memory overhead remains minor with increase in view
  2. The main contribution of LVSM is in removing the 3D inductive bottleneck. Doesnt a hard coded encoder pipeline introduce a inductive bias
  3. Multiple approaches[1,2,3] have previosuly introduced conditioning based on point cloud features . What is the advantage of current approach comapred to the current encoding schema
  4. The approach requries customized encoder for each new NVS approach which is a significant bottleneck
  5. How much does L_distill and L_carrier contribute towards performance improvement for the approach
  6. The results from Re10K dataset in appendix shows minimal improvements above N=40 views, moreover in all tables the performance decreases after the number of source views increase above 40. Can the authors provide an explanation for this
  7. The results for the performance vs PSNR is mising for DL3DV. Can the authors incclude this result.
  8. The approach failed to cite relevant work in cross domain generalization [4]

[1] Kwak, Min-Seop, et al. "Projected Representation Conditioning for High-fidelity Novel View Synthesis." arXiv preprint arXiv:2602.12003 (2026). [2] Wu, Zirui, et al. "From rays to projections: Better inputs for feed-forward view synthesis." arXiv preprint arXiv:2601.05116 (2026). [3] Chen, Kangjie, et al. "VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis." arXiv preprint arXiv:2609.33253 (2026). [4] Nair, Nithin Gopalakrishnan, et al. "Scaling transformer-based novel view synthesis with models token disentanglement and synthetic data." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors
  1. The main contribution of LVSM is in removing the 3D inductive bottleneck. Doesnt a hard coded encoder pipeline introduce a inductive bias. I woudld like to hear the authros thoughts on this.
  2. Can the authors provide an explanation for results from Re10K dataset in appendix shows minimal improvements above N=40 views
6 comments
AuthorsOpenReview ↗

Response to Reviewer uWTu (Part 1/4): Inductive Bias, View Scaling, and Memory

Q1 & W2. Inductive Bias and Representation Flexibility

We agree that TACC introduces additional inductive biases through its predefined spatial hierarchy and geometry-based allocation. While these constraints may limit representation flexibility, particularly under aggressive compression, they represent a deliberate trade-off for efficient compression and faster target-view rendering. Similar structural biases also exist in methods such as ZPressor [1], which uses camera geometry to select anchor views and aggregate support-view features.

Specifically, TACC organizes existing source tokens into reusable multi-granularity Carriers and allocates a fixed token budget according to target relevance, avoiding repeated processing of the complete source representation. Importantly, these constraints operate at the compression stage without introducing an explicit 3D representation or modifying LVSM's original rendering architecture. Only the lightweight Carrier module is trained, while the pretrained backbone remains frozen. We have clarified this trade-off in Appendix A and discuss target-conditioned regional refinement as a potential direction for improving representation flexibility.

Q2 & W6. Quality Saturation with Increasing Source Views

The observed quality saturation can be partly explained by the fixed decoder-visible token budget. Increasing the number of source views provides additional scene information without increasing the decoder's representation capacity. Consequently, the benefit of additional views depends on whether their complementary information can be preserved within the same budget, and reconstruction quality need not improve monotonically.

However, 40 source views is not a universal turning point for TACC. To further examine this behavior, we conducted additional experiments using LagerNVS on the official 140-scene DL3DV evaluation set under two fixed compression budgets, with the same Carrier checkpoint and evaluation protocol.

Source ViewsA6 PSNR ↑ (dB)A10 PSNR ↑ (dB)
2026.698027.2189
4027.015527.4253
6027.101527.5815
8026.620127.3415
10025.019226.8787

Under both budgets, PSNR continues to improve up to 60 source views, indicating that the saturation behavior depends on the experimental setting rather than a universal source-view threshold.

We thank the reviewer for raising this interesting question. Investigating the effective upper limit of useful source-view information across different datasets and experimental settings is a valuable direction for future research. Such insights could also inspire heuristic acceleration strategies that adapt computation to the amount of useful information available, avoiding unnecessary processing of increasingly redundant observations.

Manuscript location: Appendix C.2, p. 23, lines 1188–1210 (Table 11).

W1. GPU Memory Scaling

TACC exhibits slower GPU memory growth because its decoder processes a fixed number of source tokens, independent of the total source-view count. While source tokenization and Carrier construction still incur source-dependent memory costs, the decoder-visible representation remains fixed at 10,240 tokens (A=10A=10) for LVSM, compared with 122,880 tokens processed by Full Context at 120 source views.

To quantify this distinction, we conducted a stage-wise GPU memory analysis using LVSM Decoder-Only on RE10K, measuring source tokenization, Carrier construction, and decoding separately.

Source ViewsSource Tokenization (Full / TACC)Decoding (Full / TACC)TACC Tree ConstructionTACC Overall Peak
150.82 / 0.822.31 / 2.111.122.11
200.86 / 0.862.81 / 2.171.272.17
401.05 / 1.054.80 / 2.401.862.40
601.23 / 1.236.79 / 2.632.452.63
801.41 / 1.418.77 / 2.833.033.03
1001.59 / 1.5910.76 / 3.063.623.62
1201.78 / 1.7812.77 / 3.344.224.22

All memory values are in GiB. Stage-wise peaks include GPU tensors retained from preceding stages; the overall peak is the maximum across stages.

At 120 source views, TACC reduces decoder-stage peak memory from 12.77 GiB to 3.34 GiB, while its overall peak reaches 4.22 GiB during one-time Carrier construction. Thus, TACC substantially reduces decoder-stage memory growth without eliminating source-dependent preprocessing overhead. We have added these measurements to Appendix C.1 (Table 7) and Figure 5(a).

Manuscript location: Appendix C.1, p. 21, lines 1095–1120 (Table 7); Section 4.2, p. 8, Figure 5(a).

AuthorsOpenReview ↗

Response to Reviewer uWTu (Part 2/4): Structured Representations and Backbone Adaptation

W3. Comparison with Existing Structured Representations

The main advantage of TACC lies in its reusable multi-granularity source representation and target-conditioned token allocation under a fixed decoder-visible budget. While ReNoV [2], PVSM [3], and VGGT-Diff [4] introduce geometric conditioning mechanisms for novel view synthesis, TACC focuses on compressing existing source representations without modifying the pretrained rendering pathway.

To examine this distinction, we implemented three paper-inspired compression adapters within the same frozen LagerNVS backbone. All adapters operate between the Reconstructor and Renderer, producing exactly 13,690 source-side tokens (A=10A=10) through a 117 × 117 target-aligned spatial grid and one pooled source token. Their implementations are summarized below:

  1. ReNoV-inspired [2]: We associate source features with 3D positions obtained from source depth and project them onto the target-view grid. The projected features are combined with Fourier-encoded coordinates, camera information, and visibility masks to construct the compressed tokens.
  2. PVSM-inspired [3]: We construct colored 3D points from source RGB images, depth, and camera poses. Gaussian splatting then generates target-view RGB and alpha maps, which are processed by a lightweight encoder to produce compressed tokens, supplemented by a pooled source token.
  3. VGR-inspired [4]: We combine source features with back-projected 3D positions and confidence estimates from a frozen VGGT model. Depth- and confidence-guided projection constructs a near-surface hard anchor, while front- and back-surface evidence is fused through residual refinement.

Controlled comparison. All three adapters follow the same initialization protocol, training schedule, and optimization settings as TACC, using identical training data, evaluation views, and the A=10A=10 token budget. The LagerNVS backbone remains frozen, and only the respective compression modules are trained. Unlike TACC, these adapters do not employ progressive Carrier hierarchies or target-conditioned per-view quota allocation.

MethodParams80v PSNR ↑100v PSNR ↑120v PSNR ↑
ReNoV-inspired [2]0.454M23.015122.760722.4773
PVSM-inspired [3]0.155M13.531713.526513.5201
VGR-inspired [4]0.413M22.608522.326322.0373
TACC0.303M27.341526.878726.4732

TACC outperforms the strongest adapted baseline by 4.00–4.33 dB across 80–120 source views, supporting the effectiveness of reusable multi-granularity Carriers and target-conditioned allocation for fixed-budget compression. Importantly, these are compression adapters inspired by the respective methods, rather than reproductions of their complete NVS systems. We have expanded the related-work discussion in Section 2.1 and provided the adapter implementations and comparisons in Appendix C.4 (Table 16).

Manuscript location: Section 2.1, pp. 2–3, lines 104–109; Appendix C.4, pp. 25–27, lines 1347–1406 (Table 16).

W4. Backbone Adaptation and Transferability

We agree that integrating TACC into a new backbone requires adapting its source-token interface and training a corresponding Carrier module. While Carrier weights are not directly transferable across backbones, TACC reuses the same core compression design without modifying or retraining the pretrained backbone.

Specifically, TACC employs the same progressive Carrier construction and target-conditioned allocation across LagerNVS, LVSM Decoder-Only, and CLiFT. The insertion points are adapted to their respective architectures: between the Reconstructor and Renderer in LagerNVS, before the joint Transformer in LVSM, and in place of the native compression pathway in CLiFT. In all cases, the backbone remains frozen, and only the lightweight Carrier module is trained.

This distinction also applies to architecture-agnostic compression methods such as ZPressor [1], which adopts a shared compression design but trains separate compressors for different backbones. Once trained, a Carrier checkpoint can be reused across scenes, source-view counts, and target views within the evaluated settings without additional training. We have clarified the backbone-specific interfaces and independently trained Carrier checkpoints in Appendices B.1 and B.3, while acknowledging cross-backbone weight transfer as a remaining limitation.

AuthorsOpenReview ↗

Response to Reviewer uWTu (Part 3/4): Ablations and Quality–Efficiency

W5. Ablation of Training Losses

Both training objectives contribute to reconstruction quality. Removing LdistilL_{\mathrm{distil}} reduces PSNR by 0.56 dB on average, while removing LcarrierL_{\mathrm{carrier}} leads to a 0.62 dB average reduction across 80–120 source views.

To isolate their contributions, we conducted additional ablations using LagerNVS on the official 140-scene DL3DV evaluation set, with a fixed A=10A=10 budget and 20 target views per scene. Each loss term was removed independently during Carrier training, while the backbone remained frozen and all variants shared the same evaluation protocol.

Source ViewsVariantPSNR ↑SSIM ↑LPIPS ↓
80Full TACC27.34150.86490.0791
80w/o LdistilL_{\mathrm{distil}}26.69740.85170.0816
80w/o LcarrierL_{\mathrm{carrier}}26.56420.84800.0825
100Full TACC26.87870.85230.0833
100w/o LdistilL_{\mathrm{distil}}26.37660.84170.0848
100w/o LcarrierL_{\mathrm{carrier}}26.29870.84000.0851
120Full TACC26.47320.84160.0869
120w/o LdistilL_{\mathrm{distil}}25.93530.83050.0884
120w/o LcarrierL_{\mathrm{carrier}}25.97470.83200.0879

Both ablations also consistently reduce SSIM and increase LPIPS, confirming the contributions of the two objectives to reconstruction quality. These results have been added to Appendix C.4 (Table 15).

Manuscript location: Appendix C.4, p. 26, lines 1351–1363 (Table 15).

W7. Additional PSNR–Performance Results on DL3DV

We have added the requested PSNR–Effective FPS comparison on DL3DV, showing that TACC achieves the highest reconstruction quality at every matched token budget, although its Effective FPS is lower than the competing compression methods at the same budget.

We thank the reviewer for suggesting this additional evaluation. Specifically, we evaluated TACC, ToMe, DivPrune, and ZPressor under six decoder-visible budgets (A∈{2,4,6,8,10,13}A \in \{2,4,6,8,10,13\}) using LagerNVS on the official 140-scene DL3DV evaluation set, with 80 source views and 20 target views per scene.

BudgetTACCToMeDivPruneZPressor
A221.1599 / 71.8617.0749 / 99.0019.0492 / 99.1618.6672 / 101.81
A424.3268 / 52.4118.2611 / 65.0820.3359 / 64.4521.4607 / 65.66
A626.6201 / 39.5218.8116 / 47.2321.0042 / 47.2023.5503 / 47.93
A827.1159 / 32.8819.6274 / 38.0921.4596 / 37.2923.5808 / 38.15
A1027.3415 / 27.9520.0344 / 31.4321.7624 / 31.5424.1568 / 31.26
A1327.4844 / 22.4720.5562 / 24.9422.1158 / 24.9324.6789 / 25.09

Each entry reports PSNR (dB) / Effective FPS. Effective FPS includes target-conditioned allocation, Carrier readout, and rendering, while excluding one-time source encoding and Carrier construction.

Across all six budgets, TACC exceeds the strongest competing method by 2.11–3.54 dB. Notably, TACC at A=6A=6 achieves 26.6201 dB and 39.52 Effective FPS, outperforming ZPressor at A=13A=13 (24.6789 dB and 25.09 Effective FPS) by 1.94 dB in PSNR and 1.58× in target-time throughput, despite using a smaller token budget. We have included the corresponding curves in Figure 6(a) and the complete results in Appendix C.2 (Table 12).

Manuscript location: Section 4.2, p. 9, Figure 6(a); Appendix C.2, p. 23, lines 1212–1234 (Table 12).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewers' Feedback and Authors' Response

We thank all three reviewers for their careful evaluations, constructive criticism, and thoughtful suggestions. Their feedback has been valuable in strengthening our experimental evidence and clarifying the scope of our contributions.

Strengths. We appreciate the reviewers' recognition of TACC's reusable progressive Carrier hierarchy, target-conditioned token allocation, and efficient compression design. In particular, the reviewers highlighted its ability to maintain exact token budgets and spatial coverage, reuse source representations across target views, and integrate into multiple pretrained NVS backbones without modifying their original parameters. They also acknowledged the extensive evaluations across three backbones and two datasets, the reconstruction-quality improvements over matched-budget compression baselines, and the favorable quality–efficiency trade-offs under dense source-view settings.

In response to the reviewers' concerns, we have substantially strengthened the experimental evaluation and clarified several important distinctions, including the fairness of Full Context comparisons, the contributions of individual compression components, and the relationship between target-time throughput and end-to-end efficiency. The main additions and revisions are summarized below:

  • Dense-context adaptation: We clarify that Full Context uses the officially released frozen checkpoints and additionally compare TACC against an LVSM backbone fully fine-tuned for dense source contexts. This separates the effects of backbone adaptation from compression effectiveness. (Sec. 4.1, p. 6; Appendix C.1, Table 9, p. 22.)

  • Additional comparisons and ablations: We introduce ReNoV-, PVSM-, and VGGT-Diff-inspired compression adapters, extend comparisons with ToMe, DivPrune, and ZPressor, and provide controlled ablations of Top-1 full-view preservation, the entropy-adaptive coverage floor, and the training objectives. (Sec. 4.3; Appendix C.4, Tables 14–16, pp. 25–27.)

  • Quality–efficiency and source-view scaling: We add PSNR–Effective FPS comparisons across six token budgets on DL3DV and investigate reconstruction-quality saturation as the number of source views increases under fixed compression budgets. (Sec. 4.2, Fig. 6(a), p. 9; Appendix C.2, Tables 11–12, p. 23.)

  • Efficiency and memory analysis: We extend GPU memory profiling and end-to-end amortization experiments, reporting one-time Carrier construction costs, complete cache sizes, and the number of target views required to approach or achieve end-to-end acceleration. These results distinguish sustained rendering throughput from overall computational costs. (Sec. 4.2, Fig. 5 and Table 3, p. 8; Appendix C.5, Tables 17–18, pp. 28–29.)

  • Robustness and methodological clarification: We evaluate allocation robustness under pose perturbations, sparse outliers, and similar geometric relevance scores. We also clarify the inductive biases introduced by TACC, backbone-specific adaptation requirements, and its relationship to relevant prior work. (Appendix A, p. 14; Appendix C.6, Tables 20–21, p. 30.)

Together, these revisions provide stronger experimental support for TACC's ability to preserve or improve dense-context reconstruction quality while reducing target-time rendering costs under fixed token budgets. We believe the revised manuscript now offers a more rigorous and transparent assessment of the method's effectiveness, computational trade-offs, and practical applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.