acceptodds
Under review as a conference paper at ICLR 2027

AIMS: Anchor-Integrated Multi-View Synthesis for Scalable Novel View Rendering

Abstract

Feed-forward novel view synthesis methods achieve strong generalization from posed multi-view inputs, but scaling them to large input view sets remains challenging. Transformer-based approaches that jointly process all input-view tokens incur rapidly increasing computation and memory as the number of views grows, while simple view subsampling discards potentially useful observations. We introduce Anchor-Integrated Multi-View Synthesis (AIMS), a scalable framework that decouples the number of available observations from the number of views processed by the global synthesis model. AIMS selects a fixed set of spatially distributed anchor views using farthest point sampling, groups nearby observations around each anchor, and uses a lightweight learnable integrator to fuse their information into enriched anchor representations. This allows additional observations to contribute to synthesis while keeping the downstream global view budget fixed. Evaluations on RealEstate10K and ScanNet demonstrate a favorable quality–efficiency trade-off against transformer-based and Gaussian-based baselines. AIMS achieves 29.41 dB and 17.73 dB PSNR on the two datasets, respectively, with rendering averaging 7.24 ms per view.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
novel view synthesis, transformer, large model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FuZFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes AIMS for feed-forward novel view synthesis from many posed source images. It selects anchor views using farthest-point sampling and groups nearby views around each anchor. A shared View Group Encoder compresses each group into learned scene tokens, and a global decoder uses these tokens to synthesize target views. The global decoding budget is fixed when the number of anchors and tokens per group are fixed. The model is trained on RealEstate10K and evaluated on RealEstate10K and ScanNet. The paper also studies the number of anchors, the neighborhood size during inference, and how the attention patterns depend on camera poses.

Critical strengths
  1. The local-to-global design. The proposed model makes sense to me. The local groups can first gather information from nearby views, and the global decoder then takes a compact representation. The number of anchors and the neighborhood size also control different things, namely the global representation capacity and the local observations. The method and the attention cost are easy to understand.

  2. Quality and rendering time. The results are promising, with 29.41 dB on RealEstate10K and 17.73 dB on ScanNet, and 7.24 ms per target for cached rendering. The ScanNet results are useful to show the generalization beyond the training dataset. Table 3 also shows that more neighbors can help on ScanNet. I appreciate that the paper reports the one-time encoding cost separately and discusses the limitation of grouping only based on poses.

Critical weaknesses
  1. The training settings of the baselines are different. Sec. 4.2 uses released checkpoints for the baselines. According to Appendix A.1, AIMS is initialized from the LVSM encoder-decoder and then jointly trains the new VGE and pretrained decoding modules for 20 epochs. The grouping and the context distribution also change. Thus I am not sure how much of the improvement in Tables 1 and 2 comes from the proposed architecture, and how much comes from the additional training and changed input distribution. The anchor-only baseline evaluations are useful, but a baseline adapted with the same training settings would make the architectural contribution more convincing.

  2. The contribution of local grouping and the scalability claim need more evidence. Changing the anchor count also changes the number of global tokens. Table 3 changes K only during inference, while the model is trained with K=5. I did not find an anchor-only AIMS variant with K=0 trained under the same settings, or a global learned compressor with similar size and the same output-token budget. These comparisons would help understand whether local integration is the main reason for the improvement. In addition, Appendix A.5 acknowledges that the groups can overlap and some candidate views may not be used. Thus keeping the global decoder cost fixed does not necessarily show that the method effectively uses a larger input collection.

Questions for authors
  1. Could the closest LVSM encoder-decoder baseline be adapted with the same RealEstate10K scenes, source/target sampling, training duration, and objective? This would help understand the contribution of the architecture separately from the additional training.

  2. With the same decoder and global output-token budget, how does the method compare with a K=0 anchor-only variant and a non-hierarchical learned compressor, trained under matched settings?

  3. When G and K are fixed and the candidate set becomes larger, how many unique source views are actually used? It would be helpful to show this number together with quality, total encoding/grouping time, and peak memory over different candidate-set sizes.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FuZF

Thank you for the careful and constructive review.

W1 / Q1. Baseline with the same training. We agree this is the right comparison. Training LVSM (E–D) on the same inputs as AIMS is very costly, because its encoder attends jointly over the views of all groups. As an affordable alternative, we are fine-tuning LVSM (E–D) on the anchors-only setting (G = 12) with the same scenes, sampling, objective and duration. We expect it to finish within two days and will report the results here.

W2 / Q2. K = 0 and a global compressor. We are training AIMS with G = 12 and K = 0 under the same settings and will report it as soon as it finishes, which we expect within 2 days. The LVSM (E–D) run described under W1 / Q1 serves as the non-hierarchical compressor, since it compresses the same 12 anchors jointly.

Q3. Unique views used. Thank you for raising this. Using the main checkpoint (G = 12, K = 5, at most 72 slots), we varied the candidate pool N ∈ {60, 70, 80, 90} with the eight held-out targets fixed. The table shows N = 60 → 90:

DatasetUnique views usedPSNR ↑Grouping + encoding (ms)Peak GPU memory (GiB)
RealEstate10K54.78 → 66.7428.075 → 28.60988.44 → 89.581.188 → 1.318
ScanNet54.49 → 66.1917.741 → 17.76488.89 → 90.101.206 → 1.325
DL3DV55.12 → 66.6616.766 → 17.00888.46 → 89.751.206 → 1.325

With a 50% larger pool, AIMS uses about 21–22% more distinct views, while time rises by under 1.5% and memory by about 10–11%. Repeated slots fall from 23–24% to 7–8%. Quality improves on RealEstate10K and DL3DV and is roughly flat on ScanNet, which peaks at N = 80 (17.835 dB), so larger pools do not guarantee monotonic gains.

For a given setting, the number of views used is bounded by G·(K+1) = 72. This bound is a user choice: the anchor count G is designed to set how much of the collection is covered, so for larger collections the user can choose a larger G. The cost then grows with G, not with N. We will state this explicitly.

AuthorsOpenReview ↗

Response to Reviewer FuZF - W1 & W2

Thank you for your patience. All training runs have finished, and we report the comparisons promised under W1 / Q1 and W2 / Q2.

Adapted baselines and the trained anchor-only ablation. All three models use G=12, the same decoder architecture, and 3,072 global latent tokens. LVSM receive the 12 anchors, and each AIMS variant uses its stated K during both training and evaluation, making K=0 a separately trained control.

All models are trained exclusively on RealEstate10K; ScanNet and DL3DV measure zero-shot transfer. Evaluation uses identical held-out targets and matched scenes: 7,281 RealEstate10K, 843 ScanNet, and 140 DL3DV scenes.

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓
RealEstate10KFine-tuned LVSM, 12 anchors30.5960.91280.0878
RealEstate10KAIMS, K=030.0250.90510.0935
RealEstate10KAIMS, K=529.9030.90500.0895
ScanNetFine-tuned LVSM, 12 anchors16.9310.57920.5291
ScanNetAIMS, K=016.5270.56670.5409
ScanNetAIMS, K=517.9520.58430.5142
DL3DVFine-tuned LVSM, 12 anchors16.9410.45860.4371
DL3DVAIMS, K=016.6480.44490.4471
DL3DVAIMS, K=517.1940.45300.4304

Baseline adaptation. Fine-tuned LVSM performs best on RealEstate10K, confirming its strength after adaptation. Nevertheless, full AIMS retains PSNR advantages of 1.021 dB on ScanNet and 0.253 dB on DL3DV. These cross-dataset gains therefore persist after giving LVSM the same additional training duration.

Trained K=0 ablation. Relative to K=0 AIMS, K=5 AIMS improves PSNR by 1.425 dB on ScanNet and 0.546 dB on DL3DV, with improvements in SSIM and LPIPS on both datasets. The global token budget remains unchanged. On RealEstate10K, K=0 achieves slightly higher PSNR and nearly identical SSIM, while K=5 achieves better LPIPS.

Analysis of the dataset-varying gains. We measured visual similarity using one frozen DINOv2 ViT-S/14 descriptor per image, independently of either rendering model. Within-group similarity averages cosine similarity over distinct image pairs within the G=12, K=5 reference groups, counting repeated pairs only once. Between-group similarity compares the normalized mean descriptors of different groups.

DatasetWithin-group similarityBetween-group similarityAIMS K=5 PSNR gain over anchor-only LVSM
RealEstate10K0.9740.861−0.693 dB
DL3DV0.8590.770+0.253 dB
ScanNet0.7510.556+1.021 dB

Lower similarity indicates greater visual diversity across observations. On RealEstate10K, where within- and between-group similarities are highest, anchor-only LVSM achieves the best performance, followed by K=0 AIMS. In contrast, K=5 AIMS outperforms LVSM on the more visually diverse DL3DV and ScanNet datasets, with the largest gains observed on ScanNet. This trend suggests that integrating neighboring views becomes increasingly beneficial as visual diversity increases.

We will incorporate these experiments into the main paper to clarify this trade-off: anchor-only methods can be preferable when scene observations are highly similar, whereas AIMS benefits from integrating neighboring views when observations are more diverse. This highlights the advantage of our approach in generalizing to complex scenes with greater visual variation, without increasing the global latent-token budget.

Reviewer_zY5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduce Anchor-Integrated Multi-View Synthesis (AIMS) to reduce the computational costs. AIMS selects a fixed set of spatially distributed anchor views using farthest point sampling, groups nearby observations around each anchor, and uses a lightweight learnable integrator to fuse their information into enriched anchor representations. Evaluations on RealEstate10K and ScanNet demonstrate a favorable quality–efficiency trade-off.

Critical strengths

This work pays attention to reducing computational cost and memory usage in novel view rendering, an interesting and practically relevant task. Experimental results demonstrate faster inference and strong performance on the RealEstate10K and ScanNet datasets.

Critical weaknesses

The methodological novelty of this work appears limited: combining KNN with farthest point sampling (FPS) is common in point cloud networks such as PointNet++ [A] and Point Transformer, while using learnable tokens to aggregate features has also been explored in methods such as VGGT-Omega [B] and LVSM [C].

[A] Qi, Charles Ruizhongtai, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS 2017.

[B] Wang, Jianyuan, et al. VGGT-$\Omega. CVPR 2026.

[C] Jin, Haian, et al. Lvsm: A large view synthesis model with minimal 3d inductive bias. ICLR 2025.

Questions for authors

Is the proposed model fine-tuned on ScanNet? Why does it achieve substantial performance gains on RealEstate10K but only modest gains on ScanNet?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zY5X

Thank you for the review and the helpful references. We respond to both points below.

W1. Novelty. We agree that FPS, neighborhood grouping and learnable latent tokens are established components. Our contribution is how they are organized for view synthesis: local evidence integration is separated from global rendering capacity. Each pose-local group is compressed independently by a shared encoder, and the decoder receives a fixed number of group-level scene tokens. The number of anchors sets the capacity of the scene representation and the group size sets how much local evidence is used, so the user can trade quality against computational cost. We will cite PointNet++ and VGGT-Ω and clarify this in the related work.

Q1. ScanNet. AIMS is trained only on RealEstate10K; ScanNet is a zero-shot evaluation for all methods. Against LVSM (D)'s best anchor-only setting (Table 2), the gains are comparable: +1.17 dB on RealEstate10K and +0.88 dB on ScanNet. The gap looks larger on RealEstate10K in Table 1 because LVSM (D) degrades there when given all views (24.32 dB), while on ScanNet it improves (17.34 dB). We attribute this to camera variation (Figs. 2 and 5): views within a group are nearly identical in RealEstate10K but differ more in ScanNet, so additional views carry more information on ScanNet. Table 3 is consistent with this: larger groups keep improving AIMS on ScanNet (16.82 → 17.82 dB from K = 1 to K = 7).

Reviewer_zY5XOpenReview ↗

Comment

I appreciate the authors’ responses. Although this work applies FPS and learnable tokens to a different task (i.e., novel view synthesis), I still find its novelty limited, given the widespread use of FPS and the prior use of learnable tokens in VGG-Omega.

AuthorsOpenReview ↗

Response to Reviewer zY5X - Novelty

Thank you for the follow-up. We agree that FPS and learnable tokens are widely used, and we do not claim either as a contribution. To be precise about what we do claim:

Role of FPS. FPS is not our contribution. We use it only as a simple way to select anchors that are spread across the scene, so that the groups cover it well. Other selection rules can serve the same purpose: in an ablation run during the rebuttal, uniform anchor selection performs on par with FPS (within about 0.1 dB).

Difference from VGGT-Ω. In VGGT-Ω, each frame carries register tokens that exchange information across frames, while the image tokens are kept and decoded. Its representation therefore still grows with the number of frames. In AIMS, the learnable tokens replace the image tokens: only they are passed to the decoder, and they are produced per group of views. The scene representation has a fixed size regardless of how many views are available, and no attention spans all views.

What the design contributes. At the same budget of 3,072 scene tokens, AIMS trained with neighbors (K = 5) improves over AIMS trained without them (K = 0) by 1.43 dB on ScanNet and 0.55 dB on DL3DV, with no gain on RealEstate10K, where views are highly similar. We see the contribution as this local-integration design under a fixed budget, together with the evidence of when it helps.

We understand that the reviewer may weigh novelty differently, and we appreciate the careful consideration.

Reviewer_njeFOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents AIMS, a novel view synthesis approach that enables scaling feedforward NVS to a large number of input views. Existing feedforward approaches are limited by the cpability to scale to a large number of source views because of the transformer backbone. AIMS selects a fixed set of distributed views using farthest point sampling and train a lightweight learnable integrator to fuse their information into enriched anchor representations, a decoder utilizes these representations and the target pose to reconstruct the target view. AIMS scheives SOTA results in Re10K and ScanNet benchmarks while portraying extremely fast rendering times

Critical strengths
  1. The problem that the authors are trying to tackle is relevant,existing approaches suffer from
  2. The approach for choosing the anchors using farthest point sampling is interesting.
  3. The rendering speed-up while conditioned on a large number of input views is a notable improvement
Critical weaknesses
  1. The authors claim that AIMS uses a learnable lightweight layer to fuse information in the abstract however, the encoder is 12 layers and the decoder
  2. AIMS initialises the model using LVSM's pretrained checkpoints, which provide a good starting initialisation point for enhanced performance.
  3. Results for the DL3DV dataset are missing
  4. The performance of AIMS lags behind that of LVSM for 4 input views and is marginally above for 8 views, although starting from an initialised state.
  5. Sufficient ablations are missing to show the relevance of farthest point sampling to choose anchors instead of uniform sampling.
  6. Could the authors clarify what the purpose of learnable tokens in the encoder is?
Questions for authors
  1. Could the authors clarify what the purpose of learnable tokens in the encoder is?
6 comments
AuthorsOpenReview ↗

Title: Response to Reviewer njeF

Thank you for the detailed review.

W1. "Lightweight". We agree that "lightweight" does not describe a 12-layer transformer. Our intention was to emphasize the computational efficiency of the proposed architecture rather than its model size. We will revise the wording accordingly.

W2. LVSM initialization. We agree that the pretrained checkpoint helps. AIMS initializes the tokenizer and decoder from LVSM (E–D), while the View Group Encoder is trained from scratch. For a fairer comparison, we are fine-tuning LVSM (E–D) on the anchor-only setting (G = 12) with our data and training settings. We expect this to finish within two days and will report the results here.

W3. DL3DV. We agree that DL3DV would make the evaluation more complete. We will report the comparative results on DL3DV as soon as the training described under W2 finishes.

W4. AIMS vs. LVSM at 4 and 8 views. We believe this may come from comparing Fig. 3 with Table 2, and we apologize if the paper was unclear. Fig. 3 reports short 10-epoch runs used only to study the quality–cost trade-off across anchor counts; they are not fully trained models. The fully trained model reaches 29.41 dB on RealEstate10K and 17.73 dB on ScanNet. We will label Fig. 3 more clearly.

W5. FPS vs. uniform sampling. We chose FPS because a camera trajectory is not guaranteed to cover a scene evenly: some regions may be filmed for longer and others only briefly. In such cases, uniform sampling over frames over-represents the regions filmed for longer, whereas sampling in pose space spreads the anchors across the scene.

Following your suggestion, we replaced FPS with uniform anchor selection in a paired evaluation on the same scenes. The change relative to FPS is:

DatasetΔPSNRΔSSIMΔLPIPS
RealEstate10K+0.089+0.0014−0.0009
ScanNet−0.015+0.0015−0.0025
DL3DV+0.105+0.0053−0.0048

The two perform similarly (differences of about 0.1 dB or less), with uniform sampling slightly ahead on RealEstate10K and DL3DV. These datasets are continuous video captures where uniform sampling already covers the trajectory well. We will add this ablation and revise our claim about FPS accordingly.

W6 / Q1. Learnable tokens. They compress the input views into a fixed number of tokens, the same role as the latent tokens in LVSM (E–D). The difference is where they are applied. LVSM encodes all input views jointly, so its attention cost grows quickly with the number of views. AIMS divides the views into groups, and the learnable tokens compress each group separately into L tokens. The decoder therefore always receives G·L tokens.

AuthorsOpenReview ↗

Response to Reviewer njeF - W2 & W3

Thank you for your patience. The training has finished, and we report the matched LVSM comparison (W2) and the DL3DV results (W3).

Setup. We fine-tuned LVSM (E–D) from the same released checkpoint on the 12 anchor views, with the same data, objective and 20 epochs as AIMS. To match its 3,072 latent tokens, we retrained AIMS (G = 12, K = 5) with L = 256 tokens per group (12 × 256 = 3,072). Both models are trained only on RealEstate10K, so ScanNet and DL3DV are zero-shot for both.

Dataset (scenes)MethodPSNR ↑SSIM ↑LPIPS ↓
RealEstate10K (7,281)LVSM (E–D), fine-tuned30.5960.91280.0878
RealEstate10K (7,281)AIMS29.9030.90500.0895
ScanNet (843)LVSM (E–D), fine-tuned16.9310.57920.5291
ScanNet (843)AIMS17.9520.58430.5142
DL3DV (140)LVSM (E–D), fine-tuned16.9410.45860.4371
DL3DV (140)AIMS17.1940.45300.4304

On RealEstate10K, the fine-tuned LVSM is better than AIMS by about 0.7 dB. On the two zero-shot datasets, AIMS has higher PSNR: +1.02 dB on ScanNet and +0.25 dB on DL3DV, where LVSM has slightly higher SSIM.

Why the gain differs across datasets. We measured how similar the input views are, using one frozen DINOv2 ViT-S/14 descriptor per view (independent of both models). Within-group similarity is the mean cosine similarity between views of the same group. Between-group similarity is the mean cosine similarity between the average descriptors of different view groups.

DatasetWithin-group similarityBetween-group similarityAIMS PSNR gain over anchor-only LVSM
RealEstate10K0.9740.861−0.698 dB
DL3DV0.8590.770+0.253 dB
ScanNet0.7510.556+1.021 dB

In RealEstate10K, the views within a group are nearly identical and the groups resemble each other, so the neighbors add little beyond the 12 anchors. In ScanNet, both similarities are the lowest, so the neighbors show more of the scene, and AIMS gains the most. DL3DV lies in between on all three columns.

Together, these results suggest that AIMS benefits most when the views within a group are more diverse, so that the neighbors contribute scene information beyond the anchors.

We will add this experiment to the main paper and state its conclusion: anchor-only LVSM can be the better choice when the views of a scene are highly similar, while AIMS is better suited to scenes where many diverse views are needed to cover the scene.

I still fail to understand with the authors view that the learnable tokens compress information from the input views. From the flow diagram in Figure 1, these seem to be appended with the input views tokens

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewers,

Thank you again for your time and thoughtful feedback on our submission.

As the discussion period is drawing to a close, we would like to gently follow up on our responses and the additional results we have posted. We would be grateful to know whether they have addressed your concerns, and we would be happy to clarify any remaining questions.

If your concerns have been resolved, we would sincerely appreciate it if you could take this into consideration in your final assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.