- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes AIMS for feed-forward novel view synthesis from many posed source images. It selects anchor views using farthest-point sampling and groups nearby views around each anchor. A shared View Group Encoder compresses each group into learned scene tokens, and a global decoder uses these tokens to synthesize target views. The global decoding budget is fixed when the number of anchors and tokens per group are fixed. The model is trained on RealEstate10K and evaluated on RealEstate10K and ScanNet. The paper also studies the number of anchors, the neighborhood size during inference, and how the attention patterns depend on camera poses.
Critical strengths
-
The local-to-global design. The proposed model makes sense to me. The local groups can first gather information from nearby views, and the global decoder then takes a compact representation. The number of anchors and the neighborhood size also control different things, namely the global representation capacity and the local observations. The method and the attention cost are easy to understand.
-
Quality and rendering time. The results are promising, with 29.41 dB on RealEstate10K and 17.73 dB on ScanNet, and 7.24 ms per target for cached rendering. The ScanNet results are useful to show the generalization beyond the training dataset. Table 3 also shows that more neighbors can help on ScanNet. I appreciate that the paper reports the one-time encoding cost separately and discusses the limitation of grouping only based on poses.
Critical weaknesses
-
The training settings of the baselines are different. Sec. 4.2 uses released checkpoints for the baselines. According to Appendix A.1, AIMS is initialized from the LVSM encoder-decoder and then jointly trains the new VGE and pretrained decoding modules for 20 epochs. The grouping and the context distribution also change. Thus I am not sure how much of the improvement in Tables 1 and 2 comes from the proposed architecture, and how much comes from the additional training and changed input distribution. The anchor-only baseline evaluations are useful, but a baseline adapted with the same training settings would make the architectural contribution more convincing.
-
The contribution of local grouping and the scalability claim need more evidence. Changing the anchor count also changes the number of global tokens. Table 3 changes K only during inference, while the model is trained with K=5. I did not find an anchor-only AIMS variant with K=0 trained under the same settings, or a global learned compressor with similar size and the same output-token budget. These comparisons would help understand whether local integration is the main reason for the improvement. In addition, Appendix A.5 acknowledges that the groups can overlap and some candidate views may not be used. Thus keeping the global decoder cost fixed does not necessarily show that the method effectively uses a larger input collection.
Questions for authors
-
Could the closest LVSM encoder-decoder baseline be adapted with the same RealEstate10K scenes, source/target sampling, training duration, and objective? This would help understand the contribution of the architecture separately from the additional training.
-
With the same decoder and global output-token budget, how does the method compare with a K=0 anchor-only variant and a non-hierarchical learned compressor, trained under matched settings?
-
When G and K are fixed and the candidate set becomes larger, how many unique source views are actually used? It would be helpful to show this number together with quality, total encoding/grouping time, and peak memory over different candidate-set sizes.
2 comments
Response to Reviewer FuZF - W1 & W2
Thank you for your patience. All training runs have finished, and we report the comparisons promised under W1 / Q1 and W2 / Q2.
Adapted baselines and the trained anchor-only ablation. All three models use G=12, the same decoder architecture, and 3,072 global latent tokens. LVSM receive the 12 anchors, and each AIMS variant uses its stated K during both training and evaluation, making K=0 a separately trained control.
All models are trained exclusively on RealEstate10K; ScanNet and DL3DV measure zero-shot transfer. Evaluation uses identical held-out targets and matched scenes: 7,281 RealEstate10K, 843 ScanNet, and 140 DL3DV scenes.
| Dataset | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| RealEstate10K | Fine-tuned LVSM, 12 anchors | 30.596 | 0.9128 | 0.0878 |
| RealEstate10K | AIMS, K=0 | 30.025 | 0.9051 | 0.0935 |
| RealEstate10K | AIMS, K=5 | 29.903 | 0.9050 | 0.0895 |
| ScanNet | Fine-tuned LVSM, 12 anchors | 16.931 | 0.5792 | 0.5291 |
| ScanNet | AIMS, K=0 | 16.527 | 0.5667 | 0.5409 |
| ScanNet | AIMS, K=5 | 17.952 | 0.5843 | 0.5142 |
| DL3DV | Fine-tuned LVSM, 12 anchors | 16.941 | 0.4586 | 0.4371 |
| DL3DV | AIMS, K=0 | 16.648 | 0.4449 | 0.4471 |
| DL3DV | AIMS, K=5 | 17.194 | 0.4530 | 0.4304 |
Baseline adaptation. Fine-tuned LVSM performs best on RealEstate10K, confirming its strength after adaptation. Nevertheless, full AIMS retains PSNR advantages of 1.021 dB on ScanNet and 0.253 dB on DL3DV. These cross-dataset gains therefore persist after giving LVSM the same additional training duration.
Trained K=0 ablation. Relative to K=0 AIMS, K=5 AIMS improves PSNR by 1.425 dB on ScanNet and 0.546 dB on DL3DV, with improvements in SSIM and LPIPS on both datasets. The global token budget remains unchanged. On RealEstate10K, K=0 achieves slightly higher PSNR and nearly identical SSIM, while K=5 achieves better LPIPS.
Analysis of the dataset-varying gains. We measured visual similarity using one frozen DINOv2 ViT-S/14 descriptor per image, independently of either rendering model. Within-group similarity averages cosine similarity over distinct image pairs within the G=12, K=5 reference groups, counting repeated pairs only once. Between-group similarity compares the normalized mean descriptors of different groups.
| Dataset | Within-group similarity | Between-group similarity | AIMS K=5 PSNR gain over anchor-only LVSM |
|---|---|---|---|
| RealEstate10K | 0.974 | 0.861 | −0.693 dB |
| DL3DV | 0.859 | 0.770 | +0.253 dB |
| ScanNet | 0.751 | 0.556 | +1.021 dB |
Lower similarity indicates greater visual diversity across observations. On RealEstate10K, where within- and between-group similarities are highest, anchor-only LVSM achieves the best performance, followed by K=0 AIMS. In contrast, K=5 AIMS outperforms LVSM on the more visually diverse DL3DV and ScanNet datasets, with the largest gains observed on ScanNet. This trend suggests that integrating neighboring views becomes increasingly beneficial as visual diversity increases.
We will incorporate these experiments into the main paper to clarify this trade-off: anchor-only methods can be preferable when scene observations are highly similar, whereas AIMS benefits from integrating neighboring views when observations are more diverse. This highlights the advantage of our approach in generalizing to complex scenes with greater visual variation, without increasing the global latent-token budget.
Response to Reviewer FuZF
Thank you for the careful and constructive review.
W1 / Q1. Baseline with the same training. We agree this is the right comparison. Training LVSM (E–D) on the same inputs as AIMS is very costly, because its encoder attends jointly over the views of all groups. As an affordable alternative, we are fine-tuning LVSM (E–D) on the anchors-only setting (G = 12) with the same scenes, sampling, objective and duration. We expect it to finish within two days and will report the results here.
W2 / Q2. K = 0 and a global compressor. We are training AIMS with G = 12 and K = 0 under the same settings and will report it as soon as it finishes, which we expect within 2 days. The LVSM (E–D) run described under W1 / Q1 serves as the non-hierarchical compressor, since it compresses the same 12 anchors jointly.
Q3. Unique views used. Thank you for raising this. Using the main checkpoint (G = 12, K = 5, at most 72 slots), we varied the candidate pool N ∈ {60, 70, 80, 90} with the eight held-out targets fixed. The table shows N = 60 → 90:
With a 50% larger pool, AIMS uses about 21–22% more distinct views, while time rises by under 1.5% and memory by about 10–11%. Repeated slots fall from 23–24% to 7–8%. Quality improves on RealEstate10K and DL3DV and is roughly flat on ScanNet, which peaks at N = 80 (17.835 dB), so larger pools do not guarantee monotonic gains.
For a given setting, the number of views used is bounded by G·(K+1) = 72. This bound is a user choice: the anchor count G is designed to set how much of the collection is covered, so for larger collections the user can choose a larger G. The cost then grows with G, not with N. We will state this explicitly.