- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies fixed-budget frame selection for novel view synthesis given known target camera poses. It uses genetic search to optimize reconstruction quality on proxy views, then distills the selected subsets into a lightweight sequential selector through behavior cloning. The selector uses target relevance, inter-view redundancy, and image sharpness without reconstruction or rendering at inference time. Experiments across six datasets demonstrate consistent gains under fixed budgets, transfer across reconstructors, and improved quality and efficiency over full-input reconstruction with LongLRM.
Critical strengths
- The "less is more" idea in this paper is reasonable. I think it is quite similar to sparse attention. Table 11 shows higher PSNR than all external baselines across six datasets and five budgets.
- This paper presents effective search distillation and cross-reconstructor transfer.
- The paper is well written and easy to follow.
Critical weaknesses
- COVER and FisherRF rely on intermediate reconstruction states, and COVER originally adds views during reconstruction training. Section 4 does not clarify their model details. Table 5 omits their selection costs.
- Tables 5 and 8 compare 30–50 selected frames with about 130 full-input frames, while the original Long-LRM mainly trains with 32 frames. Appendix A.4 does not specify the length. So the degradation may partly come from input length mismatch rather than redundancy?
Questions for authors
- How are COVER and FisherRF adapted regarding intermediate reconstruction model? Could you report their total selection costs, including reconstruction?
- What input length was used to train the LongLRM checkpoint?
- Beyond the untrained feature aggregation in Table 10, could you evaluate a target conditioned geometric baseline that greedily applies Equation4 over the full candidate pool?
- Are there any sparse attention methods for removing redundant tokens or selecting informative ones that could be referenced or extended here? Like VSA, DSA. Have you considered extension experiments, such as training an indexer?
2 comments
Thank you for the detailed response and additional exp! I appreciate the clarification of the COVER and FisherRF pipelines, the acknowledgment of the potential input length mismatch in LongLRM, and the comparisons with the target conditioned geometric baseline and the DSA/VSA inspired selection variants. These results provide further evidence that the learned, selection conditioned strategy offers benefits beyond simple geometric coverage and one-shot selection. Overall, the response strengthens my assessment, and I will maintain my original positive score.
Response to Reviewer YZhX
Thank you for the review and the pointers to sparse attention. We will incorporate these additional results and clarifications into the revised manuscript and supplementary material.
W1/Q1. COVER and FisherRF. Both build their own 3DGS model during selection. LongLRM is used only afterwards, on the K selected frames, as for every method.
COVER starts from 10 random views and adds one view by coverage every 200 3DGS iterations.
FisherRF starts from 5 views. At each step it retrains a 3DGS and adds the frame with the highest expected information gain.
Total selection time per DL3DV scene (median), including their 3DGS:
W2/Q2. LongLRM input length. Thank you for highlighting this point. We agree that LongLRM’s training input length may affect performance: the checkpoint was trained with 32 input views, whereas our full-sequence evaluation uses approximately 130. The degradation may therefore partly reflect this input-length mismatch.
This is a central motivation for our method. Memory and computational costs constrain the number of views used during training, so feed-forward reconstructors are typically trained on substantially fewer views than a capture contains. Even at inference, LongLRM requires 17.8 GB for approximately 130 views (Table 5), while AnySplat and Wild3R run out of memory on a 40 GB GPU with the full sequence (Appendix A.4). Training imposes additional memory requirements, further restricting feasible input lengths.
Consequently, providing every captured frame is not necessarily practical or beneficial, particularly when the sequence contains redundant or low-quality observations. Our method selects an informative subset from the full capture, helping the reconstructor use its input capacity effectively while operating closer to the input lengths encountered during training. The choice of subset matters even at equal input length: at (K=30), our method achieves 22.33 dB on DL3DV, compared with 21.60 dB for Uniform.
On captures containing motion blur, our selected-frame LongLRM pipeline provides a useful alternative to standard 3DGS trained on all available input frames. Motion blur is common in casual phone videos and can degrade per-scene radiance-field optimization [Ma et al., 2022; Wang et al., 2023; Lee et al., 2024]. On 100 randomly drawn ScanNet-iPhone and ScanNet++ videos with motion blur (15 sharp test views each), we compare 3DGS (official recipe, 30k iterations, COLMAP initialization) trained on all 95 frames with LongLRM on our K frames. 3DGS also sees the 15 frames closest to the test views, which our selector may not pick. At K = 30, ours is higher (+1.27 dB on average), with higher SSIM and lower LPIPS; the margin grows at K = 40 and 50 (+1.72 dB and +1.95 dB, respectively).
Q3. Greedy Eq. 4 over the full pool. We ran it on DL3DV (PSNR / SSIM / LPIPS):
Equation 4 determines the ordering of our training labels; the subsets themselves come from reconstruction-aware genetic search. Our learned selector outperforms direct greedy selection using this geometric criterion by 0.47 dB at K = 20 and 0.27 dB at K = 30.
Q4. Sparse attention (VSA, DSA). Applied to frame selection, the two ideas become:
DSA-style: an indexer scores each candidate frame (the "token") once against the target views (the "queries"), and the top K frames are kept.
VSA-style: coarse-to-fine. Blocks of consecutive frames are scored first, then the best frames are chosen inside the best blocks.
Our selector differs in one key way: it picks frames one at a time, given those already chosen, which avoids redundant picks. We tested both variants with the same labels, network and seeds as ours (DL3DV, PSNR):
Both variants are well below ours. These experiments evaluate frame-selection adaptations inspired by DSA and VSA. With the same training labels, network and seeds, our sequential selector outperforms both the one-shot top-K and coarse-to-fine variants at K = 20 and 30. These results support selection-conditioned rescoring in our evaluated setting.
Training an indexer to match the reconstructor’s internal attention would introduce a different supervision signal and require model-specific access to attention scores. A direct full-capture implementation would also need to address the memory and computational costs of processing hundreds or thousands of frames.