acceptodds
Under review as a conference paper at ICLR 2027

Less Is More: Genetic Frame Selection for Efficient Novel View Synthesis

Abstract

Feed-forward novel view synthesis reconstructs a scene from many input images in a single forward pass, yet more views do not necessarily improve performance: redundant or poorly chosen frames increase computational cost and may even degrade reconstruction quality. We address the problem of selecting, from an already captured sequence, a fixed-size subset of input views that is most informative for reconstructing specified target viewpoints. We propose a render-free view selector that scores candidate frames based on three complementary criteria: target-view coverage, measured against observed frames that stand in for the targets, redundancy with previously selected views, and image sharpness. A lightweight scoring network then selects the most informative frames without rendering, reconstruction, or per-scene optimization at inference time. To train the selector, we distill an expensive offline search procedure in which a genetic algorithm identifies high-quality subsets by directly optimizing reconstruction performance on training scenes. The selector learns to reproduce these choices from geometric and image-level features alone. Across six datasets and multiple input budgets, our method consistently outperforms both geometric and reconstruction-aware view-selection baselines while incurring significantly lower selection costs than reconstruction-based alternatives. Moreover, carefully selected subsets can outperform feed-forward reconstruction from the full input sequence. The learned selector generalizes across diverse reconstruction paradigms (feed-forward, 3D Gaussian Splatting, and NeRF), to object-targeted reconstruction and to a cross-capture setting in which the target views come from a separate acquisition pass. More broadly, our results indicate that explicitly reasoning about target relevance and inter-view redundancy is a fundamental factor in efficient scene reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Keyframe selection, Novel View Synthesis, Sparse View Selection, Multi-View 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YZhXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fixed-budget frame selection for novel view synthesis given known target camera poses. It uses genetic search to optimize reconstruction quality on proxy views, then distills the selected subsets into a lightweight sequential selector through behavior cloning. The selector uses target relevance, inter-view redundancy, and image sharpness without reconstruction or rendering at inference time. Experiments across six datasets demonstrate consistent gains under fixed budgets, transfer across reconstructors, and improved quality and efficiency over full-input reconstruction with LongLRM.

Critical strengths
  1. The "less is more" idea in this paper is reasonable. I think it is quite similar to sparse attention. Table 11 shows higher PSNR than all external baselines across six datasets and five budgets.
  2. This paper presents effective search distillation and cross-reconstructor transfer.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. COVER and FisherRF rely on intermediate reconstruction states, and COVER originally adds views during reconstruction training. Section 4 does not clarify their model details. Table 5 omits their selection costs.
  2. Tables 5 and 8 compare 30–50 selected frames with about 130 full-input frames, while the original Long-LRM mainly trains with 32 frames. Appendix A.4 does not specify the length. So the degradation may partly come from input length mismatch rather than redundancy?
Questions for authors
  1. How are COVER and FisherRF adapted regarding intermediate reconstruction model? Could you report their total selection costs, including reconstruction?
  2. What input length was used to train the LongLRM checkpoint?
  3. Beyond the untrained feature aggregation in Table 10, could you evaluate a target conditioned geometric baseline that greedily applies Equation4 over the full candidate pool?
  4. Are there any sparse attention methods for removing redundant tokens or selecting informative ones that could be referenced or extended here? Like VSA, DSA. Have you considered extension experiments, such as training an indexer?
2 comments
AuthorsOpenReview ↗

Response to Reviewer YZhX

Thank you for the review and the pointers to sparse attention. We will incorporate these additional results and clarifications into the revised manuscript and supplementary material.

W1/Q1. COVER and FisherRF. Both build their own 3DGS model during selection. LongLRM is used only afterwards, on the K selected frames, as for every method.

  • COVER starts from 10 random views and adds one view by coverage every 200 3DGS iterations.

  • FisherRF starts from 5 views. At each step it retrains a 3DGS and adds the frame with the highest expected information gain.

Total selection time per DL3DV scene (median), including their 3DGS:

KCOVERFisherRFOurs (with appearance)
2050 s394 s0.98 s
3094 s833 s0.98 s
40143 s1,480 s0.99 s

W2/Q2. LongLRM input length. Thank you for highlighting this point. We agree that LongLRM’s training input length may affect performance: the checkpoint was trained with 32 input views, whereas our full-sequence evaluation uses approximately 130. The degradation may therefore partly reflect this input-length mismatch.

This is a central motivation for our method. Memory and computational costs constrain the number of views used during training, so feed-forward reconstructors are typically trained on substantially fewer views than a capture contains. Even at inference, LongLRM requires 17.8 GB for approximately 130 views (Table 5), while AnySplat and Wild3R run out of memory on a 40 GB GPU with the full sequence (Appendix A.4). Training imposes additional memory requirements, further restricting feasible input lengths.

Consequently, providing every captured frame is not necessarily practical or beneficial, particularly when the sequence contains redundant or low-quality observations. Our method selects an informative subset from the full capture, helping the reconstructor use its input capacity effectively while operating closer to the input lengths encountered during training. The choice of subset matters even at equal input length: at (K=30), our method achieves 22.33 dB on DL3DV, compared with 21.60 dB for Uniform.

On captures containing motion blur, our selected-frame LongLRM pipeline provides a useful alternative to standard 3DGS trained on all available input frames. Motion blur is common in casual phone videos and can degrade per-scene radiance-field optimization [Ma et al., 2022; Wang et al., 2023; Lee et al., 2024]. On 100 randomly drawn ScanNet-iPhone and ScanNet++ videos with motion blur (15 sharp test views each), we compare 3DGS (official recipe, 30k iterations, COLMAP initialization) trained on all 95 frames with LongLRM on our K frames. 3DGS also sees the 15 frames closest to the test views, which our selector may not pick. At K = 30, ours is higher (+1.27 dB on average), with higher SSIM and lower LPIPS; the margin grows at K = 40 and 50 (+1.72 dB and +1.95 dB, respectively).

Q3. Greedy Eq. 4 over the full pool. We ran it on DL3DV (PSNR / SSIM / LPIPS):

KGreedy Eq. 4Ours
2020.50 / .685 / .21420.97 / .696 / .204
3022.06 / .720 / .18322.33 / .724 / .177

Equation 4 determines the ordering of our training labels; the subsets themselves come from reconstruction-aware genetic search. Our learned selector outperforms direct greedy selection using this geometric criterion by 0.47 dB at K = 20 and 0.27 dB at K = 30.

Q4. Sparse attention (VSA, DSA). Applied to frame selection, the two ideas become:

  • DSA-style: an indexer scores each candidate frame (the "token") once against the target views (the "queries"), and the top K frames are kept.

  • VSA-style: coarse-to-fine. Blocks of consecutive frames are scored first, then the best frames are chosen inside the best blocks.

Our selector differs in one key way: it picks frames one at a time, given those already chosen, which avoids redundant picks. We tested both variants with the same labels, network and seeds as ours (DL3DV, PSNR):

MethodK = 20K = 30
DSA-style (one-shot top-K indexer)19.5221.59
VSA-style (coarse-to-fine)19.1921.26
Ours20.9722.33

Both variants are well below ours. These experiments evaluate frame-selection adaptations inspired by DSA and VSA. With the same training labels, network and seeds, our sequential selector outperforms both the one-shot top-K and coarse-to-fine variants at K = 20 and 30. These results support selection-conditioned rescoring in our evaluated setting.

Training an indexer to match the reconstructor’s internal attention would introduce a different supervision signal and require model-specific access to attention scores. A direct full-capture implementation would also need to address the memory and computational costs of processing hundreds or thousands of frames.

Thank you for the detailed response and additional exp! I appreciate the clarification of the COVER and FisherRF pipelines, the acknowledgment of the potential input length mismatch in LongLRM, and the comparisons with the target conditioned geometric baseline and the DSA/VSA inspired selection variants. These results provide further evidence that the learned, selection conditioned strategy offers benefits beyond simple geometric coverage and one-shot selection. Overall, the response strengthens my assessment, and I will maintain my original positive score.

Reviewer_YfnkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses frame selection for novel view synthesis to reduce computational cost and memory usage by selecting informative, minimally redundant views. It first employs a reconstruction-aware genetic search to identify frames that are useful for synthesizing the target views. These selections are then used to train a neural network for efficient frame selection at inference time. The paper presents sufficient experiments to evaluate the proposed method.

Critical strengths
  • The paper is well written and easy to follow.
  • It addresses frame selection, an important task in novel view synthesis that can reduce computational cost and memory usage during inference.
  • This work provides many experimental results to measure its effectiveness.
Critical weaknesses
  • The proposed approach may favor target-view rendering quality over faithful 3D reconstruction, raising the following concerns:

(1) Frame selection requires the target camera poses to be known in advance, which limits the generality of the setting. Selecting source views close to the target viewpoints also makes it unclear whether the performance gains reflect improved 3D reconstruction or primarily reliance on nearby observations (overfitting).

(2) Although reducing redundancy can improve efficiency, overlapping multi-view observations provide geometric constraints that support consistent and accurate 3D reconstruction. Removing such observations too aggressively may compromise the fidelity of the recovered scene geometry.

  • The neural network formulates frame selection as a classification task, which appears to restrict it to selecting a fixed number of views.
  • The results in Table 1 demonstrate that the proposed frame selection has limited improvement compared to the simple uniform frame sampling.
Questions for authors

Please check the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Yfnk (Part 1)

Thank you for the review. We ran new experiments on DL3DV (21 test scenes, K = 30 unless stated otherwise). We will update the paper and supplementary material.

(1) Task scope and known target poses. Our goal is novel view synthesis at specified viewpoints under a limited input budget. For example, given a video capture of an entire room, a user may wish to render a short camera path around one object. When only a small subset of the captured frames can be supplied to a feed-forward reconstructor within the available memory and computation budget, selecting observations relevant to that path is a natural part of the task.

Thus, we seek a compact set of input frames which renders higher quality frames at those viewpoints. In pose-conditioned novel view synthesis, the requested camera poses specify the views to render; our setting additionally assumes that these poses are available at frame-selection time, as with a planned rendering path, a robot trajectory, or a second capture with known poses. We use them to identify observed proxy views O; target RGB images are never accessed during selection. Target-aware selection therefore follows directly from our rendering objective.

(1)-(2) Geometry. We compared target-agnostic and target-aware methods using the same candidate pool. Target-aware baseline definitions appear in our response to E69L (W1/Q1).

Reference and scale: sparse COLMAP points triangulated from all capture frames with poses fixed (median 37k points/scene). No dense model or fitted alignment is used. we only undo its internal pose normalization, so its depths are in the units of the COLMAP model.

F-score: rendered depths with opacity > 0.5 are back-projected to 3D. Precision is the fraction of predicted points within τ of a reference point; recall is the fraction of reference points within τ of a predicted point. τ = 5% of the median reference depth seen from the targets.

Chamfer distance: the average of the two directional median nearest-neighbor distances between these point sets, normalized by the same reference depth. Medians reduce sensitivity to distant points under sparse reference coverage. Lower is better.

Region: targets = depths rendered at 15 targets, recall over the COLMAP points they see; whole scene = depths rendered at 96 evenly spaced capture frames, recall over all COLMAP points.

MethodPSNR ↑Chamfer ↓ (test views)F-score ↑ (test views)Chamfer ↓ (whole scene)F-score ↑ (whole scene)
Ours22.330.04500.5260.04280.543
Target-agnostic:
------------------
FVS21.550.04660.5200.04310.543
NeRF-Director21.460.04610.5210.04320.543
k-medoids21.330.04650.5170.04270.544
Uniform21.300.04670.5180.04300.544
DPP21.320.04700.5160.04340.540
COVER21.160.04880.5020.04550.524
Random20.500.04840.5030.04440.531
Target-aware (same O views as ours):
Uniform-O21.550.04790.5080.04520.527
FVS-O21.990.04690.5150.04450.531
NeRF-Director-O22.040.04730.5110.04500.526
COVER-O22.020.05010.4910.04770.506
MVSNet-O21.420.04810.5040.04520.522
Greedy Eq. 422.060.04820.5040.04640.515
  • Among target-aware methods, ours performs best on every reported metric. It achieves higher PSNR and better geometry both at the test views and across the whole scene. Whole-scene Chamfer is 0.0428, compared with 0.0445-0.0477 for these baselines, and F-score is 0.543, compared with 0.506-0.531.

  • At the test views, ours also leads the target-agnostic baselines. It achieves the lowest Chamfer distance (0.0450) and highest F-score (0.526), alongside the highest rendering quality (22.33 dB). This supports the effectiveness of the selected frames for our novel-view-synthesis objective.

  • Whole-scene geometry remains comparable to the strongest target-agnostic baselines under the reported sparse-reference metrics. Our Chamfer distance is 0.0428 versus the best value of 0.0427, and our F-score is 0.543 versus 0.544. Thus, the improvement at the requested viewpoints is accompanied by similar geometric quality across the scene.

Our method improves novel-view synthesis while achieving whole-scene geometric quality comparable to the strongest target-agnostic baselines. The target-aware variants have worse whole-scene geometry than their target-agnostic counterparts. Our selector achieves the best rendering quality while matching the strongest target-agnostic baselines closely.

The sparse COLMAP reference can lower absolute F-scores, since valid reconstructed points are penalized when no reference point lies nearby. All methods are evaluated against the same reference.

AuthorsOpenReview ↗

Response to Reviewer Yfnk (Part 2)

(3) Fixed number of views. The selector picks one frame at a time, given the frames already chosen; K is just the number of steps. The same network can be used for any budget, with no need of retraining. For example, on DL3DV (PSNR):

K22242628
Uniform20.6220.9021.2621.45
Best of the paper's baselines (FVS)21.0121.2821.4721.79
Best target-aware baseline (NeRF-Director-O)21.2321.5021.7621.90
Ours21.3921.7421.9822.17

Ours is first at every budget.

(4) Uniform in Table 1. Table 1 evaluates cross-capture novel view synthesis: the targets come from a held-out video, while all selectable frames come from other captures of the same scene. Uniform and k-medoids are the strongest baselines in this setting.

In this cross-capture setting, spreading frames evenly over the room is already a good strategy, even though not the best. This is why Uniform is the second strongest baseline in Table 1. Ours is still the best in every case:

  • Table 1 as in the paper: first at both budgets, +0.16 / +0.25 dB over Uniform (K = 30 / 40).

  • Uniform without the O frames: +0.36 / +0.23 dB.

  • Uniform over 10 start offsets: ours is above every offset on average, by +0.30 / +0.21 dB over their mean.

  • Standard setting (targets on the captured path, DL3DV): +0.33 to +0.79 dB over Uniform, for K = 10 to 50.

Reviewer_E69LOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of retrospective input frame selection for feed-forward novel view synthesis (NVS). Using an entire video capture can increase computational cost and degrade rendering quality due to redundant or blurry frames. To address this, the authors propose a lightweight, render-free sequential scoring network (5,505 parameters) that selects a fixed-size subset of frames based on target viewpoint relevance, redundancy, appearance similarity, and image sharpness.

The selector is trained through behavior cloning using trajectories generated by an offline genetic algorithm (GA) that optimizes rendering quality on proxy views using PSNR, SSIM, and LPIPS. Experiments across multiple datasets, different frame budgets, and reconstruction methods demonstrate improvements in rendering quality and efficiency.

Critical strengths
  1. The paper proposes to distill an expensive GA-based frame selection process into a lightweight sequential scoring network.

  2. The proposed method achieves consistent improvements across multiple datasets, frame budgets, and reconstruction methods. It also significantly reduces reconstruction time and memory usage. The zero-shot generalization to unseen datasets and different reconstruction methods is encouraging.

Critical weaknesses
  1. The proposed method explicitly uses optimization views O to guide frame selection, while the compared baselines do not appear to use the same target-region information. This may give the proposed method an advantage beyond the learned scoring network itself. Although the untrained feature-summation baseline helps evaluate the contribution of learning, comparisons with target-aware geometric selection methods are still missing. It is therefore difficult to determine how much improvement comes from the target-aware features and how much comes from the learned sequential policy.

  2. The GA produces unordered subsets, which are converted into sequential training trajectories using the camera-distance reduction criterion in Equation 4. However, multiple remaining frames may have identical distance reductions, and the paper does not explain how such ties are handled. This could introduce arbitrary ordering into the training labels and potentially affect the behavior-cloning process.

Questions for authors
  1. Can the authors provide comparisons with target-aware versions of traditional selection methods, such as COVER or Uniform, using the same optimization views O?

  2. How are ties handled in Equation 4 when multiple remaining frames have identical distance reductions? Does the choice of tie-breaking rule affect the training or final selection performance?

  3. How are camera coordinate systems aligned between the COLMAP-based target poses H and the poses predicted by unposed feed-forward reconstructors, such as AnySplat and Wild3R, in Appendix A.4?

5 comments
AuthorsOpenReview ↗

Response to Reviewer E69L

Thank you for the precise questions. The revised manuscript and appendix will include the additional findings and clarifications provided in this response.

W1/Q1. Target-aware baselines. We added six baselines. They use the same O views as guidance and pick from the same pool as ours; none may pick the O frames.

  • FVS-O. We weight each candidate by the fraction of O’s COLMAP points it observes. The first frame has the highest weight; each subsequent frame maximizes this weight multiplied by its distance to the nearest selected camera. This combines target relevance with spatial diversity.

  • Uniform-O spaces K frames evenly along the capture, weighted by the fraction of the O views' 3D points each frame sees (as in FVS-O).

  • Greedy Eq. 4 searches the whole pool. At each step it adds the frame that most lowers Eq. 4: the sum, over the O views, of the distance to the nearest chosen camera. When no frame lowers it further, it starts a new round with the remaining frames.

  • COVER-O. COVER prefers frames that see 3D points that few chosen frames have seen yet. COVER-O counts only the points the O views see.

  • NeRF-Director-O is FVS-O with NeRF-Director's distance. That distance also counts two frames as far apart when they share few 3D points.

  • MVSNet-O. MVSNet rates how well a frame pairs with a reference view: many shared 3D points, with a good angle between the two cameras. Here the O views are the reference views, and they take turns picking their best-rated frame.

DL3DV, PSNR / SSIM / LPIPS:

MethodK = 20K = 30
Uniform-O20.16 / .666 / .23721.55 / .698 / .205
FVS-O20.56 / .686 / .21721.99 / .717 / .187
Greedy Eq. 420.50 / .685 / .21422.06 / .720 / .183
COVER-O20.30 / .683 / .22422.02 / .720 / .190
NeRF-Director-O20.78 / .691 / .21222.04 / .717 / .187
MVSNet-O19.86 / .663 / .23421.42 / .701 / .197
Ours20.97 / .696 / .20422.33 / .724 / .177

With the same target information, ours is best on all metrics.

W2/Q2. Ties in Eq. 4. Eq. 4 determines the order of frames within each training subset. Once the frames already placed in the sequence provide the nearest camera to every O view, adding any remaining frame leaves the objective unchanged. These remaining frames therefore tie. We place the earliest-captured frame next.

To test sensitivity to this choice, we retrained the selector using two alternative tie-breaking rules: choose the latest-captured frame, or choose randomly among tied frames. We kept the training subsets and all other settings unchanged. PSNR on DL3DV:

KEarliest at ties (ours)Latest at tiesRandom at ties
2020.9720.9420.97
3022.3322.3222.31

The results are nearly identical: the differences are at most 0.03 dB. The tie rule has a negligible effect in these experiments.

Q3. AnySplat/Wild3R alignment. AnySplat and Wild3R predict scenes in their own coordinate system, while our target poses are expressed in COLMAP coordinates. Using only the K input cameras’ predicted and COLMAP poses, we estimate one global similarity transform: rotation from their orientations, and scale and translation from their centers. We apply this transform to every target pose before rendering. The alignment requires no iterative refinement and uses no target poses or images to estimate the transform. Target RGB images are used only for evaluation.

Thank you for your rebuttal. I now have a better understanding of how GA selects frames. However, I still have questions about optimization views O.

  1. Line 375 states: "Baselines may select from both the candidate pool P and the optimization views O, whereas our method selects only from P." I reckon this is reasonable during training, since O is used to evaluate the selected subsets. But during inference, since O is used to guide the selection of frames from P, why can't O be used as input for reconstruction? What practical application scenario motivates this restriction?

  2. Based on question 1, I would appreciate two experiments. The first is on the DL3DV dataset, where frames from O can also be selected for reconstruction. The second experiment is the performance without using the target-related features provided by O.

I also share the concern raised in YZhX's Weakness 2. While the experiments demonstrate the benefits of frame selection, they do not isolate the cause of the performance degradation when using approximately 130 frames. Could the authors please train or fine-tune the reconstruction model using approximately 130 frames of input (or another input length) and compare the performance of using all 130 frames versus selecting 30 frames?

Comments on the paper

1 comment
AuthorsOpenReview ↗

ICLR 2027 rebuttal - Submission 6415

General response

We thank the reviewers for their constructive feedback, and we appreciate that they found the paper clear, the problem important, and the results consistent across datasets, budgets, and reconstructors. Below we summarize the new experiments we ran on DL3DV (21 test scenes) to address their main questions, and we answer each reviewer in detail afterward:

  1. Target-aware baselines (E69L, YZhX): six baselines, including target-aware Uniform, FVS and COVER, with the same optimization views O and the same candidate pool as ours. Ours beats all six.

  2. Geometry (Yfnk): target-aware baselines have worse geometry than ours. Our Chamfer distance and point-cloud F-score are comparable to coverage-based selection.

  3. Ties in Eq. 4 (E69L): other tie rules change PSNR by at most 0.03 dB.

  4. Cost of COVER and FisherRF, including their 3DGS (YZhX): 94 s and 833 s per scene at K = 30, against 0.98 s for ours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.