acceptodds
Under review as a conference paper at ICLR 2027

Iterate, Don't Stack: Grounded Recursive Transformers for Generalizable Novel View Synthesis

Abstract

Existing feed-forward novel view synthesis transformers apply a fixed amount of computation to render each target view per scene, although the complexity within a single scene is not distributed evenly. By fixing the number of transformer layers, they freeze the trade-off between quality and computation. We propose Universal View Synthesis Model, UVSM, capable to adaptively allocate computation within the target image. UVSM follows a unique principle: ground the scene once and recursively refine each token corresponding to image patches in parallel until convergence. A halting probability gate acting on refined tokens determines whether they need further refinement step. The halting gate acts as a dial at inference between a fast preview and a converged render. One trained model therefore replaces a family of fixed-depth models that a quality-compute curve would otherwise require. We further show that increasing the number of refinement steps in our network generalizes better than expanding the depth of a transformer network. Under the same training protocol, UVSM beats every baseline on all seven benchmarks with the highest margins on the out-of-distribution evaluations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
novel view synthesis, adaptive computation, recurrent transformers, test-time scaling, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hZMtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UVSM, an LVSM-style feed-forward transformer for novel view synthesis. A 12-layer backbone maps posed input views and target rays to a latent representation, which a weight-tied 2-layer refiner updates up to eight times. The reported results are strong, particularly on zero-shot benchmarks. The method also incorporates per-patch halting to reduce computation, although the provided implementation does not substantiate the claimed per-patch computational savings.

Critical strengths
  1. The performance is good. They have strong zero-shot results, achieving ScanNet +3.29 dB, DL3DV-even +1.65 dB over the best baseline.

  2. The broad benchmark study, architectural ablations, and supplied code are valuable.

Critical weaknesses

The computational savings from per-patch halting are not established. Fig. 1 and Sec. 7 state that only active tokens undergo further refinement, but the submitted implementation continues processing all tokens until every patch has halted. Thus, the mean exit depth reported in Fig. 5 and Table 7 does not measure executed computation. The authors should report end-to-end FLOPs and latency across halting thresholds, including a comparison with fixed-depth inference using the same checkpoint.

Questions for authors

I have one question, corresponding to the weakness above: Please clarify how halted patches are excluded from subsequent computation, reconcile this with the submitted implementation, and quantify the resulting FLOPs and latency savings.

Reviewer_u2FSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that feed-forward NVS transformers spend a fixed budget of compute on every generated target view, even though the scene complexity is unevenly distributed within an image. Also, by fixing the network depth, the tradeoff between quality and compute is also fixed. The paper then proposes a new model, UVSM, which first grounds a scene with a backbone model. The backbone uses cross-attention between the target and input tokens, PRoPE camera encoding and per-view self-attention. A refiner model is then applied up to 8 times to the target tokens. A per-patch halting head then turns the exit probabilities into a halting distribution. Baselines (LVSM encoder-decoder and decoder-only, with and without PRoPE, SVSM and iLRM) are retrained under a shared protocol on RealEstate10K and ACID. UVSM performs best on all seven benchmarks, with the largest margins in zero-shot transfer. Another claim of the paper is that adding recurrent steps improves out-of-distribution quality, while adding distinct layers does not. The third claim is that the halting threshold acts as an inference-time dial between a fast preview and a full quality render. In my opinion, the paper is well motivated and shows some promising results. I was especially intrigued by the analysis of the halting mechanism. I also have some questions and concerns about the evaluation though. Especially, as the main results highlight baselines that were retrained with small compute budget and several of the baselines seem to fail altogether. Also, the central claim of "iterate, don't stack" seems to rest on differences of about 0.3dB in rendering quality from single runs, which does not really seems significant to me. The adaptive-compute claim is also never measured in actual compute. My current assessment is a weak reject, but I am willing to discus the paper and adapt my score if I am wrong.

Critical strengths
  1. The motivation and introduction of the problem is very well done and relevant to the community in my opinion. A fixed per-frame compute budget for scenes with varying complexity across frames seems like a real limitation of current models. I also think that making the depth tunable at inference without retraining is an attractive and valuable goal.
  2. The architecture, as presented, is quite simple and elegant. UVSM has a reasonable amount of parameters and its depth comes from reusing a small block.
  3. I like that all baselines are retrained under one protocol, with the same data, effective batch size and number of iterations, and evaluated in a single harness. The intent is right, even though I have concerns about the resulting baseline quality (see below).
  4. The ablations are structured around the right questions. The depth sweeps compare distinct layers against recurrent steps, and configurations b3 and c3 are matched on the number of layer applications per forward pass.
  5. The halting analysis is the most convincing part of the paper. At equal mean depth, adaptive halting clearly beats uniform truncation. From Fig. 5, running all tokens for four steps gives roughly 26 dB on ScanNet, while adaptive halting at a mean depth of 4.1 reaches 28.02 dB.
  6. The out-of-distribution qualitative results are very nice. In Fig. 3 and App. E, UVSM preserves thin structures and avoids the doubled, semi-transparent surfaces of the baselines.
  7. Code is included in the supplementary material which is a plus.
Critical weaknesses
  1. My main concern is that the retrained baselines are representative. The protocol uses a global batch of 24 for 100K steps at 256 resolution on 3 GPUs, far below the budgets the baselines were designed for. Some baselines visibly fail: iLRM scores 23.69 dB on RealEstate10K and in Fig. 3 goes off the rails completely and SVSM reaches only 19.98 dB on ScanNet. The paper notes that absolute scores can differ from the original papers, but LVSM’s numbers are quite a bit below those originally reported. What is more curious though is that the backbone-only ablation already beats every baseline on ScanNet and roughly matches the strongest basline on RealEstate10k. So a large part of the +3.3dB margin comes from the backbone itself or from weak baselines, not from the recurrence. The effect of recurrence is only a1 to a2. I would encourage the authors to add a sanity check, for example evaluating released checkpoints of LVSM on the same benchmarks, or reproducing a baseline’s reported numbers under the original setting.
  2. The central claim of iterate, don't stack is weaker than one might think upon reading the headline. In the comparison, c3 beats b3 on zero-shot by only 0.27–0.33 dB PSNR. The backbone sweep does not support the claim that stacking layers leaves out-of-distribution quality “flat or declining” either. On LPIPS, adding backbone layers improves all three zero-shot benchmarks. I think the claim should be weakened to what is actually supported, adding reccurence gives slightly better zero-shot PSNR at matched layer applications, but worse large-context performance.
  3. All results appear to come from single runs, without seeds, standard deviations or confidence intervals. Many of the paper’s conclusions rest on differences of 0.1–0.3 dB. Without variance estimates it is not possible to tell whether these effects are real or just noise.
  4. The paper reports no FLOPs, latency or memory for UVSM at any halting threshold, nor for the baselines. All headline numbers run at full depth with the gate unused. The gate also never lets the mean depth exceed 4.68. I would like to see a quality-vs-FLOPs plot comparing the halting sweep with separately trained models.
  5. Compute is not matched against the baselines. UVSM performs 28 block applications at full depth, each block containing both cross- and self-attention, while LVSM-dec-only uses 24 layers. The paper compares parameter counts but not FLOPs or inference time, so it is unclear whether UVSM leads at equal compute.This seems at odds with the PRoPE paper, which, as far as I know, reports gains when adding PRoPE to LVSM. An implementation or tuning issue in the baseline seems just as likely as the explanation offered in Sec. 8, which is not tested either.
  6. The claim that the refiner “resolves correspondence inside a grounded latent manifold” rests on cross-attention maps for three query patches in a single scene. Quantitative support would make it much stronger, for example attention entropy, or the distance between the attention peak and the ground-truth correspondence derived from depth, averaged over a test set.
  7. Adding PRoPE to fixed-depth LVSM drops out-of-distribution performance sharply. This seems strange and not consistent with the PRoPE paper, which, as far as I know, reports gains when adding PRoPE to LVSM. An implementation or tuning issue in the baseline might be the culprit here?
Questions for authors
  1. Can you add a sanity check for the retrained baselines, for example released LVSM checkpoints evaluated on your benchmarks, or a reproduction of reported numbers under the original setting? What do you think causes iLRM’s near-total failure?
  2. How do you reconcile the “iterate, don’t stack” claim with c3 being clearly worse than b3 on RE10K-L and ACID-L, with LPIPS improving monotonically as backbone depth increases, and with the 18 → 24 layer gain in the LVSM family?
  3. Could you report mean and standard deviation over several seeds, at least for a1–a5, b3 and c3?
  4. What are the FLOPs and wall-clock latency of UVSM at each halting threshold and full depth, compared with the baselines?
  5. Why does the gate never produce a mean depth above 4.68, and is there a way to reach the range between that and full depth?
  6. Can you check the PRoPE baselines against the PRoPE paper’s reported gains?
Reviewer_8BXQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UVSM, a feed-forward novel view synthesis framework that replaces a large stack of independent Transformer layers with a grounded backbone followed by a weight-tied recursive refiner. A per-patch halting mechanism allows the refinement depth to be adjusted at inference time. The proposed framework further incorporates relative camera positional encoding, an EMA teacher, and a KL-regularized halting objective.

The paper reports improvements over several retrained baselines across seven evaluation settings, with particularly strong results on out-of-distribution datasets. It also investigates the effects of recurrent depth, standalone backbone depth, and adaptive halting.

Critical strengths
  1. Important and well-motivated problem. The paper addresses two relevant limitations of existing Transformer-based novel view synthesis methods: their fixed computational depth and their limited robustness to distribution shifts.

  2. Strong empirical results, particularly on zero-shot transfer. UVSM achieves substantial improvements over competitive baselines on ScanNet and DL3DV. For example, it improves PSNR from 25.27 dB to 28.56 dB on ScanNet compared with the strongest baseline. The qualitative results also demonstrate improvements in structure preservation and rendering fidelity.

  3. Relatively comprehensive empirical analysis. The paper includes comparisons with multiple representative methods, component ablations, depth-scaling experiments, and intermediate refinement analyses. The observation that recurrent refinement can improve out-of-distribution performance when increasing standalone depth becomes less effective is particularly interesting.

Critical weaknesses
  1. Limited methodological novelty relative to existing recurrent and adaptive-computation frameworks.

Weight-tied Transformers, per-token adaptive computation, probabilistic halting, and test-time recurrent depth have already been investigated in prior work, including Universal Transformers, PonderNet, and recurrent-depth models. The main contribution appears to be their integration into a novel view synthesis framework rather than a fundamentally new mechanism.

While this application is meaningful, the paper should more clearly identify which components are technically novel and distinguish the proposed halting objective from existing approaches.

  1. Insufficiently controlled evidence for the central "iterate rather than stack" claim.

The depth-scaling experiments in Table 6 are useful, but the compared configurations modify both backbone depth and the number of recurrent refinement steps. Consequently, the observed improvements cannot be attributed solely to weight sharing or recurrence.

A more convincing experiment would compare a weight-tied refiner against an untied refiner while keeping the backbone, effective refinement depth, and training protocol fixed. Reporting both parameter counts and computational costs would further strengthen the conclusion.

  1. The practical efficiency benefits of adaptive halting are not adequately demonstrated.

A key motivation of the paper is adaptive computation allocation. However, the experiments primarily characterize reconstruction quality as a function of average refinement depth, without reporting actual inference latency, throughput, FLOPs, or memory usage.

Since the backbone always incurs a fixed computational cost and dynamic token removal introduces additional implementation overhead, a reduction in average refinement depth does not directly translate into proportional runtime savings.

Moreover, the perceptual-quality gap between adaptive inference and full-depth inference remains noticeable in Table 7.

The paper would benefit from actual quality-versus-runtime comparisons against fixed-depth models and alternative stopping strategies.

  1. The per-patch halting mechanism and geometric interpretation require further clarification.

The method includes per-view self-attention, which introduces dependencies between target patches. It is unclear how halted patches are handled during subsequent refinement steps and whether removing them changes the attention context available to active patches.

Furthermore, the proposed geometric-locking explanation is supported mainly by qualitative attention visualizations. Sharper attention does not necessarily imply more accurate geometric correspondence. More direct correspondence or reprojection evaluations would strengthen the claimed mechanism.

Questions for authors
  1. Can the authors compare weight-tied and untied refinement blocks with an identical backbone and effective depth, ideally also matching inference FLOPs?

  2. What are the actual inference latency, FLOPs, throughput, and memory requirements of UVSM under different halting thresholds? Are the reported early-exit results obtained through genuine dynamic execution?

  3. How are halted patches treated in subsequent self-attention operations? Are their representations retained as contextual keys/values?

  4. Can the authors provide quantitative evidence that recursive refinement improves geometric correspondence accuracy, as well as statistical variability across training runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.