acceptodds
Under review as a conference paper at ICLR 2027

PointWeave: Feed-Forward View Synthesis with Explicit Point Interface

Abstract

We introduce PointWeave, a feed-forward novel-view renderer that uses explicit 3D points to guide learned appearance reconstruction. Existing 3DGS-based methods render explicit primitives efficiently, but recovering fine detail can require many primitives. Geometric-free methods offer flexible and scalable appearance modeling, but can struggle to maintain geometric consistency under unseen camera transformations. In contrast, PointWeave combines explicit geometry with learned image synthesis. It predicts 3D points with appearance features from input views, and uses learned ray-to-point attention to aggregate their features for each target ray. The resulting feature map is combined with colors retrieved from the input views using the inferred geometry and passed through a decoder to reconstruct the target image. On RealEstate10K, PointWeave achieves 31.38 dB PSNR with 8,192 points, outperforming recent 3DGS-based methods by 1.6 dB while using fewer primitives. It matches the performance of state-of-the-art geometric-free methods with up to 4× faster rendering and substantially better geometric consistency under camera roll, field-of-view, pixel-aspect and world-scale changes. It also outperforms the baselines in zero-shot transfer to unseen datasets, with 0.54–2.02 dB PSNR gains on ACID, DL3DV, ScanNet++, and DTU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feedforward 3d reconstruction, novel view synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Wiu2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PointWeave presents a feed-forward novel-view synthesis method that combines explicit sparse 3D points with a learned image renderer. The method predicts points and appearance features from the input views, uses geometry-aware ray-to-point attention to gather features for each target ray, retrieves RGB evidence from context images, and decodes the final target image. The main idea is to use geometry for correspondence while leaving fine appearance modeling to learned features and the image decoder. The results are strong overall. On RE10K, PointWeave reaches 31.38 dB PSNR with only 8,192 points, clearly outperforming the tested 3DGS baselines and matching LagerNVS in PSNR, while rendering faster. The paper also reports good cross-dataset generalization and robustness to camera transformations.

Critical strengths
  1. The method is relatively simple and practical. The predicted point set can be reused across target views, and the ray-to-point formulation gives the renderer an explicit geometric interface.

  2. The quantitative results are quite strong. In particular, the gap over 3DGS-based methods on RE10K and DL3DV is substantial, while using far fewer primitives.

  3. The evaluation is broader than a standard NVS paper. Besides in-domain results, the authors include zero-shot transfer, camera transformation tests, runtime comparisons, scene editing, and additional point-density experiments.

Critical weaknesses
  1. My main concern is novelty. Explicit point geometry plus learned point features and a neural image decoder has a fairly long history. Neural Point-Based Graphics already combines point descriptors with a neural renderer. Pointersect also performs local point-to-ray reasoning using ray-centered geometry. More importantly, FWD predicts depth-based point clouds, attaches latent and view-dependent features, fuses multi-view information, and applies a neural refinement renderer. I could not find a discussion of FWD in the paper. The specific PointWeave design is different, but the paper needs a much sharper explanation of what is fundamentally new beyond these earlier point-based neural rendering pipelines.

  2. The ablation study does not sufficiently isolate the main technical contribution. The most important component is ray-to-point attention, but there is no convincing comparison against simple projection/splatting, distance-based weighting, or another lightweight point aggregation scheme. I would also like ablations of the ray-relative coordinates, view-dependent feature basis (A_j), background token, and the five confidence/reliability inputs.

  3. There is some backbone confounding. PointWeave starts from a strong ReSplat/DepthSplat reconstruction backbone rather than learning the representation from scratch. A controlled experiment with the same point backbone but different rendering heads would make it much clearer where the gain comes from.

  4. The “few primitives” argument is not fully established by the 131k-point experiment. The authors themselves note that the dense variant changes the neighborhoods seen by a backbone pretrained with sparse points. So this experiment does not cleanly show that dense points are unnecessary; it may simply show that the pretrained backbone does not transfer well to a different sampling density.

Questions for authors
  1. How do the authors distinguish PointWeave from FWD and earlier neural point rendering methods beyond the use of ray-to-point attention? A direct discussion or experimental comparison would help a lot.

  2. What happens if the proposed attention is replaced by simple kNN weighting, projected feature splatting, or an MLP over the nearest point? This seems like the most important missing ablation.

  3. Can the authors separately ablate the view-dependent feature basis, ray-relative coordinates, background slot, five reliability features, and context RGB retrieval?

4 comments

Thank you for the review! We are glad you find our method practical and our results strong with far fewer primitives. We address your concerns below.

Q1. Comparison with other neural point rendering methods

Thank you for pointing us to FWD, we will cite and discuss it in the revision. We do not claim "points + learned features + neural decoder" as new (Sec. 2 places PointWeave in this family). The differences are where the points come from and how pixel colours are predicted from them (table below). In FWD, a fixed rasterizer decides which points reach a pixel and with what weight. In our method, these weights are a learned, non-linear function of each point's position relative to the ray. As shown below, ours achieves better rendering quality than these baselines.

Where the points come fromHow pixel colors are predictedPer-scene fitting
NPBGgivenz-buffer rasterizationyes
Pointersectgiventransformer per ray, blends input point coloursno
FWDpredicted, one per input pixelfixed splatting per view, then cross-view fusionno
PointWeavepredicted, one per 4×4 patchlearned ray-to-point attention over neighbouring points, plus colour lookup in the inputsno

We evaluate these baselines on DTU with FWD's protocol. NPBG requires per-scene fitting, so we fit one model per test scene on its three input views. We also include NPBG++, which is a follow-up to NPBG but does generalize in a feedforward manner, for a fairer comparison. The three FWD variants are trained on DTU, while Pointersect and our zero-shot model are not. We also report our model fine-tuned on DTU for 10k steps with batch size = 4 for a fair comparison.

MethodTrained on DTUPoint cloudPSNR ↑SSIM ↑LPIPS ↓
FWD-Uyespredicted17.42.598.341
FWDyespredicted (with depth supervision during training)20.15.721.259
FWD-Dyessensor depth as input at test time (with depth supervision during training)21.98.791.208
NPBGyes, per sceneCOLMAP12.02.545.341
NPBG++noCOLMAP14.39.586.493
PointersectnoCOLMAP15.05.565.434
PointWeave, zero-shotnoours15.58.614.406
PointWeave, fine-tuned on DTUyesours22.54.799.239

Q2.Additional ablation studies

We show the ablation results below, following the same ablation protocol in Table 6. We will add the ablation studies in the camera-ready version.

ItemVariantΔ PSNRΔ SSIMΔ LPIPS
Reference (ours)30.82.921.098
aray-relative → world coordinates−1.720−.0327+.0453
bwithout AjA_j-0.206−.0045+.0018
cwithout background token−0.017−.0001.0000
dwithout the five per-pixel values-0.108-.0002+.0008
ewithout context retrieval−0.302−.0004+.0009
fattention → kNN weighting (fixed inverse-distance weights)-1.049−.0262+.0224
gattention → projected feature splatting−2.073−.0401+.0569
hattention → MLP over the nearest point−0.478−.0072+.0059

Q3. Ablation of different rendering heads

The rendering head consists of the ray-to-point attention and the decoder. Table 6 in the main paper compares different decoders, and the table below compares alternatives to the attention, with the decoder fixed.

VariantΔ PSNRΔ SSIMΔ LPIPS
Ray-to-point attention (ours)30.82.921.098
Fixed inverse-distance weights-1.049−.0262+.0224
Projected feature splatting−2.073−.0401+.0569
MLP over the nearest point to the ray−0.478−.0072+.0059

Q4. On the number of primitives

We would like to clarify. The "fewer points" claim refers to Table 1: with 8,192 points we obtain better quality than 3DGS-based methods that use 47k–393k Gaussians. The 131k-point run in the appendix only asks whether more points help our method further. There is no density mismatch: ReSplat and our method both predict depth at the same resolution and run the point transformer at the same point density. ReSplat ends with 131k Gaussians, 16 times our count, because it predicts 16 Gaussians per output point from the point transformer, while we use the output points directly.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and additional experiments. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns or if any questions remain.

Thank you for your time and consideration!

Thanks for detailed responses. My concerns are well addressed. I raise my score from weak reject to weak accept.

Thank you for your positive feedback and for raising your score! We are glad that our responses have addressed your concerns, and we sincerely appreciate your time and consideration.

Reviewer_xMxxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PointWeave, a feed-forward novel-view synthesis method that combines sparse 3D points, ray-to-point attention, and learned image decoding. It achieves competitive rendering quality and efficiency with relatively few primitives.

这篇论文提出的 PointWeave 方法具有一定新颖性,将稀疏 3D 点、ray-to-point attention 和学习式图像解码结合起来,为使用大量 Gaussian 表示复杂外观提供了一种有趣的替代方案。论文整体表达清晰,实验结果也比较突出,尤其是在渲染质量、效率、跨数据集泛化能力以及对相机变换的鲁棒性方面表现较好,因此我认为该论文具有一定的研究价值。

然而,我对论文的实验完整性仍存在一些担忧。自由视角渲染的展示还不够充分;其次,该方法依赖 ray-to-point attention 估计深度并从输入视图检索颜色,在遮挡边界附近,估计深度可能落在不同表面之间,从而产生错误的颜色检索、颜色渗漏或重影。论文目前缺乏针对遮挡边界和新可见区域的专门定量评估,我认为这一点需要进一步验证。最后,DL3DV 的全图评估包含输入的 context views,我觉得作者如果能够在相同 256×448 分辨率下单独报告 held-out target views 的结果,会更加清楚地反映真正的新视角合成性能。

总体而言,我认为 PointWeave 的方法设计有趣,实验结果也具有较强的竞争力。虽然上述问题影响了我对实验完整性的判断,但我认为这些问题总体上属于可以通过进一步实验和分析解决的问题。因此,我给出 3 分。

Critical strengths
  1. The method is novel and well-motivated. Using sparse 3D geometry to guide neural rendering provides an interesting alternative to representing complex appearance with a large number of Gaussians.
  2. The experimental results are strong, particularly in rendering quality, efficiency, cross-dataset generalization, and robustness to camera transformations.
Critical weaknesses
  1. The free-viewpoint rendering results could be more extensive. A continuous camera walkthrough with larger viewpoint changes and complex occlusions would help demonstrate the visual stability of the method.
  2. The method estimates depth through ray-to-point attention and uses it to retrieve colors from input views. Near occlusion boundaries, the estimated depth may fall between different surfaces, potentially causing incorrect color retrieval, bleeding, or ghosting. The paper lacks dedicated evaluations of these regions. Region-specific PSNR/LPIPS and visual error maps would be helpful.
  3. The full-frame evaluation on DL3DV includes input context views. Reporting results separately on held-out target views at the same 256×448 resolution would provide a clearer assessment of novel-view synthesis performance.
Questions for authors
  1. How does the method handle cases where the attention-weighted depth falls between different surfaces, particularly near occlusion boundaries?
  2. Could the authors provide quantitative results for occlusion boundaries and newly visible regions?
5 comments

Thank you for the review! We are glad you find our method novel and well motivated, and that the results show strong rendering quality, efficiency, cross-dataset generalization, and robustness to camera transformations. We address your questions below.

Q1. Free-viewpoint rendering

Please find two free-viewpoint videos (orbit.mp4, extrap.mp4) in the updated supplementary materials. Each video shows three RealEstate10K scenes (2 input views, top row) and three DL3DV scenes (6 input views, bottom row). In the extrapolation video, the camera moves along the straight line through the input cameras and continues past them by 25% of the baseline beyond each end. In the orbit video, the camera sweeps ±15° in yaw and ±5° in pitch around the scene centre.

Q2. Attention-weighted depth falls between different surfaces

This can happen. Three parts of the design limit its effect:

  1. Attention is usually sparse. The attention weights are usually concentrated on one or two points (largest weight 0.62 on average), so the depth usually lands on one surface.
  2. Occlusion signal. In the context retrieval, each retrieved colour comes with the log-depth difference to the input view's own depth at that pixel. For an in-between location this difference is non-zero, so the decoder can discount the colour.
  3. Decoder. A ray that splits its attention between two surfaces still carries the features of both, and the decoder can resolve them with neighbouring pixels.

Q3. Evaluation at boundary regions and newly visible regions

We evaluate on ScanNet++ (zero-shot from RE10K), where iPhone LiDAR depth defines both types of region. A pixel is in the boundary band if neighbouring LiDAR depths differ by more than 10 %. A pixel is observed if its 3D point agrees with the LiDAR depth of at least one input view, otherwise it is newly visible. As shown below, our method achieves higher PSNR than ReSplat and LagerNVS at both boundary and newly visible regions.

Boundary vs interior:

Region (share of pixels)MethodPSNR ↑SSIM ↑LPIPS ↓
Boundary band (4.8 %)ReSplat18.12.627.247
LagerNVS16.49.519.259
PointWeave18.39.624.250
Interior (95.2 %)ReSplat20.65.784.240
LagerNVS20.86.747.229
PointWeave22.38.806.230

Observed vs newly visible regions:

Region (share of pixels)MethodPSNR ↑SSIM ↑LPIPS ↓
Observed (75.6 %)ReSplat23.86.819.193
LagerNVS22.16.765.193
PointWeave24.48.828.190
Newly visible (17.9 %)ReSplat16.19.626.414
LagerNVS17.70.640.351
PointWeave18.25.679.378

Q4. Held-out view only results for full-frame DL3DV

Below is the full-frame comparison of Table 3 with the input views excluded from the targets. As shown we outperform all baselines.

Method2 views (PSNR / SSIM / LPIPS)4 views6 views
MVSplat17.41 / .520 / .40721.45 / .712 / .23822.74 / .767 / .197
DepthSplat19.13 / .611 / .31422.93 / .775 / .18324.05 / .818 / .151
TokenGS19.06 / .588 / .44522.85 / .736 / .32723.59 / .758 / .310
ReSplat (no refinement)18.54 / .575 / .40123.67 / .779 / .21425.55 / .834 / .166
ReSplat (2 refinements)19.19 / .591 / .38624.87 / .804 / .19326.90 / .859 / .144
CLiFT18.25 / .557 / .38123.50 / .757 / .20225.08 / .804 / .161
PointWeave (ours)22.26 / .719 / .27427.34 / .866 / .13629.67 / .910 / .097

Thank you for the detailed rebuttal and the additional experiments. The held-out view evaluation and region-specific quantitative results address most of my concerns.

I have one remaining question regarding occlusion boundaries. PointWeave achieves slightly higher PSNR than ReSplat in boundary regions (18.39 vs. 18.12), but slightly worse SSIM (0.624 vs. 0.627) and LPIPS (0.250 vs. 0.247).

Could the authors explain this discrepancy? In particular, does it suggest that color bleeding or ghosting artifacts may still occur near occlusion boundaries?

Thank you for the follow-up!

The reported ReSplat results include two test-time refinement steps. Without refinement, ReSplat achieves 18.01 / 0.612 / 0.254 (PSNR / SSIM / LPIPS) in the boundary regions. In comparison, our method outperforms across all three metrics, with improvements of +0.38 dB PSNR, +0.012 SSIM, and −0.004 LPIPS.

To further investigate colour bleeding and ghosting artifacts, we measure the percentage of predicted depths that fall into the empty space between foreground and background surfaces at occlusion boundaries. Specifically, at pixels in the boundary regions, we classify a predicted depth as a floater if it lies more than 5% behind the foreground surface and more than 5% in front of the background surface.

Floaters in boundary regions ↓PointWeaveReSplat (no refinement)ReSplat (2 refinement steps)
Backbone depth prediction8.1%10.1%10.1%
Depth from attention (ours) or Gaussians (ReSplat)7.5%8.6%7.8%

As shown in the table, while some floaters remain in both methods (about 8–10 % of boundary pixels), our method consistently achieves a lower floater rate, even compared with ReSplat after test-time refinement.

Reviewer_5LcyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method for posed feed-forward novel view synthesis that is based on depth map un-projection from context views, followed by a "ray-to-point attention" for rays from the target view that weighs point features. Further, the authors propose to un-project the predicted depth (obtained via softmax weights of the mentioned attention) along the ray and to project it back into context views to assemble additional appearance information. To predict novel views with a 2D image decoder, the collected features and appearance information are processed and the novel RGB view is predicted.

Critical strengths

The paper proposes a simple but effective baseline method for feed-forward novel view synthesis from few context views.

The proposed method does not require extensive training and is fast at both encoding and rendering time.

Critical weaknesses

The paper as-is does not provide a good justification for some of the introduced components. For example, view-dependent color modeling or the general "attention" and use of point features are not really ablated. Please see my questions for details.

The quantitative results are good but not state-of-the-art quality and, e.g., a comparison against implicit baselines on consistency seems a bit unfair given the partly explicit nature of the proposed method. When comparing against explicit (3DGS-based) methods, as shown in the suppl. materials, one can see that the method is naturally not more robust than these baselines.

While the authors also mention the use of a more performant geometry foundation model as backbone in the future work, I think that is fair to ask for a comparison with e.g. VGGT-Omega initialization, as the proposed method seems to be pretty sample-efficient and this would directly clarify whether LagerNVS's better performance can actually be attributed to a stronger prior initialization or whether there is another reason.

Questions for authors
  1. It is unclear to me where the point features are coming from. Does the pipeline include the 3D kNN attention of ReSplat or not? If yes, how / at what moment is the pixel stride in depth predictions applied (before or after kNN-aggregation)? If not, please elaborate why you didn't choose to add this step to the pipeline as 3D consolidation of point features intuitively could get better outcomes.
  2. Could you please compare the "predicted" depth through the softmax weights with actual depth maps for some frames? Generally, a nice addition to the supplemental materials would be to show some qualitative examples for the 5 per-pixel values that are fed additionally to Z, as well as the "retrieved" colors from the context views that are given in W.
  3. Can you please add peak memory measurements to the tables?
  4. The paper requires more ablations for the method design:
  • What is the effect of view-dependent appearance modeling that is proposed? (predicting and using AjA_j)? How much does the color actually change for the same observed 3D point from different views when on e.g. a reflective surface? How does performance change when not using this explicit AjA_j parameterization but still including njTdin_j^Td_i in the point features?
  • What happens to the training without the additional background slot?
  • What are the results of an even simpler baseline, where we keep unprojection to 3D points but then assume that ti0t_i^0 is the depth along a ray and use only the appearance branch with this approximate depth with the learned decoder? Basically the setting where we do not use the feature branch / ray-point-attention at all.
  1. It would be interesting to see some statistics of the "attention" weights and how the weights correlate with sideways offset from rays and / or tij−ti0t_{ij} - t_i^0. Is there some clear pattern emerging? For example, is the nearest point (depth or pixel distance) always the highest weighted in the softmax? How well can one fit a linear map between these properties and the resulting weights / logits?
  2. How well does the method compare against baselines with larger baselines in a two-view setting?
4 comments

Thank you for the detailed suggestions! We are glad you find our method simple, effective and fast. We address each point below.

Part (1/2)

Q1. Additional ablation studies

We show the ablation results below, following the same ablation protocol in Table 6. We will add the ablation studies in the camera-ready version.

ItemVariantΔ PSNRΔ SSIMΔ LPIPS
Reference (ours)30.82.921.098
awithout AjA_j-0.206−.0045+.0018
bwithout background token−0.017−.0001.0000
cassuming depth at ti0t_i^0-6.903-.1680+.1914

We measure how much the colour of the same 3D point changes between an input view and the target view on ScanNet++, using ground truth LiDAR correspondences. The table reports the median change in 8-bit levels (0-255), grouped by the change in viewing angle.

Viewing-angle change0–5°5–10°10–20°20–40°Overall median
Median colour change5.19.210.424.26.4

Q2. Consistency comparison with implicit method

The use of explicit geometry for better consistency is one of the key motivations for our design (Line 52-58 in the paper). The comparison is meant to show where our method stands: by adding an explicit 3D layer to an implicit renderer, we obtain much better rendering quality than purely explicit (3DGS-based) methods, and better 3D consistency and 4× faster rendering than implicit methods.

Q3. Consistency comparison with explicit geometry (3DGS-based) methods

3DGS-based methods render with an analytic rasterizer, so they are naturally consistent under camera transformations, whereas our method uses a learned decoder. Table 7 shows that the resulting gap is small and easy to close. Without any augmentation, our method already exceeds MVSplat and DepthSplat under field-of-view and pixel-aspect changes. After a 500-step augmentation fine-tune, it matches ReSplat under roll and pixel-aspect changes and exceeds it under field-of-view changes (27.84 / 25.42 / 26.98 vs. 27.85 / 24.70 / 27.03 dB). Our method thus reaches the consistency of explicit geometry methods at higher rendering quality and with 5-40x fewer primitives (Table 1).

Q4. Using foundation model as backbone

Following this suggestion, we tested the VGGT backbone used by LagerNVS during the rebuttal. We replaced ReSplat's cost-volume depth predictor with LagerNVS's camera-conditioned VGGT, kept our point transformer, attention and decoder, and fine-tuned jointly for 32k steps at batch size 16. The VGGT weights are initialized from LagerNVS and the rest from our trained RE10K and DL3DV models. With this backbone our method exceeds LagerNVS in PSNR and SSIM on RE10K and on DL3DV with four and six input views, and matches it with two views (tables below). These results demonstrate that our ray-to-point attention and decoder are not specific to ReSplat and can benefit from stronger geometric backbones. These runs are short (32k steps), so we will report fully trained models in the camera-ready version.

RE10K, 2 input views, 256²

MethodPSNR ↑SSIM ↑LPIPS ↓
LagerNVS31.39.926.078
Ours (ReSplat backbone, Table 1)31.38.927.093
Ours (VGGT backbone, fine-tuned 32k steps)31.63.929.091

DL3DV, square-input protocol, 256² (PSNR ↑ / SSIM ↑ / LPIPS ↓)

Method2 views4 views6 views
LagerNVS23.30 / .739 / .20327.56 / .869 / .09529.45 / .904 / .068
Ours (ReSplat backbone, Table 3)21.95 / .696 / .28527.42 / .861 / .13529.96 / .911 / .090
Ours (VGGT backbone, fine-tuned 32k steps)23.31 / .750 / .24228.32 / .884 / .11930.54 / .920 / .086

Q5. How to get point features

Yes, we use ReSplat's point transformer with 3D kNN attention (Appendix A):

  1. The backbone predicts a depth map per input view.
  2. The depth map is resized to 1/4 resolution, and each pixel is lifted to one point. Each point is associated with a 3D location pjp_j and a feature vector fjf_j extracted from the input images (Eq. 1 in ReSplat)
  3. The point transformer refines these points: 12 blocks alternating attention over the 16 nearest 3D neighbours and global attention (Eq. 2 in ReSplat)
  4. Two linear layers take the refined feature of each point output by the point transformer (qjq_j in Sec. 3.1) and map it to the base feature bjb_j and the matrix AjA_j.

ReSplat's Gaussian heads and recurrent refinement are not used. We will move this summary from App. A into Sec. 3.1.

Part(2/2)

Q6. Compare predicted depth with actual depth

We compare our depth with ReSplat's on ScanNet++ (zero-shot from RE10K), with iPhone LiDAR as ground truth. Our depth is the attention-weighted depth and is never supervised. The median-scaled columns align each prediction to the ground-truth median and therefore measure shape only.

MethodAbsRel ↓RMSE (m) ↓δ<1.25 ↑AbsRel, median-scaled ↓δ<1.25, median-scaled ↑
ReSplat (no refinement)0.2200.57776.0 %0.13184.0 %
ReSplat (2 refinements)0.2160.56476.4 %0.12684.3 %
PointWeave (attention depth)0.2440.56578.9 %0.11286.5 %

Q7. Examples of the 5 per-pixel values and retrieved colors

Thank you for your suggestion! Please find the qualitative examples in Appendix F and G in the updated supplementary materials.

Q8. Peak memory usage

We show the peak GPU memory usage for each method below, evaluated on RE10K with two input views at 256² and one target view per render, matching the setting of Table 1.

Encode peak is the GPU memory needed to build the scene representation from the context views, render peak is the memory needed to render one target view from that representation, and inference peak is the larger of the two, since the two stages run one after the other.

MethodEncode peak (GiB)Render peak, 1 view (GiB)Inference peak (GiB)
pixelSplat2.850.932.85
MVSplat1.120.211.12
DepthSplat2.541.432.54
TokenGS (1,024 tokens)1.741.681.74
TokenGS (4,096 tokens)1.060.921.06
SplatWeaver4.282.804.28
ReSplat (no refinement)1.930.931.93
ReSplat (2 refinements)3.910.993.91
LVSM (decoder-only)—1.051.05
CLiFT0.450.610.61
LagerNVS6.044.006.04
PointWeave, K=20K=202.041.112.04
PointWeave, K=5K=52.041.102.04

Q9. Attention statistics

Thank you for the suggestion! We sampled 641,568 target rays from the RE10K test set (K=20K=20), and here are our findings:

  1. The attention is concentrated. The largest weight of a ray is 0.62 on average.
  2. The highest-weighted point is usually the nearest in pixel distance (90.4 % chance).
  3. Depth decides between nearby candidates. When two candidates lie at a similar pixel distance (both within 3 px) and their depths differ by more than 10 %, the one in front receives the larger weight in 74.9 % of the cases (82.4 % for a difference above 20 %).
  4. Neither dependence is linear. A linear fit of the logits gives R² 0.30 from pixel distance and 0.004 from depth offset.

We also include the plots of attention weights against the two distances with more detailed discussion in Appendix H in the updated supplementary.

Q10. Two-view performance with larger gap

Following NoPoSplat, we group the two-view RE10K test pairs by the overlap of the two input images, and compare with LagerNVS and ReSplat under the evaluation protocol of Table 1. Our method is ahead of ReSplat in every bin; LagerNVS is ahead at small and medium overlap and our method is ahead at large overlap.

MethodSmall overlap (PSNR / SSIM / LPIPS)Medium overlapLarge overlap
ReSplat (2 refinements)24.63 / .828 / .17427.52 / .876 / .13030.63 / .920 / .093
LagerNVS26.17 / .853 / .13829.25 / .897 / .10232.32 / .933 / .073
PointWeave25.66 / .848 / .16829.03 / .894 / .12232.51 / .935 / .085

Thank you for your detailed responses. My questions and concerns have been addressed. I am willing to raise my score to a weak accept. When open-sourcing the work, please consider sharing the VGGT-based model checkpoint for full reproducibility and the best performance.

Thank you for your positive feedback and for raising your score! We appreciate your constructive comments and suggestions throughout the review process. We will also make sure to release the VGGT-based model checkpoint alongside our code to support full reproducibility and the best performance.

Reviewer_q6HnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PointWeave predicts sparse 3D points and appearance features from an input image with camera parameters, aggregates target pixel features through geometry-guided ray-to-point attention, and generates new perspectives by combining input colors retrieved according to the predicted geometry. Its appearance representation separates cacheable point features from lightweight perspective-related adjustments. The paper validates image quality, rendering speed, and geometric consistency on RE10K, DL3DV, four cross-dataset test sets, and camera transform benchmarks.

Critical strengths
  1. The method presents an efficient engineering design between explicit geometry and learned image reconstruction. Compared to scene-by-scene optimized point rendering, single-pass feedforward prediction and reusable point appearances reduce the processing cost of new scenes and perspectives; local geometry is used to select evidence, while details are provided jointly by learned features and the input image. In Table 1, 8,192 points achieved 31.38 dB, and the version with K=5 still achieved 31.32 dB, with a single-view rendering time of 7.0 ms, demonstrating a valuable trade-off between quality and speed.

  2. Validation is not limited to the standard test set. The PSNR of the four cross-dataset tests in Table 4 is superior to the listed baselines; Table 5 shows that it surpasses PVSM across four camera or scale transformation classes without using additional transform enhancements. Table 6 also reveals the trade-off between improved reconstruction quality from neighborhood image decoding and a slight reduction in transform consistency, making the method's gains and limitations easier to understand.

Critical weaknesses
  1. The criteria and conditions for "matching the quality of the optimal method" need to be more explicitly defined. In Table 1, the PSNR is nearly identical to that of LagerNVS, yet the LPIPS score is 0.093 compared to 0.078 for the latter. Under the square-input conditions in Table 3, the dual-view PSNR is 21.95 dB—lower than the 23.30 dB achieved by the baseline—and the LPIPS scores for two, four, and six views also lag behind. While the authors acknowledge the limitations regarding sparse views and occluded regions, these results suggest that the method’s primary advantage lies in a trade-off between speed, PSNR, and transformation stability, rather than consistently maintaining equivalent perceptual quality. This constrains the conclusions that can be drawn regarding the method's applicability to few-view scenarios.

  2. The comparison in Table 2, which claims "five times fewer training samples," cannot be solely attributed to the data efficiency of the method itself. Section 4.1 explicitly utilizes a pre-trained ReSplat encoder and depth predictor; the original ReSplat paper also includes training on the RE10K dataset, whereas LagerNVS employs a different 3D pre-trained initialization. The current table only accounts for the steps multiplied by the batch size during the subsequent training phase, failing to incorporate the inherited training history. Consequently, while the 9.6M versus 51.2M figure indicates a smaller training budget for this specific phase, it does not directly support the claim that the entire training pipeline is five times more sample-efficient. This claim should be qualified, and the training costs associated with the initialization models should be explicitly stated.

Questions for authors

Does the count in Table 2 cover only the current training phase for both PointWeave and the baselines? What data and training steps were involved in the ReSplat initialization used, and how would the conclusions regarding sample efficiency change if this prior training were factored in?

2 comments

Thank you for the review! We are glad you find our quality–speed trade-off valuable and our evaluation broad. We address your concerns below.

Q1. Matching the quality of LagerNVS?

We agree that LagerNVS performs better with two input views in Table 3 when our method uses the ReSplat backbone. During the rebuttal, we gained access to GPUs with enough memory to test the VGGT backbone used by LagerNVS, as suggested by Reviewer 5Lcy. We replaced ReSplat's cost-volume depth predictor with LagerNVS's camera-conditioned VGGT, kept our point transformer, attention and decoder, and fine-tune jointly for 32k steps at batch size 16. The VGGT weights are initialized from LagerNVS and the rest from our trained RE10K and DL3DV models. With this backbone our method exceeds LagerNVS in PSNR and SSIM on RE10K and on DL3DV with four and six input views, and matches it with two views (tables below). These results demonstrate that our ray-to-point attention and decoder are not specific to ReSplat and can benefit from stronger geometric backbones. These runs are short (32k steps), so we will report fully trained models in the camera-ready version.

RE10K, 2 input views, 256², matching Table 1

MethodPSNR ↑SSIM ↑LPIPS ↓
LagerNVS31.39.926.078
Ours (ReSplat backbone, Table 1)31.38.927.093
Ours (VGGT backbone, fine-tuned 32k steps)31.63.929.091

DL3DV, square-input protocol, 256² (PSNR ↑ / SSIM ↑ / LPIPS ↓), matching Table 3

Method2 views4 views6 views
LagerNVS23.30 / .739 / .20327.56 / .869 / .09529.45 / .904 / .068
Ours (ReSplat backbone, Table 3)21.95 / .696 / .28527.42 / .861 / .13529.96 / .911 / .090
Ours (VGGT backbone, fine-tuned 32k steps)23.31 / .750 / .24228.32 / .884 / .11930.54 / .920 / .086

We do notice that LagerNVS keeps a better LPIPS in general, and we think part of the gap is the perceptual loss weight. LagerNVS uses 0.5 as the perceptual loss weight, which is 10 times larger than the weight we used (0.05). We finetuned our model in Table 1 for 30k steps with a perceptual loss weight 0.5, the LPIPS goes from 0.093 to 0.085, with a 0.12 dB decrease in PSNR, suggesting additional hyperparameter tuning may be helpful to balance the performance on different metrics.

Q2. Data efficiency of 3D backbone pre-training

Thank you for pointing this out. We agree that the pre-training phase should be counted for a fairer comparison. The ReSplat initialization has two stages: (1) pre-training a depth model, and (2) training the full model from those depth weights. The table below counts both stages as Phase 1 for ours. As shown, we match the PSNR of the best baseline, LagerNVS, while using about half the training samples. We will include this in the camera-ready version.

MethodBackboneTraining Samples (Phase 1)Training Samples (Phase 2)TotalPSNR ↑SSIM ↑LPIPS ↓
LVSM (decoder-only)None06.4M6.4M28.89.894.108
LVSM (decoder-only)None051.2M51.2M29.68.906.098
LagerNVSVGGT56.0M6.4M62.4M30.48.918.086
LagerNVSVGGT56.0M51.2M107.2M31.39.926.078
Ours, batch 4ReSplat48.0M1.2M49.2M30.82.921.098
Ours, batch 16ReSplat48.0M4.8M52.8M31.12.925.095
Ours, batch 16, 600k (full)ReSplat48.0M9.6M57.6M31.38.927.093

Dear Reviewer,

We would like to kindly follow up on our rebuttal and additional experiments. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns or if any questions remain.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.