- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The work proposes Pano123, which is a 3D reconstruction framework from a single panorama input. The approach is proposed in a feed-forward setting. The main observation assumes that only the disoccluded regions have to be filled in such a setup, and those are typically sparse. To this end, the authors propose a 3-stage approach. Firstly, an initial Gaussian Splats estimation is performed with an off-the-shelf depth prediction as the initialisation, with an additional predicted refinement step. The next step predicts a set of disocclusion candidates. Finally, a set of candidates behind the occlusion surface is updated with a learnable decoder.
Critical strengths
I believe the paper's motivation and main design choice are compelling. Predicting only the candidate disoccluded Gaussians, while preserving the visible ones intact, is a good, interesting idea. It is shown in the paper that the approach works as described, and quantitative results show substantial improvements over the compared baselines. Similarly, qualitative samples show convincing results with the scenes fully completed, as opposed to producing holes (as is shown in the baselines). The parametrisation in spherical coordinates is also a good design choice, which is well motivated in the paper (namely, preventing the stretched splats). Finally, it is sensible to restrict stage 3 attention to nearest neighbours, which should help with the prediction speed.
Critical weaknesses
While the results are convincing, most of the very scarce visual samples present rather flat regions (e.g. walls). Without a generative prior, the method can struggle with severe object occlusions; the results could get way worse if the disoccluded region cannot be smoothly interpolated (this also ties to initialising queries with the average of neighbouring Gaussians).
The evaluation setup is not very extensive. The only ablation is Stage I vs Full Model. While this is a good ablation, one could ablate many more parameters. Starting with the dependence on the initial depth prediction quality, stage I refiner (with and w/o), stage II trained vs a simple heuristic, extent of query candidates, etc. Similarly, the authors do not provide many baseline comparisons; one such could be Stage I + inpainting. Further, the authors cite Scene4U, which even though have not released the code yet, could be compared by reusing the same dataset and protocol (and notably, the method solves a very similar problem). Similarly, the cited LayerPano3D could possibly be adapted to the task. There should be nothing preventing testing PanSplat on a single input view. Finally, CylinderSplat explicitly offers a single panorama setup. Conversely, I am not convinced about PanoSplatt3R as a baseline with adaptation to multi-view by input view duplication.
One important work, not included, is PERF [1], which is a single-panorama to a radiance field approach. Further, I would also refer to PanoGRF [2] as an important prior work.
[1] Guangcong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang, Chen Change Loy, Ziwei Liu, "PERF: Panoramic Neural Radiance Field from a Single Panorama", TPAMI, 2024. [2] Zheng Chen, Yan-Pei Cao, Yuan-Chen Guo, Chen Wang, Ying Shan, Song-Hai Zhang, "PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas", NeurIPS, 2023.
Questions for authors
- Have the authors considered using a baseline of Stage I plus an off-the-shelf inpainting method (in low-opacity or missing regions)? It could help to assess the performance of Stage III.
- It would be interesting to see results of splatting only the candidate stage 3 Gaussians, without refinement.
- Why do "near" and "far" distances overlap? There could be samples in "near" that are farther than some in "far".
- It would be good to prove that the cosine scale modulation and spherical residual parametrisation are important by ablating them.
- Have the authors tried using a depth gradient in place of the learned disocclusion detector?
- What is the method's sensitivity to the number of candidate depths?
Please engage with the authors.
Dear reviewers,
The authors have submitted detailed responses to your reviews.
Could you please review their feedback, engage with the authors if any points remain unclear, and update your evaluation and score if their response has addressed your concerns?
Best regards, AC