acceptodds
Withdrawn from ICLR 2027

Active Gaussian Scene Reconstruction via Policy Learning on Hybrid Voxel Maps

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Active reconstruction requires an agent to decide where to move and where to look, so as to explore unknown regions and refine poorly reconstructed ones. However, existing methods struggle to balance the gains of exploration and refinement and typically select the next view greedily over the full camera pose space, which limits efficient exploration and refinement. To this end, we present a reinforcement learning framework for active reconstruction driven by a hybrid voxel map that couples exploration state with reconstruction uncertainty. This single spatial representation serves both the exploration of unknown regions and the refinement of poorly reconstructed regions, with refinement guided by the Gaussian Fisher uncertainty it carries. We further separate target-position selection from viewing-direction selection, in which the policy network reads its observation from the hybrid voxel map and outputs the target position, while the viewpoint scoring selects the viewing direction on the same map. This separation avoids the prohibitive action space, accelerates training convergence, and simplifies the exploration task. Experiments on the indoor scenes show that our method outperforms state-of-the-art active reconstruction methods under the same observation budget and reconstruction configuration.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Gaussian splatting, reinforcement learning, 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QpWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an active RGB-D scene reconstruction framework that combines incremental 2D Gaussian Splatting with learned position selection. It maintains a hybrid voxel representation of occupancy and reconstruction uncertainty, with the latter obtained by aggregating per-Gaussian Fisher-based estimates. A PPO policy uses local voxel maps and recent motion history to select target positions, while a separate view scorer chooses viewing directions based on exploration gain and expected uncertainty reduction. This separation reduces the action space handled by the policy. Training rewards encourage surface coverage, spatial exploration, and reconstruction refinement. Experiments under a fixed observation budget show improvements on InteriorGS and competitive transfer to Replica without fine-tuning.

Critical strengths
  1. The framework addresses a useful planning problem with a sensible design. Active reconstruction needs to balance discovering new surfaces with revisiting poorly reconstructed regions. Combining occupancy and reconstruction uncertainty gives the policy access to both signals. Separating position prediction from viewing-direction selection also makes the decision process easier to formulate and reduces the action space handled by the learned policy.
  2. The reported results demonstrate useful improvements at the system level. On InteriorGS, the method improves both geometric coverage and rendering quality, with coverage reaching 0.692 and PSNR reaching 24.37 dB. The intermediate-budget results help show how reconstruction develops as observations accumulate. Testing on Replica without fine-tuning is also valuable, as it examines whether the learned policy remains useful outside its training dataset.
Critical weaknesses
  1. The contribution appears incremental. Voxel-based mapping and Fisher-based view scoring are established ideas, and the added value of PPO is not clearly isolated. The current ablation changes policy inputs, rewards, and view scoring together. My main concern is whether the gains require the learned position policy. A comparison with a greedy or frontier-based position selector using the same maps and view scorer would help answer this.
  2. Initialization is unclear. The paper mentions a "random valid pose" without explaining how it is sampled or validated. Starting poses and initial observations can affect subsequent exploration. Please clarify the procedure and examine how sensitive performance is to these choices.
  3. Voxel-resolution sensitivity is missing. The voxel size and its selection are not reported. Coarse voxels may average away uncertainty in small, poorly reconstructed regions. A sensitivity study would help explain how the chosen resolution balances fine-detail refinement, runtime, and memory.
  4. Efficiency is only partially evaluated. The reported 5.2% concerns uncertainty updates within mapping, rather than the full pipeline. Please provide absolute timings that include policy inference, candidate-view scoring, and path search, alongside a baseline comparison. Offline training cost should be reported separately.
Questions for authors
  1. Initialization. How are the "random valid poses" sampled and validated, and what initial observations are acquired? Is the same initialization used across methods during evaluation?
  2. Voxel resolution. What voxel size is used, how was it selected, and is it fixed across scenes? How sensitive is reconstruction quality to reasonable changes in this resolution?
  3. Efficiency. What is the absolute online runtime, including candidate-view scoring and path search, and how does it compare with the baselines? What is the separate offline training cost ?
Reviewer_JdjJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an active Gaussian reconstruction framework that combines exploration states and reconstruction uncertainty in voxel maps. A learned policy selects target positions, while candidate viewing directions are scored using exploration and refinement gains.

Critical strengths

The framework connects exploration and reconstruction refinement through a shared spatial representation. Results on InteriorGS show substantial improvements under equal observation budgets. Evaluation on Replica without fine-tuning provides useful evidence of transfer.

Critical weaknesses

A greedy position-selection baseline using the same hybrid maps and view scoring is missing. This makes it difficult to determine whether the gains come from the learned policy or the improved reconstruction signals.

The ablation study covers only one scene. Adding uncertainty also changes the policy input, reward, and view scoring simultaneously, leaving their individual contributions unclear. On Replica, results are competitive with ActiveGS rather than consistently better.

Questions for authors

Could the authors compare against greedy position selection using the same maps and scoring function?

Could the uncertainty ablations be separated and evaluated across multiple scenes?

Reviewer_bZNiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a reinforcement learning framework for active indoor reconstruction using 2D Gaussian Splatting. Its central representation consists of aligned voxel maps encoding exploration states and reconstruction uncertainty, with the latter derived from per-Gaussian Fisher information. A PPO-trained policy selects target positions, while candidate viewing directions are scored using unknown-space discovery and expected uncertainty reduction. Training rewards combine surface coverage, spatial exploration, and uncertainty reduction.

Validation includes reconstruction comparisons on eight held-out InteriorGS scenes, transfer to Replica without fine-tuning, performance at different observation budgets, a single-scene ablation study, and a breakdown of mapping time. Results show improvements on InteriorGS and competitive, although mixed, performance on Replica.

Critical strengths
  1. The framework addresses an important distinction between observing a region and reconstructing it reliably. Combining exploration states with reconstruction uncertainty provides a coherent basis for allocating observations between discovering unseen regions and refining existing surfaces. Using this information across policy inputs, view scoring, and rewards is a technically meaningful design that could improve observation efficiency.

  2. The reported results provide encouraging evidence of system-level utility. Under the reported equal-observation protocol, the method improves both geometric and rendering metrics on InteriorGS. Transfer to Replica without fine-tuning provides additional evidence that the learned policy is not restricted to its training dataset. These results justify further investigation, although the limited reporting prevents a strong conclusion about robustness.

Critical weaknesses
  1. The experiments do not isolate or substantiate the central methodological claims. In Table 3, moving from “Cov.+Exp.” to “Full” simultaneously introduces uncertainty into the policy observation, reward, and view scoring. Consequently, the improvement cannot establish the individual contribution of these components or the benefit of using a shared uncertainty representation throughout the framework. Moreover, this ablation covers only one scene and reports no variability. There is no controlled comparison against greedy position selection using the same maps and view scoring, nor against joint position-and-direction prediction. Thus, neither the claimed long-horizon advantage nor the claimed learning benefit of action decomposition is directly demonstrated. Better performance at later observation budgets does not resolve this attribution problem. The paper also lacks direct evidence that its uncertainty estimates predict actual reconstruction error or the benefit of additional observations. These omissions concern the principal contributions, rather than peripheral implementation choices.

  2. The evaluation is insufficient to establish robustness, competitiveness, and computational efficiency. Table 1 reports dataset-level summaries without per-scene results, repeated-run statistics, or confidence intervals. InteriorGS evaluation covers only eight scenes, while the Replica scene count and selection are not clearly specified. This makes it difficult to assess consistency across environments and sensitivity to initialization. The baselines also omit directly relevant approaches discussed in the paper, particularly NextBestPath and MAGICIAN; comparison primarily against coverage-driven or single-view methods does not establish an advantage over existing long-horizon planning. Finally, the efficiency analysis reports only relative mapping-time percentages on one scene. It excludes a complete accounting of candidate-view evaluation, policy inference, path planning, and end-to-end runtime. Equal observation budgets therefore support claims about observation efficiency, but do not establish computational or operational efficiency.

Questions for authors
  1. Clarification of the evaluation protocol: Which exact scenes were evaluated on each dataset, how were the eight InteriorGS scenes selected, and were methods evaluated from identical initial poses? Please provide per-scene results and specify the number of independent runs and their variability. These details would materially affect my assessment of robustness.

  2. Evidence for the central components: Do controlled results exist that separately remove uncertainty from the policy input, reward, and view scoring, with appropriate retraining? Are there matched comparisons against greedy position selection and joint position-and-direction prediction?

  3. Positioning against relevant methods: Why were NextBestPath and MAGICIAN omitted from the experimental comparison despite being discussed as related work? Are there concrete incompatibilities in sensing, action space, or reconstruction settings that prevent a meaningful comparison?

  4. Clarification of computational cost: What are the absolute runtimes for candidate-view scoring, policy inference, path planning, and mapping, and what is the total runtime relative to the baselines on the same hardware? Does the reported 5.2% include the Fisher computations required to score all candidate views, or only uncertainty updates after observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.