- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents InfoField3D, which converts task-relevant surface responses from a semantic point cloud into a differentiable field over 3D observation positions. The method combines local geometric partitioning, low-order spherical harmonics for compact directional representations, and Gaussian occupancy models for approximate occlusion handling. Experiments evaluate representation fidelity, viewpoint optimization efficiency, and downstream robotic task performance.
Critical strengths
- Clear motivation and a coherent design. Evaluating task relevance and visibility point by point can be computationally expensive. The proposed compact representation and analytic formulation address this issue directly, providing a practical interface for continuous viewpoint optimization.
- Broad experimental evaluation. The evaluation goes beyond reconstruction error and runtime to examine the quality of selected viewpoints, using original surface points and mesh-based occlusion as references. Downstream experiments with a frozen policy and matched-displacement viewpoint controls provide additional evidence of practical utility.
Critical weaknesses
- The relationship between the proposed observation score and task requirements needs further clarification. The score primarily reflects task relevance, surface orientation, and occlusion. However, fine manipulation may depend on resolving a small critical region rather than maximizing aggregate surface response. The downstream results support the score’s usefulness on the evaluated tasks, but its applicability to other visual requirements remains unclear. Representative failure cases would help establish when the score does or does not predict improved execution.
- The operating limits of the geometric and occlusion approximations are not sufficiently characterized. Low-order spherical harmonics and smooth occupancy models support efficient, differentiable computation, but narrow gaps, thin shells, concave structures, or spatially concentrated task targets may produce approximation errors that affect viewpoint ranking. The overall fidelity results are encouraging. A targeted analysis of several challenging geometries, including sensitivity to partition granularity and key occlusion parameters, would strengthen the evaluation.
Questions for authors
- How are task-relevance weights generated and normalized? How sensitive are the recommended viewpoints to inaccurate semantic localization?
- In which cases does a higher field response fail to improve task execution? Are these failures primarily attributable to the scoring function, geometric approximation, or downstream policy?
- Could the authors provide the partitioning and occlusion parameters used in the experiments and clarify whether they remain fixed across scenes?
4 comments
Response to Reviewer dTfq (2)
W2: The operating limits of the low-order SH and smooth-occupancy approximations are not characterized: narrow gaps, thin shells, concave structures, or spatially concentrated targets may cause errors that affect viewpoint ranking. A targeted analysis of such geometries, including sensitivity to partition granularity and key occlusion parameters, is needed.
A: Thank you for your suggestion. We constructed synthetic scenes with four types of geometries where the approximations are most likely to fail, to examine their operating limits. The key experimental designs and results are summarized below, detailed settings and results are given in Appendix H.3 of the revised version:
- Narrow gap: the target is inside a closed box and can only be seen through a vertical slit in the front wall, with the slit width progressively narrowed.
- Thin shell: a thin plate stands in front of the target, blocking the line of sight on one side, with the shell thickness progressively reduced.
- Concave structure: the target is the bottom of an open cylindrical cup, with the ratio of cup depth to cup diameter progressively increased.
- Spatially concentrated target: the target is a small patch on one face of a cuboid, with its size progressively reduced.
Change in approximation fidelity from the easiest to the hardest tested setting.
| Geometry | Geometric condition | Optimal-value attainment (pp) | Spearman’s ρ |
|---|---|---|---|
| Narrow gap | Gap width: 80 → 5 mm | −13.12 | −0.1481 |
| Thin shell | Thickness: 20 → 1 mm | −2.94 | −0.0727 |
| Concave structure | Depth-to-diameter ratio: 0.25 → 3 | −0.04 | −0.0096 |
| Concentrated target | Side-length ratio: 100% → 5% | −1.23 | −0.0554 |
Effect of reducing the minimum Gaussian width in the hardest thin-shell and narrow-gap settings.
| Thin shell | Narrow gap | |||
|---|---|---|---|---|
| Minimum width relative to default | Attainment change (pp) | Spearman’s ρ change | Attainment change (pp) | Spearman’s ρ change |
| 1 | – | – | – | – |
| 1/4 | +8.41 | +0.1598 | +0.06 | +0.0514 |
| 1/8 | +8.21 | +0.2429 | +0.13 | +0.0504 |
As shown in the tables, concave structures and spatially concentrated targets have limited impact on approximation fidelity, whereas performance gradually degrades for increasingly extreme narrow-gap and thin-shell geometries (which we speculate occurs when the structural scale approaches or falls below the Gaussian smoothing scale). Further experiments show that this degradation can be alleviated to some extent by reducing the minimum Gaussian width. We also conducted additional ablation experiments to identify the source of the error, finding that it mainly arises from the smoothing approximation of occlusion rather than spherical harmonic truncation.
Sensitivity to partition granularity and occlusion parameters. We conducted a dedicated sensitivity analysis. The results show that the average quality of the recommended viewpoints is not sensitive to parameter variations within a reasonable range. A noticeable drop occurs only when the Gaussian width is increased enough to cause over-smoothing, consistent with the findings from the challenging-geometry experiments above. The key results are shown below; see Appendix H.3.2 of the revised version for experimental details.
Sensitivity to partition granularity and occlusion parameters across 80 scenes. Changes in mean task-region visibility are relative to the default setting.
| Parameter | Adjustment relative to default | Change (pp) |
|---|---|---|
| Number of units | 0.8–2× | −0.30 to +0.28 |
| Minimum Gaussian width | 0.5–2× | +0.31 to −3.25 |
| Attenuation strength κ | 0.5–2× | −0.38 to +0.92 |
In summary, the approximations of InfoField3D are reliable for common geometries and within reasonable parameter ranges. Its operating limits appear on narrow gaps and thin shells whose structural scale is smaller than the Gaussian smoothing scale, and the degradation is gradual and can be partly alleviated by refining the parameters.
Response to Reviewer dTfq (3)
Q1: How are task-relevance weights generated and normalized? How sensitive are the recommended viewpoints to inaccurate semantic localization?
A: Thank you for your question. We answer from two aspects.
(1) Generation and normalization of task-relevance weights. Task relevance can be generated by a model or specified by a human. In our experiments, GPT-4o generates weighted text queries from the task description and multi-view images. Modified-SAM and OpenCLIP are then used to compute matching scores between image regions and the queries. These scores are projected onto the 3D surface using depth information to obtain a pointwise nonnegative task-relevance map (see Figure C1 in Appendix C.5).
For each task role, we divide its response by its total area-weighted relevance. The normalized responses of multiple roles are then combined using an equally weighted geometric mean (see Appendix C.2). Because the normalization factor is independent of the viewpoint, the score cannot increase merely because less irrelevant geometry is observed.
(2) Sensitivity of the recommended viewpoints to inaccurate semantic localization. When this localization contains errors, can the recommended viewpoints still maintain their quality? To investigate this, we conducted simulation experiments based on several common types of localization errors encountered in practice.
Across all scenes used in Section 4.2 of the paper, we applied three types of simulated perception errors to the task-relevance map: noise, where local false-positive patches appear on background surfaces; spill, where the target region expands to surrounding surfaces; and erosion, where contiguous parts of the target region are missed. Each error type was tested at different strength levels. Each method searched for viewpoints using only the perturbed task-relevance map, and the recommended viewpoints were then evaluated on the unperturbed map using hard-ray Task-region visibility to assess their actual quality. The key experimental results are summarized below, with further details provided in Appendix H.4 of the revised version.
Mean loss for each method under the strongest simulated perturbation (total variation 0.50) of each error type, measured in percentage points.
| Method | Noise ↓ | Spill ↓ | Erosion ↓ |
|---|---|---|---|
| InfoField3D | 1.55 | 4.00 | 1.27 |
| GP-BO | 9.94 | 0.78 | 1.39 |
| CIM | 12.63 | 2.89 | 4.91 |
| CMA-ES | 8.38 | 1.92 | 2.58 |
| GradientNBV | 13.30 | 4.87 | 2.77 |
| Sobol | 6.34 | 1.00 | 1.26 |
| Random | 6.88 | 0.82 | 1.38 |
As shown in the table, InfoField3D is more robust to background noise than all baselines and achieves robustness to local misses close to that of the best baseline, but is more sensitive to target-region spill than to the other two error types. We speculate that this difference arises from the spatial structure of the errors: spill introduces a spatially coherent region of erroneous relevance near the true target, systematically influencing the recommended viewpoint. In contrast, scattered background false positives and local misses have less impact on the main observation direction of the target.
Overall, InfoField3D shows good robustness to noise and local misses, while its higher sensitivity to target-region spill reflects how its objective function responds to the spatial distribution of task relevance.
Response to Reviewer dTfq (4)
Q2: In which cases does a higher field response fail to improve task execution? Are these failures primarily attributable to the scoring function, geometric approximation, or downstream policy?
A: Thank you for your question. The cases in which a higher score does not bring improved execution have been answered in point (2) of W1; here we focus on the attribution. We examined the three possible causes one by one, and believe that such cases are mainly related to the downstream policy, rather than to the geometric approximation or the scoring function itself.
(1) Geometric approximation. For cases where the score improved but execution did not, we re-evaluated the recommended viewpoints with full-scene hard rays that do not rely on smoothing approximations, and the visible fraction of the task region remains clearly higher than at the initial viewpoint. This shows that the viewpoint has indeed improved, and the improvement is not an artifact caused by approximation error.
(2) Scoring function. InfoField3D score measures the weighted observation sufficiency of the task region; it is determined by the task-relevance map and is not tailored to any specific VLA policy. In the representative cases we inspected, the visibility of the key targets has indeed improved, whether judged by geometric metrics, GPT-4o score, or human observation, indicating that the score reflects the quantity it is intended to measure. What it does not model is the specific policy’s preference over viewpoints, which is exactly what point (3) below discusses.
(3) Downstream policy. We believe this may be the main cause, primarily involving the two situations discussed in point (2) of W1: first, observation itself is not the bottleneck of task execution; second, the recommended viewpoint may deviate from the viewpoint distribution familiar to the policy.
(4) Regarding (3), ideal viewpoint search and the needs of a specific policy are not mutually exclusive. Interestingly, InfoField3D’s design makes it easy to incorporate the preferences of specific embodied policies, allowing viewpoint recommendations to be tailored to each policy. For example, if a policy’s viewpoint preference can be represented as a spatial distribution, it can be directly incorporated as a multiplicative term in the InfoField3D objective. Additionally, if the policy’s operating mode (e.g., robots using suction cups versus grippers may perform the same task differently) can be provided in text form to the model responsible for task understanding, the task-relevance map can account for the policy’s specific preferences from the outset.
In summary, the failure of recommended viewpoints to improve VLA execution may be primarily attributed to the downstream policy’s limited ability to translate improved observation into better task performance. To address this limitation, one possible solution is to incorporate the policy’s preferences into the objective function.
Q3: Could the authors provide the partitioning and occlusion parameters used in the experiments and clarify whether they remain fixed across scenes?
A: Thank you for your question. All scenes use the same partitioning and occlusion settings, with no per-scene tuning. The key details are summarized below, with further information provided in Appendix H.5 of the revised version. We will soon release the key code and example data to provide an accessible reference implementation.
| Item | Setting |
|---|---|
| Partitioning | |
| Scheme | Recursive principal-axis bisection (Section 3.4) |
| Unit to split | Largest area-weighted spread |
| Split position | Area-weighted median along the principal axis |
| Number of units | 224, 448, or 896 by a fixed rule (about 600 on average) |
| Occlusion | |
| Occupancy model | One anisotropic Gaussian per unit (Eqs. (9)–(11)) |
| Minimum Gaussian width | 2 mm |
| Attenuation strength | Centerline optical depth of 12, the same for all units |
| Occluders | All other units, without spatial truncation |
The centerline optical depth is measured through the Gaussian center along its narrowest principal axis, so a line of sight crossing the thinnest part of a unit retains only of the response. The number of units increases with the surface area and the lateral extent of the scene, using the same thresholds for all scenes.
Response to Reviewer dTfq (1)
W1: The score mainly reflects task relevance, surface orientation, and occlusion, but fine manipulation may depend on resolving a small critical region rather than maximizing aggregate surface response, so its applicability to other visual requirements is unclear; representative failure cases are needed to show when the score does or does not predict improved execution.
A: Thank you for your comment and for raising this question. We answer in two parts.
(1) InfoField3D does not maximize aggregate surface response, but rather task-relevance-weighted surface response. Even a very small critical region can dominate the score when the relevance weights are concentrated on it.
For example, opening a drawer primarily requires observing its grip bar, whereas removing its handle may require observing the support rods and mounting rings. Thus, even for the same object in the same scene, different tasks may require substantially different optimal viewpoints (Appendix H.1 of the revised version provides a visualization of InfoField3D’s viewpoint recommendations for this example). In our experiments, to translate such an understanding of task-specific visual requirements into viewpoint selection, InfoField3D takes as input a task-relevance map over the 3D scene, produced by an upstream multimodal model based on its understanding of the task, and uses this map to recommend optimal viewpoints.
In this sense, in principle, as long as the upstream stage (whether a model or a human) can express visual requirements as a spatial distribution of regions and their relative importance, InfoField3D can recommend appropriate viewpoints accordingly.
Regarding the potential conflict you raised between "resolving a small critical region" and "maximizing aggregate surface response", an intuitive explanation is that when a task depends on a very small critical region, the relevance weights are concentrated on that region. InfoField3D’s objective can therefore be approximately understood as "maximizing the surface response of this small critical region." Thus, "very small" and "maximizing" are not contradictory.
(2) When the score improvement can, and cannot, translate into improved execution. Based on the current results, this mainly depends on two factors.
a. Whether observation is the bottleneck of task execution. Figure 6(A.1) reports results stratified by initial task completion. In the tier with the lowest initial completion, the recommended viewpoint brings an improvement of 42.4 percentage points. As initial completion increases, the improvement gradually diminishes, even turning into a decrease of 2.4 percentage points in the highest tier. We hypothesize that when the policy can already complete the task well from the initial viewpoint, observation is no longer the bottleneck, leaving little room for a better viewpoint to translate into improved task performance.
b. Whether the recommended viewpoint deviates from the viewpoint distribution familiar to the policy. As shown in the table below, when recommended viewpoints are divided into two groups based on whether they are closer to or farther from the default camera position than the initial viewpoints, the closer group shows a substantially greater mean improvement in task completion than the farther group.
Take the task “pick up the orange juice and place it in the basket” in scene 2087 as an example: the observation sufficiency of the task-critical region from the recommended viewpoint is better than from the initial viewpoint, whether judged by geometric metrics, GPT-4o score, or human observation, but it is farther from the default camera position than the initial viewpoint, and the task completion of the frozen policy instead decreases slightly. We believe that although such viewpoints improve the visibility of the task region, they deviate from the viewpoint distribution familiar to this VLA policy, and the policy’s ability to use the information in them decreases accordingly. See Appendix H.2 of the revised version for a detailed analysis.
In summary, InfoField3D can optimize the observation sufficiency of task-critical regions according to specific task requirements, but whether better viewpoints lead to higher task completion may also depend on whether observation is a bottleneck to task execution and whether the policy can effectively utilize the information provided by the new viewpoint.