acceptodds
Withdrawn from ICLR 2027

Policy-Dependent View Acquisition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting camera frames before transmission requires a robot to estimate the value of observations it has not yet received. That value depends on both the views and the policy that consumes them. We formulate conditional view-set value and distinguish decision-level interaction from excess interaction induced by a policy. Selective substitutability training (SST) penalizes excess interaction together with subset regret and action-uncertainty mismatch; a residual coalition scheduler (RCS) selects fresh views from low-rate camera sketches under a per-decision byte cap. Across eight MuJoCo/robosuite manipulation tasks and three training seeds, SST+RCS achieves 69.58% episode success at a 55% communication cap, compared with 62.92% for all-subset training with a Set Transformer. The paired difference is 6.67 percentage points, with a 95% hierarchical bootstrap interval of [2.78,10.56]. A separate eight-seed evaluation shows a similar intermediate-budget gain. An expanded physical evaluation yields a +7.81-point paired difference across 576 trials per method. In the main frontier, SST+RCS loses at the smallest cap, ties at full view and has slightly higher mean latency than the reference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, active perception, multi-view acquisition, bandwidth-constrained control, conditional view-set value, selective substitutability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_utmuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes decision-level view interaction from excess interaction induced by a policy. It proposes selective substitutability training and a sketch-conditioned scheduler for acquiring camera frames under a communication budget, with evaluations in simulated manipulation and physical-robot settings.

Critical strengths

Distinguishing decision-level interaction from policy-induced excess interaction is an interesting conceptual contribution. It provides a useful way to examine whether additional observations address task ambiguity or limitations of the consuming policy.

The experiments report a positive paired success difference at intermediate budgets, separate policy and router effects, and account for sketch costs. Explicit reporting of unfavorable budget and transfer conditions makes the empirical scope credible.

Critical weaknesses

The added value of the central excess-interaction penalty remains unclear. Equations (7)–(8) combine control, subset-regret, excess-interaction, and calibration losses. Figure 6c studies several components, but does not show a matched (β=0\beta=0) condition retaining the other losses. Consequently, the overall improvement supports the combined training framework more directly than the specific contribution of the proposed penalty.

The mechanism-level evidence is less conclusive than the closed-loop evidence. Section 5.2 reports excess decreasing from 0.01049 to 0.00940 and subset regret from 0.11757 to 0.11150 relative to all-subset training, while Figures 3–4 provide no uncertainty estimates. Sections 3.2 and 4.2 also acknowledge approximation and shared-bias concerns in the decision references. The reliability of these mechanistic differences therefore requires further characterization.

Questions for authors
  1. What happens when (β=0\beta=0) while the other losses, training budget, and router remain unchanged? Does the excess-interaction penalty provide an additional closed-loop success benefit?
  2. Are the reported excess and subset-regret reductions consistent across training seeds? Please provide or identify paired uncertainty estimates for these differences.
  3. What empirical evidence shows that the excess reduction and the benefit of the penalty remain stable across reference choices or on cases where the approximate references disagree?

I would raise my rating if a matched ablation and reference-sensitivity analysis establish the added value of the excess-interaction penalty, and lower it if new evidence substantially weakens its claimed role in the reported improvements.

Reviewer_SYneOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies view acquisition for robot manipulation under a communication budget. Rather than always transmitting all camera observations, the robot must decide which full-resolution views are worth refreshing before they are received, while unselected cameras retain stale observations. The paper argues that the usefulness of a set of views depends not only on the observations themselves, but also on the policy consuming them. It introduces a distinction between decision-level interaction, where multiple views are genuinely complementary for the underlying decision, and excess policy interaction, where a policy unnecessarily relies on multiple views because it fails to use information already available from a smaller subset. Based on this formulation, the authors propose Selective Substitutability Training (SST), which combines control supervision with subset regret, excess-interaction regularization, and action calibration. They further introduce a Residual Coalition Scheduler (RCS), which predicts the value of feasible camera subsets from low-rate sketches and history, and selects full-resolution views under a per-decision byte budget. Experiments cover eight MuJoCo/robosuite manipulation tasks. At a 55% communication cap, SST+RCS reaches 69.58% success compared with 62.92% for the main all-subset + Set Transformer reference, corresponding to a paired improvement of 6.67 percentage points with a 95% interval of [2.78, 10.56]. The paper also reports acquisition diagnostics, transfer tests, latency measurements, and physical-robot experiments.

Critical strengths

(a) The distinction between intrinsic view complementarity and policy-induced multi-view dependence is conceptually interesting. The paper asks a more nuanced question than standard view selection: whether a policy needs multiple views because the task truly requires them, or because the policy itself fails to exploit a smaller sufficient observation set. The relation between interaction differences and subset regret gives this idea a reasonably clean mathematical interpretation.

  1. The simulation evaluation is more careful than a simple single-number comparison. The authors report multiple communication budgets, separate policy and scheduler effects, latency, camera loss, and layout shift. They also explicitly acknowledge cases where SST+RCS does not win rather than hiding them. This is useful.
Critical weaknesses
  1. I am not convinced that the proposed “excess interaction” quantity is reliably identifiable in practice. The central distinction of the paper depends on comparing the learned policy against an ideal decision-level reference VdecV_{dec}. In practice, however, VdecV_{dec} is unavailable and is replaced with approximate references. The authors themselves state that agreement between these approximate references does not establish correctness and cannot rule out shared bias. This is important because the approximate reference is not only used for analysis; it directly affects SST training. If the reference incorrectly labels genuine complementarity as policy-induced excess, the method may explicitly suppress useful multi-view behavior. I do not think the current experiments sufficiently establish robustness to this issue.

  2. The empirical gains are narrow and are not consistently preserved across budgets or distribution shifts. The headline result is a +6.67-point improvement at the 55% communication cap. However, the proposed method is worse than the reference at the 30% cap, ties it at full view, and the SST advantage becomes −1 point under the reported layout shift. This makes the contribution feel more like an improvement in a particular intermediate-budget regime than a generally better observation-acquisition principle. The policy/router crossing also suggests that the contributions are not cleanly additive: with an all-subset policy, replacing the Set Transformer with RCS actually reduces performance from 62.92% to 61.67%, while most of the gain appears after changing the policy training to SST.

  3. The claimed physical-robot validation is not sufficiently documented in the provided manuscript. The main text reports a small physical comparison with 80 trials per method, where SST achieves 44/80 successes versus 38/80 for the budgeted baseline; the authors explicitly note that this difference is uncertain. The manuscript then claims a much larger physical study with 576 trials per method and a +7.81-point difference, but provides only the aggregate number and refers the reader to Appendix H. In the PDF, Appendix H is not present, and I cannot find the hardware setup, camera configuration, task setup, or uncertainty analysis for this larger experiment. Given that the motivation is explicitly robotic deployment under communication constraints, I consider this a significant gap in the current submission.

  4. The camera-selection setting is small and somewhat specialized. The main experiments use only four cameras, for which the feasible subsets can be directly enumerated. At the 55% cap, there are only ten candidate subsets. This avoids the main combinatorial difficulty of set-valued view acquisition. It is therefore unclear whether RCS remains useful or computationally practical for richer sensor rigs, where exhaustive feasible-set evaluation is impossible.

Questions for authors
  1. Can the authors provide the complete physical-robot evaluation details? The current manuscript reports an expanded study with 576 trials per method, but I could not find the corresponding robot hardware, camera placement, tasks, communication setup, per-task results, or statistical intervals in the provided PDF. These details are important for assessing whether the method transfers beyond simulation.

  2. How sensitive is SST to errors in the approximate decision reference? Please report results when SST is trained separately with each approximate reference, and ideally when controlled noise/bias is introduced into the reference. This would clarify whether the proposed regularization remains beneficial when VdecV_{dec} is imperfect.

  3. Why does the method reverse under layout shift? The reported SST margin changes from positive in-distribution to −1 percentage point under layout shift. Is this due to the policy, the value predictor, or the view-selection distribution? This seems important because the motivation suggests that view value should adapt to changing decision context.

  4. How would the acquisition procedure scale to substantially more cameras? With four cameras the feasible set is small enough to enumerate. What is the intended inference procedure for 8, 16, or more camera streams, and how does its cost scale?

  5. The related-work discussion on active perception and learned view control could be broadened:

[1] Peihao Chen, Dongyu Ji, Kunyang Lin, Weiwen Hu, Wenbing Huang, Thomas H. Li, Mingkui Tan, and Chuang Gan. Learning Active Camera for Multi-Object Navigation. Advances in Neural Information Processing Systems (NeurIPS), 35:28670–28682, 2022. NeurIPS Proceedings

[2] Xuechao Zhang, Dong Wang, Sun Han, Weichuang Li, Bin Zhao, Zhigang Wang, Xiaoming Duan, Chongrong Fang, Xuelong Li, and Jianping He. Affordance-Driven Next-Best-View Planning for Robotic Grasping. Conference on Robot Learning (CoRL), 2023.

[3] Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, and Shuran Song. Vision in Action: Learning Active Perception from Human Demonstrations. Conference on Robot Learning (CoRL), 2025.

Reviewer_jnaCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of selecting camera viewpoints for robots before transmitting visual observations, the core challenge lies in the robot’s need to estimate the value of observations that have not yet been received. The paper propose Selective Substitutability Training (SST), which penalizes excessive interaction while incorporating subset regret and action uncertainty mismatch. Meanwhile, the paper design a Residual Coalition Scheduler (RCS) to select new views from low-bitrate camera sketches under the per-decision byte budget constraint. The experiments were conducted on eight MuJoCo/robosuite manipulation tasks(static, insertion, occlusion, handover families), 400 demonstrations per task, with three training seeds.

Critical strengths

None

Critical weaknesses

The "single/pairwise term + residual" structure of RCS and SST regularization (subset regret + excess + calibration) form a reasonable combination. However, the individual components (Set Transformer, modality dropout, diffusion strategy) are all existing techniques, with limited innovation at the combination level.

Questions for authors

Fix the SST strategy, artificially modify the feasible set under the 55% cap (e.g., unify camera costs), and quantify the contribution of the discrete feasibility structure to the gain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.