acceptodds
Withdrawn from ICLR 2027

Beyond Relevance: Rethinking What Robots Need to See Across Tasks and States

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual relevance does not determine observation precision or visual capacity. Motivated by action sensitivity to state errors, we study action-conditioned measurement sufficiency beyond the Base policy’s existing context. Controlled ACT analyses show task- and state-dependent location–precision allocation under a fixed budget, no uniformly monotonic benefit from increasing the precision ceiling or residual capacity, and different preferred configurations across tasks. We propose Action–Visual Precision Matching: action context jointly ranks location–precision candidates, Global Top-K selects them under a shared budget, and only selected high-precision observations are encoded. Their feature residuals relative to base-aligned references refine native action generation. The main ACT configuration improves mean success rate by 2.65 percentage points over its matched Base and outperforms same-budget random allocation. A post-hoc task-wise configuration combination reaches 84.20%, exceeding Base and the best uniform configuration by 4.03 and 1.38 percentage points. Diffusion Policy and π0 further test integration across action generation structures. These results support matching visual resources to current action requirements rather than uniformly maximizing them.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning, visual precision, conditional visual refinement, action refinement, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_iBdvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how visual precision and capacity should be allocated according to action requirements in robot manipulation. The authors propose Action-Visual Precision Matching, which jointly selects locations and precision levels under a fixed budget and refines the base policy using visual feature residuals. Experiments on LIBERO demonstrate improvements over the base policy, particularly for ACT, and show that optimal visual configurations vary across tasks and execution states.

Critical strengths
  • The paper presents an interesting perspective on adaptive visual perception by distinguishing visual relevance from action-dependent precision requirements.
  • The proposed method provides a coherent framework for joint location-precision selection and residual-based action refinement.
Critical weaknesses
  • Insufficient Evidence for Action-Visual Precision Alignment. The central claim of the paper is that visual precision should be matched to action requirements. However, conditioning visual allocation on action representations does not guarantee that the selected precision levels reflect actual action demands. While Figure 2 shows that precision allocation varies across tasks and timesteps, the paper provides limited analysis of why these variations occur or how they relate to action sensitivity, task progress, or manipulation stages. As a result, it remains unclear whether the learned allocation genuinely captures action-visual precision matching or merely exploits task-specific heuristics to improve performance.

  • Limited Justification for Joint Location-Precision Selection. In Table 3, random allocation already yields a substantial improvement over the base policy (80.17% -> 81.77%), whereas learned allocation provides a relatively modest additional gain (82.82%). Also, the effects of location and precision selection are not disentangled. Ablating these components individually is necessary to justify the joint selection design and quantify each mechanism's contribution.

  • Inconsistent Gains Across Policy Architectures. Performance improvements are heavily concentrated in ACT (+2.65), whereas Diffusion Policy (+0.43) and π0\pi_0 (+0.12) exhibit only marginal gains. This discrepancy weakens the claim that the proposed method provides consistent benefits across diverse policy architectures.

  • Presentation. The paper uses terms such as "visual precision," "visual capacity," and "measurement sufficiency," but their distinctions from established concepts such as spatial resolution, patch scale, and token budget are not sufficiently clarified.

Questions for authors
  • How can the authors verify that the learned precision allocation genuinely reflects action requirements? Could the authors provide an analysis relating the allocation patterns in Figure 2 to action sensitivity, task progress, or manipulation stages?

  • What are the individual contributions of location and precision selection? Could the authors provide separate ablations under the same visual budget?

  • Why are the performance improvements substantially smaller for Diffusion Policy and π0 compared to ACT?

Reviewer_3oz9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose to add additional visual precision beyond the base policy's context. Specifically, it use the action context to rank the location-position candidates and supply the model with high-precision feature residuals. The method is evaluated on LIBERO with different polices. Performance are improved minorly.

Critical strengths
  • The evaluation is conducted on 500 episodes per suite per seed, which provides stable results.
Critical weaknesses
  • Overall, the gain on different models is minor. Specifically, the average gains on DP and π₀ are only 0.43 and 0.12 percentage points. Even though large rollout number is used, this provides weak evidence across architectures. The only noticable advantage comes from ACT. However, it still comes from a starved base with 64x64 inputs. Also, more benchmark should be included rather than just using the LIBERO one.
  • The core assumption is not justified. The paper never shows that allocation follows action sensitivity rather than relevance. Specifically, any router that looks at the image will pick different patches when the image changes. So this cannot tell why it picks them. Some of it is not learned: under MAX256/K48, the router simply takes all 32 of the 128-precision candidates in almost every task, making the composition at about 66.7% / 33.3%.
Questions for authors

N/A

Reviewer_dfJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how robot policies should allocate additional visual information beyond their existing observations. Its central argument is that identifying a relevant image region does not determine the spatial precision needed for control. The proposed method uses base-policy action representations to rank candidates jointly across locations and precision levels, selects them through a shared Top-K budget, and encodes only the selected fine patches. Differences between these features and spatially aligned references derived from the base image then refine the frozen policy’s action generation.

Experiments cover the four LIBERO suites using ACT, Diffusion Policy, and π0. The main ACT configuration improves the average success, while improvements for Diffusion Policy and π0 are smaller. The paper also analyzes allocation across tasks and execution states, compares precision–capacity configurations, and reports a retrospective task-wise configuration oracle.

Critical strengths
  1. The paper makes a useful control-oriented question concrete. Conditioning visual refinement on what the policy is preparing to do is a useful design choice. The base-aligned residual construction also fits the motivation of adding information beyond what the base policy already sees. This formulation could be useful for designing visual interfaces for robot policies even beyond the particular implementation here.

  2. The ACT study contains interesting results and several careful experimental choices. The main configuration improves average success in all four suites. Matching base checkpoints across fine configurations and constructing the image pyramid from the same demonstration states help make the comparisons interpretable.

Critical weaknesses
  • The central comparison does not yet isolate the benefit of learning action-conditioned precision allocation. As described in Appendix C.6, the random control replaces the learned routing at evaluation while keeping all parameters unchanged. The fine encoder and refinement modules therefore appear to have been trained with the learned router’s selections and subsequently evaluated with a different selection distribution. The resulting performance drop can reflect sensitivity to that change, as well as the quality of the allocation itself. Table 3 is informative about the trained system’s dependence on its router, but it does not establish the advantage over a refinement pathway trained and evaluated with random allocation under the same budget.

  • The paper acknowledges that this intervention changes location and precision jointly. There is also no comparison isolating the contribution of action context relative to a router using the visual context alone. The allocation visualizations show that selections vary with state, but they do not establish that those variations correspond to control-critical precision requirements.

  • The strongest improvement is over an ACT base using 64×64 images and eight base spatial tokens. The main method adds 24 residual tokens, with both high-detail and reference patches passed through the fine encoder. The fixed-budget comparisons are useful, but there is no comparison with a straightforward higher-resolution policy or another simple refinement baseline at a comparable measured computational cost. Appendix B correctly notes that token ratios do not imply equal FLOPs; without latency or compute measurements, it is difficult to judge the benefit of this particular allocation strategy.

Questions for authors
  • Clarification of the random control. Is Table 3 obtained by replacing routing only at inference in the same learned checkpoint, as Appendix C.6 appears to indicate? If so, do you have a comparison where the fine pathway is trained and evaluated with random routing using the same data, update budget, and residual capacity? This would directly affect my interpretation of the 1.05-point advantage.

  • Evidence for action conditioning. What evidence isolates the contribution of the action representations used for routing? An otherwise matched router trained using visual context and candidate geometry, without action representations, would be a particularly informative control. Does the advantage persist in such a comparison?

  • Resource accounting and the practical alternative. What are the measured inference latency and computational overhead relative to Base, including candidate scoring, high/reference encoding, and residual readout? Is there a higher-resolution or fixed-allocation baseline at a comparable cost? This would help establish whether the proposed mechanism is a useful way to spend the additional resources.

  • All evaluations also remain within LIBERO. Although standard LIBERO approaches ceiling performance for some strong VLAs, the ACT setup here is not itself saturated. My concern is whether the observed allocation benefits and configuration preferences persist under different visual demands—for example, changed camera viewpoints, object scales, or placement tolerances. An independent benchmark or a targeted evaluation under held-out visual conditions would strengthen that conclusion.

Reviewer_inCFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Nearly all adaptive-visual work for robot policies asks which tokens or regions to keep. This paper asks a second, independent axis: at what precision, and with how much additional capacity. Working with ACT as a controlled platform, it defines precision as spatial sampling density at a fixed local input size, lets an action-conditioned router rank location–precision candidates jointly, selects them with a single global top-k under a shared budget, and encodes only the chosen high-precision patches. Their feature residuals against a base-aligned reference form a memory that refines the policy's native action generation. The base policy stays frozen, and the branch is integrated into three different action-generation structures.

Critical strengths
  1. The question is well posed and the axis is genuinely understudied. Nearly all adaptive-visual work for robot policies asks which tokens or regions to keep. This paper asks the second, independent axis — at what precision and with how much additional capacity — and that axis is real and largely untouched.
  2. Clean framing; the contribution is clearly located. The four-way framing in the first figure is a clean way to locate the contribution within the surrounding literature.
  3. Writing is clear; related work is broad and accurate.
Critical weaknesses
  1. The paper studies allocating a resource, but the resource is never defined. The base policy consumes a low-resolution image, while a high-precision patch is cropped from a much higher-resolution one. Both readings are problematic. If the high-resolution input truly comes from a higher-resolution source, the pipeline requires sensor input far beyond what the base policy needs — and none of that is inside the budget, which counts only encoded patches. If instead it is upsampled from the same low-resolution frame, then no information above the base resolution exists anywhere in the pipeline. A paper about allocating a resource cannot be reviewed when the resource is undefined.

  2. No cost model: the method allocates a resource while measuring none. No FLOPs, no wall-clock time, no memory, no bandwidth, and no statement of the source image resolution. Since all candidates are encoded at fixed size, patches at different precisions cost the same to encode — so the precision axis is free at encoding time and the only real cost sits in the sensor and cropping pipeline, which is never measured or even specified.

  3. The conceptual core (the Jacobian) is neither used nor verified. The conceptual core is a local sensitivity relation, yet the paper states plainly that the Jacobian is neither estimated nor optimized, only used to explain. So the central claim — that the learned allocation really does match precision to action sensitivity — is never tested, even though perturbing selected versus unselected locations and measuring the resulting action change is straightforward with the tools already present.

  4. The abstract's wording does not match the scope of the evidence. Six checkpoints, one benchmark family, simulation only. The abstract promises a general statement about the practice, while the evidence supports a statement about these policies on this suite at this ceiling proportion — and the ceiling proportion is itself policy-dependent.

  5. **The decisive control is missing, and it costs one experiment.**Run the base at higher input resolutions with no router, no residual memory, and no added module at all. If a higher-resolution base matches or exceeds the reported result, the method is an elaborate partial implementation of raising input resolution. If it does worse, that is a genuinely surprising and publishable finding — but it must be shown, not assumed. That it was not run is the single largest gap in the experimental design.

Questions for authors
  1. What is the source resolution of the high-resolution input, and is it the same sensor stream that produces the base's low-resolution input? If it is a higher-resolution render, please state the factor and count it in the budget.
  2. Will you run the base at higher input resolutions with no module added? What is the result?
  3. On the strongest base the result is null with most suites negative. Should the cross-architecture integration claim in the abstract be qualified?
  4. Are there any compute, latency, FLOP, or bandwidth measurements? If all precisions cost the same to encode, where is the allocated resource?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.