- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies when a visuomotor reinforcement learning policy should use an expensive visual modality. The paper first examines several in-loop gating mechanisms and identifies several failure modes, including teacher-relative value gaps, gate-critic co-evolution, and cold-start credit lock-in. It then proposes post-hoc counterfactual gating, where an any-gate policy is trained with mask dropout and kept fixed during inference, while visual usage is determined through value or action counterfactuals. The paper also derives a performance-difference bound for action counterfactual gating and evaluates the approach on Franka reach environments, a mixed-domain setting, and a PiPER robot. While the proposed idea is interesting, the current experiments do not provide sufficient evidence for the paper's broader claim that in-loop gating generally fails. The evaluation mainly focuses on custom manipulation environments and PPO, with limited comparison to existing information-gating methods and limited validation across different tasks, RL algorithms, and partially observable settings.
Critical strengths
- The paper addresses an important and practically relevant problem. Reducing the computational cost of visual observations in visuomotor RL is valuable, particularly when visual processing is substantially more expensive than proprioceptive processing.
- The paper provides a useful analysis of failure modes in in-loop gating. The discussion of teacher-relative value gaps, gate-critic co-evolution, and cold-start credit lock-in offers plausible explanations for the unstable behavior of several learned gating mechanisms.
- The mixed-domain setting is a useful experimental design. It tests whether a gating mechanism can distinguish between episodes where vision is necessary and those where vision is redundant within the same training process.
- The action-counterfactual formulation has a meaningful theoretical motivation. The performance-difference bound provides a principled connection between changes in the policy action distribution and the potential performance impact of removing visual information.
- The paper includes several diagnostic experiments and ablations. Rather than only reporting the final performance, the paper attempts to identify why different in-loop gating mechanisms fail.
Critical weaknesses
- The main claim is broader than what the experiments support. The paper argues that learned in-loop gating mechanisms generally fail and motivates post-hoc gating as a more reliable alternative. However, most of the evidence comes from a relatively small set of custom manipulation environments. This experimental scope is too narrow to support such a broad conclusion about visuomotor RL.
- The task diversity is limited. Most experiments focus on Franka reach and related manipulation scenarios. Although the PiPER experiment provides additional validation on a different robot platform, it remains within the same general task family. The paper therefore provides limited evidence that the conclusions generalize to substantially different settings such as navigation, locomotion, or long-horizon interaction.
- The evaluation relies heavily on PPO. The observed failure modes may depend on the optimization dynamics of PPO, the policy architecture, or other training choices. The paper provides limited evidence that the same conclusions hold for substantially different RL algorithms.
- The evaluation does not sufficiently consider partial observability. The paper does not provide experiments with recurrent policies or explicit POMDP settings. Such settings are particularly relevant because the usefulness of visual information can depend on temporal context and hidden state.
- The practical computational advantage is sensitive to the cost model. The paper reports that the visual forward pass takes only 0.88 ms on the reported CPU in one latency analysis, and the latency-based objective favors always-on execution. This raises concerns about how robust the reported computational benefits are to different hardware, visual encoders, and definitions of computational cost.
- The reported compute savings may not directly translate to practical speedups. The evaluation does not sufficiently account for factors such as memory transfer, GPU utilization, batching, energy consumption, and wall-clock latency. These factors can affect the actual benefit of selectively executing the visual branch.
- The generalization analysis is limited. The experiments provide evidence that the method works under the selected environments and policy architectures, but they do not establish robustness to substantially different visual encoders, policy architectures, observation spaces, or task distributions.
- The controlled environments may not fully reflect realistic modality redundancy. The Leak and Mixed settings are useful for isolating specific failure modes, but their controlled construction may make the distinction between vision-essential and vision-redundant cases relatively explicit. More realistic and naturally occurring sources of visual redundancy would provide stronger evidence for the proposed approach.
- The theoretical analysis does not fully address the empirical limitations. The performance-difference bound provides useful theoretical motivation for the action-counterfactual criterion, but it does not by itself establish that this criterion is consistently reliable or preferable to alternative gating strategies in practical RL systems.
Questions for authors
- Do the observed failures of in-loop gating persist with RL algorithms other than PPO?
- How does the proposed method perform with recurrent policies or in POMDP settings?
- Can the authors evaluate the method on substantially different task families beyond manipulation?
- How sensitive are the reported compute savings to the visual encoder, hardware, and cost model?
- What is the computational overhead of estimating the proposed counterfactual gating score?
- Do the reported compute savings remain under a realistic wall-clock or energy-based cost model?
- How does the method perform with larger visual encoders or transformer-based cross-attention architectures?
2 comments
Official Comment by Reviewer gW5y
Thank you for the substantial revision and for adding the frozen-critic control, the PushCube experiments, the recurrent-policy analysis, and the encoder-latency measurements. These additions strengthen the paper and clarify the proposed gate–critic feedback mechanism. However, several concerns remain only partially addressed. The experiments do not establish whether the findings generalize beyond PPO or manipulation, the recurrent-policy result is theoretical rather than empirical, and the practical compute benefits are not validated with end-to-end wall-clock or energy measurements. The added latency figures and architectural discussion also do not directly evaluate the method with larger encoders or cross-attention policies. I appreciate the authors’ efforts, but given these remaining gaps, I will maintain my original score.
Thank you for the careful review. Below, "retained" is return as a percentage of the always-on policy, and "saved" is the percentage of computation skipped.
Claim broader than the evidence. We agree that the conclusion should state what we tested. The revision does (the four families, PPO, our settings), and we propose the title "Infer When to Look" for the final version. We also tested the proposed cause directly. When the hard gate's label comes from a frozen critic, the collapse disappears: the gate stays open in all three seeds (0.38–0.72 where vision is needed, 0.23–0.48 where it is redundant), against an open rate of 0 with the loop intact.
Task diversity (Q3). The evidence spans two robots, Franka and an independent 6-DoF PiPER arm with strong domain randomization, and conditions in which vision is essential, provably redundant, or mixed within one run. On ManiSkill3 PushCube, a standard benchmark, no learned gate tells the two kinds of episodes apart: the hard gate closes on both, and soft attention and InfoGating stay open on both. Navigation and locomotion are outside this revision.
Dependence on PPO (Q1). The loop we identify is not specific to PPO: the gate's label comes from a critic trained on states the gate itself selected. An off-policy learner trains on a replay buffer that the gated policy fills, so the same loop exists. Consistent with this, an independently trained off-policy critic, probed offline, separates the domains by only 3.8–4.4×, against 33× inside the loop (Sec. 5.3).
Partial observability (Q2). For a recurrent policy, a skip also changes later actions through the hidden state, so the bound is stated along the observation history ht (new Remark 4.8): J(πo)−J(πg) ≤ 1−γ2Amax Eht∼ρπo[TV(πo(⋅∣ht), πg(⋅∣ht))], where πo is the always-on policy and πg the gated one. For memoryless policies it reduces to Prop. 4.5.
Cost model, wall-clock, and encoders (Q4, Q6, Q7). The savings we report are fractions of the policy's computation, so they do not depend on hardware; the speedup does. If a fraction o of steps runs the encoder and r is the ratio of encoder time to the rest of the step, the per-step speedup is 1+o⋅r1+r, approaching 1/o as the encoder dominates.
Overhead of the gating score (Q5). Computing the signal at every step would save nothing, because the signal needs the visual pass (Remark 4.11 of the revision). We use two realizations.
Leak and Mixed are constructed. By design: they make visual redundancy provable (Lemma J.1), so a gate's decisions can be scored against ground truth. In Cond A and B the redundancy is not constructed: which episodes can skip vision is not known in advance, and the gate's choice beats random and periodic skipping at the same savings by 5.7–22.2 percentage points.
What the theory establishes.
It is a certificate rather than a tight estimate of the loss.
Which of these matters most for your assessment? We can still add analysis within the discussion period.