acceptodds
Under review as a conference paper at ICLR 2027

Don't Learn When to Look: Post-hoc Counterfactual Gating for Visuomotor RL

Abstract

A visuomotor policy that also senses its own body pays for the camera at every step in our architecture, 98.8% of policy computation, whether or not vision is useful right now. The natural fix is a learned gate that decides when to look. We show that every in-loop version of this idea fails, and that a simple change works immediately: don’t learn when to look; infer it from the frozen policy. The diagnosis comes first. The gradient-corruption premise behind gated fusion is unsupported. Teacher-relative value gaps measure how well the student learned, not whether vision helps, and no threshold repairs them. A corrected counterfactual label produces striking domain separation which mixed-domain training, a protocol we propose, exposes as gate-critic coevolution. Three further mechanisms (soft attention, expert routing, a reward-level observation cost) either cannot discriminate, or weight vision inversely to its utility, a cold-start lock-in that changing visual difficulty does not move. Then the remedy. Train one any-gate policy with mask dropout; gate it post hoc with counterfactual signals read from the frozen network. A performance-difference bound attaches to the action counterfactual and yields a frontier with guaranteed loss. Empirically, across three seeds and four domains, post-hoc gating sits above the always-on/quota convex hull, beats a random-episode control by 12–27pp at matched savings where vision is essential, is return-neutral (98%) at 17% savings, and in mixed domains outperforms even a domain-label reference; a distilled proprioception-only predictor gates without looking at 39% real savings and no loss. No labels, no retraining. When to look is not something a policy should learn; it is something to infer from one after training.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, robot learning, visuomotor policies, conditional computation, active perception

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gW5yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when a visuomotor reinforcement learning policy should use an expensive visual modality. The paper first examines several in-loop gating mechanisms and identifies several failure modes, including teacher-relative value gaps, gate-critic co-evolution, and cold-start credit lock-in. It then proposes post-hoc counterfactual gating, where an any-gate policy is trained with mask dropout and kept fixed during inference, while visual usage is determined through value or action counterfactuals. The paper also derives a performance-difference bound for action counterfactual gating and evaluates the approach on Franka reach environments, a mixed-domain setting, and a PiPER robot. While the proposed idea is interesting, the current experiments do not provide sufficient evidence for the paper's broader claim that in-loop gating generally fails. The evaluation mainly focuses on custom manipulation environments and PPO, with limited comparison to existing information-gating methods and limited validation across different tasks, RL algorithms, and partially observable settings.

Critical strengths
  1. The paper addresses an important and practically relevant problem. Reducing the computational cost of visual observations in visuomotor RL is valuable, particularly when visual processing is substantially more expensive than proprioceptive processing.
  2. The paper provides a useful analysis of failure modes in in-loop gating. The discussion of teacher-relative value gaps, gate-critic co-evolution, and cold-start credit lock-in offers plausible explanations for the unstable behavior of several learned gating mechanisms.
  3. The mixed-domain setting is a useful experimental design. It tests whether a gating mechanism can distinguish between episodes where vision is necessary and those where vision is redundant within the same training process.
  4. The action-counterfactual formulation has a meaningful theoretical motivation. The performance-difference bound provides a principled connection between changes in the policy action distribution and the potential performance impact of removing visual information.
  5. The paper includes several diagnostic experiments and ablations. Rather than only reporting the final performance, the paper attempts to identify why different in-loop gating mechanisms fail.
Critical weaknesses
  1. The main claim is broader than what the experiments support. The paper argues that learned in-loop gating mechanisms generally fail and motivates post-hoc gating as a more reliable alternative. However, most of the evidence comes from a relatively small set of custom manipulation environments. This experimental scope is too narrow to support such a broad conclusion about visuomotor RL.
  2. The task diversity is limited. Most experiments focus on Franka reach and related manipulation scenarios. Although the PiPER experiment provides additional validation on a different robot platform, it remains within the same general task family. The paper therefore provides limited evidence that the conclusions generalize to substantially different settings such as navigation, locomotion, or long-horizon interaction.
  3. The evaluation relies heavily on PPO. The observed failure modes may depend on the optimization dynamics of PPO, the policy architecture, or other training choices. The paper provides limited evidence that the same conclusions hold for substantially different RL algorithms.
  4. The evaluation does not sufficiently consider partial observability. The paper does not provide experiments with recurrent policies or explicit POMDP settings. Such settings are particularly relevant because the usefulness of visual information can depend on temporal context and hidden state.
  5. The practical computational advantage is sensitive to the cost model. The paper reports that the visual forward pass takes only 0.88 ms on the reported CPU in one latency analysis, and the latency-based objective favors always-on execution. This raises concerns about how robust the reported computational benefits are to different hardware, visual encoders, and definitions of computational cost.
  6. The reported compute savings may not directly translate to practical speedups. The evaluation does not sufficiently account for factors such as memory transfer, GPU utilization, batching, energy consumption, and wall-clock latency. These factors can affect the actual benefit of selectively executing the visual branch.
  7. The generalization analysis is limited. The experiments provide evidence that the method works under the selected environments and policy architectures, but they do not establish robustness to substantially different visual encoders, policy architectures, observation spaces, or task distributions.
  8. The controlled environments may not fully reflect realistic modality redundancy. The Leak and Mixed settings are useful for isolating specific failure modes, but their controlled construction may make the distinction between vision-essential and vision-redundant cases relatively explicit. More realistic and naturally occurring sources of visual redundancy would provide stronger evidence for the proposed approach.
  9. The theoretical analysis does not fully address the empirical limitations. The performance-difference bound provides useful theoretical motivation for the action-counterfactual criterion, but it does not by itself establish that this criterion is consistently reliable or preferable to alternative gating strategies in practical RL systems.
Questions for authors
  1. Do the observed failures of in-loop gating persist with RL algorithms other than PPO?
  2. How does the proposed method perform with recurrent policies or in POMDP settings?
  3. Can the authors evaluate the method on substantially different task families beyond manipulation?
  4. How sensitive are the reported compute savings to the visual encoder, hardware, and cost model?
  5. What is the computational overhead of estimating the proposed counterfactual gating score?
  6. Do the reported compute savings remain under a realistic wall-clock or energy-based cost model?
  7. How does the method perform with larger visual encoders or transformer-based cross-attention architectures?
2 comments

Thank you for the careful review. Below, "retained" is return as a percentage of the always-on policy, and "saved" is the percentage of computation skipped.

Claim broader than the evidence. We agree that the conclusion should state what we tested. The revision does (the four families, PPO, our settings), and we propose the title "Infer When to Look" for the final version. We also tested the proposed cause directly. When the hard gate's label comes from a frozen critic, the collapse disappears: the gate stays open in all three seeds (0.38–0.72 where vision is needed, 0.23–0.48 where it is redundant), against an open rate of 0 with the loop intact.

Task diversity (Q3). The evidence spans two robots, Franka and an independent 6-DoF PiPER arm with strong domain randomization, and conditions in which vision is essential, provably redundant, or mixed within one run. On ManiSkill3 PushCube, a standard benchmark, no learned gate tells the two kinds of episodes apart: the hard gate closes on both, and soft attention and InfoGating stay open on both. Navigation and locomotion are outside this revision.

Dependence on PPO (Q1). The loop we identify is not specific to PPO: the gate's label comes from a critic trained on states the gate itself selected. An off-policy learner trains on a replay buffer that the gated policy fills, so the same loop exists. Consistent with this, an independently trained off-policy critic, probed offline, separates the domains by only 3.8–4.4×, against 33× inside the loop (Sec. 5.3).

Partial observability (Q2). For a recurrent policy, a skip also changes later actions through the hidden state, so the bound is stated along the observation history hth_t (new Remark 4.8): J(πo)−J(πg) ≤ 2Amax⁡1−γ Eht∼ρπo[TV(πo(⋅∣ht), πg(⋅∣ht))],J(\pi_o)-J(\pi_g)\ \le\ \frac{2A_{\max}}{1-\gamma}\ \mathbb{E}_{h_t\sim\rho^{\pi_o}}\Big[\mathrm{TV}\big(\pi_o(\cdot\mid h_t),\ \pi_g(\cdot\mid h_t)\big)\Big], where πo\pi_o is the always-on policy and πg\pi_g the gated one. For memoryless policies it reduces to Prop. 4.5.

Cost model, wall-clock, and encoders (Q4, Q6, Q7). The savings we report are fractions of the policy's computation, so they do not depend on hardware; the speedup does. If a fraction oo of steps runs the encoder and rr is the ratio of encoder time to the rest of the step, the per-step speedup is 1+r1+o⋅r\frac{1+r}{1+o\cdot r}, approaching 1/o1/o as the encoder dominates.

  • Measured latency per step at batch size 1, with rendering and host-to-device transfer as separate items: our CNN takes 0.35 ms on GPU and 0.54 ms on one CPU thread, ResNet-50 takes 5.3 ms and 81 ms, and ViT-B/14 takes 5.0 ms and 313 ms.
  • Speedup. With our small encoder the gain is small, as you note (Sec. 6). With standard encoders, in Cond A (vision needed) it is 1.1–1.3× at 20% saved and 1.3–2.1× at 52% saved. On PiPER, the predictor saves 62% with 99.8% retained, about 2.6× for ResNet-50 on one CPU thread.
  • Batching. Skipped images can be removed from a batch, because the decision is made before the visual pass.
  • Architectures. The method needs only a policy that can act with its visual input masked, so it does not depend on the encoder or the fusion design (with cross-attention, the visual keys and values are masked).

Overhead of the gating score (Q5). Computing the signal at every step would save nothing, because the signal needs the visual pass (Remark 4.11 of the revision). We use two realizations.

  • Probe and commit. The signal is computed only on the first 5 steps of an episode (and every 50 steps when re-deciding). These steps count as executed in all reported savings.
  • Predictor. A small network predicts the signal from the joint state alone, so the decision needs no image.

Leak and Mixed are constructed. By design: they make visual redundancy provable (Lemma J.1), so a gate's decisions can be scored against ground truth. In Cond A and B the redundancy is not constructed: which episodes can skip vision is not known in advance, and the gate's choice beats random and periodic skipping at the same savings by 5.7–22.2 percentage points.

What the theory establishes.

  • (i) A certificate that holds for every gate and every threshold on a frozen policy, now extended to recurrent policies.
  • (ii) The optimality of thresholding the action counterfactual under that certificate (new Remark 4.7): no gate with the same certified loss closes more of the always-on state distribution.
  • (iii) An ordering the experiments confirm: across the thresholds of our gates, the rank correlation between the quantity the bound charges and the realized loss is 0.96, 0.97, 0.85, and 0.96 in Cond A, Cond B, Mixed, and Leak. On PiPER no threshold loses return, so there is nothing to order.

It is a certificate rather than a tight estimate of the loss.

Which of these matters most for your assessment? We can still add analysis within the discussion period.

Reviewer_gW5yOpenReview ↗

Official Comment by Reviewer gW5y

Thank you for the substantial revision and for adding the frozen-critic control, the PushCube experiments, the recurrent-policy analysis, and the encoder-latency measurements. These additions strengthen the paper and clarify the proposed gate–critic feedback mechanism. However, several concerns remain only partially addressed. The experiments do not establish whether the findings generalize beyond PPO or manipulation, the recurrent-policy result is theoretical rather than empirical, and the practical compute benefits are not validated with end-to-end wall-clock or energy measurements. The added latency figures and architectural discussion also do not directly evaluate the method with larger encoders or cross-attention policies. I appreciate the authors’ efforts, but given these remaining gaps, I will maintain my original score.

Reviewer_TsqPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether vision is needed or not for visuomotor policy and proposes a few gating heuristics.

Critical strengths

The paper did some interesting diagnosis regarding policy vision inputs.

The paper honestly report negative results of their heuristic methods.

Critical weaknesses

a. The paper is difficult to follow, which is the main obstacle to assessing its methodology and contribution. The abstract starts with a claim specific to the author's own architecture (“98.8% of policy computation”), which doesn’t by itself establish a general motivation. The abstract then introduces several concepts without context (“gradient-corruption premise,” “teacher-relative value gaps,” “Cond A/B, Leak, Mixed”) that only make sense after reading Sections 3–5. The introduction has the same issue. The “four mechanism families (label-supervised hard gates, soft attention, expert routing, reward-level observation costs)” appear without any context, so it is hard to tell what point is being made or why these four were chosen. I’d suggest improving writing so that the problem, the proposed method, and the main result are stated clearly.

b. The experiments are conducted with a single algorithm (PPO) on a small set of reach tasks. It is too narrow to support the broad claims in the title and conclusion about when vision is or isn’t needed, or that in-loop gating fails in general. The negative result is reported honestly (e.g., "all in-loop gating fails”), but without evidence on standard benchmarks or other RL algorithms it’s unclear whether it reflects a general conclusion or just findings of this particular setup. The comparisons are mostly against the authors’ own heuristic variants (quota mixtures, random-episode rules, entropy/speed heuristics) rather than established baselines; a comparison to published methods would make the work more convincing.

Questions for authors

see weaknesses

3 comments

Thank you for the specific suggestions. Below, "points" are percentage points of return retained relative to the always-on policy.

"Proposes a few gating heuristics." The paper proposes one gate, and it is not a heuristic. It thresholds the action counterfactual of a frozen policy, the quantity that the performance-difference bound of Prop. 4.5 charges. Because the bound is taken over the always-on state distribution, minimizing visual computation subject to it is a fractional knapsack whose exact solution is this threshold rule (new Remark 4.7).

(a) Presentation. Thank you for the precise diagnosis; we followed it point by point in the revised PDF.

  • The motivation now comes from the cost structure of visuomotor policies; the 98.8% figure moves to Sec. 3 as a measurement of our network.
  • "Gradient-corruption premise", "teacher-relative value gaps", and the domain names no longer appear in the abstract or introduction.
  • The four families come with the reason for choosing them: they are four ways a gating decision can enter an RL agent (a supervised label, a differentiable weight, a routing choice, a rewarded action).

(b) Scope. We agree that the evidence supports a claim about the families and settings we test, not about in-loop gating in general, and the revised conclusion says so. We also propose the title "Infer When to Look" for the final version. The failure has a structural cause: when a gate's training signal comes from a critic trained on data the gate itself shaped, label and gate form a feedback loop, and nothing in this argument is specific to PPO or to reaching. Three results support this.

  • A causal test (new). We retrained the hard gate in the mixed domain with its label taken from a frozen critic. With the loop intact, the gate closes on both kinds of episodes (open rate 0). With the loop cut, it stays open in all three seeds (0.38–0.72 where vision is needed, 0.23–0.48 where it is redundant).
  • A standard benchmark (new). On ManiSkill3 PushCube, the hard gate again closes on both kinds of episodes (open rate 0 in both seeds) and then fails the episodes that need vision (success 0.08–0.11, against 0.95–0.96 for always-on). Soft attention stays open on both kinds (weight 0.95–0.97). On Franka it separated them in the wrong direction; here it does not separate them at all.
  • Real computation. All four in-loop mechanisms compute their decision from visual features, so they can skip at most 1% of the computation.

(b) Comparisons. Our main baselines are the learned gates, which follow published formulations and are reimplemented in one codebase so that differences are not implementation artifacts. In Cond A, relative to the always-on policy trained with the same recipe:

MethodReturn retainedComputation actually saved
Always-on100%0%
Learned hard gate78.6%below 1%
Soft attention45.1%below 1%
Mixture-of-experts router (Jacobs et al., 1991; Shazeer et al., 2017)92.6%below 1%
Observation cost (AMRL, Bellinger et al., 2020; Act-Then-Measure, Krale et al., 2023)93.4%below 1%
Ours, decided per episode96.7%20.4%
Ours, re-decided every 50 steps92.4%51.6%

Every learned gate does worse than simply leaving vision on: it loses return and saves no computation (Appendix Table 11 and the corrected Table 2). The random rule and the fixed-rate mixtures are controls, not methods. Two further comparisons:

  • A lower camera rate (new). Looking only every 2, 5, 10, or 20 steps is the standard way to cut visual cost. At the same savings, it and random skipping are 5.7–22.2 points below our gate where vision is needed (Cond A and B) and 1.5–2.8 points below on PiPER (paired 95% intervals above zero in all cases but one). In Leak and Mixed, where skipping costs little, the differences are within noise.
  • An event trigger (new). Looking again whenever the joint state has changed enough is a strong simple rule on the same frozen policy. It ties our gate in Cond A (at 20% savings, the range it reached), Cond B, and Leak, is 1.0–1.4 points lower on PiPER, and 1–7 points higher in Mixed. It carries no guarantee, but it shows that simple rules work once the decision is taken out of training. The revision reports it and drops the claim that cheap signals cannot replace the counterfactual.
  • InfoGating (Tomar et al., 2023) computes a soft mask from the full observation under an L1 penalty. Because the mask is computed from the image, it cannot skip the visual pass. We tested its objective at the modality level on PushCube with two penalty weights a factor of 10 apart. The gate stays fully open on both kinds of episodes (weight above 0.99), so it does not discriminate either.

If a specific published baseline beyond these would change your assessment, please tell us and we will try to add it within the discussion period.

Reviewer_TsqPOpenReview ↗

comments

We thank the authors for adding experiments and rewriting the paper. I agree with other reviewers regarding the weaknesses of the paper and I find the paper still difficult to read and follow after rewriting. I keep the rating.

Thank you for reading the revision and for your reply. We would like to make the paper easier to follow in the time that remains, and two pieces of information would help us most.

  1. Which parts are still hard to follow? For example: the abstract and introduction, the setup and the names of the settings (Sec. 3), the method (Sec. 4), or the results (Sec. 5). We will revise the parts you name and post the changes before the discussion closes. In the meantime we are replacing the shorthand labels in the tables and in Sec. 5 with plain names and adding a short table of terms.

  2. Which remaining weakness weighs most for you? The other reviews raised the scope of the conclusion, the range of tasks, and the baselines. The revision answers these in its last appendix: a causal test of the failure (Table 12), a second benchmark, ManiSkill3 PushCube (Table 13), and comparisons with random and periodic skipping and an event trigger (Table 14).

In case it helps, here is the paper in three sentences.

  • Learning a gate inside RL training fails in the four ways we test. A learned hard gate closes on episodes that need vision as well as on those that do not, and this stops when the critic that labels it is frozen.
  • We therefore decide after training. We freeze a policy that can act with or without the image and skip the image where it would barely change the action, with a bound on the return this can lose.
  • On the main reaching task this saves 20% of the computation while keeping 96.7% of the return, and on a second robot 44–62% while keeping the full return. It beats learned gates and random or periodic skipping. A simple event trigger performs comparably: it ties in three settings, is slightly behind on the second robot, and ahead in the mixed domain.
Reviewer_CHXmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes failures of learned visual gating in visuomotor RL and proposes post-hoc gating based on counterfactual signals from a frozen policy.

Critical strengths

None

Critical weaknesses
  1. The writing is confusing and difficult to follow.

  2. The experiments are limited to simulation, with no evaluation on a real robot.

  3. The main conclusion is too broad for the evidence provided. Failures of several gating methods in a limited set of environments do not establish that learning when to look is generally ineffective.

  4. The practical benefit is unclear. Reduced computation does not necessarily translate into lower latency or better deployment performance, and the paper does not sufficiently demonstrate these benefits.

  5. Beating trivial baselines is not sufficiently convincing.

Questions for authors

See weaknesses.

1 comment

Thank you for the review. We take your points in order; the general response has the full results. Below, "retained" is return as a percentage of the always-on policy, "saved" is the percentage of computation skipped, and "points" are percentage points of retained return.

1. Writing. The abstract and introduction are rewritten so that the problem, the method, and the result come first, with every term defined where it appears. Changed text is in blue in the revised PDF.

2. Simulation only. Our claims concern whether a gate's decisions track the real need for vision, and that can be scored only where the need is known: in Leak (target positions given in the state), vision is provably redundant (Lemma J.1), and in Mixed each episode's type is known. A real robot cannot provide this ground truth, so simulation is where these claims can be tested. A real-robot run would test transfer, a separate question outside this revision.

3. Conclusion too broad. We agree that the conclusion should state what we tested. The revision does (the four families, PPO, our settings), and we propose the title "Infer When to Look: Post-hoc Counterfactual Gating for Visuomotor RL" for the final version. The failure itself has a structural cause rather than a setup-specific one: whenever a gate's training signal comes from a learner trained on data the gate shapes, the two co-evolve. We have now tested this directly. When the gate's label comes from a frozen critic, so that the gate cannot change its own label, the collapse disappears: the gate stays open in all three seeds, against an open rate of 0 with the loop intact. The same collapse appears on a second benchmark, ManiSkill3 PushCube: the hard gate closes on both kinds of episodes and then fails those that need vision (success 0.08–0.11, against 0.95 for always-on). The main contribution, a gate that decides after training with a certified loss bound, does not depend on the in-loop claim.

4. Practical benefit. We measured it (batch size 1, median):

EncoderGPUCPU, 1 thread
Our 64×64 CNN0.35 ms0.54 ms
ResNet-505.3 ms81 ms
ViT-B/145.0 ms313 ms

Reduced computation does translate into lower latency once the visual pass dominates the step. If a fraction oo of steps runs the encoder and rr is the ratio of encoder time to the rest of the step, the per-step speedup is 1+r1+o⋅r\frac{1+r}{1+o\cdot r}, close to 1/o1/o for standard encoders. In Cond A (vision needed) it is 1.1–1.3× at 20% saved and 1.3–2.1× at 52% saved. On PiPER, the predictor, which decides without the image, saves 62% with 99.8% retained, about 2.6× for ResNet-50 on one CPU thread. With our small CNN on a GPU the gain is small, as Sec. 6 states.

5. Baselines. Our main baselines are the learned gates, since they are what one would otherwise use. In Cond A, relative to the always-on policy trained with the same recipe:

MethodReturn retainedComputation actually saved
Always-on100%0%
Learned hard gate78.6%below 1%
Soft attention45.1%below 1%
Mixture-of-experts router92.6%below 1%
Observation cost (as in AMRL)93.4%below 1%
Ours, decided per episode96.7%20.4%
Ours, re-decided every 50 steps92.4%51.6%

Every learned gate does worse than simply leaving vision on. It loses return, and it saves no computation because it reads visual features to decide (Appendix Table 11; the hard gate is re-evaluated with paired episodes). Our gate is the only one that saves computation. Two simpler references confirm this:

  • A lower camera rate (new). Looking only every k-th step is the standard way to cut visual cost. Where vision is needed, it and random skipping are 5.7–22.2 points below our gate at the same savings (paired 95% intervals above zero in all cases but one).
  • Fixed-rate mixtures. The best mixture of always-on and fixed-rate gating is 7.9–16.5 points below our gate, and the learned hard gate is 15.3 points below that mixture.
  • An event trigger (new). Looking again whenever the joint state has changed enough is a strong simple rule on the same frozen policy. It ties our gate in Cond A (at 20% savings, the range it reached), Cond B, and Leak, is 1.0–1.4 points lower on PiPER, and 1–7 points higher in Mixed. It carries no guarantee, but it shows that simple rules work once the decision is taken out of training, where learned gates do not.

Do the measured speedups and the comparison with learned gates address your concerns about practical benefit and baseline strength?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new experiments, changes in the revision, and re-evaluated results

We thank the reviewers. Changed text in the revised PDF is in blue.

New experiments

  • Cutting the feedback loop removes the collapse. We retrained the learned hard gate in the mixed domain with its label taken from a frozen critic. With the loop intact, the gate closes on both kinds of episodes (open rate 0). With the loop cut, it stays open in all three seeds (0.38–0.72 where vision is needed, 0.23–0.48 where it is redundant). This confirms the cause proposed in Sec. 5.3.
  • A second benchmark. On ManiSkill3 PushCube, no learned gate tells the two kinds of episodes apart. The hard gate closes on both (open rate 0) and then fails the episodes that need vision (success 0.08–0.11, against 0.95 for always-on). Soft attention and an InfoGating-style gate stay open on both.
  • A simple post-hoc rule. An event trigger on joint-state change, applied to the same frozen policy, performs about as well as our gate: it ties in Cond A (at 20% savings, the range it reached), Cond B, and Leak, is 1.0–1.4 points lower on PiPER, and 1–7 points higher in Mixed. Simple rules work once the decision is taken out of training; ours is the one with a guarantee. The revision reports this and drops the claim that cheap signals cannot replace the counterfactual.

What the revision changes

  • Writing and scope. The abstract and introduction are rewritten: problem, method, and result first, every term defined where it appears. The conclusion states the scope we test (the four families, PPO, our settings), and we propose the title "Infer When to Look: Post-hoc Counterfactual Gating for Visuomotor RL" for the final version.
  • Theory. New Remark 4.7: our threshold rule is the optimal gate under the bound of Prop. 4.5. New Remark 4.8 extends the bound to recurrent policies.
  • Evaluation protocol. robosuite draws the cube's yaw from its own random generator, which our evaluation seeds did not control, so compared runs saw different episodes. The revision seeds it and re-evaluates all post-hoc results (3 seeds, 100 paired episodes; 300 for the PiPER value-counterfactual row).
  • Table 2. For the training-time gates, the computation column reported the fraction of gated steps (10.9%, 40.4%, 64.5%), not computation saved. Their real saving is at most 1%: all four in-loop mechanisms and the quota gates compute their decision from visual features, so the encoder runs at every step.

Re-evaluated results. Retained is return as a percentage of the always-on policy on the same episodes; saved is the percentage of computation skipped; points are percentage points of retained return. Vision is needed in Cond A and B and provably redundant in Leak; Mixed has both kinds; PiPER is a second robot.

SettingGateSubmission: retainedsavedRevision: retainedsaved
Cond AAction counterfactual, per episode98.3%17.3%96.7 ± 0.6%20.4%
Cond AAction counterfactual, every 50 steps91.8%45.9%92.4 ± 1.6%51.6%
Cond BValue counterfactual, per episode85.8%25.9%94.9%21.7%
PiPERValue counterfactual, every 50 steps99.4%39.5%100.4 ± 0.8%43.7%
LeakPredictor (joint state only)105.5%36.4%100.0%36%
MixedPredictor (joint state only)104.1%39.3%94.6%48%
PiPERPredictor (joint state only)——99.8%62%

Values above 100% reflect unpaired episodes (submission) or seed variation (revision), not gains. In Cond A, our gate is now 8.6 and 18.4 points above random skipping (submission: 18.8 and 26.8) and 7.9 and 16.5 points above the best mixture of always-on and fixed-rate gating (submission: 7 and 13); the learned gate is 15.3 points below that mixture (submission: 12.3). In Mixed, our gate ties the domain-label reference (return 226.2 vs. 226.7; submission: 231.0 vs. 213.4), and the predictor retains 94.6%, not the full return.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.