acceptodds
Under review as a conference paper at ICLR 2027

WHETHER VS. WHEN: SEPARATING SHORTCUT FORMATION FROM EVIDENCE-TIMING SENSITIVITY IN VALUE-BASED ONLINE REINFORCEMENT LEARNING

Abstract

Deep reinforcement learning agents can overweight early experience, a phenomenon studied as primacy bias, but whether a shortcut forms and how strongly its formation depends on the timing of disambiguating evidence are distinct properties that this framing can conflate. We separate them in a controlled visual task where a salient, spurious feature and a subtler, reward-relevant feature agree except at planted evidence cells. Holding the content and quantity of evidence fixed, we relocate the same 40 evidence-cell visits from the first to the last 400 steps of a 4000-step value-based online Double DQN training window. This temporal shift changes shortcut formation from 0 of 30 to 30 of 30 paired seeds, with a mean paired margin difference of −6.17 (95% CI [−6.41, −5.93]). The split survives interventions targeting action coverage, exploration, evidence dose, integration time, replay-sampling exposure, and optimizer-update count, although inward placement and closer matching of integration time attenuate it. The effect also persists across a smaller vision transformer, a structurally distinct corridor environment, and fully organic, action-driven visitation (0/30 vs. 30/30), though realized exposure there is endogenous and unequal rather than matched. In contrast, matched fixed-dataset supervised and bootstrapped-TD learners show substantially smaller timing effects, while a matched on-policy PPO agent shows no timing-dependent split despite forming the same shortcut. Removing experience replay also closes the split, but batch size and gradient variance change simultaneously, so this does not identify replay as the unique cause. A rewind intervention further points to the role of accumulated network state in how evidence is incorporated. Together, these results show that shortcut formation is not unique to online reinforcement learning, but its dependence on evidence timing is substantially amplified in the value-based online-RL regime studied here, while leaving the precise mechanism unresolved.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, deep reinforcement learning, shortcut learning, primacy bias, continual learning, experience replay, value-based reinforcement learning, representation learning, nonstationarity

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_3HfyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether reliance on a spurious cue and its sensitivity to the timing of disambiguating evidence can differ across learning protocols. In a controlled task, it moves the same 40 evidence-cell visits between early and late windows and measures relative reliance on the spurious and reward-relevant features. Under its pre-shift formation criterion, Double DQN exhibits a 0/30 versus 30/30 split, whereas the tested fixed-dataset learners show smaller differences and a single PPO configuration forms shortcuts under both schedules; additional architectures, environments, and interventions examine the scope of this observation.

Critical strengths
  1. The task makes the learned shortcut interpretable: the spurious and reward-relevant cues are separately manipulated, and behavioral reliance is measured rather than inferred from reward alone. The corrected paired construction matches visited-cell content and evidence count, making the within-protocol comparison informative and easier to audit.

  2. The paper tests several alternative explanations and demonstrates a repeatable empirical contrast within its studied settings. The DQN timing gap persists under forced action coverage, and it is attenuated but remains substantial under interior windows and approximately matched integration time. TinyViT and the corridor task provide additional evidence across architecture and environment. These controls strengthen the empirical characterization without identifying a unique mechanism.

Critical weaknesses
  1. The no-replay control appears to change whether EARLY evidence reaches optimization at all, in addition to changing replay and batch size. Appendix Q retains a 500-step warmup and uses only the transition produced at each update step; it reports 11,501 updates over 12,000 steps, with no transition reuse. Section 3.1 places all EARLY evidence visits in [0,400). Under this description, those evidence transitions would never enter a gradient update, whereas LATE evidence could be used. If confirmed, the 29/30 versus 29/30 outcome would not establish what happens when both schedules' evidence actually reaches optimization without replay. This confound goes beyond the acknowledged batch-size/gradient-variance change and limits the interpretation of a control highlighted in the Abstract, Contribution 3, and Sections 4.2 and 5. Its impact is confined to this auxiliary control; the inference does not invalidate the main replay-based DQN result. The implementation needs clarification before treating this as a confirmed defect.

  2. The breadth of the protocol contrast remains limited. The main positive evidence is from Double DQN; the single PPO configuration does not reproduce the timing gap, and switching CNN to TinyViT tests architecture rather than a second value-based algorithm. The grid and Corridor provide controlled and structurally different tasks, but do not establish transfer to an independently established benchmark. This restricts the evidence for broader algorithmic or task-level applicability when assessing significance. I acknowledge that the paper explicitly scopes the finding to its tested protocols; the limitation does not invalidate those observations or show that PPO generally lacks timing sensitivity.

Presentation comment: Figures 2–4 are difficult to read at the displayed page scale because the panels, axis labels, condition labels, and annotations are small. In particular, Figure 3(b) places the original-run curve, the RNG-corrected point, and the EARLY reference together with several annotations in a compact panel; although the caption explains these distinctions, they are easy to miss visually. Figure 4 combines paradigm, architecture, and environment changes in one list of small labels, making the comparison structure harder to follow.

Questions for authors
  1. No-replay control: Did any EARLY evidence transition from steps [0,400) enter a gradient update, given Appendix Q's retained 500-step warmup and current-transition-only updates? Please report the number of evidence transitions actually used by optimization in each schedule. If the EARLY count is zero, please revise the interpretation of this control or, if available, provide a comparison in which both schedules' evidence reaches optimization.

  2. Interpretation of the PPO result: Appendix O reports no detected EARLY/LATE difference on both binary formation and the continuous margin, and provides useful post-shift learning checks. Is there an existing zero-evidence PPO comparison, or another Phase-I comparison, showing whether the 40 evidence visits affect its continuous reliance measure at all? This would help distinguish evidence that has an effect without a timing difference from evidence with no detected effect at this dose. The answer would affect how informative the PPO boundary is; without it, the result remains a valid but limited observation for the tested configuration. This does not imply that PPO lacks primacy bias or that the DQN findings are unsound.

  3. Positioning relative to critical-period and reward-transition studies: Sections 1–2 already distinguish this paper from general primacy-bias research, which I acknowledge. However, Achille et al. (2019, Figure 1C) move a fixed-duration, 40-epoch input-deficit window, and PPO curriculum/reward-transition studies also investigate when an intervention changes subsequent learning (de Kleijn et al., 2022, Park et al., 2024). Could you qualify the broad "None holds a fixed evidence quantity constant while varying only timing" statement by distinguishing a fixed input-deficit window, reward/cost-schedule transitions, and the present fixed positive disambiguating-evidence visits? A short comparison of what is held fixed and what outcome is measured would clarify the incremental knowledge provided by the feature-reliance and cross-protocol analysis. These prior studies are related context, not demonstrations of the same experiment.

  4. Algorithmic scope and external validity: The positive timing result primarily concerns Double DQN, while the single tested PPO configuration has no detected timing gap; TinyViT varies the architecture rather than the RL algorithm, and the grid and Corridor are controlled, designed tasks (Table 3; Appendices C, K and O). Do you have existing evidence from another value-based algorithm/configuration or a broader PPO setting, or from an established benchmark adapted to retain separately testable spurious and reward-relevant cues?

References for Question 3 :

2 comments

Thank you for the detailed and constructive review. We particularly appreciate your recognition of the matched evidence construction and the distinction between establishing a timing effect and identifying its mechanism. Your questions prompted us to revisit several results directly, including the submitted implementation.

On the no-replay control: You identified a genuine problem. We audited the submitted code and confirmed that optimizer updates begin at step 499, while all 40 EARLY evidence visits occur within steps [0, 400). Because the no-replay learner updates only on the current transition, none of those EARLY transitions contributes to a gradient update. In contrast, all 40 LATE evidence transitions are used once. We therefore withdraw the original no-replay ablation's interpretation that removing replay eliminates timing sensitivity. You were right that this issue goes beyond the batch-size and gradient-variance differences we had acknowledged.

We also have a separately preregistered follow-up experiment from October 7 in which both evidence windows occur after warmup, and all 40 evidence transitions in each condition receive one gradient update. Across 30 paired seeds, the timing difference remains: LATE − EARLY B-consistent accuracy is −0.271 (95% CI [−0.344, −0.197]). This uses a modified protocol, so we present it as additional evidence rather than a direct correction of the submitted ablation. The main replay-based Double DQN result is unaffected by the identified defect.

On PPO: We agree that observing shortcut formation in both conditions does not establish an intrinsic absence of timing sensitivity. We inspected the continuous reliance measurements and found no detectable effect of the 40 evidence visits at either timing: the LATE-window change was +0.043 (95% CI [−0.041, +0.126]), and the EARLY-window effect was −0.091 (95% CI [−0.218, +0.036]). There is no separate zero-evidence PPO run. These results suggest that the chosen evidence dose may simply have been insufficient to change PPO's reliance substantially. We will narrow the claim accordingly and avoid interpreting PPO as generally insensitive to evidence timing.

On the formation criterion: We checked the individual logged checkpoints rather than relying only on the five-checkpoint average. At step 4000, the original replay-based DQN comparison still shows a large difference: 0/30 EARLY versus 24/30 LATE shortcut-forming seeds, with a paired margin difference of +4.585 (95% CI [4.229, 4.942]). This is a post-shift-boundary diagnostic, not a replacement for the registered pre-shift criterion, but it shows that the main qualitative result is not solely produced by averaging earlier checkpoints.

On novelty and related work: Your comparison with critical learning periods and reward-transition studies is helpful. We agree that our earlier statement about prior studies was too broad. The narrower distinction is that we hold the number and content of positive, feature-disambiguating evidence visits fixed while moving their timing, and measure reliance on the spurious versus reward-relevant feature. This is different from moving an input-deficit window or changing a reward curriculum, but we do not claim that timing interventions themselves are new.

On scope and presentation: We agree that one Double DQN configuration, architectural variants and controlled environments cannot establish a general property of value-based RL. Likewise, the fixed-data TD experiment is a deliberately restricted self-loop control, not a representative offline-RL benchmark. We will clarify these boundaries and improve Figures 2–4, particularly the label sizes and the separation of different comparison types.

Thank you again. Your review helped us identify a real problem in an auxiliary control while also making the scope of the main finding much clearer.

I appreciate the detailed response and the audit of the code. I appreciate that you confirmed the no-replay issue, and I agree the main replay-based DQN result is unaffected.

I will keep my score. The response is to narrow down the claims rather than to solve the main concerns:

  1. Novelty: Moving a fixed set of evidence visits in time is now the only contribution. This means that the gain over critical-period and primacy-bias work is much smaller than claimed in the paper.
  2. PPO: Evidence has no effect on PPO at either timing, so PPO results are insensitive to timing.
  3. No-replay: The no-replay follow-up results show that the timing gap still exists without replay. Therefore, the claim in Abstract and Contribution 3 that removing replay closes the split is no longer supported and a revision of the manuscript is required.
  4. Scope: Evidence still comes from one value-based algorithm and there is no comparison on any established benchmark.

Thank you for planning to improve the figures.

Reviewer_DyEyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the impact of timing, i.e., when evidence for a rule arrives during training, on an RL agent's tendency to learn spurious features. The study tests this effect by constructing a task where observations contain an obvious color and subtler stripes. The correct action depends on the stripes. They give the agent 40 instances where color and stripes disagree, within 4,000 training steps, and move these instances from the beginning to the end of training. For DDQN, early counterexamples prevent the shortcut, while PPO learns the color shortcut under both orders.

Critical strengths

I think this is an interesting effect to study, especially w.r.t. the primacy bias, since it was not obvious to me why the same counterexamples can prevent a shortcut early but fail to correct it later.

Critical weaknesses
  • The paper is extremely difficult to follow. Most sentences are hard to parse, and the paper uses unnecessary jargon. Most the empirical insights are unclear and therefore hard for me to assess. I only got the motivation with a lot of effort, and then most of hte things afterwards were lost to me.
Questions for authors

Please see weaknesses

1 comment

Thank you for reading the paper and for your candid feedback. We particularly appreciate your comment that the difference between preventing a shortcut early and correcting it later is an interesting question.

We agree that the writing made the paper considerably harder to follow than it needed to be. Looking back, we introduced too much terminology and placed several experimental comparisons close together without clearly explaining what each was meant to establish. That made it difficult to see the main result through the details.

The central finding is simpler than our presentation suggested. We gave a Double DQN learner the same 40 examples in which an obvious visual cue conflicts with the feature that actually determines reward. When those examples appeared early, the learner did not develop the shortcut. When they appeared late, it generally retained the shortcut, despite receiving the same evidence.

The experiments then ask whether this difference can be explained solely by exploration, action coverage, training duration or other individual factors. Those controls help characterize the effect, but they do not identify a unique mechanism. We should have made that distinction much more obvious.

We will revise the presentation around this simpler progression, use shorter sentences, remove unnecessary terminology and make the key comparisons easier to read in the figures. We will also make clear which experiments establish the main observation and which are limited exploratory controls.

Your feedback is valuable because a result that requires so much effort to understand is not being communicated effectively, regardless of how carefully it was measured. Thank you for pointing this out.

Reviewer_yetXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts in a controlled environment to conceptually decouple “shortcut feature learning” and “timing sensitivity of presenting counterfactual evidence.” By shifting a fixed amount of counterexample evidence across different time windows during Double DQN training, it shows extreme representation differentiation under early versus late presentation.

Critical strengths

The paper contains enough experimental details.

Critical weaknesses
  1. As for motivation, the paper studies shortcut learning, but it neither proposes a method to solve shortcuts nor provides a way to identify false features. The experiments preset “real features” and “false features,” which departs from the real reinforcement learning situation where evidence appears randomly and lacks practical guidance.
  2. For the experiment, the authors, after finding that a single-variable intervention cannot eliminate the gap, attribute the reason to broad “path dependence”; this explanation is too drifting and vague. A single-factor check may not be able to rule out multi-factor joint effects.
  3. About writing, the paper is full of many obscure, complex long sentences, stiff terminology, and obvious signs of AI writing, which increases the reader’s comprehension burden.
Questions for authors

Please refer to the “Weakness” section for related questions.

1 comment

Thank you for the direct feedback. We appreciate the concerns about motivation, the interpretation of path dependence, and the readability of the paper. These are important points, and we agree that our presentation did not explain the contribution clearly enough.

Our intention was not to propose a general method for detecting spurious features or removing shortcuts from arbitrary RL environments. The controlled task deliberately makes the spurious and reward-relevant features known, allowing us to ask a narrower question: if two learners receive the same corrective evidence, can its timing alone substantially change what they learn?

The answer in our Double DQN setting is yes. Moving the same 40 counterexamples from early to late training changes shortcut formation from 0/30 to 30/30 seeds under the registered criterion. We believe the significance is that the quantity of corrective evidence alone does not determine its effectiveness; the learner's prior training history also matters. This does not establish a general mitigation method, and we will make that distinction clearer.

We also agree with your criticism of our use of "path dependence." The individual interventions show that several proposed factors do not independently explain the entire timing gap. They do not rule out interactions between those factors, nor do they identify a unique mechanism. We intended path dependence as a descriptive term for that unresolved dependence on training history, but we should have stated this more plainly rather than allowing it to sound like a mechanistic explanation.

Finally, we agree that the writing is unnecessarily dense. We will simplify the terminology, shorten the sentences and reorganize the results so readers can see what each experiment actually demonstrates. We will also distinguish the controlled scientific question from the broader practical challenge of finding and correcting shortcuts in realistic environments.

Thank you for highlighting these gaps. They help us clarify both the value and the limits of the current work.

Reviewer_2zPJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes between two questions: whether a model relies on a flawed shortcut, and whether this reliance depends on when counterexamples are encountered. In visual tasks, salient features often align with the features that truly determine rewards, with only a small number of samples revealing a conflict. The authors show that by scheduling the same set of counterexamples to appear either early or late during training, the shortcut formation rate for Double DQN shifts from 0/30 to 30/30. In contrast, the supervised learning, fixed-data TD, and PPO baselines do not exhibit a temporal sensitivity of the same magnitude.

Critical strengths
  • The experimental design is robust; the main experiments utilize 30 different random seeds, with careful verification that the content and quantity of counterexamples are identical across both conditions.
  • PPO develops shortcuts in both the "early" and "late" conditions without showing a divergence based on timing, a finding that intuitively supports the idea that "shortcut formation" and "sensitivity to the timing of counterexamples" are distinct phenomena.
Critical weaknesses
  • In the no-replay ablation, a 500-step warmup period is used, after which only the current transition is updated at each step. Consequently, all 40 counterexamples in the "EARLY" condition occur within the first 400 steps—meaning none of them are ever used for gradient updates. According to the supplementary code, the first update occurs at the very beginning, while all "LATE" counterexamples appear after training has commenced. This not only introduces the differences in batch size and gradient variance already acknowledged by the authors, but also undermines the comparability of the effective number of counterexamples used for training between the two conditions.
  • The fixed-data TD baseline employs a task with self-loops and no terminal states, which is not representative of general offline reinforcement learning.
  • Placing 40 counterexamples within specific early or late windows serves as an experimental tool rather than a general-purpose solution.
  • The current results are insufficient to conclude that counterexamples must be provided as early as possible or that "PPO is less prone to learning shortcuts than DQN."
Questions for authors
  1. Please confirm that, in the "no-replay" setting, the "EARLY" counterexamples are never included in the training process. Additionally, please report the actual number of counterexamples that were used for gradient updates in each condition. Do the original ablation findings still hold if both sets of counterexamples are introduced only after the update process has begun, or if the warmup phase is removed?
  2. Is it possible to jointly control the exploration rate, action coverage, and the duration of training following the introduction of counterexamples? Furthermore, how should the total training duration be handled in such a setup?
1 comment

Thank you for your careful review and for recognizing the effort to match the counterexample content and quantity across the two timing conditions. Your observation about the no-replay experiment was correct, and we appreciate you identifying it.

We went back to the exact submitted code and all 60 no-replay runs. Updates begin at step 499, after the entire EARLY evidence window [0, 400). Since each update uses only the current transition, 0/40 EARLY evidence transitions were used for optimization, compared with 40/40 LATE transitions. This is a genuine confound that we had not adequately accounted for.

We therefore withdraw our original interpretation that removing replay closes the timing gap. This finding cannot support a conclusion about replay's causal role. The defect is confined to that auxiliary control and does not affect the main replay-based comparison.

We can also report a relevant result from a separately preregistered follow-up conducted after submission. In that experiment, both evidence windows occur after optimizer warmup, and every counterexample receives exactly one gradient update. Across 30 paired seeds, the timing effect persists: LATE − EARLY B-consistent accuracy is −0.271 (95% CI [−0.344, −0.197]). We want to be clear that this is a modified protocol, not a repair of the original experiment, but it directly addresses your question about whether the effect survives when both groups actually train on their evidence.

Regarding your question about controlling exploration, action coverage and post-evidence training duration jointly: the paper tests several of these factors separately and includes an approximately matched integration-time comparison. We agree that these controls do not establish what happens when all factors are jointly matched. Doing so also requires deciding which quantity to hold fixed: the global training horizon, the number of updates after evidence, or both through a longer schedule. These answer related but different questions, and we should not treat them as interchangeable.

We will make that distinction clearer and narrow our conclusions to what the existing interventions establish. In particular, we do not claim that counterexamples should always be provided as early as possible, or that PPO is generally less prone to shortcuts than DQN.

Thank you again for catching the no-replay issue. It was an important correction, and your questions helped us distinguish the evidence the learner encounters from the evidence that actually reaches optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.