- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether reliance on a spurious cue and its sensitivity to the timing of disambiguating evidence can differ across learning protocols. In a controlled task, it moves the same 40 evidence-cell visits between early and late windows and measures relative reliance on the spurious and reward-relevant features. Under its pre-shift formation criterion, Double DQN exhibits a 0/30 versus 30/30 split, whereas the tested fixed-dataset learners show smaller differences and a single PPO configuration forms shortcuts under both schedules; additional architectures, environments, and interventions examine the scope of this observation.
Critical strengths
-
The task makes the learned shortcut interpretable: the spurious and reward-relevant cues are separately manipulated, and behavioral reliance is measured rather than inferred from reward alone. The corrected paired construction matches visited-cell content and evidence count, making the within-protocol comparison informative and easier to audit.
-
The paper tests several alternative explanations and demonstrates a repeatable empirical contrast within its studied settings. The DQN timing gap persists under forced action coverage, and it is attenuated but remains substantial under interior windows and approximately matched integration time. TinyViT and the corridor task provide additional evidence across architecture and environment. These controls strengthen the empirical characterization without identifying a unique mechanism.
Critical weaknesses
-
The no-replay control appears to change whether EARLY evidence reaches optimization at all, in addition to changing replay and batch size. Appendix Q retains a 500-step warmup and uses only the transition produced at each update step; it reports 11,501 updates over 12,000 steps, with no transition reuse. Section 3.1 places all EARLY evidence visits in [0,400). Under this description, those evidence transitions would never enter a gradient update, whereas LATE evidence could be used. If confirmed, the 29/30 versus 29/30 outcome would not establish what happens when both schedules' evidence actually reaches optimization without replay. This confound goes beyond the acknowledged batch-size/gradient-variance change and limits the interpretation of a control highlighted in the Abstract, Contribution 3, and Sections 4.2 and 5. Its impact is confined to this auxiliary control; the inference does not invalidate the main replay-based DQN result. The implementation needs clarification before treating this as a confirmed defect.
-
The breadth of the protocol contrast remains limited. The main positive evidence is from Double DQN; the single PPO configuration does not reproduce the timing gap, and switching CNN to TinyViT tests architecture rather than a second value-based algorithm. The grid and Corridor provide controlled and structurally different tasks, but do not establish transfer to an independently established benchmark. This restricts the evidence for broader algorithmic or task-level applicability when assessing significance. I acknowledge that the paper explicitly scopes the finding to its tested protocols; the limitation does not invalidate those observations or show that PPO generally lacks timing sensitivity.
Presentation comment: Figures 2–4 are difficult to read at the displayed page scale because the panels, axis labels, condition labels, and annotations are small. In particular, Figure 3(b) places the original-run curve, the RNG-corrected point, and the EARLY reference together with several annotations in a compact panel; although the caption explains these distinctions, they are easy to miss visually. Figure 4 combines paradigm, architecture, and environment changes in one list of small labels, making the comparison structure harder to follow.
Questions for authors
-
No-replay control: Did any EARLY evidence transition from steps [0,400) enter a gradient update, given Appendix Q's retained 500-step warmup and current-transition-only updates? Please report the number of evidence transitions actually used by optimization in each schedule. If the EARLY count is zero, please revise the interpretation of this control or, if available, provide a comparison in which both schedules' evidence reaches optimization.
-
Interpretation of the PPO result: Appendix O reports no detected EARLY/LATE difference on both binary formation and the continuous margin, and provides useful post-shift learning checks. Is there an existing zero-evidence PPO comparison, or another Phase-I comparison, showing whether the 40 evidence visits affect its continuous reliance measure at all? This would help distinguish evidence that has an effect without a timing difference from evidence with no detected effect at this dose. The answer would affect how informative the PPO boundary is; without it, the result remains a valid but limited observation for the tested configuration. This does not imply that PPO lacks primacy bias or that the DQN findings are unsound.
-
Positioning relative to critical-period and reward-transition studies: Sections 1–2 already distinguish this paper from general primacy-bias research, which I acknowledge. However, Achille et al. (2019, Figure 1C) move a fixed-duration, 40-epoch input-deficit window, and PPO curriculum/reward-transition studies also investigate when an intervention changes subsequent learning (de Kleijn et al., 2022, Park et al., 2024). Could you qualify the broad "None holds a fixed evidence quantity constant while varying only timing" statement by distinguishing a fixed input-deficit window, reward/cost-schedule transitions, and the present fixed positive disambiguating-evidence visits? A short comparison of what is held fixed and what outcome is measured would clarify the incremental knowledge provided by the feature-reliance and cross-protocol analysis. These prior studies are related context, not demonstrations of the same experiment.
-
Algorithmic scope and external validity: The positive timing result primarily concerns Double DQN, while the single tested PPO configuration has no detected timing gap; TinyViT varies the architecture rather than the RL algorithm, and the grid and Corridor are controlled, designed tasks (Table 3; Appendices C, K and O). Do you have existing evidence from another value-based algorithm/configuration or a broader PPO setting, or from an established benchmark adapted to retain separately testable spurious and reward-relevant cues?
References for Question 3 :
- Nikishin et al., The Primacy Bias in Deep Reinforcement Learning, ICML 2022, Already cited. https://proceedings.mlr.press/v162/nikishin22a/nikishin22a.pdf
- Achille et al., Critical Learning Periods in Deep Networks, ICLR 2019, Already cited. https://arxiv.org/html/1711.08856v3
- de Kleijn et al., A Critical Period for Robust Curriculum-Based Deep Reinforcement Learning of Sequential Action in a Robot Arm, 2022, https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12595
- Park et al., Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning, AAAI 2024, https://arxiv.org/html/2403.06880v1#A3.SS1
2 comments
I appreciate the detailed response and the audit of the code. I appreciate that you confirmed the no-replay issue, and I agree the main replay-based DQN result is unaffected.
I will keep my score. The response is to narrow down the claims rather than to solve the main concerns:
- Novelty: Moving a fixed set of evidence visits in time is now the only contribution. This means that the gain over critical-period and primacy-bias work is much smaller than claimed in the paper.
- PPO: Evidence has no effect on PPO at either timing, so PPO results are insensitive to timing.
- No-replay: The no-replay follow-up results show that the timing gap still exists without replay. Therefore, the claim in Abstract and Contribution 3 that removing replay closes the split is no longer supported and a revision of the manuscript is required.
- Scope: Evidence still comes from one value-based algorithm and there is no comparison on any established benchmark.
Thank you for planning to improve the figures.
Thank you for the detailed and constructive review. We particularly appreciate your recognition of the matched evidence construction and the distinction between establishing a timing effect and identifying its mechanism. Your questions prompted us to revisit several results directly, including the submitted implementation.
On the no-replay control: You identified a genuine problem. We audited the submitted code and confirmed that optimizer updates begin at step 499, while all 40 EARLY evidence visits occur within steps [0, 400). Because the no-replay learner updates only on the current transition, none of those EARLY transitions contributes to a gradient update. In contrast, all 40 LATE evidence transitions are used once. We therefore withdraw the original no-replay ablation's interpretation that removing replay eliminates timing sensitivity. You were right that this issue goes beyond the batch-size and gradient-variance differences we had acknowledged.
We also have a separately preregistered follow-up experiment from October 7 in which both evidence windows occur after warmup, and all 40 evidence transitions in each condition receive one gradient update. Across 30 paired seeds, the timing difference remains: LATE − EARLY B-consistent accuracy is −0.271 (95% CI [−0.344, −0.197]). This uses a modified protocol, so we present it as additional evidence rather than a direct correction of the submitted ablation. The main replay-based Double DQN result is unaffected by the identified defect.
On PPO: We agree that observing shortcut formation in both conditions does not establish an intrinsic absence of timing sensitivity. We inspected the continuous reliance measurements and found no detectable effect of the 40 evidence visits at either timing: the LATE-window change was +0.043 (95% CI [−0.041, +0.126]), and the EARLY-window effect was −0.091 (95% CI [−0.218, +0.036]). There is no separate zero-evidence PPO run. These results suggest that the chosen evidence dose may simply have been insufficient to change PPO's reliance substantially. We will narrow the claim accordingly and avoid interpreting PPO as generally insensitive to evidence timing.
On the formation criterion: We checked the individual logged checkpoints rather than relying only on the five-checkpoint average. At step 4000, the original replay-based DQN comparison still shows a large difference: 0/30 EARLY versus 24/30 LATE shortcut-forming seeds, with a paired margin difference of +4.585 (95% CI [4.229, 4.942]). This is a post-shift-boundary diagnostic, not a replacement for the registered pre-shift criterion, but it shows that the main qualitative result is not solely produced by averaging earlier checkpoints.
On novelty and related work: Your comparison with critical learning periods and reward-transition studies is helpful. We agree that our earlier statement about prior studies was too broad. The narrower distinction is that we hold the number and content of positive, feature-disambiguating evidence visits fixed while moving their timing, and measure reliance on the spurious versus reward-relevant feature. This is different from moving an input-deficit window or changing a reward curriculum, but we do not claim that timing interventions themselves are new.
On scope and presentation: We agree that one Double DQN configuration, architectural variants and controlled environments cannot establish a general property of value-based RL. Likewise, the fixed-data TD experiment is a deliberately restricted self-loop control, not a representative offline-RL benchmark. We will clarify these boundaries and improve Figures 2–4, particularly the label sizes and the separation of different comparison types.
Thank you again. Your review helped us identify a real problem in an auxiliary control while also making the scope of the main finding much clearer.