- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper analyzes “filtering-induced bias,” which can arise when filtering weights depend on noisy value estimates in offline RL. Focusing on IQL, the authors propose delayed and decoupled filtering, which constructs filtering weights from delayed Q/V snapshots of a different estimator than the one being updated. The proposed method achieves substantial empirical gains under high bias-stress settings.
Critical strengths
- The paper investigates filtering-induced bias in a reasonable problem setting.
The paper investigates filtering-induced bias through mathematical analysis. The reasoning is coherent and easy to follow, and the experiments are designed to further examine this phenomenon.
Critical weaknesses
- The practical significance of filtering-induced bias needs clarification.
Prior IQL results show that larger expectiles can improve performance on long-horizon tasks requiring trajectory stitching. While this does not rule out filtering-induced bias, it raises the question of when harmful bias outweighs beneficial policy improvement. The experiments do not sufficiently clarify this trade-off or establish whether appropriate expectile tuning could address the observed degradation.
- The contribution of delayed filtering beyond existing stabilization techniques is unclear.
Delayed updates are widely used: DQN periodically synchronizes its target Q-network [1], while continuous-control methods use soft target updates to stabilize value learning [2]. Although the proposed method applies delay to filtering weights and additionally introduces cross-estimator decoupling, the benefit of its delayed component beyond existing target-network mechanisms remains unclear. Comparisons against slower soft updates or periodic target synchronization would help establish this contribution.
- The experimental comparisons are not fair.
The table1 does not adequately support the authors’ claims. First, increasing the number of ensemble models can improve performance [3], but the paper uses a different number of models from the baseline algorithm, making the comparison unfair. Second, following the paper’s reasoning, slower soft target updates could also be effective. However, the paper does not compare against smaller soft-update rates. Therefore, the experimental support for the authors’ claims is weak.
- The paper’s readability is poor.
The terms “filtered expectation” and “filtering weights” are introduced in the introduction, but they are difficult to understand because they are not defined until Section 5.
Table 1 contains many numbers. Would it be clearer to report success rates as percentages rather than raw values?
[1] Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." nature 518.7540 (2015): 529-533.
[2] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. Pmlr, 2018.
[3] An, Gaon, et al. "Uncertainty-based offline reinforcement learning with diversified q-ensemble." Advances in neural information processing systems 34 (2021): 7436-7447.
Questions for authors
- How does the proposed method differ from existing methods that use soft target updates or periodic critic synchronization?
- Could you compare the proposed method against a target network updated more slowly using a smaller soft-update rate?
- Why do you use more ensemble networks than the baseline?
7 comments
[W3]
We acknowledge that our original comparisons alone do not fully isolate the effects of filtering from those of critic ensemble size. To address this concern, we conducted additional ablation experiments. Specifically, we evaluated IQL with critics and ReBRAC with critics, matching the critic counts used in our main experiments. All results are averaged over the same set of default OGBench tasks under the corresponding evaluation settings.
As shown in the results below, DD-IQL achieves higher mean scores than matched-size IQL controls across all four settings, and DD-ReBRAC achieves higher mean scores than matched-size ReBRAC controls across the two settings. These results show that the performance advantage persists when the baselines use the same critic counts.
| Setting | IQL (τ=0.9) | IQL (τ=0.99) | IQL (N=20, τ=0.9) | IQL (N=20, τ=0.99) | DD-IQL (τ=0.9) | DD-IQL (τ=0.99) |
|---|---|---|---|---|---|---|
| Original γ | 19% | 4% | 12% | 16% | 21% | 28% |
| γ=0.999 | 15% | 5% | 17% | 15% | 36% | 35% |
| Setting | ReBRAC | ReBRAC (N=50) | DD-ReBRAC |
|---|---|---|---|
| Original γ | 30% | 11% | 48% |
| γ=0.999 | 44% | 16% | 63% |
[W4]
We thank the reviewer for these helpful suggestions. We will revise the Introduction and the presentation of the experimental results to improve readability.
First, we will explain “filtered expectation” and “filtering weights” in the Introduction, immediately after their first mention. Specifically, we will add:
“A filtered expectation is a weighted average of action-value estimates, normalized by the total weight. The filtering weights determine how strongly each action contributes to this average. In IQL’s upper-expectile regression (), actions whose estimated Q-values exceed the current state-value estimate receive greater weight.”
Second, we will revise Table 1 to report success rates as percentages, multiplying both the means and standard deviations by 100. For example, will be presented as . We will clearly indicate the percentage units in the table caption.
We will incorporate the additional experimental results and clarifications presented in this rebuttal into the revised manuscript.
Thank you for the additional experiments. I have one follow-up question.
In the new control, IQL with does not improve performance at :
| Setting | IQL (, ) | IQL (, ) |
|---|---|---|
| Original | 4% | 1% |
| 5% | 2% |
Could you report the value-estimate trajectories (as in Fig. 1) and the mean filtering weight for this control?
- If the smaller update rate suppresses the value drift while performance remains near zero, this would suggest that the failure of IQL at is not primarily caused by filtering-induced bias. It may instead stem from the highly concentrated weights (mean weight in Table 2).
- Separately, the slight degradation under may reflect slower value propagation within the 1M-step budget rather than a property of the filtering mechanism.
We thank the reviewer for carefully reading our manuscript and for the constructive feedback.
[W1]
We agree that larger expectiles can facilitate trajectory stitching and improve policy performance. The practical concern is that stronger emphasis on high-valued actions can also increase sensitivity to positive estimation errors. Our goal is to make this use of larger expectiles more robust through delayed and decoupled filtering.
Our standard configuration follows the previously tuned IQL baseline in Park et al. (2025b), which used τ=0.9. DD-IQL inherits the same baseline hyperparameters without additional retuning. We additionally evaluate τ=0.99 to examine the effect of stronger filtering under both the original and increased discount factors.
The N=2 ablation in Table 3 illustrates the different responses to this expectile increase. Under the original discount factors, increasing τ from 0.9 to 0.99 reduces IQL’s average score from 0.19 to 0.04, whereas DD-IQL with D=100 improves from 0.16 to 0.25. With γ=0.999, IQL’s score decreases from 0.15 to 0.05, while DD-IQL remains stable at 0.28 and 0.29. These results are consistent with bias mitigation helping preserve the potential benefits of larger expectiles.
[W2]
The distinction concerns which quantities are delayed and how the filtering weights are constructed. Conventional soft or periodic hard target updates control the evolution of the target critic. In the value loss below, this corresponds to Qθˉi, shown in red, which provides the regression target for Vψi. Our delayed decoupling method retains this standard target-network mechanism and additionally controls the construction of the filtering weight wi, shown in blue:
LVi(ψi)=E(s,a)∼D[wi(s,a)(Qθˉi(s,a)−Vψi(s))2].
Specifically, DD-IQL constructs wi from delayed snapshots of a different estimator's critic and value function:
w_i(s,a) = \left\lvert \tau- \mathbf{1} \left\\{ \widetilde Q_j(s,a)-\widetilde V_j(s) < 0 \right\\} \right\rvert, \qquad j\neq i.
The filtering source is refreshed every D updates. Between refreshes, the filtering weight for a fixed state-action pair remains unchanged, while the receiving critic and value function continue learning. Meanwhile, the target critic follows the standard soft-update rule:
θˉi←(1−ρ)θˉi+ρθi.
Thus, ρ controls target-critic tracking, whereas D controls the refresh interval of the separate filtering source. Importantly, DD-IQL applies delayed filtering alongside the standard soft target update, rather than replacing it. The two mechanisms operate on different quantities; therefore, delaying the filtering source is fundamentally different from simply reducing the target-network update rate. Furthermore, DD-IQL introduces cross-estimator decoupling by constructing the filtering weights from a different estimator (j=i).
Following the reviewer's suggestion, we evaluated two additional IQL controls: a tenfold smaller soft-update rate, ρ=0.0005 instead of 0.005, and periodic hard target synchronization, denoted IQL-PS. These changes to the target-network update schedule do not provide consistent improvements across the tested bias-stress settings. In contrast, the filtering ablations in Table 3 show that delayed filtering and cross-estimator decoupling can improve performance under these conditions.
All results are reported as success rates (%).
The DF-IQL ablation in Table 3 further isolates the effect of delayed filtering. DF-IQL preserves the standard IQL architecture and soft target-update rate, modifying only the refresh interval of the filtering weights. At D=1, DF-IQL recovers standard IQL; at sufficiently large D, it achieves higher average performance under the evaluated bias-stress settings. These results suggest that delayed filtering can provide additional benefits beyond conventional target-network updates, even within the standard IQL architecture.