acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Filtering-Induced Bias in Offline Reinforcement Learning

Abstract

Value-based reinforcement learning can suffer from overestimation bias when noisy value estimates are repeatedly used in bootstrapped updates. This issue becomes particularly pronounced with large discount factors or long decision horizons, where estimation errors can persist and accumulate through repeated Bellman backups. We analyze this bias propagation mechanism and further show that, even without explicit action maximization, adaptive filtering can introduce bias when filtering decisions become coupled with value-estimation errors. To mitigate this coupling, we propose two mechanisms: delayed filtering and decoupled filtering. We instantiate these mechanisms in IQL, yielding DD-IQL. Empirically, DD-IQL exhibits reduced filter–error correlation and improved value stability. Across a range of offline RL environments, DD-IQL also improves policy performance over standard IQL, with particularly large gains under high bias-stress settings.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Overestimation Bias, Bias Propagation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dr6nOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper analyzes “filtering-induced bias,” which can arise when filtering weights depend on noisy value estimates in offline RL. Focusing on IQL, the authors propose delayed and decoupled filtering, which constructs filtering weights from delayed Q/V snapshots of a different estimator than the one being updated. The proposed method achieves substantial empirical gains under high bias-stress settings.

Critical strengths
  1. The paper investigates filtering-induced bias in a reasonable problem setting.

The paper investigates filtering-induced bias through mathematical analysis. The reasoning is coherent and easy to follow, and the experiments are designed to further examine this phenomenon.

Critical weaknesses
  1. The practical significance of filtering-induced bias needs clarification.

Prior IQL results show that larger expectiles can improve performance on long-horizon tasks requiring trajectory stitching. While this does not rule out filtering-induced bias, it raises the question of when harmful bias outweighs beneficial policy improvement. The experiments do not sufficiently clarify this trade-off or establish whether appropriate expectile tuning could address the observed degradation.

  1. The contribution of delayed filtering beyond existing stabilization techniques is unclear.

Delayed updates are widely used: DQN periodically synchronizes its target Q-network [1], while continuous-control methods use soft target updates to stabilize value learning [2]. Although the proposed method applies delay to filtering weights and additionally introduces cross-estimator decoupling, the benefit of its delayed component beyond existing target-network mechanisms remains unclear. Comparisons against slower soft updates or periodic target synchronization would help establish this contribution.

  1. The experimental comparisons are not fair.

The table1 does not adequately support the authors’ claims. First, increasing the number of ensemble models can improve performance [3], but the paper uses a different number of models from the baseline algorithm, making the comparison unfair. Second, following the paper’s reasoning, slower soft target updates could also be effective. However, the paper does not compare against smaller soft-update rates. Therefore, the experimental support for the authors’ claims is weak.

  1. The paper’s readability is poor.

The terms “filtered expectation” and “filtering weights” are introduced in the introduction, but they are difficult to understand because they are not defined until Section 5.

Table 1 contains many numbers. Would it be clearer to report success rates as percentages rather than raw values?

[1] Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." nature 518.7540 (2015): 529-533.

[2] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. Pmlr, 2018.

[3] An, Gaon, et al. "Uncertainty-based offline reinforcement learning with diversified q-ensemble." Advances in neural information processing systems 34 (2021): 7436-7447.

Questions for authors
  1. How does the proposed method differ from existing methods that use soft target updates or periodic critic synchronization?
  2. Could you compare the proposed method against a target network updated more slowly using a smaller soft-update rate?
  3. Why do you use more ensemble networks than the baseline?
7 comments

We thank the reviewer for carefully reading our manuscript and for the constructive feedback.

[W1]

We agree that larger expectiles can facilitate trajectory stitching and improve policy performance. The practical concern is that stronger emphasis on high-valued actions can also increase sensitivity to positive estimation errors. Our goal is to make this use of larger expectiles more robust through delayed and decoupled filtering.

Our standard configuration follows the previously tuned IQL baseline in Park et al. (2025b), which used τ=0.9\tau=0.9. DD-IQL inherits the same baseline hyperparameters without additional retuning. We additionally evaluate τ=0.99\tau=0.99 to examine the effect of stronger filtering under both the original and increased discount factors.

The N=2N=2 ablation in Table 3 illustrates the different responses to this expectile increase. Under the original discount factors, increasing τ\tau from 0.90.9 to 0.990.99 reduces IQL’s average score from 0.190.19 to 0.040.04, whereas DD-IQL with D=100D=100 improves from 0.160.16 to 0.250.25. With γ=0.999\gamma=0.999, IQL’s score decreases from 0.150.15 to 0.050.05, while DD-IQL remains stable at 0.280.28 and 0.290.29. These results are consistent with bias mitigation helping preserve the potential benefits of larger expectiles.

[W2]

The distinction concerns which quantities are delayed and how the filtering weights are constructed. Conventional soft or periodic hard target updates control the evolution of the target critic. In the value loss below, this corresponds to QθˉiQ_{\bar{\theta}_i}, shown in red, which provides the regression target for VψiV_{\psi_i}. Our delayed decoupling method retains this standard target-network mechanism and additionally controls the construction of the filtering weight wiw_i, shown in blue:

LVi(ψi)=E(s,a)∼D[wi(s,a)(Qθˉi(s,a)−Vψi(s))2].\mathcal{L}_{V_i}(\psi_i) = \mathbb{E}_{(s,a)\sim\mathcal{D}} \left[ \textcolor{blue}{w_i(s,a)} \left( \textcolor{red}{Q_{\bar{\theta}_i}(s,a)} - V_{\psi_i}(s) \right)^2 \right].

Specifically, DD-IQL constructs wiw_i from delayed snapshots of a different estimator's critic and value function:

w_i(s,a) = \left\lvert \tau- \mathbf{1} \left\\{ \widetilde Q_j(s,a)-\widetilde V_j(s) < 0 \right\\} \right\rvert, \qquad j\neq i.

The filtering source is refreshed every DD updates. Between refreshes, the filtering weight for a fixed state-action pair remains unchanged, while the receiving critic and value function continue learning. Meanwhile, the target critic follows the standard soft-update rule:

θˉi←(1−ρ)θˉi+ρθi.\bar{\theta}_i \leftarrow (1-\rho)\bar{\theta}_i+\rho\theta_i.

Thus, ρ\rho controls target-critic tracking, whereas DD controls the refresh interval of the separate filtering source. Importantly, DD-IQL applies delayed filtering alongside the standard soft target update, rather than replacing it. The two mechanisms operate on different quantities; therefore, delaying the filtering source is fundamentally different from simply reducing the target-network update rate. Furthermore, DD-IQL introduces cross-estimator decoupling by constructing the filtering weights from a different estimator (j≠ij\neq i).

Following the reviewer's suggestion, we evaluated two additional IQL controls: a tenfold smaller soft-update rate, ρ=0.0005\rho=0.0005 instead of 0.0050.005, and periodic hard target synchronization, denoted IQL-PS. These changes to the target-network update schedule do not provide consistent improvements across the tested bias-stress settings. In contrast, the filtering ablations in Table 3 show that delayed filtering and cross-estimator decoupling can improve performance under these conditions.

SettingIQL (τ=0.9)IQL (τ=0.99)IQL (ρ=0.0005, τ=0.9)IQL (ρ=0.0005, τ=0.99)IQL-PS (τ=0.9)IQL-PS (τ=0.99)DD-IQL (N=2, τ=0.9)DD-IQL (N=2, τ=0.99)
Original γ19%4%18%1%19%4%16%25%
γ=0.99915%5%19%2%15%8%28%29%

All results are reported as success rates (%).

The DF-IQL ablation in Table 3 further isolates the effect of delayed filtering. DF-IQL preserves the standard IQL architecture and soft target-update rate, modifying only the refresh interval of the filtering weights. At D=1D=1, DF-IQL recovers standard IQL; at sufficiently large DD, it achieves higher average performance under the evaluated bias-stress settings. These results suggest that delayed filtering can provide additional benefits beyond conventional target-network updates, even within the standard IQL architecture.

[W3]

We acknowledge that our original comparisons alone do not fully isolate the effects of filtering from those of critic ensemble size. To address this concern, we conducted additional ablation experiments. Specifically, we evaluated IQL with N=20N=20 critics and ReBRAC with N=50N=50 critics, matching the critic counts used in our main experiments. All results are averaged over the same set of default OGBench tasks under the corresponding evaluation settings.

As shown in the results below, DD-IQL achieves higher mean scores than matched-size IQL controls across all four (γ,τ)(\gamma,\tau) settings, and DD-ReBRAC achieves higher mean scores than matched-size ReBRAC controls across the two γ\gamma settings. These results show that the performance advantage persists when the baselines use the same critic counts.

SettingIQL (τ=0.9)IQL (τ=0.99)IQL (N=20, τ=0.9)IQL (N=20, τ=0.99)DD-IQL (τ=0.9)DD-IQL (τ=0.99)
Original γ19%4%12%16%21%28%
γ=0.99915%5%17%15%36%35%
SettingReBRACReBRAC (N=50)DD-ReBRAC
Original γ30%11%48%
γ=0.99944%16%63%

[W4]

We thank the reviewer for these helpful suggestions. We will revise the Introduction and the presentation of the experimental results to improve readability.

First, we will explain “filtered expectation” and “filtering weights” in the Introduction, immediately after their first mention. Specifically, we will add:

“A filtered expectation is a weighted average of action-value estimates, normalized by the total weight. The filtering weights determine how strongly each action contributes to this average. In IQL’s upper-expectile regression (τ>1/2\tau>1/2), actions whose estimated Q-values exceed the current state-value estimate receive greater weight.”

Second, we will revise Table 1 to report success rates as percentages, multiplying both the means and standard deviations by 100. For example, 0.14±0.030.14\pm0.03 will be presented as (14±3)%(14\pm3)\%. We will clearly indicate the percentage units in the table caption.

We will incorporate the additional experimental results and clarifications presented in this rebuttal into the revised manuscript.

Thank you for the additional experiments. I have one follow-up question.

In the new control, IQL with ρ=0.0005\rho = 0.0005 does not improve performance at τ=0.99\tau = 0.99:

SettingIQL (ρ=0.005\rho = 0.005, τ=0.99\tau = 0.99)IQL (ρ=0.0005\rho = 0.0005, τ=0.99\tau = 0.99)
Original γ\gamma4%1%
γ=0.999\gamma = 0.9995%2%

Could you report the value-estimate trajectories (as in Fig. 1) and the mean filtering weight for this control?

  1. If the smaller update rate suppresses the value drift while performance remains near zero, this would suggest that the failure of IQL at τ=0.99\tau = 0.99 is not primarily caused by filtering-induced bias. It may instead stem from the highly concentrated weights (mean weight ≈0.05\approx 0.05 in Table 2).
  2. Separately, the slight degradation under ρ=0.0005\rho = 0.0005 may reflect slower value propagation within the 1M-step budget rather than a property of the filtering mechanism.
Reviewer_RYwJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies filtering-induced estimation bias in offline reinforcement learning. It interprets IQL's expectile update as a filtered expectation and explains how using noisy value estimates to construct filtering weights can couple those weights to estimation errors, even without explicit action maximization. It proposes delayed filtering from periodically refreshed snapshots and decoupled filtering across estimators, yielding Delayed Decoupled IQL (DD-IQL), and also evaluates a related ReBRAC extension. Experiments on nine OGBench environments compare original and high-discount/high-expectile settings, with additional task variants, component ablations, correlation diagnostics, and computational profiling. The reported results show more stable values and improved performance relative to IQL, particularly under bias stress. The diagnostics use TD residuals and ensemble disagreement as proxies rather than direct measurements of value error.

Critical strengths
  1. The filtered-expectation view is useful. It explains why value-dependent weighting can introduce bias even without explicit action maximization.
  2. Delay and cross-estimator filtering are practical ideas with a clear rationale. The paper also acknowledges their limits, including persistent and correlated errors.
  3. The evaluation goes beyond returns. It includes component ablations, stability checks, coupling diagnostics, and computational measurements. The bias-stress results are encouraging.
Critical weaknesses
  1. The evidence for bias reduction is still indirect. Weaker coupling and more stable values do not necessarily mean more accurate estimates. A direct test would help.
  2. The main comparisons also change ensemble capacity and estimator structure. More closely matched controls are needed to show what filtering itself contributes.
  3. The added value of delay after decoupling is unclear. Its benefits outside stress settings also need explanation. I lean slightly negative for now. If these concerns are resolved, I would consider raising my score.
Questions for authors
  1. Can you measure bias against a known noise-free filtering target in a controlled example? Please check underestimation as well. Do the coupling diagnostics track actual accuracy?
  2. How do ensemble IQL, self-delayed filtering, and DD-IQL compare with matched capacity, structure, and training budget? A representative matched ReBRAC comparison would also help.
  3. What does delay add once filtering is decoupled? The delay ablation looks fairly stable across several settings. Where does delay help beyond decoupling alone, including under standard conditions?
4 comments

We thank the reviewer for carefully reading our manuscript and for the constructive feedback.

[W1]

To provide direct evidence of bias reduction, we added a controlled experiment in a known tabular MDP. We model critic errors using either IID noise or a synthetic noise field with spatial and temporal correlations, the latter serving as a simplified model of neural network approximation errors. Further details of the controlled MDP setup and the complete simulation code are available at the link below.

https://anonymous.4open.science/r/iclr2027_rebuttal-8647/

We compare tabular value-learning analogues of the DF-IQL, SD-IQL, and DD-IQL variants in Table 3 of our main paper. DF-IQL with D=1D=1 corresponds to the tabular analogue of the standard IQL value-learning update. We directly measure the signed difference between the normalized filtering output and the known noise-free target. Results are reported as mean ±\pm sample standard deviation across 20 training seeds.

DD-IQL achieves a substantially smaller magnitude of mean filtering-output bias than DF-IQL and SD-IQL across the reported conditions. Increasing the delay generally reduces the positive bias of both SD-IQL and DD-IQL. DF-IQL, however, transitions from overestimation to substantial underestimation at larger delays. This pattern is consistent with the pessimistic effect of minimum aggregation becoming dominant as delay attenuates positive filtering-induced bias. Once overestimation is effectively controlled, retaining minimum aggregation can introduce excessive pessimism. It also supports our design choice to avoid minimum aggregation in our proposed methods.

IID-noise

MethodD=1D=10D=100D=1000
DF-IQL+534.73±1.21-394.52±0.74-458.31±1.29-448.57±3.49
SD-IQL+1188.69±0.96+117.96±0.85+38.72±0.93+13.60±1.77
DD-IQL+12.87±1.23+1.27±0.88+0.10±1.28-0.32±1.41

NN-like-noise

MethodD=1D=10D=100D=1000
DF-IQL+509.02±3.89+449.26±3.99+58.78±6.72-374.77±10.07
SD-IQL+1144.75±5.60+1090.85±5.85+673.24±9.21+112.59±6.36
DD-IQL+61.61±5.43+59.52±5.79+30.10±5.16+1.28±4.70

We additionally measure coupling as the mean absolute temporal Pearson correlation between filtering weights and receiver readout errors, using the same evaluation checkpoints as the filtering-bias measurements. For DF-IQL and SD-IQL, coupling generally decreases as DD increases, alongside a reduction in positive filtering-output bias. DD-IQL maintains low coupling across the tested delay periods.

IID-noise

MethodD=1D=10D=100D=1000
DF-IQL0.496±0.0240.103±0.0200.075±0.0130.084±0.013
SD-IQL0.543±0.0140.108±0.0140.083±0.0080.080±0.011
DD-IQL0.082±0.0090.083±0.0070.081±0.0140.083±0.010

NN-like-noise

MethodD=1D=10D=100D=1000
DF-IQL0.497±0.0210.490±0.0240.364±0.0200.094±0.020
SD-IQL0.536±0.0180.528±0.0180.406±0.0270.094±0.014
DD-IQL0.081±0.0130.083±0.0110.084±0.0130.080±0.008

[W2]

We acknowledge that our original comparisons alone do not fully isolate the effects of filtering from those of critic ensemble size and aggregation. To address this concern, we conducted additional ablation experiments. Specifically, we evaluated IQL with N=20N=20 critics and ReBRAC with N=50N=50 critics, matching the critic counts used in our main experiments, along with their corresponding mean-aggregation variants, denoted IQL-M and ReBRAC-M, respectively. All results are averaged over the same set of default OGBench tasks under the corresponding evaluation settings.

As shown in the results below, DD-IQL achieves higher mean scores than both matched-size IQL controls across all four γ,τ\gamma,\tau settings, and DD-ReBRAC achieves higher mean scores than both matched-size ReBRAC controls across the two γ\gamma settings. These results show that the performance advantage persists when the baselines use the same critic counts and when mean aggregation is included as an additional control.

SettingIQL (τ=0.9\tau=0.9)IQL (τ=0.99\tau=0.99)IQL (N=20,τ=0.9N=20,\tau=0.9)IQL (N=20,τ=0.99N=20,\tau=0.99)IQL-M (N=20,τ=0.9N=20,\tau=0.9)IQL-M (N=20,τ=0.99N=20,\tau=0.99)DD-IQL (τ=0.9\tau=0.9)DD-IQL (τ=0.99\tau=0.99)
Original γ\gamma0.190.040.120.160.190.020.210.28
γ=0.999\gamma=0.9990.150.050.170.150.100.040.360.35
SettingReBRACReBRAC (N=50N=50)ReBRAC-M (N=50N=50)DD-ReBRAC
Original γ\gamma0.300.110.410.48
γ=0.999\gamma=0.9990.440.160.460.63

In addition, Table 3 of our main paper compares DF-IQL, SD-IQL, and DD-IQL using N=2N=2 critics, the same number of training updates, and the same batch size. DF-IQL applies delayed filtering to the standard IQL architecture and recovers IQL at D=1D=1. Its delay sweep therefore evaluates the effect of delayed filtering while keeping the estimator structure fixed. SD-IQL and DD-IQL share the same paired Q/V architecture and, at a matched DD, differ only in whether filtering weights are constructed from the same estimator or another estimator. Their comparison isolates the contribution of cross-estimator decoupling. Under bias-stress settings involving a higher discount factor, a higher expectile level, or both, DF-IQL and SD-IQL show improved performance at larger delay periods DD, whereas DD-IQL maintains strong and stable performance across all evaluated delay periods.

[W3]

We agree that DD-IQL’s policy performance in Table 3 is relatively insensitive to the delay period. Much of its observed advantage under the bias-stress settings is already present at D=1D=1, indicating that cross-estimator decoupling provides a substantial benefit without additional filtering delay. However, this does not imply that decoupling eliminates all residual value bias.

Decoupling and delay introduce different forms of separation. Decoupling changes which estimator supplies the filtering signal, whereas delay limits how quickly changes in the learned Q/V estimates are reflected in that signal. Using another estimator does not necessarily remove dependencies arising from shared data and bootstrapped learning, so temporal separation can still provide an additional benefit.

The controlled MDP experiment demonstrates such an additional effect on bias. At τ=0.99\tau=0.99, DD-IQL’s mean filtering-output bias decreases from +12.87 ±\pm 1.23 at D=1D=1 to -0.32 ±\pm 1.41 at D=1000D=1000 under IID noise, and from +61.61 ±\pm 5.43 to +1.28 ±\pm 4.70 under correlated noise. Thus, although decoupling already produces substantially smaller bias than the other methods, increasing the delay further reduces the remaining bias in this controlled setting.

Under the standard benchmark setting, however, we do not observe a consistent incremental policy-performance gain from increasing the delay after decoupling. We will clarify this distinction: our results demonstrate an additional bias-reduction benefit of delay in the controlled experiment, while Table 3 shows limited additional gains in aggregate policy performance once filtering is decoupled.

We will incorporate the additional experimental results and clarifications presented in this rebuttal into the revised manuscript.

Thank you again for your constructive feedback. Following our responses and additional experiments above, we would welcome any further questions or remaining concerns, particularly regarding the direct bias measurements, comparisons with matched critic counts, or the role of delay after decoupling. Please let us know if any aspect would benefit from further clarification. We would be happy to address it and will incorporate the additional results and clarifications into the revised manuscript.

We thank the reviewer again for the constructive feedback. We have uploaded the revised manuscript incorporating the additional experiments and clarifications discussed in our rebuttal. For ease of reference, the revised text is highlighted in blue.

[W1] We added Section 7.6 (Figure 2) and Appendix E.7 (Figure 4) to directly measure max-QQ and filtering-output bias against known noise-free references, including underestimation.

[W2] We added matched-critic and aggregation controls in Section 7.4 (Table 3) and clarified the matched-structure comparisons among DF-IQL, SD-IQL, and DD-IQL in Section 7.5 (Table 4) and Appendix D.3.

[W3] We revised Section 7.5 to clarify that additional delay provides limited policy-performance gains after decoupling on OGBench environments, while Section 7.6 demonstrates further bias reduction with increasing delay in the controlled MDP.

We sincerely appreciate the reviewer's suggestions, which helped improve the clarity and experimental support of our work. Please feel free to let us know if you have any further questions or concerns. We would be happy to address them.

Reviewer_J5QDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies bias caused by using noisy value estimates to construct adaptive filtering weights. It proposes delayed and cross-estimator filtering and implements them in DD-IQL, with an additional ReBRAC variant. Experiments on OGBench report improved value stability and policy performance, particularly at high discount factors and expectile levels.

Critical strengths
  1. The paper identifies a useful limitation of in-sample value learning: avoiding explicit maximization does not remove the dependence between selection weights and estimation errors. The filtered-expectation formulation makes this issue clear and gives a direct motivation for separating weight construction from the estimator being updated.

  2. The experiments include standard settings, deliberately difficult settings, and additional task variants. The two-critic ablations are particularly useful because they show that cross-estimator filtering can help under high-discount or high-expectile settings without increasing the number of critics.

Critical weaknesses
  1. The main comparisons do not fully isolate filtering from ensemble size and aggregation. DD-IQL uses 20 paired estimators, while DD-ReBRAC uses 50 critics against two-critic baselines. DD-ReBRAC also replaces minimum aggregation with a mean. The two-critic ablations help, but DD-IQL does not improve over IQL in the standard setting there. Controls at the main ensemble sizes are needed to establish how much of the headline gain comes from the proposed filtering mechanisms.

  2. The diagnostics establish weaker coupling to learned error proxies, but do not directly measure reduced value bias. TD residuals, temporal drift, and leave-one-out ensemble disagreement are not independent reference values. Since the estimators share data and bootstrapped targets, reduced correlation does not establish the conditional unbiasedness used in the analysis. This leaves the proposed explanation of the performance gains only partly validated.

Questions for authors

Please address the concerns listed in the Weaknesses section above.

4 comments

We thank the reviewer for carefully reading our manuscript and for the constructive feedback.

[W1]

We acknowledge that our original comparisons alone do not fully isolate the effects of filtering from those of critic ensemble size and aggregation. To address this concern, we conducted additional ablation experiments. Specifically, we evaluated IQL with N=20N=20 critics and ReBRAC with N=50N=50 critics, matching the critic counts used in our main experiments, along with their corresponding mean-aggregation variants, denoted IQL-M and ReBRAC-M, respectively. All results are averaged over the same set of default OGBench tasks under the corresponding evaluation settings.

As shown in the results below, DD-IQL achieves higher mean scores than both matched-size IQL controls across all four γ,τ\gamma,\tau settings, and DD-ReBRAC achieves higher mean scores than both matched-size ReBRAC controls across the two γ\gamma settings. These results show that the performance advantage persists when the baselines use the same critic counts and when mean aggregation is included as an additional control.

SettingIQL (τ=0.9\tau=0.9)IQL (τ=0.99\tau=0.99)IQL (N=20,τ=0.9N=20,\tau=0.9)IQL (N=20,τ=0.99N=20,\tau=0.99)IQL-M (N=20,τ=0.9N=20,\tau=0.9)IQL-M (N=20,τ=0.99N=20,\tau=0.99)DD-IQL (τ=0.9\tau=0.9)DD-IQL (τ=0.99\tau=0.99)
Original γ\gamma0.190.040.120.160.190.020.210.28
γ=0.999\gamma=0.9990.150.050.170.150.100.040.360.35
SettingReBRACReBRAC (N=50N=50)ReBRAC-M (N=50N=50)DD-ReBRAC
Original γ\gamma0.300.110.410.48
γ=0.999\gamma=0.9990.440.160.460.63

We also acknowledge the reviewer’s observation that, with N=2N=2, the original discount factors, and τ=0.9\tau=0.9, DD-IQL has a lower mean score than IQL: 0.160.16 versus 0.190.19. However, DD-IQL achieves higher mean scores by larger margins in the other three matched settings: 0.250.25 versus 0.040.04 with a higher expectile, 0.280.28 versus 0.150.15 with a higher discount factor, and 0.290.29 versus 0.050.05 when both are increased. Together with the matched-critic comparisons, these results support our main claim that delayed and decoupled filtering improves robustness under the bias-stress settings examined in this study.

MethodOriginal γ\gamma, τ=0.9\tau=0.9Original γ\gamma, τ=0.99\tau=0.99γ=0.999\gamma=0.999, τ=0.9\tau=0.9γ=0.999\gamma=0.999, τ=0.99\tau=0.99
IQL (N=2N=2)0.190.040.150.05
IQL (N=20N=20)0.120.160.170.15
DD-IQL (N=2N=2)0.160.250.280.29
DD-IQL (N=20N=20)0.210.280.360.35

[W2]

To provide direct evidence of bias reduction, we added a controlled experiment in a known tabular MDP. We model critic errors using either IID noise or a synthetic noise field with spatial and temporal correlations, the latter serving as a simplified model of neural network approximation errors. Further details of the controlled MDP setup and the complete simulation code are available at the link below.

https://anonymous.4open.science/r/iclr2027_rebuttal-8647/

We compare tabular value-learning analogues of the DF-IQL, SD-IQL, and DD-IQL variants in Table 3 of our main paper. DF-IQL with D=1D=1 corresponds to the tabular analogue of the standard IQL value-learning update. The tables report Max-QQ bias after 10610^6 updates. Results are reported as mean ±\pm sample standard deviation across 20 training seeds.

DD-IQL achieves a substantially smaller magnitude of mean Max-QQ bias than DF-IQL and SD-IQL across the reported conditions. Increasing the delay generally reduces the positive bias of both SD-IQL and DD-IQL. DF-IQL, however, transitions from overestimation to substantial underestimation at larger delays. This pattern is consistent with the pessimistic effect of minimum aggregation becoming dominant as delay attenuates positive filtering-induced bias. Once overestimation is effectively controlled, retaining minimum aggregation can introduce excessive pessimism. This observation supports our design choice to avoid minimum aggregation in our proposed methods.

IID-noise

MethodD=1D=10D=100D=1000
DF-IQL+549.13 ± 1.25-400.02 ± 0.87-464.81 ± 1.22-457.69 ± 3.43
SD-IQL+1217.05 ± 0.97+119.87 ± 0.88+39.50 ± 0.84+13.85 ± 1.75
DD-IQL+13.14 ± 1.22+1.36 ± 0.98+0.16 ± 1.19-0.22 ± 1.29

NN-like Noise

MethodD=1D=10D=100D=1000
DF-IQL+522.77 ± 4.00+461.53 ± 4.12+59.88 ± 6.65-382.38 ± 11.02
SD-IQL+1172.28 ± 5.56+1117.05 ± 5.82+687.78 ± 9.00+114.21 ± 6.07
DD-IQL+62.12 ± 6.02+59.83 ± 6.41+30.07 ± 5.91+1.68 ± 5.13

We will incorporate the additional experimental results and clarifications presented in this rebuttal into the revised manuscript.

Thank you for the additional experiments and clarifications. My main concerns have been substantially addressed. My rating will therefore be increased from 2 to 3.

Thank you for carefully considering our additional experiments and clarifications, and for updating your assessment. Your constructive feedback has helped us strengthen the experimental comparisons and provide more direct evidence of bias reduction. We will incorporate the comparisons with matched critic counts, aggregation ablations, and known-MDP results into the revised manuscript, together with the corresponding clarifications. Please feel free to share any further questions or suggestions; we would be happy to address them.

We thank the reviewer again for the constructive feedback. We have uploaded the revised manuscript incorporating the additional experiments and clarifications discussed in our rebuttal. For ease of reference, the revised text is highlighted in blue.

[W1] We added Section 7.4 (Table 3) to report matched-critic and aggregation controls for IQL and ReBRAC. We also revised Section 7.5 (Table 4) to clarify the two-critic ablations and explicitly acknowledge the setting where DD-IQL does not outperform standard IQL.

[W2] We added Section 7.6 (Figure 2) and Appendix E.7 (Figure 4) to directly measure signed max-Q bias against known ground truth. We also revised Section 7.2 to clarify that correlation-based diagnostics measure dependence on an error proxy rather than true value error.

We sincerely appreciate the reviewer's suggestions, which helped improve the clarity and experimental support of our work. Please feel free to let us know if you have any further questions or concerns. We would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.