- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DAET, which identifies redundant actions by comparing their return distributions while accounting for uncertainty in the learned estimates. DAET-Mask combines distributional RL, ensembles, two screening gates, and a practical-equivalence criterion to mask redundant actions. The paper provides conditional value-preservation bounds and evaluates the method in a diagnostic GridWorld and a warehouse scheduling task.
Critical strengths
-
The paper is generally clear and easy to follow. The distinction between uncertainty in outcomes and uncertainty in their estimates provides an intuitive motivation.
-
The proposed mechanism is conceptually simple and easy to implement.
-
The experiments combine a useful diagnostic environment with a meaningful industrial-data application. Ten-seed GridWorld results, sensitivity analyses, and paired statistical tests on 100 warehouse instances improve the transparency of the evaluation.
Critical weaknesses
-
The theoretical guarantee does not directly cover the implemented policy. Theorem 1 constructs a policy that transfers each masked action’s probability to its retained representative. Equation (17), however, renormalizes probabilities over all remaining actions. These operations differ. Moreover, Corollary 1 assumes bounded estimation errors without establishing that the practical gates enforce those bounds. The results therefore provide conditional action-set preservation, rather than a safety guarantee for the actual learning process.
-
Important controlled comparisons are missing. DAET introduces ensembling, uncertainty gating, and in RMFS, an auxiliary distributional loss. Comparisons against the same architecture without masking and against mean-based masking with the same uncertainty mechanism are needed to isolate the contribution of distributional testing.
-
The evidence for improved learning efficiency remains limited. The warehouse results demonstrate better final scheduling quality, but do not report learning curves or training cost. Runtime increases from 5.06 s to 6.73 s relative to SOAR, while the action-space ratio decreases from 9.08% to 8.79%.
-
The positioning against previous work is overly broad. The contribution should be positioned around distributional equivalence testing, rather than suggesting that previous approaches generally assume directly accessible ground truth.
Questions for authors
Please refer to Weaknesses.
3 comments
Reply to Reviewer tycy [2/2]
W3: The evidence for improved learning efficiency remains limited. The warehouse results demonstrate better final scheduling quality, but do not report learning curves or training cost. Runtime increases from 5.06 s to 6.73 s relative to SOAR, while the action-space ratio decreases from 9.08% to 8.79%.
RMFS is a challenging combinatorial optimization (OR) problem where efficient exploration is critical. The efficiency of exploration is directly related to the quality of the final solution. Moreover, a 3.2% action reduction greatly decreases subsequent combination choices. We have added the learning curves in the anonymous repository:
- https://anonymous.4open.science/r/daet-mask_public/rmfs/makespan_iter_1_all_smooth10_DAET.png
- https://anonymous.4open.science/r/daet-mask_public/rmfs/completed_time_iter_1_all_smooth10_DAET.png
- https://anonymous.4open.science/r/daet-mask_public/rmfs/makespan_iter_1_all_smooth10_SOAR.png
- https://anonymous.4open.science/r/daet-mask_public/rmfs/completed_time_iter_1_all_smooth10_SOAR.png
The learning curves show that DAET-Mask converges better in fewer iterations.
W4: The positioning against previous work is overly broad. The contribution should be positioned around distributional equivalence testing, rather than suggesting that previous approaches generally assume directly accessible ground truth.
Thank you for the suggestion. We agree that our core contribution is indeed distributional equivalence testing.
Our original intent was to highlight that previous methods typically evaluate action equivalence through deterministic comparisons of point estimates (e.g., mean Q-values or one-step dynamics). They inherently ignore the estimation uncertainty associated with these learned values in online RL. Our proposed distributional equivalence testing directly resolves this limitation by shifting the evaluation from deterministic comparisons to a rigorous statistical inference framework. Therefore, we introduce DAET, a novel formulation that casts action equivalence in RL as a hypothesis testing problem, jointly accounting for aleatoric uncertainty in long-term outcomes and epistemic uncertainty in their learned estimates.
Dear Reviewer,
We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!
Regards,
Authors
Reply to Reviewer tycy [1/2]
Thank you for the detailed and constructive feedback. Below, we provide detailed responses to all the comments, and changes in the revised PDF are highlighted in blue.
W1: The theoretical guarantee does not directly cover the implemented policy. Theorem 1 constructs a policy that transfers each masked action’s probability to its retained representative. Equation (17), however, renormalizes probabilities over all remaining actions. These operations differ. Moreover, Corollary 1 assumes bounded estimation errors without establishing that the practical gates enforce those bounds. The results therefore provide conditional action-set preservation, rather than a safety guarantee for the actual learning process.
Theorem 1 guarantees the existence of a near-optimal policy within the reduced action space after masking. However, online RL algorithms learn through continuous exploration, and the current policy is typically not optimal during online learning. Therefore, we do not require the agent to explicitly transfer probabilities to the retained representative. Instead, we preserve the natural learning dynamics (Eq. 17) to continuously update the policy and gradually converge toward the optimum.
Moreover, Corollary 1 provides a conditional safety guarantee: If local estimation errors are bounded (Assumption i), masking is safe. In online RL, since the ground truth is inaccessible, estimation errors are inevitable with finite data. The dual-gating and ensemble ROPE rules are designed to push toward this assumption. By avoiding equivalence testing in regions of high epistemic uncertainty, these mechanisms ensure that masking is only triggered in well-explored regions where local estimation errors are practically controlled. Conceptually, as K→∞, the ensemble yields a stabilized empirical distribution that bounds the function-space variance. To balance estimation stability and computational overhead in practice, we set K=10 as the default and have added its sensitivity study in Appendix I.5.
W2: Important controlled comparisons are missing. DAET introduces ensembling, uncertainty gating, and in RMFS, an auxiliary distributional loss. Comparisons against the same architecture without masking and against mean-based masking with the same uncertainty mechanism are needed to isolate the contribution of distributional testing.
Thank you for your suggestion. We add an ablation experiment.
DAET-Mask w/o maskvariant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, with only the dynamic masking disabled.DAET-Mask w/ mean-based maskingvariant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, and further incorporates the same epistemic uncertainty gate as DAET-Mask, but replaces the distributional testing with mean-based masking.The code has been updated in the anonymous repository https://anonymous.4open.science/r/daet-mask_public/rmfs/algorithms/ensemble_qrdqn_hetgat_mean.py. We have added the ablation study in Section 4.2 in the revised version.