acceptodds
Under review as a conference paper at ICLR 2027

DAET: Distributional Action Equivalence Testing Integrating Aleatoric and Epistemic Uncertainty

Abstract

Modern reinforcement learning (RL) algorithms typically treat actions as distinct during learning, even when different actions lead to nearly identical long-term consequences. This action redundancy unnecessarily inflates the effective exploration space and severely degrades exploration efficiency. This motivates the need for a principled criterion for identifying actions that can be safely treated as equivalent. However, existing equivalence criteria are inadequate for this purpose: transition-based equivalence is overly restrictive, while mean-based equivalence is too permissive and insensitive to risk. More importantly, both paradigms treat action equivalence as a property of directly accessible ground truth, whereas in online RL, it must be inferred through learned estimates from finite data. To bridge this gap, we introduce Distributional Action Equivalence Testing (DAET), a novel formulation that casts action equivalence as a statistical inference problem over return distributions. DAET jointly accounts for aleatoric uncertainty in long-term outcomes and epistemic uncertainty in their learned estimates, requiring sufficient statistical evidence before equivalence is accepted. Building on this formulation, we propose DAET-Mask, an uncertainty-aware framework that operationalizes DAET for online RL. It integrates a distributional RL backbone with Deep Ensembles, a dual-gating mechanism, and an uncertainty-guided ROPE decision rule to realize safe dynamic action masking without relying on hand-crafted rules. Theoretically, we show that distributional action masking is near-optimal in the ideal case and remains robust under bounded estimation error. Experiments in both controlled and realistic environments demonstrate that DAET-Mask reliably identifies and masks redundant actions, thereby improving exploration efficiency.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, action equivalence, hypothesis testing, epistemic uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tycyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DAET, which identifies redundant actions by comparing their return distributions while accounting for uncertainty in the learned estimates. DAET-Mask combines distributional RL, ensembles, two screening gates, and a practical-equivalence criterion to mask redundant actions. The paper provides conditional value-preservation bounds and evaluates the method in a diagnostic GridWorld and a warehouse scheduling task.

Critical strengths
  1. The paper is generally clear and easy to follow. The distinction between uncertainty in outcomes and uncertainty in their estimates provides an intuitive motivation.

  2. The proposed mechanism is conceptually simple and easy to implement.

  3. The experiments combine a useful diagnostic environment with a meaningful industrial-data application. Ten-seed GridWorld results, sensitivity analyses, and paired statistical tests on 100 warehouse instances improve the transparency of the evaluation.

Critical weaknesses
  1. The theoretical guarantee does not directly cover the implemented policy. Theorem 1 constructs a policy that transfers each masked action’s probability to its retained representative. Equation (17), however, renormalizes probabilities over all remaining actions. These operations differ. Moreover, Corollary 1 assumes bounded estimation errors without establishing that the practical gates enforce those bounds. The results therefore provide conditional action-set preservation, rather than a safety guarantee for the actual learning process.

  2. Important controlled comparisons are missing. DAET introduces ensembling, uncertainty gating, and in RMFS, an auxiliary distributional loss. Comparisons against the same architecture without masking and against mean-based masking with the same uncertainty mechanism are needed to isolate the contribution of distributional testing.

  3. The evidence for improved learning efficiency remains limited. The warehouse results demonstrate better final scheduling quality, but do not report learning curves or training cost. Runtime increases from 5.06 s to 6.73 s relative to SOAR, while the action-space ratio decreases from 9.08% to 8.79%.

  4. The positioning against previous work is overly broad. The contribution should be positioned around distributional equivalence testing, rather than suggesting that previous approaches generally assume directly accessible ground truth.

Questions for authors

Please refer to Weaknesses.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer tycy [1/2]

Thank you for the detailed and constructive feedback. Below, we provide detailed responses to all the comments, and changes in the revised PDF are highlighted in blue.

W1: The theoretical guarantee does not directly cover the implemented policy. Theorem 1 constructs a policy that transfers each masked action’s probability to its retained representative. Equation (17), however, renormalizes probabilities over all remaining actions. These operations differ. Moreover, Corollary 1 assumes bounded estimation errors without establishing that the practical gates enforce those bounds. The results therefore provide conditional action-set preservation, rather than a safety guarantee for the actual learning process.

Theorem 1 guarantees the existence of a near-optimal policy within the reduced action space after masking. However, online RL algorithms learn through continuous exploration, and the current policy is typically not optimal during online learning. Therefore, we do not require the agent to explicitly transfer probabilities to the retained representative. Instead, we preserve the natural learning dynamics (Eq. 17) to continuously update the policy and gradually converge toward the optimum.

Moreover, Corollary 1 provides a conditional safety guarantee: If local estimation errors are bounded (Assumption i), masking is safe. In online RL, since the ground truth is inaccessible, estimation errors are inevitable with finite data. The dual-gating and ensemble ROPE rules are designed to push toward this assumption. By avoiding equivalence testing in regions of high epistemic uncertainty, these mechanisms ensure that masking is only triggered in well-explored regions where local estimation errors are practically controlled. Conceptually, as K→∞\mathrm{K} \to \infty, the ensemble yields a stabilized empirical distribution that bounds the function-space variance. To balance estimation stability and computational overhead in practice, we set K=10 as the default and have added its sensitivity study in Appendix I.5.

W2: Important controlled comparisons are missing. DAET introduces ensembling, uncertainty gating, and in RMFS, an auxiliary distributional loss. Comparisons against the same architecture without masking and against mean-based masking with the same uncertainty mechanism are needed to isolate the contribution of distributional testing.

Thank you for your suggestion. We add an ablation experiment.

  • The DAET-Mask w/o mask variant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, with only the dynamic masking disabled.
  • The DAET-Mask w/ mean-based masking variant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, and further incorporates the same epistemic uncertainty gate as DAET-Mask, but replaces the distributional testing with mean-based masking.
MethodObj (Makespan) ↓\downarrowCompT ↓\downarrowASR ↓\downarrow
SOAR406.79 (base)152.36 (base)9.08%
DAET-Mask w/o masking403.38 (↓\downarrow 0.8%)150.78 (↓\downarrow 1.0%)9.08% (unchanged)
DAET-Mask w/ mean-based masking406.12 (↓\downarrow 0.2%)152.60 (↑\uparrow 0.2%)7.50% (↓\downarrow 17.4%)
DAET-Mask (default)394.10 (↓\downarrow 3.1%)145.65 (↓\downarrow 4.4%)8.79% (↓\downarrow 3.2%)

The code has been updated in the anonymous repository https://anonymous.4open.science/r/daet-mask_public/rmfs/algorithms/ensemble_qrdqn_hetgat_mean.py. We have added the ablation study in Section 4.2 in the revised version.

Ablation study. To isolate the contribution of distributional equivalence masking and validate our architectural choices, we evaluate four variants of DAET-Mask on the RMFS task (Table 2). First, the DAET-Mask w/o masking variant disables dynamic masking and yields only modest improvements over the SOAR baseline. This confirms that the main performance gains of DAET-Mask come from our distribution equivalence masking. Second, the DAET-Mask w/ mean-based masking variant replaces the distributional ROPE test with a mean-based criterion while retaining the same epistemic uncertainty gate. It reduces ASR to 7.50% but increases CompT to 152.60. This aligns with our GridWorld findings that mean-based equivalence is risk-blind and prematurely prunes essential exploratory actions. Finally, we examine two variants that alter the policy architecture. Specifically, the DAET-Mask w/o ensemble-mean logits variant uses a standard PPO actor and scalar value head, while the DAET-Mask w/o greedy continuation policy variant constructs quantile TD targets using the PPO actor's next-action probabilities instead of the greedy action. Both variants exhibit slightly worse scheduling performance. These results confirm that our specific architectural choices are beneficial, but the core distributional masking drives the overall efficiency gains.

AuthorsOpenReview ↗

Reply to Reviewer tycy [2/2]

W3: The evidence for improved learning efficiency remains limited. The warehouse results demonstrate better final scheduling quality, but do not report learning curves or training cost. Runtime increases from 5.06 s to 6.73 s relative to SOAR, while the action-space ratio decreases from 9.08% to 8.79%.

RMFS is a challenging combinatorial optimization (OR) problem where efficient exploration is critical. The efficiency of exploration is directly related to the quality of the final solution. Moreover, a 3.2% action reduction greatly decreases subsequent combination choices. We have added the learning curves in the anonymous repository:

The learning curves show that DAET-Mask converges better in fewer iterations.

W4: The positioning against previous work is overly broad. The contribution should be positioned around distributional equivalence testing, rather than suggesting that previous approaches generally assume directly accessible ground truth.

Thank you for the suggestion. We agree that our core contribution is indeed distributional equivalence testing.

Our original intent was to highlight that previous methods typically evaluate action equivalence through deterministic comparisons of point estimates (e.g., mean Q-values or one-step dynamics). They inherently ignore the estimation uncertainty associated with these learned values in online RL. Our proposed distributional equivalence testing directly resolves this limitation by shifting the evaluation from deterministic comparisons to a rigorous statistical inference framework. Therefore, we introduce DAET, a novel formulation that casts action equivalence in RL as a hypothesis testing problem, jointly accounting for aleatoric uncertainty in long-term outcomes and epistemic uncertainty in their learned estimates.

Dear Reviewer,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Regards,

Authors

Reviewer_FHfqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles action redundancy in RL: when distinct actions lead to nearly identical long-term outcomes, treating them as separate inflates the exploration space. The authors argue existing criteria are inadequate. Transition based equivalence is too strict and mean based equivalence is too permissive and risk-blind (equal expected returns can hide very different risk profiles). Their fix is to compare the full distribution of possible returns for each action, not just the average. Two actions count as equivalent if their return distributions are close enough (measured by a distance called the 1-Wasserstein distance). But since the agent only has limited data and can't know the true distributions, they turn this into a statistical test: declare two actions equivalent only when there's enough evidence that they're really close, not just close by luck.

Critical strengths
  1. Casting equivalence as a distributional, uncertainty-aware test resolves a genuine and well-articulated gap.
  2. Gate 1 is principled, not heuristic.
Critical weaknesses
  1. Empirical validation is too narrow to support the generality claimed.
  2. Reliability of the epistemic quantification is unestablished. The central safety argument depends on the ensemble-quantile ROPE rule being trustworthy, but K is small, there is no calibration study, and recall is only 0.67 even in the toy setting.
Questions for authors
  1. What is the ensemble size K, and how sensitive are results to it? Do you have any calibration evidence (e.g., coverage) that the 1−α ensemble quantile approximates a real credible bound?
  2. Can you report on at least one standard discrete-action benchmark to show the method generalises beyond a bespoke GridWorld and one OR task?
  3. What is the wall-clock / compute overhead of ensembles × quantile heads × pairwise testing per decision, relative to QR-DQN?
  4. How were ΔQ, τ_ep, ε, T_mask chosen per environment?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer FHfq [1/3]

Thank you for the detailed and constructive feedback. Below, we provide detailed responses to all the comments, and changes in the revised PDF are highlighted in blue.

W1 & Q2: Empirical validation is too narrow to support the generality claimed. Can you report on at least one standard discrete-action benchmark to show the method generalises beyond a bespoke GridWorld and one OR task?

To better validate the generality of our method, we add a new experiment on MinAtar/Breakout-v0 (using the standard 6-action space), a widely used discrete-action RL benchmark. This environment naturally tests action abstraction in a standard setting: while left and right are critical for paddle control, actions such as up, down, noop, and fire (when the ball is already in play) lead to identical long-term return distributions.

Using the QR-DQN backbone (K=10\mathrm K=10 heads, N=32\mathrm N=32 quantiles), we compare DAET-Mask against QR-DQN (unmasked), Mean-Q Masking, Forward-Dynamics, and Inverse-Dynamics across 10 random seeds for 2,000,000 frames (ΔQ=0.2,ϵ=0.2,τep=0.3\Delta_Q=0.2, \epsilon=0.2, \tau_\text{ep}=0.3).

Table 7. Comparison on MinAtar/Breakout-v0 (10 seeds, 2M frames).

MethodFinal Return (↑\uparrow)Active Actions (↓\downarrow)Precision (↑\uparrow)Recall (↑\uparrow)
QR-DQN41.63 ±\pm 9.816.00 ±\pm 0.00N/AN/A
Mean-Q Masking34.71 ±\pm 9.202.59 ±\pm 0.250.75 ±\pm 0.020.50 ±\pm 0.03
Forward-Dynamics41.64 ±\pm 11.785.31 ±\pm 0.130.98 ±\pm 0.010.25 ±\pm 0.06
Inverse-Dynamics38.17 ±\pm 7.415.33 ±\pm 0.110.98 ±\pm 0.010.07 ±\pm 0.10
DAET-Mask (Ours)59.49 ±\pm 8.622.96 ±\pm 0.200.92 ±\pm 0.020.50 ±\pm 0.03

We have updated the code in the anonymous repository https://anonymous.4open.science/r/daet-mask_public/train_minatar.py. We have also added these results in Appendix K of the revised version.

As shown in Table 7, DAET-Mask effectively abstracts the action space to an average of 2.962.96 active actions, which closely aligns with the theoretically optimal action space size for this task (approximately 3, corresponding to left, right, and one active counterpart for fire/noop). By reliably filtering out redundant exploratory moves while maintaining a high masking precision of 0.920.92, DAET-Mask substantially improves exploration efficiency and outperforms the unmasked QR-DQN baseline in final return (59.4959.49 vs. 41.6341.63). In contrast, the baseline methods exhibit two distinct failure modes: severe over-masking and ineffective under-masking. Mean-Q Masking relies solely on expected returns and fails to distinguish underlying risk, resulting in an aggressive over-reduction of active actions to 2.592.59. This inevitably prunes essential paddle movements, directly causing a performance collapse to 34.7134.71. Conversely, both transition-based baselines, i.e., Forward-Dynamics and Inverse-Dynamics, are overly conservative, barely reducing the action space (≥5.31\ge 5.31) and yielding negligible performance gains. These results further demonstrate that DAET-Mask can safely discard redundancy to accelerate learning without compromising decision quality.

AuthorsOpenReview ↗

Reply to Reviewer FHfq [2/3]

W2 & Q1: Reliability of the epistemic quantification is unestablished. The central safety argument depends on the ensemble-quantile ROPE rule being trustworthy, but K is small, there is no calibration study, and recall is only 0.67 even in the toy setting. What is the ensemble size K, and how sensitive are results to it? Do you have any calibration evidence (e.g., coverage) that the 1−α ensemble quantile approximates a real credible bound?

Ensemble size. In our experiments, the ensemble size is K=10. It represents a trade-off between estimation stability and computational overhead in online RL. To address your concern, we have added an sensitivity study on K (K=5, 10, 15, 20, 25). We have updated the results in Appendix I.5 in the revised version.

We further analyze the sensitivity of DAET-Mask to the number of ensemble heads K\mathrm{K} by varying K∈{5,10,15,20,25}\mathrm{K} \in \{5,10,15,20,25\}. As shown in Figure 8, DAET-Mask exhibits stable learning curves across different values of K\mathrm{K} and converges to comparable episodic returns. To balance estimation stability and computational overhead in online RL, we set K=10\mathrm{K}=10 as the default in the main text.

Calibration. Executing rigorous statistical inference online during RL training is computationally prohibitive. To bridge this gap, DAET-Mask uses Deep Ensembles to construct an empirical distribution. The ensemble discrepancy quantile acts as an empirical uncertainty score, and the tolerance parameter α\alpha​ acts as a practical gating threshold. We have revised Section 3 to tighten the statements of our practical rule:

  1. "rigorous statistical inference"->"uncertainty-aware empirical approximation";

  2. "reliably small"->"empirically bounded within the ROPE".

Moreover, we add a calibration analysis in the GridWorld environment, using K=10 ensemble members across 10 independent training runs. Under frozen continuation policies, we construct reference return distributions from the environment and measure how often the reference Wasserstein distance falls below the empirical 0.95 ensemble discrepancy quantile. We evaluate both unfiltered action pairs and pairs passing the dual-gating mechanism, obtaining coverage of 59.27% and 88.28%, respectively. The results support the reliability of our epistemic quantification in practice, with the dual-gating mechanism substantially improving empirical coverage. The code is updated in the anonymous repository (https://anonymous.4open.science/r/daet-mask_public/calibrate_daet.py).

Recall. In RL action masking, safety takes priority over efficiency.

  • False Positives (FP): Masking a valid, non-redundant action, which may cause catastrophic performance collapse (as observed in the Mean-Q Masking baseline).
  • False Negatives (FN): Failing to mask a redundant action, which may cost some exploration efficiency but remains safe for policy learning.

Therefore, the outcome of low precision is much severer than low recall. DAET-Mask obtains a near-perfect Precision of 1.0 paired with a Recall of 0.67. It improves exploration efficiency while maintaining safety. Moreover, compared to other baselines, DAET-Mask achieves the highest precision while maintaining a much higher recall.

AuthorsOpenReview ↗

Reply to Reviewer FHfq [3/3]

Q3: What is the wall-clock / compute overhead of ensembles × quantile heads × pairwise testing per decision, relative to QR-DQN?

We measured 0.85 ms per decision for DAET-Mask versus 0.11 ms for QR-DQN, excluding environment interactions and backpropagation. These measurements were averaged over 10,000 environment steps. Both methods use N=32 quantiles per action, and DAET-Mask uses K=10 ensemble members. Masking runs every 200 steps, with dual gating filtering candidate comparisons.

Specifically, DAET-Mask’s cost comprises 0.64 ms for ensemble-based action selection and 0.21 ms for distributional testing. The pairwise testing overhead is computationally cheaper than the network forward pass. More generally, assuming QR-DQN operates with an O(N)\mathcal{O}(\mathrm{N}) time complexity, DAET-Mask scales at O(KN)\mathcal{O}(\mathrm{KN}). In practice, the empirical time multiplier is smaller than K because GPU parallelization prevents the ensemble cost from scaling exactly linearly. We verified this by profiling K∈{5,10,15,20,25}\mathrm{K} \in \{5, 10, 15, 20, 25\}, confirming the overall cost consistently remains a predictable constant-factor O(KN)\mathcal{O}(\mathrm{KN}) complexity.

The profiling code is updated in the anonymous repository: https://anonymous.4open.science/r/daet-mask_public/train_timing.py.

Q4: How were ΔQ, τ_ep, ε, T_mask chosen per environment?

How to choose ΔQ,τep,ϵ\Delta_Q, \tau_\text{ep}, \epsilon ? These thresholds were chosen based on sensitivity analyses to balance effective action-space reduction with learning stability. The detailed sensitivity results are provided in Appendix I.5 (GridWorld) and J.5 (RMFS).

  • GridWorld: We clarified the detailed experimental setting in Appendix I.3.

    The DAET-Mask pipeline uses the following hyperparameters: epistemic CV threshold τep\tau_\text{ep} = 0.3, Q-value margin ΔQ\Delta_Q = 0.3, Wasserstein ROPE radius ϵ\epsilon = 0.3, and tolerance error level α\alpha = 0.05.

  • RMFS: We have updated Appendix J.4 to explicitly state these defaults values.

    The DAET-Mask pipeline uses the following hyperparameters: epistemic CV threshold τep\tau_\text{ep} = 0.015, Q-value margin ΔQ\Delta_Q = 0.5, Wasserstein ROPE radius ϵ\epsilon = 0.2, and tolerance error level α\alpha = 0.05.

How to choose TmaskT_\text{mask} ? The masking horizon TmaskT_\text{mask} is not tuned empirically, but structurally determined by the environment's state recurrence properties.

  • GridWorld: We explained how to choose TmaskT_\text{mask} in GridWorld in Appendix I.3.

    Regarding the dynamic masking horizon, we set Tmask=∞T_\text{mask} = \infty. In this controlled GridWorld environment, the state space is small and discrete (10×10 = 100 states), meaning states are frequently revisited. Furthermore, the manually injected redundant actions are exact duplicates, rendering their distributional equivalence stationary under any continuation policy. Therefore, once an action is confidently identified as redundant at a specific state, maintaining a persistent state-indexed mask is highly efficient.

  • RMFS: We explained how to choose TmaskT_\text{mask} in RMFS in Appendix J.4.

    Regarding the dynamic masking horizon, we set Tmask=1T_\text{mask} = 1. More generally, DAET-Mask supports both persistent and transient masking depending on the structure of the state space. Persistent state-indexed masks are suitable for discrete or frequently revisited states, while transient masks are preferable when exact state recurrence is rare. The RMFS state is a high-dimensional, event-dependent combination of robots, orders, shelves, inventories, and workstation queues. Its graph-structured state and event-dependent candidate sets are almost never repeated exactly. Consequently, maintaining a persistent state-indexed mask table would introduce substantial storage and lookup overhead while being frequently unused. We therefore construct a transient mask from the current state at every decision step, apply it exclusively to that step's action selection, and immediately expire it (Tmask=1T_{\mathrm{mask}}=1). All candidate actions are reconsidered at the next decision event. This implementation accommodates the asynchronous nature of RMFS and avoids carrying masking decisions across distinct high-dimensional states.

Dear Reviewer,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Regards,

Authors

Reviewer_NM2sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Distributional Action Equivalence Testing (DAET), which identifies actions with similar long-term return distributions while accounting for uncertainty in their learned estimates. Its practical framework, DAET-Mask, combines quantile-based distributional reinforcement learning with an ensemble of predictors. Candidate actions are compared with a retained greedy action after passing expected-value proximity and ensemble-confidence gates. An action is masked when an upper empirical quantile of the ensemble-wise Wasserstein distances falls within a specified equivalence region.

The paper establishes a conditional value-preservation bound: if each masked action has a retained representative with a sufficiently similar true return distribution under a reference policy, the restricted action space contains a policy that approximately preserves that reference policy’s expected value. An optimal-policy specialization and an extension incorporating bounded estimation errors are also provided.

Experiments use a controlled GridWorld containing duplicate, multi-step-equivalent, and risk-distinct actions, together with a warehouse order-allocation and robot-scheduling task based on industrial data. The results demonstrate action-space reduction, better masking accuracy than the tested alternatives, and improved scheduling objectives.

Critical strengths
  1. The distributional criterion captures a meaningful distinction that expected-value comparisons miss. The GridWorld includes both equal-mean actions with different reward variability and actions that take different immediate transitions but merge into identical longer-term outcomes. This makes the motivation concrete and illustrates the intended benefit over the tested mean- and transition-based criteria.

  2. The framework provides an explicit mechanism for applying distributional comparisons during online action selection. Comparing candidates with a retained greedy representative avoids exhaustive pairwise testing, while the gates reduce unnecessary comparisons. The conditional theorem also gives a clear account of when restricting the action space preserves reference-policy value, including how estimation errors affect the bound.

  3. The scheduling evaluation provides credible evidence of practical benefit. It uses chronological training, validation, and test splits and reports results on sufficient number of test instances.

Critical weaknesses
  1. The relationship between the described PPO method and the released implementation needs clarification. Section 3.1 describes an integration through auxiliary quantile heads that can leave the original actor intact. In the released RMFS implementation, however, action logits are constructed directly from ensemble-mean quantile predictions, together with heuristic scores. The distributional heads therefore participate in the policy parameterization rather than serving only as auxiliary estimates for masking. Their targets also use greedy next-action bootstrapping. It is unclear which implementation produced the reported results and how its continuation policy corresponds to the return distributions used in the formulation. This matters because the scheduling improvement is attributed to distributional equivalence masking, while the released implementation appears to involve a broader change to the policy architecture.

  2. The statistical language is stronger than the justification provided. The practical rule uses an empirical quantile across ensemble predictions, but the paper does not establish that these predictions provide calibrated uncertainty about the true distributional discrepancy. Ensemble agreement can coexist with shared estimation error. The manuscript does acknowledge this limitation for the shared-backbone implementation, which is helpful, but descriptions such as “rigorous statistical inference” and acceptance only when a discrepancy is “reliably small” remain too strong without a clearer interpretation. The rule can be a useful uncertainty-aware approximation without being a calibrated equivalence test. Similarly, the conditional theorem does not establish its estimation-error assumptions from passing the gates or the ensemble test.

  3. I think the masking-accuracy claims need correction. Section 4.1 states precision of 1.0, whereas Table 3 reports mean precision of 0.94 ± 0.05 and 3.20 ± 2.93 false positives. Eight of ten seeds have at least one false positive. The results still favor DAET-Mask, but the distinction matters when assessing claims of safe masking. Appendix I.4 also reverses the earlier meanings of TP, FP, and FN by describing retained valid actions rather than correctly masked redundancies. The released GridWorld entry point additionally imports modules absent from the visible repository, and its defaults differ from the reported experimental settings.

Questions for authors
  1. Which continuation policy do the learned return distributions represent in each implementation? In particular, how does greedy bootstrapping of the RMFS quantile heads relate to the stochastic PPO policy and the policy-relative equivalence definition?

  2. What interpretation should be attached to the ensemble discrepancy quantile and its tolerance parameter? Is this intended as an empirical uncertainty score or as a calibrated credible bound? Please distinguish the assurance supplied by the practical rule from the assumptions of the conditional theorem.

  3. Please reconcile the precision claim in Section 4.1 with Table 3, confirm the definitions used to compute TP, FP, and FN, and provide the missing dependencies and commands needed to reproduce the reported GridWorld settings.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer NM2s [1/3]

Thank you for the detailed and constructive feedback. Below, we provide detailed responses to all the comments, and changes in the revised PDF are highlighted in blue.

W1 & Q1: RMFS implementation, continuation policy, and ablation contribution.

Integration with PPO. In Section 3.1, we aim to illustrate that the Quantile Regression (QR) parameterization is not restricted to value-based RL, but compatible with policy-based RL. In PPO setup, the distributional estimates can be generated using auxiliary quantile heads, but this does not necessarily require leaving the original actor and scalar value head intact. The original actor and scalar value head can remain unchanged or can be assisted by ensemble heads. We have revised Section 3.1 more clearly:

Crucially, our framework can be integrated with both value-based methods (e.g., QR-DQN) and policy-based methods (e.g., PPO) using quantile-based return-distribution estimates.

In the Table below, we present the results obtained using these two different ways. They both performed the SOTA baseline, and we adopted a more effective implementation way to couple the actor with the distributional heads.

Continuation policy. Both the GridWorld and RMFS implementations construct quantile targets using greedy next-action bootstrapping. In RMFS, PPO determines how actions are stochastically sampled during online interaction, while the greedy rule determines how the TD targets for the quantile heads are constructed. This off-policy evaluation paradigm decouples exploration from equivalence assessment: the distributional Bellman operator validly estimates the target policy's return distributions independent of the stochastic behavioral policy. Consequently, the learned return distributions represent the greedy continuation policy. Two actions are deemed equivalent if they lead to identical long-term distributions under greedy subsequent decision-making. We have clarified this continuation policy and its distinction from PPO action sampling in Appendix J.4 of the revised version.

In the Table below, we present the results of replacing the greedy continuation policy with the PPO stochastic policy. It is slightly worse than primary DAET-Mask due to exploration noise and randomness.

Ablation. To further isolate the contribution of the distributional equivalence masking, we add an ablation experiment.

  • The DAET-Mask w/o ensemble-mean logits variant uses a standard, independent PPO actor and scalar value head instead of constructing action logits from ensemble-mean quantile predictions.
  • The DAET-Mask w/o greedy continuation policy variant constructs the quantile TD targets by taking the expectation over the next-action probabilities given by the PPO actor, rather than bootstrapping from the greedy action.
  • The DAET-Mask w/o mask variant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, with only the dynamic masking disabled.
  • The DAET-Mask w/ mean-based masking variant retains the same ensemble quantile heads, policy parameterization, QR loss, and greedy bootstrap targets as DAET-Mask, and further incorporates the same epistemic uncertainty gate as DAET-Mask, but replaces the distributional testing with mean-based masking.
MethodObj (Makespan) ↓\downarrowCompT ↓\downarrowASR ↓\downarrow
SOAR406.79 (base)152.36 (base)9.08%
DAET-Mask w/o ensemble-mean logits396.32 (↓\downarrow 2.6%)147.54 (↓\downarrow 3.2%)8.82% (↓\downarrow 2.9%)
DAET-Mask w/o greedy continuation policy395.89 (↓\downarrow 2.7%)147.02 (↓\downarrow 3.5%)8.83% (↓\downarrow 2.8%)
DAET-Mask w/o masking403.38 (↓\downarrow 0.8%)150.78 (↓\downarrow 1.0%)9.08% (unchanged)
DAET-Mask w/ mean-based masking406.12 (↓\downarrow 0.2%)152.60 (↑\uparrow 0.2%)7.50% (↓\downarrow 17.4%)
DAET-Mask (default)394.10 (↓\downarrow 3.1%)145.65 (↓\downarrow 4.4%)8.79% (↓\downarrow 3.2%)

These variants have been updated in the anonymous repository.

We have added the ablation study in Section 4.2 in the revised version.

Ablation study. To isolate the contribution of distributional equivalence masking and validate our architectural choices, we evaluate four variants of DAET-Mask on the RMFS task (Table 2). First, ... Second, ... Finally, ... These results confirm that our specific architectural choices are beneficial, but the core distributional masking drives the overall efficiency gains. (detailed in the revised paper)

AuthorsOpenReview ↗

Reply to Reviewer NM2s [2/3]

W2 & Q2: Statistical language.

Statistical Formulation vs. Practical Rule. In Section 2, we established a rigorous hypothesis testing formulation DAET by mathematically defining the precise null hypothesis and the equivalence conditions based on true return distributions. Our implementation DAET-Mask was guided by this principle. However, executing rigorous statistical inference online during RL training is computationally prohibitive. To bridge this gap, DAET-Mask uses Deep Ensembles to construct an empirical distribution. The ensemble discrepancy quantile acts as an empirical uncertainty score, and the tolerance parameter α\alpha acts as a practical gating threshold. We have revised Section 3 to tighten the statements of our practical rule:

  1. "rigorous statistical inference"->"uncertainty-aware empirical approximation";

  2. "reliably small"->"empirically bounded within the ROPE".

Calibration. Moreover, we add a calibration analysis in the GridWorld environment, using K=10 ensemble members across 10 independent training runs. Under frozen continuation policies, we construct reference return distributions from the environment and measure how often the reference Wasserstein distance falls below the empirical 0.95 ensemble discrepancy quantile. We evaluate both unfiltered action pairs and pairs passing the dual-gating mechanism, obtaining coverage of 59.27% and 88.28%, respectively. The results support the reliability of our epistemic quantification in practice, with the dual-gating mechanism substantially improving empirical coverage. The code is updated in the anonymous repository (https://anonymous.4open.science/r/daet-mask_public/calibrate_daet.py).

Corollary. Corollary 1 provides a conditional safety guarantee: If local estimation errors are bounded (Assumption i), masking is safe. However, since the ground truth is inaccessible during online learning, we cannot absolutely guarantee this assumption. The dual-gating and ensemble ROPE rules are designed to push toward this assumption. Specifically, they avoid equivalence testing in regions of high epistemic uncertainty, instead performing in well-explored regions where estimation errors are practically controlled. Conceptually, as K$\to\infty$, the ensemble yields a stabilized empirical distribution, capturing the function-space variance. In practice, to balance estimation stability and computational overhead in online RL, we set K=10 as the default in the main text. We have added its sensitivity study in Appendix I.5.

We further analyze the sensitivity of DAET-Mask to the number of ensemble heads K\mathrm{K} by varying K∈{5,10,15,20,25}\mathrm{K} \in \{5,10,15,20,25\}. As shown in Figure 8, DAET-Mask exhibits stable learning curves across different values of K\mathrm{K} and converges to comparable episodic returns. To balance estimation stability and computational overhead in online RL, we set K=10\mathrm{K}=10 as the default in the main text.

AuthorsOpenReview ↗

Reply to Reviewer NM2s [3/3]

W3 & Q3: Metrics and implementation details.

Precision. In Section 4.1, we reported "Figures 2c–2e further reveal the masking accuracy, with DAET-Mask achieving nearly perfect precision of 1.0 while maintaining a recall of 0.67." It doesn't conflict with Table 3 in the Appendix. We have revised this statement more rigorously, and Table 3 in the Appendix records the detailed results.

Figures 2c–2e further reveal the masking accuracy, with DAET-Mask achieving a high precision of 0.94 while maintaining a recall of 0.67.

TP, FP, FN. Thank you for your clarification. All empirical results reported in tables and figures were computed using the definitions provided in Appendix I.1. We have updated the descriptions of TP, FP, FN in Appendix I.4 in the revised version.

Following the definitions of TP, FP, and FN defined in Eq.(74) and Eq.(76), TP counts redundant actions correctly masked while retaining at least one representative per equivalence class. FP counts over-masking errors that eliminate the last representative of an equivalence class or mask a singleton action. FN counts redundant actions that remain unmasked beyond one representative per class.

Reproducibility. We apologize for the difficulty in running the code. We have updated the anonymous repository: https://anonymous.4open.science/r/daet-mask_public. The following is what has been changed.

Two imports influenced the code running.

  1. from daet_mask_agent import DAETMaskAgent: DAETMaskAgent is the first version using MC Dropout[1]. We ultimately adopt Deep Ensembles as an alternative approach to estimate epistemic uncertainty and construct an empirical distribution of the 1-Wasserstein distance for DAET (see ensemble_daet_agent.py). Therefore, this import can be deleted directly without any influence. We also uploaded this file for your reference: https://anonymous.4open.science/r/daet-mask_public/daet_mask_agent.py.
  2. from plot_heatmap import plot_pruning_heatmap: This is a diagnostic utility to visualize action masks during development. Retaining or deleting it doesn't influence the final performance. We have added this file in https://anonymous.4open.science/r/daet-mask_public/plot_heatmap.py.

Dependencies and commands.

We have exported all dependencies to daet.yml (https://anonymous.4open.science/r/daet-mask_public/daet.yml). But this way may be affected by system differences. Therefore, I recommend you to create the environment using requirements.txt (https://anonymous.4open.science/r/daet-mask_public/requirements.txt). I have tried it successfully on my device.

  1. Ensure you have Python 3.9+ installed.
  2. Then, install the core dependencies via pip: pip install -r requirements.txt.
  3. Run python run.py --num_seeds 10 to obtain the GridWorld results.
  4. If you need to run the RMFS experiment, run pip install -r rmfs/requirements.txt for additional installation.

[1] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML'16).

Dear Reviewer,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their constructive feedback and time. During the discussion period, we uploaded a revised manuscript (with changes highlighted in blue) and updated our anonymous code repository. Below is a concise summary of the new experiments, analyses, and clarifications added to address the reviewers' comments:

1. Validation on Standard Discrete-Action Benchmark (Addressing FHfq)

  • MinAtar/Breakout-v0 (10 seeds, 2M frames): Added evaluations on a standard discrete benchmark (Appendix K). DAET-Mask abstracts the 6-action space to 2.96 active actions (aligning with the 3 non-redundant actions) with 0.92 precision, boosting the final return to 59.49 compared to 41.63 for unmasked QR-DQN and 34.71 for Mean-Q Masking (which collapses due to over-masking). The results support the generality of our method.

2. Controlled Ablations & Training Efficiency on Industrial RMFS (Addressing NM2s, tycy)

  • Architectural & Masking Ablations (Table 2): Evaluated four controlled variants on the real-world RMFS scheduling task. Disabling dynamic masking (Obj: 403.38, CompT: 150.78) or replacing distributional testing with mean-based masking (Obj: 406.12, CompT: 152.60) degrades performance compared to full DAET-Mask (Obj: 394.10, CompT: 145.65), isolating distributional equivalence masking as the primary driver of the gains.
  • PPO Integration & Learning Curves: Clarified how ensemble quantile heads couple with PPO via greedy next-action bootstrapping (off-policy target evaluation) and added RMFS learning curves demonstrating faster convergence in fewer iterations.

3. Uncertainty Calibration, Sensitivity & Compute Overhead (Addressing NM2s, FHfq)

  • Empirical Calibration Study: Evaluated empirical coverage in GridWorld (K=10, 10 seeds). Passing the dual-gating mechanism increases the empirical coverage of the 0.95 ensemble quantile bound from 59.27% to 88.28%, supporting the practical reliability of our uncertainty-guided ROPE rule.
  • Ensemble Size (K) & Wall-Clock Profiling: Added sensitivity analyses across K∈{5,10,15,20,25}\mathrm K \in \{5, 10, 15, 20, 25\} (Figure 8), showing stable returns across K. Profiled per-decision runtime (0.85 ms vs. 0.11 ms for QR-DQN), confirming predictable O(KN)\mathcal{O}(\mathrm{KN}) scaling.
  • Refined Statistical Framing: Tightened Section 3 to clearly distinguish the formal hypothesis-testing formulation (Section 2) from its practical empirical approximation via Deep Ensembles.

4. Theoretical Clarifications, Metric Alignment & Reproducibility (Addressing NM2s, FHfq, tycy)

  • Theory-to-Practice Link: Clarified how Theorem 1 and Corollary 1 guarantee value preservation in the reduced action space while standard softmax exploration (Eq. 17) drives policy learning.
  • Metric Consistency & Codebase: Aligned precision reporting in Section 4.1 and the TP/FP/FN descriptions in Appendix I.4, and updated the anonymous repository with all ablation scripts, calibration/profiling code, and full dependency instructions (requirements.txt).

We believe these updates directly address all the reviewers' concerns. They aim to better present our main insight: In reinforcement learning, action equivalence should be framed as an uncertainty-aware distributional testing problem, jointly accounting for intrinsic outcome risk (aleatoric uncertainty) and finite-sample estimation error (epistemic uncertainty), rather than relying on deterministic point comparisons. DAET-Mask is one practical way to concretize this perspective for safe dynamic action masking. Overall, we believe this distributional testing viewpoint offers a useful foundation for efficient exploration and hope to share it with the community through this work. Thank you again for your time and consideration.

Regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.