acceptodds
Under review as a conference paper at ICLR 2027

Do Faithful Explanations Survive Disruption? Interpretable RL in Carbon‑Aware Dual‑Source Inventory Control

Abstract

The aim of interpretable reinforcement learning is to achieve safer and more reliable decision-making through the use of transparent policies. We look at this assumption in the case of carbon-aware dual-source inventory control, where a product can be restocked quickly by means of high-emission air freight or more slowly by using lower-emission rail. We have developed a simulator which incorporates ordering costs, holding costs, backlog penalties, and partial Scope 3 emissions, and we evaluate the policies under a number of disruption scenarios such as demand spikes, supplier outages, lead-time shocks, low starting inventory, and longer planning horizons. The policy architectures employed for training PPO consist of a dense MLP, a scalar Neural Additive Model (NAM), and an interaction-NAM. With respect to the NAM-based policies, it is possible to carry out an exact decomposition of each action into its named feature and interaction components. We examine the robustness of these attributions for different training seeds and for two inventory configurations, and we compare the learned policies with a tuned dual-index heuristic, the nominal and service-target model predictive control, and a downside-penalized PPO baseline on a separate robustness grid. In the experiments, the interaction-NAM policies give rise to good action decompositions and generate interaction-importance rankings which are fairly stable regardless of the seed used. Nevertheless, the dual-index heuristic is still competitive or performs better in many disruption scenarios, and neither the interaction-NAM nor the downside-penalized PPO is able to close the gap. Furthermore, a post-training carbon-accounting sensitivity analysis indicates that the reported emissions are materially influenced by the accounting assumptions even when the policies remain unchanged. All of this suggests that accurate explanations do allow one to examine the trade-offs in this inventory setting, but they do not by themselves ensure robust control when disruptions occur.

Primary area:
reinforcement learning
Keywords:
interpretable reinforcement learning, neural additive models, inventory control, robustness, dual sourcing, carbon-aware supply chains

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cqwbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether transparent policies are safer to deploy. In a dual-source inventory control experiment, the authors show that exact attribution does not guarantee robustness, stable interaction rankings, or operational relevance.

Critical strengths
  1. The motivation in this paper is interesting.
  2. The structural-zero interaction analysis is convincing.
Critical weaknesses
  1. The paper only shows several observations that have already been discussed in previous research [1-3]
  2. Table 6 evaluates the training reward function on held-out disruptions. However, it does not distinguish insufficient optimization from poor out-of-distribution generalization.
  3. The test is not realistic. Specifically, the reference trajectory uses a constant action that differs substantially from the learned policies' behavior.

[1] Sundararajan, Mukund, and Amir Najmi. "The many Shapley values for model explanation." International conference on machine learning. PMLR, 2020.

[2] Beechey, Daniel, Thomas MS Smith, and Özgür Şimşek. "Explaining reinforcement learning with shapley values." International Conference on Machine Learning. PMLR, 2023.

[3] Lengerich, Benjamin, et al. "Purifying interaction effects with the functional anova: An efficient algorithm for recovering identifiable additive models." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.

Questions for authors
  1. What will happen if increasing the PPO training budget?
Reviewer_ebUQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether exactly faithful explanations from interpretable reinforcement learning models are also useful for understanding model performance and behavior. The authors use an inventory management problem and compare MLP, NAM, and interaction-NAM policies with a rule-based method and MPC. They study model performance under different disruptions, explanation stability across training seeds, the identifiability of interaction terms, and the effect of changing the state distribution used to calculate feature importance. The results show that exact reconstruction of a model’s action mean does not guarantee robustness, stable explanations, identifiable interactions, or a relationship between attributed importance and operational effect.

Critical strengths
  • The paper studies an important difference between faithfulness and usefulness. An explanation may exactly reconstruct a model output while still providing limited information about model behavior, robustness, or the effects of individual terms. This is a useful warning for evaluating interpretable models.

  • The paper examines several possible problems with the explanations, including variation across training seeds, structurally zero interactions, sensitivity to the evaluation states, and the relationship between attributed magnitude and operational effect. The authors also clearly acknowledge many limitations of their experiments and avoid claiming that interpretable models are generally worse than other models.

Critical weaknesses
  • The learned policies perform worse than the rule-based method, including on their own training objective. This suggests that the learned models may be under-trained or insufficiently tuned. Since the experiments use a fixed and relatively limited training budget, it is unclear whether the robustness result would remain when the learned models achieve stronger performance.

  • Some of the main findings depend on specific choices in the explanation pipeline. One interaction is always zero because its two features cannot be non-zero together, and the reported change in the most important interaction mainly appears when comparing a fixed reference trajectory with on-policy states. The four on-policy evaluations agree on the most important interaction. In addition, the explanation reconstructs the action mean but does not include the later clipping operation. These choices make it unclear how general the findings are beyond this particular inventory system and evaluation procedure.

Questions for authors
  • How often are the policy actions clipped, and how much does clipping change the final actions? Would the explanation results change if clipping were included in the explanation?

  • Would the robustness result remain if the learned models were trained for longer or tuned until their performance was closer to the rule-based method?

  • Can the authors show that similar stability or ranking problems occur for interactions whose features have overlapping support and across different realistic on-policy evaluation states?

Reviewer_YcZUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits exactly‑faithful attributions of Neural Additive Models in a dual‑sourcing inventory reinforcement‑learning setting. It finds that even when outputs can be perfectly decomposed, issues including model fragility, unidentifiable explanations, evaluation‑distribution‑dependent outputs, and misalignment between attribution magnitude and real‑world operational effect can emerge. The work demonstrates that exact faithfulness is only a necessary condition and insufficient to guarantee reliable and useful explanations.

Critical strengths

This work adopts an audit‑style empirical evaluation paradigm. Instead of merely focusing on model predictive performance, it conducts comprehensive quantitative examinations targeting the attribution explanations themselves.

Critical weaknesses

1.The work consists entirely of a simulation-based statistical audit, with no human-subject study to validate its core claims about practitioner behavior. While the paper argues that inflated Spearman correlation values and spurious importance scores will mislead practitioners, there is no empirical evidence that real engineers or operators would actually make these interpretive mistakes. All inferences about human judgment are extrapolated solely from simulation results, which undermines the real-world relevance of the risks identified in this work.

2.The comparison between Interaction-NAM and Scalar-NAM relies on two separately trained models initialized from scratch, rather than masking interaction terms within a single fully trained Interaction-NAM. This means observed performance gaps may be confounded by training stochasticity and divergent optimization trajectories, and cannot be cleanly attributed to the effect of interaction terms alone. While multi-seed runs and bootstrap sampling help mitigate variability, this experimental design still weakens the core evidence supporting the claimed decoupling between attribution magnitude and operational effect.

3.The p-values from the multiple permutation tests are not adjusted for multiple comparisons (e.g., via Holm-Bonferroni correction). Furthermore, several bootstrap confidence intervals cross zero, indicating high uncertainty around the corresponding effect sizes, yet some key conclusions appear to depend excessively on point estimates rather than interval-based statistical evidence.

Questions for authors

1.The paper’s claim that practitioners are prone to being misled by spurious attribution statistics is not backed by human-subject experiments. Could the authors provide additional evidence, either empirical or from prior literature, to justify this inference? If running additional human studies is infeasible, please consider reframing these claims as hypothetical potential risks rather than verified effects. It would also be valuable to include an analysis of how much of the reported risk can be mitigated by standard sanity-check practices commonly used in industry.

2.When validating the dissociation between attribution magnitude and operational effect, the analysis compares two independently trained models, which introduces training-related noise into the comparison. Why was the approach of masking interaction terms within a single trained model not adopted for this analysis? Adding this as an ablation experiment would substantially strengthen the paper’s core findings; if this is not possible, please elaborate on how this methodological limitation impacts the robustness of your central conclusions.

3.The permutation-test p-values lack multiple-comparison correction, and some conclusions draw heavily on point estimates even when their associated confidence intervals span zero. Please report corrected p-values and verify whether your original conclusions remain statistically valid after adjustment. Please also revise the manuscript to clearly distinguish findings that are robustly supported by confidence intervals from trends that are only observable in point estimates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.