- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies robust policy learning for the reject inference problem under the setting of distribution shift. The authors use a distributionally robust optimization framework with a Kullback–Leibler ambiguity set to learn policies that are robust to such shifts. Theoretically, the author establish a robust Bellman equation to characterize the optimal policy, contraction and truncation error bounds for the backward approximation of the value function, and estimation error of the effective discount factor. Simulation and read data application verify the performance of the proposed framework over baselines.
Critical strengths
The paper is motivated by distribution shift in the reject inference problem, which is practically relevant, particularly in sequential decision-making. In my opinion, using DRO to address this distributional uncertainty is a natural and appropriate approach. The authors frame the problem well, deriving a robust Bellman equation and develop a corresponding backward approximation algorithm within this framework, which represents a substantial technical effort.
Critical weaknesses
My major concern is about the convergence of the learn policy. Although the authors establish some theoretical guarantees, the key result of convergence to the true optimal policy is not explicitly proved. In classical DRO, one should define the true underlying distribution and show that the distribution at the center of the ambiguity set converges to it. However, the authors do not establish this convergence for their posterior-based center, as well as the convergence of the policy learnt.
Moreover, the radius should be assumed to shrink to zero at an appropriate rate as the center converges to the true distribution. In experiment, with a fixed radius, the learnt policy could still sit in the neighbour from the true policy, even as the center converges to the true distribution.
Questions for authors
Could the authors clarify what the true underlying distribution is and whether the chosen center, posterio mean, of the ambiguity set converges to it? as well as the covergen guarantee of the learnt policy.
Another common choice for constructing the ambiguity set is the Wasserstein distance. Could the authors clarify the motivation for choosing KL? Perhaps a Wasserstein-based formulation could be more computationally friendly?
I would also be interested in seeing the computational cost compared with the non-DRO case, as the additional optimization required by DRO could substantially increase the computational burden.
I am also concerned about the an overly conservative decision policy. Could the radius adapt over time based on a measure of uncertainty or variability?
Response to Reviewer ui1G
We thank the reviewer for the questions. We have uploaded the revised manuscript, with all revisions highlighted in blue, to address your concerns and would greatly appreciate your reconsideration.
Q1 and W1. We have added Proposition 4 in Section 3.4 (pp. 6–7), with a complete proof in Appendix B.6 (pp. 23–29) to prove convergence of the learned policy. The main result is as follows. If the underlying environment is stationary and each state receives sufficiently many accepted observations, the posterior estimate of the success probability converges to its true value. When the ambiguity radius and estimated effective discount factor also converge, the corresponding ambiguity sets and robust value functions converge. Furthermore, provided that the optimal decision is not exactly at the acceptance threshold, we prove that the learned policy eventually makes the same decisions as the corresponding oracle policy.
We distinguish between two convergence objectives. With a fixed positive ambiguity radius, the learned policy converges to the robust oracle, which may differ from the nominal optimal policy because it continues to protect against distributional uncertainty. When the radius shrinks to zero, the learned policy instead converges to the nominal oracle based on the true success probabilities.
We also provide an explicit error bound and a possible shrinking-radius schedule in Corollary 9, together with numerical approximation conditions in Corollary 10. Finally, we emphasize that sufficient feedback is essential: if a state is persistently rejected, its posterior may not converge. Our consistency guarantee therefore holds under sufficient feedback rather than unconditionally for the original policy.
Q2. We use KL divergence because it provides a natural way to describe uncertainty in Bernoulli success probabilities, allowing the range of possible deviations to depend on the current estimated probability. Nevertheless, we agree that Wasserstein distance is a reasonable alternative and can be computationally simpler. To address this question, we have added Appendix B.7 (p. 29), which explains how Wasserstein ambiguity can be incorporated into our robust dynamic programming framework. In particular, the corresponding worst-case probabilities can be computed directly without the numerical root-finding required by KL divergence. We have also added an experimental comparison in Appendix C.7 (Table 14). The results show that Wasserstein achieves slightly higher nominal reward (96.55 versus 95.03), whereas KL performs slightly better under a severe distribution shift (44.26 versus 43.87). These findings suggest that our proposed framework is not restricted to KL divergence, while different ambiguity measures may lead to different trade-offs between nominal performance and robustness.
Q3. We have added a direct comparison between DRRI and the nominal dynamic programming method SL-OPT in Appendix C.11 (Table 18, pp. 46–47). The results show that DRRI requires only slightly more computation than SL-OPT. Specifically, DRRI takes 0.896 seconds per replication, compared with 0.864 seconds for SL-OPT, representing a 3.7% increase in runtime. Their memory usage is also similar (197.2 versus 194.8 MiB). This relatively small difference is partly because our implementation precomputes the worst-case probability bounds, avoiding repeated optimization during policy evaluation. We also investigate how runtime and memory usage change with the number of states, the belief-grid size, and the truncation depth in Table 17. These results show that computational costs increase with problem size.
Q4. We agree that a fixed ambiguity radius may lead to overly conservative decisions, particularly as more observations become available and uncertainty about the success probabilities decreases. To address this concern, we have added an uncertainty-adaptive ambiguity radius in Appendix B.8, which gradually decreases as more accepted observations are collected while maintaining a minimum level of protection against potential distribution shifts. The adaptive formulation can be incorporated into our existing robust dynamic programming framework without changing its fundamental structure.
We have also added experiments comparing fixed and adaptive ambiguity radii in Appendix C.8 (Table 15). The results show that adaptive radii can improve nominal performance while retaining much of the robustness of the fixed-radius approach. For example, an adaptive radius with a minimum value of 0.005 increases nominal reward from 95.03 to 98.15, while achieving comparable performance under severe shifts (42.81 versus 44.26). However, allowing the radius to decay toward zero improves nominal reward further (103.88) but substantially reduces performance under severe shifts (37.92). These results demonstrate that adaptive radii offer a flexible way to balance nominal performance and robustness, rather than uniformly improving both.