- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper provides a method to switch between autonomous AI decision-making and human-AI decision-making. Each has its benefits and drawbacks: AI minimizes operating costs but can cascade risks, while humans are needed to mitigate those risks but are more costly. The paper proposes a Hawkes-PPO policy gradient method.
Critical strengths
- The paper models decision-making as an RL problem and solves it with Hawkes-PPO. The resulting delegation is shown to be better than pure human or AI controllers.
- The paper is theoretically sound and is backed by experiments. The experiments test multiple different configurations.
Critical weaknesses
- The paper is very difficult to read and follow. The terms used are not common for an ML reader, and the key insights from the theoretical results are not mentioned. The paper also lacks good illustrative figures that explain the method. I am afraid the key results are buried inside the many mathematical details, whereas the latter should serve the former.
- The Hawkes-PPO method is not compared to the regular PPO to see if there are any considerable advantages of using it.
- I think the paper would need a new writeup for clarity.
Questions for authors
- AI agents are becoming more powerful by the day. Does your formulation account for the level of AI intelligence? Does your formualtion for any level of complexity and size of the model?
- How would model-based methods fare in this setting? What about other policy learning methods, like SAC and DDPG?
3 comments
The reviewers have addressed my concerns, and I shall raise my score. However, I need to reiterate my limited knowledge in this domain.
Thank you for taking the time to read our revision and update your assessment. Your comments on the presentation helped us make the paper clearer, and we appreciate your constructive feedback.
We thank the referee for the very useful comment, especially regarding the writing. We have uploaded a new version taking into account all these comments and highlighted the main changes in blue (including modifications suggested by other reviewers)
A clearer presentation: We have substantially reorganized the manuscript to introduce the supervisory problem and its operational interpretation before the mathematical details. The abstract and the list of contributions have been updated to reflect the revised presentation and the new empirical analyses.
Section 3.1 now presents a concrete coding-agent example motivating the decisions of when to intervene, how much checking to perform, and when to restore autonomy. Sections 3.2–3.5 explain incident memory, the roles of the model components, and the objective in intuitive terms alongside the equations. In particular, we distinguish background uncertainty, incident-related uncertainty, direct project losses, and persistent errors in AI recommendations.
We have also added Figure 1 to illustrate the supervision loop and Algorithm 1 to present Hawkes-PPO in the main text. Section 4.1 explains how incident-memory filters and PPO work together, and summarizes the theoretical results: the approximation controls the error introduced by replacing incident history with finitely many filters, while a separate result bounds the loss from restricted observations under additional assumptions. These guarantees are distinguished from the empirical performance of the trained policy. Finally, the numerical section is organized around explicit questions: whether adaptive delegation improves the objective, whether incident-history filters help, and how risk and costs affect supervision. The principal PPO comparison now appears in main-text Table 2.
Comparison Hawkes-PPO and PPO: We thank the referee for pointing it out; this comparison was part of the original submission in the Appendix only (Appendix B.6, Table 5 of that version, referred to in its Section 4.3). As it is something important, we now make it easier to find and add it in the main-text in the revised version (Table 2) and discuss it in Section 4.4.
Hawkes-PPO achieves better performance. Each selected policy is evaluated on 4,096 fresh paths per initial regime using common random streams. The corresponding 90% Monte Carlo intervals do not overlap in either comparison. These results support the benefit of incident-memory filters for PPO in the tested environment and training configuration. The intervals quantify evaluation uncertainty for the selected policies, rather than variability across independent training seeds. Appendix B.6, Table 6, also reports SAC, DDPG, and CT-DDPG with and without the same filters; the filters do not improve every algorithm under the common training budget.
The formulation does not require a particular AI architecture or model size. The AI enters as a frozen black-box policy whose relevant characteristics include its baseline incident rates, the strength and persistence of incident propagation, its operating cost, and the quality of its nominal recommendations. Capability is therefore represented indirectly through these task- and model-dependent characteristics, rather than through an explicit intelligence or parameter-count variable.
The formulation can accommodate different capability profiles through these quantities. However, architecture independence does not imply that one parameterization or trained supervisory policy applies unchanged to every AI system. Applying the framework to a different model or task would require assessing its incident dynamics and operating characteristics. Section 4.5 and Appendix B.7, Tables 7–8, illustrate how supervisory decisions vary with incident propagation. From an AI start, the separately trained policies spend 0.9%, 46.4%, and 73.6% of the horizon under Human control at excitation strengths of 0.5, 1, and 1.5 times the baseline, respectively. These experiments demonstrate sensitivity to propagation strength, rather than a universal relationship between model size and supervision needs. Within each configuration, the policy uses observed incident history to inform its decisions.
Model-based methods, SAC and DDPG: SAC, DDPG and CT-DDPG, each with and without the filters, are reported in Appendix B.6, Table 6, and were also part of the original submission. Regarding model-based methods, the paper includes an approximate full-information benchmark that knows the model coefficients and solves the control problem numerically (HJB/DGM benchmark in Table 2, Appendix B.4). Hawkes-PPO’s evaluated objective is within 0.35 (AI start) and 0.45 (Human start) of this benchmark. We do not compare with model-based reinforcement learning methods that learn a dynamics model from data. Such a method would have to learn the history-dependent incident dynamics, and we regard this comparison as future work.