acceptodds
Under review as a conference paper at ICLR 2027

When Should a Human Take Back Control? Optimal Delegation under Turbulent AI Risk

Abstract

Deploying AI systems requires deciding when to delegate tasks and when humans should intervene to monitor and mitigate risk induced by AI operations. These decisions become particularly challenging when failures cluster: a hallucination or harmful output can trigger further errors, creating periods of elevated risk. We introduce a continuous-time framework for learning adaptive human oversight under such turbulent AI risk. Existing oversight and delegation formulations condition on history but do not model incident clustering, or its suppression by supervision effort, jointly with the delegation decision and this study fixes this gap. The self-exciting dynamics capture how risk events increase the likelihood of subsequent events, making their timing and history central to decision-making. We formulate a stochastic control problem that combines human actions, monitoring effort, and switching between human–AI-assisted operation and full AI delegation, balancing operational rewards against oversight costs, and cascading AI-failures and induced uncertainty. Human participation is therefore an endogenous component of risk management: the policy determines both when oversight is needed and how much effort to allocate. We study a relaxed switching formulation and propose Hawkes-PPO, a policy-gradient method that uses a bank of exponential filters of observed incident times as a finite-dimensional summary of the history. In a synthetic environment it attains a higher risk-adjusted objective than either fixed regime and approaches an approximate full-information oracle. We illustrate our results with numerical simulations by examining how cascade risks influence intervention and delegation, connecting reinforcement learning with adaptive human oversight of AI systems. In particular, we illustrate the benefit of our switching strategy and Hawkes-PPO algorithm to monitor the project efficiently along time, reducing turbulent risks occurrences and costs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, AI Hallucination, AI risk modeling, Reinforcement Learning, Hawkes-PPO, Stochastic Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qnSXOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper provides a method to switch between autonomous AI decision-making and human-AI decision-making. Each has its benefits and drawbacks: AI minimizes operating costs but can cascade risks, while humans are needed to mitigate those risks but are more costly. The paper proposes a Hawkes-PPO policy gradient method.

Critical strengths
  • The paper models decision-making as an RL problem and solves it with Hawkes-PPO. The resulting delegation is shown to be better than pure human or AI controllers.
  • The paper is theoretically sound and is backed by experiments. The experiments test multiple different configurations.
Critical weaknesses
  • The paper is very difficult to read and follow. The terms used are not common for an ML reader, and the key insights from the theoretical results are not mentioned. The paper also lacks good illustrative figures that explain the method. I am afraid the key results are buried inside the many mathematical details, whereas the latter should serve the former.
  • The Hawkes-PPO method is not compared to the regular PPO to see if there are any considerable advantages of using it.
  • I think the paper would need a new writeup for clarity.
Questions for authors
  • AI agents are becoming more powerful by the day. Does your formulation account for the level of AI intelligence? Does your formualtion for any level of complexity and size of the model?
  • How would model-based methods fare in this setting? What about other policy learning methods, like SAC and DDPG?
3 comments

We thank the referee for the very useful comment, especially regarding the writing. We have uploaded a new version taking into account all these comments and highlighted the main changes in blue (including modifications suggested by other reviewers)

A clearer presentation: We have substantially reorganized the manuscript to introduce the supervisory problem and its operational interpretation before the mathematical details. The abstract and the list of contributions have been updated to reflect the revised presentation and the new empirical analyses.

Section 3.1 now presents a concrete coding-agent example motivating the decisions of when to intervene, how much checking to perform, and when to restore autonomy. Sections 3.2–3.5 explain incident memory, the roles of the model components, and the objective in intuitive terms alongside the equations. In particular, we distinguish background uncertainty, incident-related uncertainty, direct project losses, and persistent errors in AI recommendations.

We have also added Figure 1 to illustrate the supervision loop and Algorithm 1 to present Hawkes-PPO in the main text. Section 4.1 explains how incident-memory filters and PPO work together, and summarizes the theoretical results: the approximation controls the error introduced by replacing incident history with finitely many filters, while a separate result bounds the loss from restricted observations under additional assumptions. These guarantees are distinguished from the empirical performance of the trained policy. Finally, the numerical section is organized around explicit questions: whether adaptive delegation improves the objective, whether incident-history filters help, and how risk and costs affect supervision. The principal PPO comparison now appears in main-text Table 2.

Comparison Hawkes-PPO and PPO: We thank the referee for pointing it out; this comparison was part of the original submission in the Appendix only (Appendix B.6, Table 5 of that version, referred to in its Section 4.3). As it is something important, we now make it easier to find and add it in the main-text in the revised version (Table 2) and discuss it in Section 4.4.

Hawkes-PPO achieves better performance. Each selected policy is evaluated on 4,096 fresh paths per initial regime using common random streams. The corresponding 90% Monte Carlo intervals do not overlap in either comparison. These results support the benefit of incident-memory filters for PPO in the tested environment and training configuration. The intervals quantify evaluation uncertainty for the selected policies, rather than variability across independent training seeds. Appendix B.6, Table 6, also reports SAC, DDPG, and CT-DDPG with and without the same filters; the filters do not improve every algorithm under the common training budget.

The formulation does not require a particular AI architecture or model size. The AI enters as a frozen black-box policy whose relevant characteristics include its baseline incident rates, the strength and persistence of incident propagation, its operating cost, and the quality of its nominal recommendations. Capability is therefore represented indirectly through these task- and model-dependent characteristics, rather than through an explicit intelligence or parameter-count variable.

The formulation can accommodate different capability profiles through these quantities. However, architecture independence does not imply that one parameterization or trained supervisory policy applies unchanged to every AI system. Applying the framework to a different model or task would require assessing its incident dynamics and operating characteristics. Section 4.5 and Appendix B.7, Tables 7–8, illustrate how supervisory decisions vary with incident propagation. From an AI start, the separately trained policies spend 0.9%, 46.4%, and 73.6% of the horizon under Human control at excitation strengths of 0.5, 1, and 1.5 times the baseline, respectively. These experiments demonstrate sensitivity to propagation strength, rather than a universal relationship between model size and supervision needs. Within each configuration, the policy uses observed incident history to inform its decisions.

Model-based methods, SAC and DDPG: SAC, DDPG and CT-DDPG, each with and without the filters, are reported in Appendix B.6, Table 6, and were also part of the original submission. Regarding model-based methods, the paper includes an approximate full-information benchmark that knows the model coefficients and solves the control problem numerically (HJB/DGM benchmark in Table 2, Appendix B.4). Hawkes-PPO’s evaluated objective is within 0.35 (AI start) and 0.45 (Human start) of this benchmark. We do not compare with model-based reinforcement learning methods that learn a dynamics model from data. Such a method would have to learn the history-dependent incident dynamics, and we regard this comparison as future work.

The reviewers have addressed my concerns, and I shall raise my score. However, I need to reiterate my limited knowledge in this domain.

Thank you for taking the time to read our revision and update your assessment. Your comments on the presentation helped us make the paper clearer, and we appreciate your constructive feedback.

Reviewer_18bwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The present work is concerned with optimizing delegation between human-assisted and autonomous AI operation over long-horizon tasks. The authors formulate this as a relaxed switching problem and introduce Hawkes-PPO along with the collaboration history (i.e., states, rewards, and past incidents) to learn a switching policy. The proposed framework models cascading incidents and features an objective designed to tradeoff project benefits against operating and switching costs, and risk. Numerical simulations in a synthetic environment are used to demonstrate that the learned switching policy achieves performance approaching those of an approximate full-information oracle, outperforming the AI-only baseline.

Critical strengths

1. The problem formulation is compelling. The authors do a good job motivating the problem of human delegation in mixed teams. In particular, I appreciate that their modeling approach captures a rich environment, considering incident propagation, supervision effort that affects future risk, and switching between operating regimes with different costs.

Critical weaknesses

1. Certain assumptions/modeling choices would benefit from additional justification. For example, considering only incident occurrence times rather than modeling detection lag, and the decision to use a relaxed switching controller could benefit from further justification.

Questions for authors
  1. Have you considered models which incorporate some detection lag (e.g., where the policy may only react to the incident after some time)? It would seem like the lag between incident and detection would be important to model as it could have a significant impact on the project state and success of the controller.

  2. Why is stochastic switching the appropriate formulation for human oversight, especially if a detected incident (like the Replit one) could require immediate intervention?

  3. A key challenge in human-AI teaming is modeling the behavior of the human agents, yet in this setting we assume that monitoring effort cannot increase incident excitation. Have you considered modeling deterioration caused by human error? Can you speculate on how that might affect the design of your controller?

1 comment

We thank the referee for these questions. We have uploaded a new version taking into account all these comments and highlighted the main changes in blue (including modifications suggested by other reviewers)

About the lag: Our current model assumes that incidents are observed when they occur. We make this assumption to isolate how incident history affects delegation and supervision decisions. Detection delays would change the information available to the controller. For example, with a fixed delay (d), incident observations at time (t) would cover occurrences only up to (t-d), alongside any other currently observable project information. A controller could then use this delayed history to estimate unobserved incident activity and choose its supervision and takeover decisions accordingly. Operationally, detection lag can allow further incidents to accumulate before reactive takeover, potentially increasing the value of sustained supervision. The resulting balance depends on the delay relative to the persistence of incident effects and on supervision costs. Hawkes-PPO could be trained with delayed observations by modifying the simulator’s observation mechanism and constructing its memory inputs from information available at each decision time. Extending the theoretical analysis would also require an explicit delayed-information formulation. The present experiments and guarantees concern timely observation.

We have added a paragraph at the end of Section 3.2 explaining the timely-observation assumption, discussing how detection delays could affect supervision and takeover decisions, and outlining the changes to the observation model needed to incorporate them. We have left this for future direction in the conclusion.

Stochastic switching (relaxed formulation): We thank the reviewer for asking us to clarify this modeling choice. We use rate-based switching to represent the urgency of handing control between autonomous AI operation and human-directed operation. The supervisor chooses a switching intensity: a higher intensity increases the probability of a prompt handoff. This provides a continuous control that can be learned jointly with productive action and checking effort by the reinforcement-learning policy. The formulation is intended for sustained supervisory decisions, where intervention balances evolving incident risk against supervision and handoff costs. It does not represent a guaranteed instantaneous emergency stop. We agree that a detected critical incident, such as an unauthorized destructive action, may require immediate intervention. Such an emergency override would complement the supervisory policy but is outside the switching mechanism evaluated here.

Section 3.4 now explains that the controller chooses the switching rate rather than the exact switching time, and that the realized handoff remains stochastic even at a high rate. This clarifies both the interpretation of the relaxed formulation and its distinction from immediate intervention.

Modeling deterioration caused by human error and speculate on how that might affect the design of your controller: We thank the reviewer for raising this important extension. Human-directed operation in our model remains AI-assisted, and incidents can occur in either regime. In the numerical specification, checking suppresses the history-dependent contribution to incident rates but does not eliminate baseline incidents. However, we do not separately model human-induced errors or fatigue: checking is assumed to reduce incident risk, while its operating cost captures the resources required for supervision. Human-induced deterioration could be represented by an additional incident channel, allowing human errors to propagate to subsequent state or action incidents. Alternatively, incident rates could depend nonmonotonically on checking effort, reflecting benefits at moderate effort and deterioration under excessive workload. Modeling fatigue explicitly would require an additional state tracking workload or time under human control. These extensions would introduce a trade-off between the protective effect of supervision and the risk generated by the supervisor.

This extension remains out of the scope of our current modeling since it emphasizes economic behavior rather than operational cost. This is an interesting question that can be developed in a follow-up study.

Reviewer_M2kcOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors consider a system management problem, wherein management of the underlying system can be performed by a human expert or delegated to an AI agent. Whilst requiring less effort, the agent may make harmful mistakes that compound over time. This creates a delegation problem, in which management of the system must be delegated efficiently. The authors formalise this as a stochastic control problem in which the goal is to learn a policy for delegating control over time. The state of the system is represented by a real-valued stochastic process XX. Higher values of XX are considered more desirable. The policy manages switches from human to AI management as well as the actions and effort of the human when they are responsible for management.

Unsafe AI incidents are modelled by a two-dimensional Hawkes process. Self-excitation of the Hawkes process captures the compounding nature of errors made by the AI system. The first dimension of the process correspond to incidents pertaining to incorrect modelling of the state or environment whilst the second dimension corresponds to incidents where the agent reports an inconsistent or infeasible action. Building off the aforementioned Hawkes process, the dynamics of the system XX are defined as a sum of four processes. The first is a drift term that captures how the project evolves under the selected policy, which does not directly depend on the Hawkes process. The second is a Brownian motion term that captures any uncertainty in XX that is unrelated to AI failures. Like the first term, the second term does not depend on the Hawkes process. In contrast, the third term is a Brownian motion that directly depends on the Hawkes process and captures uncertainty in system state due to compounding AI errors. The last term captures a direct losses suffered immediately after a state incident occurs, and thus also depends on the Hawkes process directly.

Given the stochastic process for XX, the authors define a learning objective comprised of four terms: (i) a terminal reward for the final state of the process, (ii) the running reward accumulated through time minus an operating cost, (iii) the quadratic variation of X, and (iv) the accumulated costs incurred by switching between human and AI supervision. The authors then propose the use of Hawkes PPO to compute an optimal control policy. The policy dictates the human's actions when they are responsible for supervision, and when control is exchanged between the human and AI agent. The authors assume that the AI system optimises a stochastic process to selects its own policy when it is delegated authority. The paper concludes with some synthetic experiments.

Critical strengths

The paper makes a genuine attempt formally model the Human-AI delegation problem using stochastic processes. The Hawkes process seems like a reasonable choice to model compounding errors of an AI system.

Critical weaknesses

This paper completely lacks an empirical grounding. The authors have adopted a very complicated system of stochastic processes to model Human-AI delegation, whilst providing extremely little evidence that this captures the real dynamics observed in practical settings. Many of the design choices seem to be made out of statistical convenience, and there are no empirical experiments on actual AI systems to validate any of the modelling assumptions made.

The written presentation of the paper is extremely poor. In fact I was unable to understand the paper's contributions without using a language model to summarise and simplify the paper for me. The language used is firmly rooted in stochastic processes, and the purpose of different terms and modelling choices are never explained intuitively or contextualised for someone with an AI safety background. The paper repeatedly uses terms like excitation and turbulence without really explaining what these intuitively mean in the context of AI safety. Notation is also poorly explained. For example, even minor things like stating that ϕij\phi_{ij} is an element of the matrix Φ\Phi would help readability. More significantly, terms like those introduced on line 321 are never intuitively explained. What is bb intended to model and capture, in contrast to σ\sigma for example? The central algorithm, Hawkes-PPO, is not even described in the main body but deferred to the appendix. This is reflective of a greater trend, where highly niche tools from very recent papers with few citations are introduced without definition or explanation with the expectation that the reader is already familiar with them. Example 3.3 is intended to be informative but introduces a bunch of new terms and notation in about three lines. There is no explanation of the role of each term, or any discussion about the example that helps elucidate the modelling choices of the authors.

Questions for authors

My main technical concern is summarised by the following question: **Why are the modelling choices made throughout the paper appropriate, and how have they been justified and validated against real-world empirical settings of Human-AI delegation?" Until this question is answered, there is no reason to believe that the process model presented is a reasonable model for Human-AI delegation.

Furthermore I have serious concerns regarding the written presentation of the paper. To be honest, I am sceptical that these concerns can be addressed during the rebuttal period, and I would encourage authors to work on a revised version for another venue that incorporates empirical experiments and provides a clearer explanation/justification of their model.

EDIT: I have raised my score to weak reject in light of the rebuttal.

3 comments

Part 1 of 2: Empirical grounding.

We thank the reviewer for highlighting the importance of empirical grounding. We have strengthened the revised paper with two complementary sources of evidence supporting the use of incident-history models for AI risk: experimental studies documenting error propagation in AI systems, and our own empirical comparisons of Hawkes-type models against matched baselines without incident-history dependence. We have uploaded a new version taking into account all these comments and highlighted the main changes in blue (including modifications suggested by other reviewers)

First, existing studies already document cascading failures in AI systems, for example: [1] experimentally demonstrate a malicious prompt propagating between AI-powered email assistants: an affected application compromises additional applications, producing a cascade of prompt injections and malicious actions. This is a particularly strong reference for incident contagion: one compromise generates further compromises (what we call excitation). It is an experimental demonstration, rather than evidence from a naturally occurring outbreak. [2] Across six multi-agent frameworks, the authors study how small errors propagate through message dependencies and become system-wide false consensus. They demonstrate that injecting a single error can trigger widespread failure, and identify cascade amplification, sensitivity to network structure, and persistence of false consensus. We have updated Section 2 “Failure propagation in language-model agents” accordingly.

Second, we have added empirical analyses of two archived datasets of actual AI-agent executions, reported in Section 3.3, Table 1, and Appendix C. These analyses directly test whether incident history improves prediction beyond contextual differences already captured by a baseline model.

For AgentLogs, we analyze 163 coding-agent sessions from five repositories, distinguishing action errors from incorrect representations of the project state. We fit continuous-time incident models and evaluate them on 78 held-out sessions containing 52 incidents. The comparator is a context-dependent Poisson model with the same baseline covariates, including repository, incident type, model indicators, and session age. Adding Hawkes-type incident-history filters improves held-out event-timing and timing-and-type log likelihood. The joint timing-and-type improvement is 15.252 nats, with interval [6.088, 25.210]. Thus, the gain concerns prediction of when incidents occur, as well as their type.

For ODCV-Bench, we analyze 950 agent runs comprising 9,017 scored decisions and 270 execution/protocol failures. Because per-step timestamps are unavailable, we evaluate failure occurrence in decision time using a Bernoulli model. Across five grouped held-out folds, adding incident-history filters improves log likelihood by 4.227 nats per 1,000 decisions, with a conditional paired-bootstrap 90% interval of [1.894, 7.044], relative to a baseline with the same model, scenario, prompt, and decision-context covariates. In both analyses, fitted parameters remain fixed during evaluation, and only preceding incidents update the history filters. These exploratory results provide consistent evidence across two datasets and observation schemes that incident history contains predictive information beyond the specified contextual baselines. Together with the experimental propagation studies above, they provide empirical motivation for the central modeling choice: retaining incident memory when assessing risk and deciding when supervision is warranted. Appendix C documents event construction, estimation, evaluation protocols, and the scope of the conditional uncertainty intervals

[1] Cohen, Stav, Ron Bitton, and Ben Nassi. "Here comes the AI worm: Preventing the propagation of adversarial self-replicating prompts within GenAI ecosystems." Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 2025. [2] Xie, Yizhe, et al. "From spark to fire: Modeling and mitigating error cascades in LLM-based multi-agent collaboration." arXiv preprint arXiv:2603.04474 (2026).

Part 2 of 2: Written presentation.

Following the suggestion of the referee we have reduced the mathematical section to the essential to emphasize the modeling aspect of our contribution.

We have clarified the terms “excitation” and “turbulent AI risk” in the Introduction and Section 3.2. Excitation describes how a past incident increases the likelihood of subsequent incidents, either of the same type or of another type. For example, an incorrect assessment of the environment may lead to an inappropriate action, whose consequences then generate further errors. We use “turbulent AI risk” to describe this incident-driven dependence and the resulting periods of elevated risk. The revised presentation also distinguishes incident propagation from its consequences for project quality: beyond direct losses, our model assumes that higher incident intensity increases volatility, representing additional uncertainty in the project’s evolution.

We thank the reviewer for highlighting the notations to be clarified. We have revised the notation and added intuitive explanations alongside the model equations. In Section 3.2, immediately after Equation (1), we now explicitly identify ϕij\phi_{ij} as the (i,j)(i,j) entry of the kernel matrix Φ\Phi, where (j) denotes the type of a past incident and (i) the type of a subsequent incident. We also give a concrete interpretation: ϕSA\phi_{SA} describes how an action error can increase the subsequent rate of state errors.

Section 3.4 now explains the distinct roles of the terms in the project dynamics. The drift bb describes how the productive action changes expected progress. The coefficient σ\sigma represents ordinary execution uncertainty, whereas γ\gamma represents additional uncertainty associated with elevated incident intensity. These continuous fluctuations are distinguished from the discrete project loss caused by a state incident.

In Section 4.2, we clarify that XtX_t measures project quality, whereas RtR_t measures error in the AI recommendation. Action incidents perturb RtR_t, making subsequent recommendations less reliable; these recommendations then affect project quality through the implemented action. We explain the roles of recovery, incident-driven shocks, and intensity-dependent uncertainty in the evolution of RtR_t.

Finally, Section 3.5 explains each component of the objective, including the distinction between the running cost of maintaining a high switching rate and the cost paid when a handoff actually occurs. These additions connect the mathematical notation to its operational meaning in human–AI supervision.

We agree that the central algorithm should be understandable from the main text. We have therefore added Section 4.1, which introduces Hawkes-PPO step by step, together with Algorithm 1 describing its training and deployment procedure and Figure 1 illustrating the supervision loop.

The revised explanation distinguishes the two components: exponential memory filters summarize the timing and types of previous incidents, while PPO learns productive action, checking effort, and switching rates from the resulting observations and rewards. We explain that each incident increments its corresponding filters, which subsequently decay at different rates, providing a compact representation of recent and longer-term incident history. We also specify which quantities the policy observes and emphasize that it does not receive the true incident intensities or excitation coefficients. Detailed losses and implementation settings remain in Appendix B.3, but the method’s inputs, decisions, and learning procedure are now presented in the main body.

We have reorganized and expanded the explanation of the former Example 3.3. The main text now introduces a concrete coding-agent example in Section 3.1, explains incident rates and self- and cross-excitation in Section 3.2, and introduces exponential memory filters in Section 4.1 before presenting the algorithm. The exponential-kernel example now appears in Appendix D as Example D.1. We explain how each incident updates its memory variable, how that memory decays, and how the coefficients translate past incidents into subsequent incident rates. We also illustrate the mechanism with an action error that increases the risk of both further action errors and incorrect assessments of the project state. This separates the operational intuition from the mathematical construction and explains the role of each term.

We thank the reviewer for prompting these improvements. The revised manuscript now includes empirical analyses of AI-agent traces, clearer explanations of the modeling choices and notation, and a main-text presentation of Hawkes-PPO with an illustrative figure and pseudocode. The abstract and the list of contributions have been updated to reflect these additions. These changes are incorporated in the updated PDF, and we hope they address the reviewer’s concerns and make the contribution easier to assess.

Dear Reviewer M2kc,

Thank you again for your detailed feedback. We wanted to follow up on whether our revision addresses your main concerns, while there is still time for us to clarify any remaining questions.

Empirical grounding: Section 3.3, Table 1, and Appendix C now present held-out comparisons using two datasets of real agent runs, AgentLogs and ODCV-Bench. These experiments examine whether incident history helps predict subsequent incidents and provide empirical support for the incident-memory component of our model. Section 3.4 also clarifies how supervision is represented in the model.

Presentation and accessibility: Section 3 now introduces the model through a coding-agent example and explains its terms in plain language. We have also moved the description of Hawkes-PPO into the main text, including Section 4.1, Figure 1, and Algorithm 1.

We would be grateful for any further feedback and would be happy to clarify anything that remains unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.