Optimizing Human-in-the-Loop Systems under Dynamic Trust: An Adherence-Aware Reinforcement Learning Framework

Many real-world decision systems adopt a human-in-the-loop architecture, where AI provides recommendations while human decision makers retain final authority. As a result, the implemented action may differ from the recommendation. Existing adherence-aware methods account for this mismatch, but typically model human adherence as a fixed probability. In practice, however, human reliance can evolve with previous interaction outcomes, making adherence endogenous to the sequential decision process. We propose an adherence-aware reinforcement learning framework that models dynamic adherence via a Belief Markov Decision Process (MDP). By augmenting the state space with a belief state over human trust, our approach simultaneously tracks dynamic adherence probabilities and adaptively calibrates recommendations to maximize expected long-term rewards despite human deviations. Based on the resulting Belief MDP, we derive an adherence-aware Bellman operator that marginalizes over human acceptance and fallback behavior, together with a scalable reinforcement learning implementation. Theoretically, we characterize the suboptimality induced by belief and human-model estimation errors. Evaluations in sepsis treatment, highway lane changing and supply chain inventory management tasks demonstrate that proactively modeling dynamic trust and making adherence-aware recommendations enhance overall system performance while simultaneously providing interpretable insights into how recommendations should adapt to varying levels of human trust.