acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Human-in-the-Loop Systems under Dynamic Trust: An Adherence-Aware Reinforcement Learning Framework

Abstract

Many real-world decision systems adopt a human-in-the-loop architecture, where AI provides recommendations while human decision makers retain final authority. As a result, the implemented action may differ from the recommendation. Existing adherence-aware methods account for this mismatch, but typically model human adherence as a fixed probability. In practice, however, human reliance can evolve with previous interaction outcomes, making adherence endogenous to the sequential decision process. We propose an adherence-aware reinforcement learning framework that models dynamic adherence via a Belief Markov Decision Process (MDP). By augmenting the state space with a belief state over human trust, our approach simultaneously tracks dynamic adherence probabilities and adaptively calibrates recommendations to maximize expected long-term rewards despite human deviations. Based on the resulting Belief MDP, we derive an adherence-aware Bellman operator that marginalizes over human acceptance and fallback behavior, together with a scalable reinforcement learning implementation. Theoretically, we characterize the suboptimality induced by belief and human-model estimation errors. Evaluations in sepsis treatment, highway lane changing and supply chain inventory management tasks demonstrate that proactively modeling dynamic trust and making adherence-aware recommendations enhance overall system performance while simultaneously providing interpretable insights into how recommendations should adapt to varying levels of human trust.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, adherence-aware decisioin making, human-in-the-loop, Belief MDP

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y5bXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies sequential decision support in which an AI recommends an action and a human either accepts it or executes a known fallback action. Prior adherence-aware MDPs assume a fixed acceptance probability. This paper instead lets acceptance depend on a latent trust state that evolves over time, and it tracks trust with a Bayesian filter (jointly with an unknown transition parameter β\beta).

The method learns a value function over the executed action. It scores each recommendation by mixing the accept and reject branches analytically, weighted by the predicted acceptance probability. This is implemented with DQN for lane changing and with a GNN actor–critic for inventory management.

The theory covers contraction of the operator, error bounds for belief and parameter estimation, and bounds under misspecification. Experiments use three simulated humans (sepsis, highway lane changing, inventory management). The method obtains the highest average reward in all three, and its recommendations change with the estimated trust level.

Critical strengths

S1. A relevant problem with a clear formulation. Treating adherence as endogenous is well motivated. Separating recommendation, human response and executed action exposes a real trade-off: a recommendation affects both the current outcome and future acceptance. The trust-conditioned policies (Tables 1b, 7) make this trade-off easy to see.

S2. A simple, implementable design. Learning QQ over the executed action and marginalizing over the accept/reject branches analytically fits into standard deep RL. The inventory ablation (Table 4: 4298 vs. 4117) suggests that this design helps.

Critical weaknesses

W1. The human model, which is the core of the problem, is assumed known, is never specified in the paper, and the stated formulation does not match the simulators. The trust-conditioned behaviors reported as findings are built into the simulators.

a. The human models are hand-written, and the agent receives exact copies. App. B assumes the acceptance model P(o∣x,θ,ur)P(o\mid x,\theta,u^r) and the fallback policy gbg^b are known, but the paper never states them for any task. In the supplementary code, the agent’s filter uses exactly the parameters of the simulated human. In the lane-change code, belief.py says “For now we intentionally let them match”. In the lane-change task, the simulator also passes the human’s internal good/bad grade of each recommendation (rec_good) directly to the filter. In sepsis and lane changing, acceptance is P(o=1∣x,θ,ur)=θP(o=1\mid x,\theta,u^r)=\theta, which does not depend on the state or on the recommendation.

b. Eq. (1) does not describe the simulators. Eq. (1) states bt+1−(θ′)=∑θPβ(θ′∣θ,xt,ut) bt(θ)b^-_{t+1}(\theta')=\sum_{\theta}P_\beta(\theta'\mid\theta,x_t,u_t)\,b_t(\theta). In the code, trust transitions depend on other quantities: sepsis: on acceptance and on whether the patient improved, i.e., on xt+1x_{t+1}; lane changing: on the recommended action, even when it is rejected, averaged over a 3-step window; inventory: on acceptance and on recent rewards.

c. As a result, Q(x,b,u)Q(x,b,u) is not a function of (x,b,u)(x,b,u) alone in these simulators. In sepsis, “conservative recommended and accepted” and “aggressive recommended, rejected, conservative fallback executed” have the same executed action. Yet after an improvement, trust rises with probability 0.8 in the first case and stays unchanged in the second. In lane changing, the reject-branch value Q(x,brej,gb(x))Q(x,b^{rej},g^b(x)) cannot depend on which recommendation was rejected, even though that recommendation changes trust.

W2. The evidence that modeling dynamic adherence improves performance is weak, and the baselines are not comparable.

Questions for authors
  1. Model consistency. Please restate the trust transition so that it matches the simulators, i.e., conditioning on oto_t, utru^r_t, the outcome or reward, and the history window. How does the executed-action decomposition change? Is the lane-change target in train_belief.py, which uses the same belief in both branches rather than baccb^{acc}/brejb^{rej}, intended to implement Eq. (7)?

  2. Misspecified acceptance model. Table 9 perturbs only the trust transition. What happens when the acceptance model or the fallback policy is wrong, for example when the true acceptance depends on context but the filter assumes P=θP=\theta? What happens when the human model is learned from data?

3.Fair baselines. Were all methods in Table 2 run against the same β\beta grid? Can you add a static-adherence learner that shares your architecture, target and human-model knowledge?

  1. Statistics. Please report: what ± means in each table; the collision rate for lane changing; the full Table 9, including the nominal run and the unobserved-signal run.

  2. Theory. Prop. 1: given (x,b,u)(x,b,u), bt+1−b^-_{t+1} is deterministic. So the TV distance between the two information-state kernels equals 1 whenever the two models predict different beliefs, and the bound 2γRmax⁡/(1−γ)22\gamma R_{\max}/(1-\gamma)^2 exceeds the trivial bound when γ>1/2\gamma>1/2 (the paper uses γ=0.99\gamma=0.99). Can the bound be restated using Lipschitz continuity of the value in the belief?

Corollary 1: please define EFQI\mathcal E_{\mathrm{FQI}} and state which conditions of Fan et al. (2020) are verified.

6.Novelty. What exactly is new relative to Chen et al. (2018, 2020), Bayes-adaptive POMDPs, and trust-aware bandits (Cai & Zhang, 2024)?

10 comments
AuthorsOpenReview ↗

Response to Reviewer Y5bX (Part 1)

We thank the reviewer for their detailed feedback. We address each point below.

1. Restating the Trust Transition [W1, Q1]

We thank the reviewer for pointing out that the description of the state information is incomplete. To make explicit the information available to the agent, we denote the full observable state by x~t=(xt,ht)\tilde{x}_t=(x_t,h_t), where hth_t collects additional observable interaction signals and relevant historical information required to characterize trust evolution. Accordingly, the trust transition in Eq. (1) is more precisely expressed as

Pβ(θt+1∣θt,x~t,ut)P_\beta(\theta_{t+1}\mid\theta_t,\tilde{x}_t,u_t).

The specific components of hth_t vary across environments:

  • Sepsis: hth_t includes the acceptance decision and the observed patient improvement signal.
  • Highway lane changing: hth_t includes the current and two preceding recommendation-quality signals.
  • Inventory management: hth_t includes the acceptance information and recent reward history used in the trust update.

All these signals are observable to the agent. By incorporating them into x~t\tilde{x}_t, the trust transition is Markovian with respect to the full observable state and executed action.

AuthorsOpenReview ↗

Response to Reviewer Y5bX (Part 2)

2. Robustness to Human-Model Misspecification [Q2]

We further evaluate robustness to fallback-policy and observation-model misspecification.

Fallback policy. The fallback policy can be approximated via supervised learning (e.g., an MLP classifier). Replacing the oracle with the learned policy causes only marginal degradation, and our method still outperforms both baselines.

MethodOracle fallback: Return / SpeedLearned fallback: Return / Speed
Proposed196.71 ± 0.75 / 29.45 ± 1.16196.30 ± 0.81 / 29.33 ± 1.13
Est-θ194.83 ± 0.78 / 28.28 ± 2.41194.35 ± 0.80 / 28.09 ± 2.40
No-belief194.71 ± 0.65 / 28.45 ± 1.71193.89 ± 0.77 / 28.12 ± 2.02

Observation model. We additionally evaluate robustness to misspecified observation models in both scenarios.

  1. Lane changing The agent assumes P(ot=1∣θt)=θtP(o_t=1\mid\theta_t)=\theta_t, while the true acceptance models incorporate three modifications: (i) quality-dependent additive acceptance, p=clip⁡(θ+0.4q~,0.01,0.99)p=\operatorname{clip}(\theta+0.4\tilde q,0.01,0.99); (ii) risk-sensitive acceptance, p=clip⁡(θ−0.8r,0.01,0.99)p=\operatorname{clip}(\theta-0.8r,0.01,0.99), where rr measures traffic risk; and (iii) a quality-trust interaction, p=clip⁡(θ+0.6(1−θ)q~,0.01,0.99)p=\operatorname{clip}(\theta+0.6(1-\theta)\tilde q,0.01,0.99). The results show that our method consistently achieves higher returns and speeds than both baselines across all three misspecification settings.
True human modelMethodReturn ↑Speed ↑
Quality additiveProposed196.71 ± 0.6429.21 ± 1.50
Est-θ194.07 ± 0.8527.70 ± 2.67
No-belief194.19 ± 0.8228.16 ± 2.11
Risk-sensitiveProposed196.41 ± 0.8029.40 ± 1.20
Est-θ194.58 ± 0.7728.23 ± 2.42
No-belief194.04 ± 0.7628.32 ± 1.80
Quality-trust interactionProposed196.41 ± 0.7929.25 ± 1.46
Est-θ193.80 ± 0.8727.58 ± 2.77
No-belief194.49 ± 0.8028.29 ± 2.01
  1. Supply chain inventory management We consider three misspecifications, each modifying a different aspect of the assumed observation model: (1) Constant tolerance removes the trust dependence of deviation sensitivity; (2) Wrong curvature uses an incorrect deviation exponent (ν=1.5\nu=1.5); and (3) Deviation sensitivity scales the tolerance width by 0.6, systematically underestimating human tolerance while preserving the original functional form.
Observation modelAssumed human behaviorReward ↑Served demand ↑
CorrectOriginal observation model4443.97 ± 752.62533.13 ± 91.59
Constant toleranceTrust-independent tolerance4064.08 ± 1019.39453.34 ± 121.62
Wrong curvatureIncorrect deviation exponent4108.87 ± 857.51498.32 ± 106.94
Deviation sensitivityUnderestimated tolerance4370.20 ± 687.70494.61 ± 78.52

The results suggest that the impact of observation-model misspecification depends on its type.

  • Wrong curvature: Despite using an incorrect functional form, the model preserves the key qualitative relationships between trust, recommendation deviation, and acceptance probability. Consequently, its performance degradation is relatively limited.
  • Deviation sensitivity: Although tolerance is underestimated, the model preserves its trust-dependent structure and induces a conservative policy that achieves performance close to the correctly specified model. This suggests that the induced conservative policy happens to reach a similarly favorable operating regime.
  • Constant tolerance: Removing the trust dependence of human tolerance fails to capture an important structural relationship in the true observation model, resulting in a more noticeable performance degradation compared with the other two types of misspecification.

These results suggest that preserving the key qualitative relationships in the human response model may be more important than accurately specifying its exact functional form.

Further discussions: Stable value learning

Figure Value-model effects examines the impact of observation-model misspecification on value learning. Panel A shows that misspecification distorts recommendation-level value estimates through errors in acceptance probabilities and posterior updates. In contrast, Panel B shows similar value landscapes across critics trained under different observation models, suggesting that executed-action value learning Q(x,b,u)Q(x,b,u) is relatively robust. This is because the critic evaluates the action actually executed based on environmental rewards and transitions, avoiding the ambiguity associated with directly learning the value of a recommendation that may lead to different executed actions.

Overall, our method remains effective under both fallback-policy and observation-model misspecification, consistently outperforming simpler adherence baselines.

AuthorsOpenReview ↗

Response to Reviewer Y5bX (Part 3)

3. Proposition 1 and Corollary 1 [Q5]

Proposition 1. We thank the reviewer for pointing out this issue. We replace the kernel TV discrepancy with the one-step predictive-belief discrepancy:

εb(M⋆,M):=sup⁡x,b,u∥TM⋆(b;x,u)−TM(b;x,u)∥1.\varepsilon_b(M^\star,M):=\sup_{x,b,u}\left\|\mathcal T_{M^\star}(b;x,u)-\mathcal T_M(b;x,u)\right\|_1.

We redefine M^\dagger as the candidate model minimizing this discrepancy. The revised bound follows from the Lipschitz continuity of the value function in the predictive belief, established in the proof of Theorem 1:

∣GMπ(x,b1−,ur)−GMπ(x,b2−,ur)∣≤Rmax⁡1−γ∥b1−−b2−∥1.|G_M^\pi(x,b_1^-,u^r)-G_M^\pi(x,b_2^-,u^r)|\leq\frac{R_{\max}}{1-\gamma}\|b_1^--b_2^-\|_1.

Specifically, this property bounds the one-step Bellman mismatch by γRmax⁡1−γεb†\frac{\gamma R_{\max}}{1-\gamma}\varepsilon_b^\dagger. Bellman contraction then gives:

∥GM⋆π−GM†π∥∞≤γRmax⁡(1−γ)2εb†.\|G_{M^\star}^\pi-G_{M^\dagger}^\pi\|\infty\leq\frac{\gamma R_{\max}}{(1-\gamma)^2}\varepsilon_b^\dagger.

Corollary 1. We define ϵFQI(n,K)\epsilon_{\mathrm{FQI}}(n,K) as the expected greedy suboptimality of the K-th FQI iterate, including statistical, approximation, and finite-iteration errors.

Regarding the conditions of Fan et al. (2020), Lemmas 1–4 establish the key structural properties required for FQI error propagation under our modified Bellman operator, including contraction, greedy consistency, Markov transition representation, and conditional unbiasedness of sampled Bellman targets. The remaining statistical regularity conditions are inherited from Fan et al. and imposed on the induced belief MDP. We will clarify these conditions and their correspondence in the revision.

4. Clarify novelty and contributions [Q6] We thank the reviewer for pointing out these related works. Our joint belief over latent trust and unknown transition parameters builds upon the Bayesian principle of Bayes-adaptive POMDPs, which provides the foundation for jointly tracking trust and identifying its dynamics.

  • Chen et al. (2018, 2020): These works learn trust and human behavior models and use them for POMDP-based planning in human-robot collaboration. In contrast, we focus specifically on adherence-aware decision-making in broader human-in-the-loop systems. We directly learn recommendation policies through reinforcement learning, rather than relying on belief-space planning. To this end, we develop a dedicated adherence-aware Bellman operator that explicitly marginalizes over human acceptance and fallback execution, providing a learning formulation consistent with scalable RL implementations.
  • Cai & Zhang (2024): This work studies dynamic trust in stochastic multi-armed bandits and develops a UCB algorithm with minimax regret guarantees. We extend the scope to general state-dependent MDPs, where complex environment dynamics and high-dimensional state representations call for scalable function-approximation-based RL. We further establish guarantees on belief and parameter estimation as well as model misspecification.

5. Other clarifications [W1, Q3, Q4]

(1) Human-model knowledge. We respectfully clarify that the agent is not provided with exact copies of the simulated human's latent state or model parameters. Rather, we assume that the functional structure of the human model is known, while the key trust-transition parameter, like β\beta is unknown and jointly inferred. The nominal experiments use a correctly specified model family to isolate the effect of model misspecification. We further investigate trust-transition model misspecification in Table 9, and conduct additional experiments to examine misspecification of the acceptance model and fallback policy.

(2) The rec_good signal. We clarify that rec_good is not an additional internal human state. It is a constructed feedback signal computed from observable information using a prespecified, known quality-scoring model. Thus, it does not provide the agent with privileged access to the human's internal state. We will make its construction and observable inputs explicit in the revised formulation.

(3) Trust-conditioned behaviors. The simulator specifies the underlying trust-transition mechanism, but does not prescribe any trust-conditioned recommendation policy. The trust-conditioned behaviors reported in Appendix A.3 emerge from the learned Q-function.

(4) Baseline comparability. All methods were intended to use identical environment and human-response dynamics, differing only in their modeling and learning formulations. To ensure that all reported results are based on identical experimental settings and implementations consistent with the proposed formulation, we have rerun the experiments. The updated results are presented in the table in Answer 1.

(5) Notations. We clarify that ± denotes the standard deviation across evaluation episodes. We will add collision rates for the lane-changing experiments and expand Table 9 to include the nominal and unobserved-signal settings.

We thank the reviewer and will incorporate the clarifications in the revision.

Reviewer_qbqiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a nicely written manuscript describing the theoreatical set up for adherence-aware RL where adherence is modeled as a hidden dynamic state. The motivation for this more advanced set-up is well articulated and mathematical derivation to reach the adherence-aware Bellman operator is clear. However, the experiment section is less developed. With my expertise in healthcare AI, I could not tell if the sepsis use case is a right choice for testing this new set-up because no observation of actual adherence data was captured in MIMIC dataset.

Critical strengths
  1. The motivation to model adherence as a hidden and dynamic state is sound.
  2. Introduction of the related work to ground the proposed approach is also comprehensive.
  3. The mathematical derviation to adherence-aware Bellman operator is clear and laid out nicely.
Critical weaknesses

The biggest weakness is the experimental results to support the critical questions such as identifiablity and advantage of the proposed approach. The use case around sepsis is highly questionable given that no adherence data was captured in the dataset used.

Questions for authors
  1. It is understable if a dataset that is suitable for testing the proposed algorithm cannot yet be identified. In this case, it will be great to articulate how such a dataset should be developed and what data elements will be essential to capture.
  2. If the other two use cases indeed are suitable for testing the proposed algorithm, can you improve Experiment section by providing more relevant backgrouind to explain the datasets used and experiment set up so others can have a chance to reproduce this work.
4 comments

We thank the reviewer for recognizing the contributions of our work and for highlighting practical considerations. We address the comments below.

Response to Question 1

We agree that publicly available datasets with the complete interaction trajectories required by our framework are limited. For example, MIMIC records patient states, treatments, and clinical outcomes, but not whether AI recommendations were presented to clinicians or accepted or rejected. We therefore outline a practical pipeline for developing such datasets.

A practical pipeline for developing such datasets. We envision the following process:

  1. Human interaction data collection: Conduct repeated human-subject experiments in an interactive simulator or decision-support interface. An initial heuristic or baseline policy generates AI recommendations, while human participants retain the authority to accept, reject, or modify them.
  2. Human model identification: Use the collected trajectories to estimate the human behavior models. The fallback policy can be learned through supervised learning from human decisions when recommendations are rejected. The observation and trust-transition models can be estimated using likelihood-based methods (e.g., EM), without requiring directly observed trust labels. Periodic self-reported trust measurements can also serve as auxiliary signals.
  3. Human-in-the-loop simulator and RL training: Integrate the learned human models into the task simulator to construct a data-driven human-in-the-loop simulator. RL agents can then be trained through repeated interactions with this simulator.

Essential data elements. To identify the human behavior models, we would collect longitudinal interaction records containing:

  • Decision context: The environmental state and task information available before each recommendation.
  • AI recommendation: The recommended action, together with any explanation, confidence information, or other feedback presented to the human.
  • Human response and executed action: Whether the recommendation was accepted, rejected, or modified, and the actual action ultimately executed. Recording the human's alternative action is particularly important for learning the fallback policy.
  • Post-decision outcomes: The subsequent environmental state, task performance, and outcome feedback available to the human, which may influence future trust.

Response to Question 2

We discuss the suitability of each application scenario for evaluating our framework, as well as potential opportunities for collecting real human-interaction data.

  • Sepsis treatment: Collecting real clinician-adherence trajectories is currently challenging, given the limited deployment of AI-assisted clinical decision systems in routine practice. Nevertheless, healthcare remains a particularly relevant application because of its high-stakes nature and the need for clinicians to retain final decision authority. Existing research has explored RL-based clinical decision support, and Sivaraman et al. (2023) conducted a study in which 24 ICU clinicians made AI-assisted decisions on real sepsis cases. We therefore view sepsis as a promising direction for future real-world validation.
  • Lane changing: Driving simulators provides a controllable environment for collecting repeated human-driver interactions. Human participants can receive lane-changing recommendations under different traffic conditions and decide whether to follow them. Their acceptance, rejection, actual driving maneuvers, and subsequent driving outcomes can be recorded.
  • Supply chain inventory management: In inventory operations, human buyers can traditionally retain discretion over AI-assisted replenishment. Liu et al. (2023) describe Alibaba's system, where buyers could ignore AI recommendations and choose order quantities. Related field evidence from Alibaba's warehouse operations also documents workers' deviations from algorithmic prescriptions (Sun et al., 2022). These studies indicate that recommendation, override, and execution data can be collected in operational settings.

We will clarify the relevant background, provide more detailed descriptions of the experimental environments, and discuss the above data-collection pipeline in the revised manuscript to improve the reproducibility and practical interpretation of our experiments.

[1]. Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., and Perer, A., Ignore, trust, or negotiate: Understanding clinician acceptance of AI-based treatment recommendations in health care, CHI, 2023.

[2]. Liu, J., Lin, S., Xin, L., and Zhang, Y., AI vs. human buyers: A study of Alibaba's inventory replenishment system, INFORMS Journal on Applied Analytics, 2023.

[3]. Sun, J., Zhang, D. J., Hu, H., and Van Mieghem, J. A., Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations, Management Science, 2022.

Dear Reviewer qbqi,

As the reviewer discussion phase is approaching its end, we wanted to kindly follow up on our rebuttal. We sincerely appreciate your helpful comments and thoughtful follow-up. We have revised the manuscript accordingly and summarize our main responses below:

  • Data collection feasibility: We discuss two potential approaches to collecting human-interaction data: (1) real-world data collection and (2) human-in-the-loop experiments. For example, driving simulators provide a feasible and controllable environment for collecting human-driver interaction data.
  • Overall experimental pipeline: We clarify the general experimental pipeline, which consists of three stages: (1) collecting human-interaction data, (2) identifying the human model from the collected data, and (3) constructing a simulator based on the identified model and implementing our adherence-aware RL algorithm.
  • Application scenarios: We further state the practical significance of application scenarios and the feasibility of collecting human-interaction data in these settings.

We sincerely appreciate your thoughtful feedback and your recognition of the value of modeling dynamic adherence, as well as the theoretical contribution of our adherence-aware Bellman operator.

We would be happy to address any further questions or concerns. Thank you again for your time and consideration.

Best regards,

The Authors of Submission 57490

Thanks for your futher reply. I hope that othe reviewers will help check the theoreatical derivations. I agree that this paper's contribution is theorey for learning from human-in-loop situation. My original concerns about lacking experimental evidence is still valid but won't be a showstopper if the theoreatical contribution is solid so my score will remain as three.

We sincerely thank you for your thoughtful reassessment. We agree that the soundness of the theoretical derivations is an important part and we welcome further scrutiny of the theoretical results. Thank you again for your careful consideration and constructive feedback!

Reviewer_PUseOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a human-in-the-loop sequential decision-making problem by modeling human trust as a hidden state in an MDP. It establishes the contraction property of the proposed Bellman operator and demonstrates theoretical and empirical advantages over existing static-adherence models.

Critical strengths
  1. It looks interesting and novel to model humans' trust in the human-in-the-loop system by viewing it as an updatable hidden state.
  2. The paper also characterizes the mis-specification error of static-adherence models for applications with dynamic trust updates.
Critical weaknesses

One main concern is how to characterize trust in practice. Although it is reasonable to model trust as an additional hidden state, the state space itself may be difficult to characterize in practice. For example, trust may be task-dependent: high trust in GPT for grammar checking does not necessarily imply high trust in its mathematical proofs. In such cases, defining the trust parameter itself is actually challenging.

Another concern is the misspecification error. Although the authors investigate model misspecification in Table 9, the experimental setup is not sufficiently clear to me. Could misspecification of the trust state space lead to larger errors than those of a simpler static-adherence model? It would be helpful to understand how robust the proposed framework is to such structural misspecification.

Questions for authors
  1. How to characterize the state space of human belief theta in practice? If the corresponding space is too large, then the MDP will be hardly learnable.

  2. Also, in the experiments, what is the exact formulation of trust? What is the exact way to model the misspecification?

Some minor things: m1. x_t is used in Section 3 but is defined in Section 4. m2. What is q in (3) and (4)? Are they Q?

5 comments
AuthorsOpenReview ↗

Response to Reviewer PUse (Part 1)

We thank the reviewer for recognizing value of this work and for the insightful feedback. we will respond to each question as below.

1. Modelling of trust state and adherence probability [W1, Q1] We thank the reviewer for the opportunity to clarify our formulation. We provide a more detailed explanation below.

General trust and context-dependent acceptance. In our formulation, we model human trust as a latent state that captures the human's general propensity to rely on the AI system. This trust is endogenous and evolves over time according to previous interactions and their realized outcomes.

Importantly, we distinguish the latent trust state from the context-dependent acceptance behavior. The latter is captured by the observation model P(ot=1∣xt,θt,utr)P(o_t=1\mid x_t,\theta_t,u_t^r), which allows the acceptance probability to vary across different environmental states and recommendations. For example, the latent state θt\theta_t captures the user's general trust in ChatGPT, while the task context (included in xtx_t) allows the observation model to capture different acceptance probabilities across these tasks.

Robustness to trust-state discretization. The latent trust state is represented as a scalar and discretized into a small finite set of levels for computational tractability. Since human trust is inherently latent, its discretized representation may not perfectly capture the underlying trust in practice. We therefore conduct an additional experiment to examine the robustness of our framework to such representation mismatch: the true trust process has five levels, {0.1,0.3,0.5,0.7,0.9}, while the agent assumes a coarser three-level representation, {0.1,0.5,0.9}.

Trust representationReward ↑Served demand ↑Trust estimation MAE ↓
5 levels (correct)4358.5 ± 868.1528.9 ± 113.60.082
3 levels (misspecified)4268.3 ± 890.8517.2 ± 119.90.119
static-adherence baseline3846.1 ± 1350.4442.9 ± 180.60.296

The coarser representation leads to a modest performance degradation and a larger trust estimation error, however, while still outperforming baseline methods.

Extension to more complex systems. In more complex multi-task human-in-the-loop systems, trust may evolve independently across different tasks. A natural extension is to introduce task-specific latent trust states θt=(θt1,…,θtK)\boldsymbol{\theta}_t=(\theta_t^1,\ldots,\theta_t^K) together with a global trust state. If the task-specific trust processes are conditionally independent given global trust, we can maintain a structured belief of the form

bt(θt)=bt(θtg)∏k=1Kbt(θtk∣θtg)b_t(\boldsymbol{\theta_t})=b_t(\theta_t^g)\prod_{k=1}^K b_t(\theta_t^k\mid\theta_t^g).

With mgm_g global trust levels and msm_s task-specific trust levels, this reduces the belief representation from O(mgmsK)O(m_g m_s^K) to O(mg+Kmgms)O(m_g+K m_g m_s), avoiding exponential growth in the number of tasks, which may alleviate the scalability issues.

AuthorsOpenReview ↗

Response to Reviewer PUse (Part 2)

2. Clarifications on trust formulations and model misspecification[Q2]

To further clarify how latent trust is characterized in practice, we specify the trust-state representation and the corresponding observation models across our experimental applications.

  1. Highway Lane Changing. We use Θ={0.3,0.5,0.7,0.9}\Theta=\{0.3,0.5,0.7,0.9\}. Human acceptance follows P(ot=1∣θt)=θtP(o_t=1\mid\theta_t)=\theta_t.
  2. Supply Chain Inventory Management. We use Θ={0.1,0.5,0.9}\Theta=\{0.1,0.5,0.9\}. Acceptance additionally depends on the deviation dtd_t from the human's fallback decision P(ot=1∣θt,dt)=(0.7+0.25θt)exp⁡[(dt0.1+0.9θt)2]P(o_t=1\mid\theta_t,d_t)=(0.7+0.25\theta_t)\exp[(\frac{d_t}{0.1+0.9\theta_t})^2]. Higher trust increases baseline acceptance and tolerance to deviations.

We also provide details of the specific model misspecifications considered in our robustness analysis. We consider three types of trust-transition misspecification:

  1. Transition Parameter Misspecification (β\beta-scale). The filter overestimates trust responsiveness by scaling its candidate values as βfilter=min⁡(2β,1)\beta_{\mathrm{filter}}=\min(2\beta,1), changing {0.3,0.8} to {0.6,1.0}.
  2. Signal Interpretation Misspecification (Strict Human). Let qˉt\bar q_t denote the average recommendation quality over the latest three steps. The system uses thresholds 2.0 and -0.8, whereas the human uses stricter thresholds 2.8 and -0.2, respectively. Signals are neutral otherwise.
  3. Signal Source Misspecification (Reward-driven Human). The human updates trust based on adjusted rewards rather than recommendation quality. For accepted recommendations, let rˉt\bar r_t be the average adjusted reward over the latest three accepted recommendations. The signal is positive if rˉt≥0.98\bar r_t\geq 0.98, negative if rˉt≤0.93\bar r_t\leq 0.93, and neutral otherwise, while the filter assumes quality-based signals.

3. Complemetary experiments [Q2] To examine whether misspecification of dynamic trust modeling can lead to larger performance losses than a simpler static-adherence model, we compare our method with SA-RL under the above three perturbed trust-transition settings. The results suggests that explicitly modeling dynamic trust remains beneficial even when the assumed transition dynamics are imperfect.

SettingProposed ReturnSA-RL ReturnProposed SpeedSA-RL Speed
Nominal197.33 ± 0.55194.64 ± 0.8229.52 ± 1.0728.19 ± 2.54
β-scale197.70 ± 0.37194.64 ± 0.8229.62 ± 0.6228.19 ± 2.54
Strict human196.10 ± 0.81194.45 ± 0.7629.28 ± 1.3128.22 ± 2.36
Reward-driven196.86 ± 0.75194.73 ± 0.8129.44 ± 1.1128.31 ± 2.27

Additional experiments evaluating robustness to misspecified observation models and fallback policies are presented in Answer 2 to Reviewer 3. For brevity, we refer the reviewer to that response.

Regarding the remaining minor comments, we thank the reviewer for pointing out the notation issues. We will define xtx_t before its first use in Section 3. We also clarify that QQ denotes the executed-action value function, whereas qq represents an arbitrary bounded candidate value function to which the Bellman operator is applied, as specified in Section 4.1.

We hope our responses above address your concerns. Please let us know if any further clarification is needed. Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.