- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies sequential decision support in which an AI recommends an action and a human either accepts it or executes a known fallback action. Prior adherence-aware MDPs assume a fixed acceptance probability. This paper instead lets acceptance depend on a latent trust state that evolves over time, and it tracks trust with a Bayesian filter (jointly with an unknown transition parameter ).
The method learns a value function over the executed action. It scores each recommendation by mixing the accept and reject branches analytically, weighted by the predicted acceptance probability. This is implemented with DQN for lane changing and with a GNN actor–critic for inventory management.
The theory covers contraction of the operator, error bounds for belief and parameter estimation, and bounds under misspecification. Experiments use three simulated humans (sepsis, highway lane changing, inventory management). The method obtains the highest average reward in all three, and its recommendations change with the estimated trust level.
Critical strengths
S1. A relevant problem with a clear formulation. Treating adherence as endogenous is well motivated. Separating recommendation, human response and executed action exposes a real trade-off: a recommendation affects both the current outcome and future acceptance. The trust-conditioned policies (Tables 1b, 7) make this trade-off easy to see.
S2. A simple, implementable design. Learning over the executed action and marginalizing over the accept/reject branches analytically fits into standard deep RL. The inventory ablation (Table 4: 4298 vs. 4117) suggests that this design helps.
Critical weaknesses
W1. The human model, which is the core of the problem, is assumed known, is never specified in the paper, and the stated formulation does not match the simulators. The trust-conditioned behaviors reported as findings are built into the simulators.
a. The human models are hand-written, and the agent receives exact copies. App. B assumes the acceptance model and the fallback policy are known, but the paper never states them for any task. In the supplementary code, the agent’s filter uses exactly the parameters of the simulated human. In the lane-change code, belief.py says “For now we intentionally let them match”. In the lane-change task, the simulator also passes the human’s internal good/bad grade of each recommendation (rec_good) directly to the filter. In sepsis and lane changing, acceptance is , which does not depend on the state or on the recommendation.
b. Eq. (1) does not describe the simulators. Eq. (1) states . In the code, trust transitions depend on other quantities: sepsis: on acceptance and on whether the patient improved, i.e., on ; lane changing: on the recommended action, even when it is rejected, averaged over a 3-step window; inventory: on acceptance and on recent rewards.
c. As a result, is not a function of alone in these simulators. In sepsis, “conservative recommended and accepted” and “aggressive recommended, rejected, conservative fallback executed” have the same executed action. Yet after an improvement, trust rises with probability 0.8 in the first case and stays unchanged in the second. In lane changing, the reject-branch value cannot depend on which recommendation was rejected, even though that recommendation changes trust.
W2. The evidence that modeling dynamic adherence improves performance is weak, and the baselines are not comparable.
Questions for authors
-
Model consistency. Please restate the trust transition so that it matches the simulators, i.e., conditioning on , , the outcome or reward, and the history window. How does the executed-action decomposition change? Is the lane-change target in train_belief.py, which uses the same belief in both branches rather than /, intended to implement Eq. (7)?
-
Misspecified acceptance model. Table 9 perturbs only the trust transition. What happens when the acceptance model or the fallback policy is wrong, for example when the true acceptance depends on context but the filter assumes ? What happens when the human model is learned from data?
3.Fair baselines. Were all methods in Table 2 run against the same grid? Can you add a static-adherence learner that shares your architecture, target and human-model knowledge?
-
Statistics. Please report: what ± means in each table; the collision rate for lane changing; the full Table 9, including the nominal run and the unobserved-signal run.
-
Theory. Prop. 1: given , is deterministic. So the TV distance between the two information-state kernels equals 1 whenever the two models predict different beliefs, and the bound exceeds the trivial bound when (the paper uses ). Can the bound be restated using Lipschitz continuity of the value in the belief?
Corollary 1: please define and state which conditions of Fan et al. (2020) are verified.
6.Novelty. What exactly is new relative to Chen et al. (2018, 2020), Bayes-adaptive POMDPs, and trust-aware bandits (Cai & Zhang, 2024)?
10 comments
Response to Reviewer Y5bX (Part 2)
2. Robustness to Human-Model Misspecification [Q2]
We further evaluate robustness to fallback-policy and observation-model misspecification.
Fallback policy. The fallback policy can be approximated via supervised learning (e.g., an MLP classifier). Replacing the oracle with the learned policy causes only marginal degradation, and our method still outperforms both baselines.
| Method | Oracle fallback: Return / Speed | Learned fallback: Return / Speed |
|---|---|---|
| Proposed | 196.71 ± 0.75 / 29.45 ± 1.16 | 196.30 ± 0.81 / 29.33 ± 1.13 |
| Est-θ | 194.83 ± 0.78 / 28.28 ± 2.41 | 194.35 ± 0.80 / 28.09 ± 2.40 |
| No-belief | 194.71 ± 0.65 / 28.45 ± 1.71 | 193.89 ± 0.77 / 28.12 ± 2.02 |
Observation model. We additionally evaluate robustness to misspecified observation models in both scenarios.
- Lane changing The agent assumes , while the true acceptance models incorporate three modifications: (i) quality-dependent additive acceptance, ; (ii) risk-sensitive acceptance, , where measures traffic risk; and (iii) a quality-trust interaction, . The results show that our method consistently achieves higher returns and speeds than both baselines across all three misspecification settings.
| True human model | Method | Return ↑ | Speed ↑ |
|---|---|---|---|
| Quality additive | Proposed | 196.71 ± 0.64 | 29.21 ± 1.50 |
| Est-θ | 194.07 ± 0.85 | 27.70 ± 2.67 | |
| No-belief | 194.19 ± 0.82 | 28.16 ± 2.11 | |
| Risk-sensitive | Proposed | 196.41 ± 0.80 | 29.40 ± 1.20 |
| Est-θ | 194.58 ± 0.77 | 28.23 ± 2.42 | |
| No-belief | 194.04 ± 0.76 | 28.32 ± 1.80 | |
| Quality-trust interaction | Proposed | 196.41 ± 0.79 | 29.25 ± 1.46 |
| Est-θ | 193.80 ± 0.87 | 27.58 ± 2.77 | |
| No-belief | 194.49 ± 0.80 | 28.29 ± 2.01 |
- Supply chain inventory management We consider three misspecifications, each modifying a different aspect of the assumed observation model: (1) Constant tolerance removes the trust dependence of deviation sensitivity; (2) Wrong curvature uses an incorrect deviation exponent (); and (3) Deviation sensitivity scales the tolerance width by 0.6, systematically underestimating human tolerance while preserving the original functional form.
| Observation model | Assumed human behavior | Reward ↑ | Served demand ↑ |
|---|---|---|---|
| Correct | Original observation model | 4443.97 ± 752.62 | 533.13 ± 91.59 |
| Constant tolerance | Trust-independent tolerance | 4064.08 ± 1019.39 | 453.34 ± 121.62 |
| Wrong curvature | Incorrect deviation exponent | 4108.87 ± 857.51 | 498.32 ± 106.94 |
| Deviation sensitivity | Underestimated tolerance | 4370.20 ± 687.70 | 494.61 ± 78.52 |
The results suggest that the impact of observation-model misspecification depends on its type.
- Wrong curvature: Despite using an incorrect functional form, the model preserves the key qualitative relationships between trust, recommendation deviation, and acceptance probability. Consequently, its performance degradation is relatively limited.
- Deviation sensitivity: Although tolerance is underestimated, the model preserves its trust-dependent structure and induces a conservative policy that achieves performance close to the correctly specified model. This suggests that the induced conservative policy happens to reach a similarly favorable operating regime.
- Constant tolerance: Removing the trust dependence of human tolerance fails to capture an important structural relationship in the true observation model, resulting in a more noticeable performance degradation compared with the other two types of misspecification.
These results suggest that preserving the key qualitative relationships in the human response model may be more important than accurately specifying its exact functional form.
Further discussions: Stable value learning
Figure Value-model effects examines the impact of observation-model misspecification on value learning. Panel A shows that misspecification distorts recommendation-level value estimates through errors in acceptance probabilities and posterior updates. In contrast, Panel B shows similar value landscapes across critics trained under different observation models, suggesting that executed-action value learning is relatively robust. This is because the critic evaluates the action actually executed based on environmental rewards and transitions, avoiding the ambiguity associated with directly learning the value of a recommendation that may lead to different executed actions.
Overall, our method remains effective under both fallback-policy and observation-model misspecification, consistently outperforming simpler adherence baselines.
Response to Reviewer Y5bX (Part 3)
3. Proposition 1 and Corollary 1 [Q5]
Proposition 1. We thank the reviewer for pointing out this issue. We replace the kernel TV discrepancy with the one-step predictive-belief discrepancy:
We redefine M^\dagger as the candidate model minimizing this discrepancy. The revised bound follows from the Lipschitz continuity of the value function in the predictive belief, established in the proof of Theorem 1:
Specifically, this property bounds the one-step Bellman mismatch by . Bellman contraction then gives:
Corollary 1. We define as the expected greedy suboptimality of the K-th FQI iterate, including statistical, approximation, and finite-iteration errors.
Regarding the conditions of Fan et al. (2020), Lemmas 1–4 establish the key structural properties required for FQI error propagation under our modified Bellman operator, including contraction, greedy consistency, Markov transition representation, and conditional unbiasedness of sampled Bellman targets. The remaining statistical regularity conditions are inherited from Fan et al. and imposed on the induced belief MDP. We will clarify these conditions and their correspondence in the revision.
4. Clarify novelty and contributions [Q6] We thank the reviewer for pointing out these related works. Our joint belief over latent trust and unknown transition parameters builds upon the Bayesian principle of Bayes-adaptive POMDPs, which provides the foundation for jointly tracking trust and identifying its dynamics.
- Chen et al. (2018, 2020): These works learn trust and human behavior models and use them for POMDP-based planning in human-robot collaboration. In contrast, we focus specifically on adherence-aware decision-making in broader human-in-the-loop systems. We directly learn recommendation policies through reinforcement learning, rather than relying on belief-space planning. To this end, we develop a dedicated adherence-aware Bellman operator that explicitly marginalizes over human acceptance and fallback execution, providing a learning formulation consistent with scalable RL implementations.
- Cai & Zhang (2024): This work studies dynamic trust in stochastic multi-armed bandits and develops a UCB algorithm with minimax regret guarantees. We extend the scope to general state-dependent MDPs, where complex environment dynamics and high-dimensional state representations call for scalable function-approximation-based RL. We further establish guarantees on belief and parameter estimation as well as model misspecification.
5. Other clarifications [W1, Q3, Q4]
(1) Human-model knowledge. We respectfully clarify that the agent is not provided with exact copies of the simulated human's latent state or model parameters. Rather, we assume that the functional structure of the human model is known, while the key trust-transition parameter, like is unknown and jointly inferred. The nominal experiments use a correctly specified model family to isolate the effect of model misspecification. We further investigate trust-transition model misspecification in Table 9, and conduct additional experiments to examine misspecification of the acceptance model and fallback policy.
(2) The rec_good signal. We clarify that rec_good is not an additional internal human state. It is a constructed feedback signal computed from observable information using a prespecified, known quality-scoring model. Thus, it does not provide the agent with privileged access to the human's internal state. We will make its construction and observable inputs explicit in the revised formulation.
(3) Trust-conditioned behaviors. The simulator specifies the underlying trust-transition mechanism, but does not prescribe any trust-conditioned recommendation policy. The trust-conditioned behaviors reported in Appendix A.3 emerge from the learned Q-function.
(4) Baseline comparability. All methods were intended to use identical environment and human-response dynamics, differing only in their modeling and learning formulations. To ensure that all reported results are based on identical experimental settings and implementations consistent with the proposed formulation, we have rerun the experiments. The updated results are presented in the table in Answer 1.
(5) Notations. We clarify that ± denotes the standard deviation across evaluation episodes. We will add collision rates for the lane-changing experiments and expand Table 9 to include the nominal and unobserved-signal settings.
We thank the reviewer and will incorporate the clarifications in the revision.
Response to Reviewer Y5bX (Part 1)
We thank the reviewer for their detailed feedback. We address each point below.
1. Restating the Trust Transition [W1, Q1]
We thank the reviewer for pointing out that the description of the state information is incomplete. To make explicit the information available to the agent, we denote the full observable state by x~t=(xt,ht), where ht collects additional observable interaction signals and relevant historical information required to characterize trust evolution. Accordingly, the trust transition in Eq. (1) is more precisely expressed as
Pβ(θt+1∣θt,x~t,ut).
The specific components of ht vary across environments:
All these signals are observable to the agent. By incorporating them into x~t, the trust transition is Markovian with respect to the full observable state and executed action.