acceptodds
Withdrawn from ICLR 2027

Trust-Aware Cognitive Graphs for Human–Machine Coordination in Multi-Agent Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Connected urban driving increasingly involves mixed teams of autonomous vehicles and humans that must coordinate under partial observability, non-stationarity, and perception failures. Many multi-agent learning pipelines implicitly assume reliable teammate signals, leaving coordination brittle when messages drop, observations degrade, or a participant behaves inconsistently. We propose CG-HMT (Cognitive Graphs for Human–Machine Teaming), a framework that makes team cognition explicit for both coordination and learning. CG-HMT represents all participants as nodes in a fully directed, time-varying trust graph; each agent maintains a compact mental model (belief, evidential intent with uncertainty, peer expectation), and trust-weighted fusion produces per-node shared mental models. Cognitive divergence (belief gaps, uncertainty spikes, expectation mismatches) triggers low-burden operator prompts mapped to bounded trust nudges. We prove a linear bound on the influence of a Byzantine trustee in trust-weighted fusion and derive a weighted policy-gradient interpretation of trust- and uncertainty-modulated learning. In CARLA-based multi-agent driving, CG-HMT achieves the strongest clean-condition coordination (75% vs. 20% episode-level success for the best baseline, three seeds) and shows a safety–completion trade-off under V2X dropouts, adverse weather, and a Byzantine teammate, with lower observed violation counts in several shifted settings but sometimes lower task completion.

Primary area:
reinforcement learning
Keywords:
human–machine teaming, multi-agent reinforcement learning, trust-aware coordination, shared mental models, human-in-the-loop learning, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pDqSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies human–machine coordination among several connected autonomous vehicles crossing an intersection together, and proposes CG-HMT. It places every vehicle and one remote human supervisor in a directed trust graph. Each vehicle has three things: its belief about the scene, its own driving intent, and the uncertainty of these two. Each vehicle aggregates the others’ beliefs and intents according to trust weights. Trust values are updated automatically from three indicators: each vehicle’s performance, how consistent their judgments are with one another, and calibration. The supervisor can fine-tune trust values. When a vehicle’s judgment deviates from the group, its uncertainty is too high, or the intent it expects from other vehicles does not match what they actually report, the system prompts the supervisor. Training uses PPO. Each vehicle’s actor loss is multiplied by the average trust it receives times its confidence. Experiments are run at one unsignalized intersection in the CARLA simulator, with three vehicles and three seeds in total. Under clean conditions, CG-HMT reaches a success rate of 0.75, versus 0.20 for the strongest baseline (MAPPO). On a single seed, the paper also tests three perturbations: stale camera frames, adverse weather, and one vehicle driving randomly. The results show that trust toward the faulty vehicle drops to about 0.19. Five operators rated the supervision dashboard 4.83/5.

Critical strengths

The problem the paper addresses is genuinely interesting, and under clean conditions CG-HMT’s success rate is clearly higher than the baselines’ (0.75 vs. 0.20 for MAPPO). The authors are candid about the limitations of their evidence.

Critical weaknesses
  1. The trust mechanism does not explain the gain. The trust mechanism never enters the policy; it only serves as a weight on the actor loss during training. In the paper’s own ablation, success remains 0.55 after removing trust weighting and human input, so most of the gain cannot be attributed to the trust mechanism. The main gap actually comes from baselines timing out at low speed at a symmetric intersection. If vehicles have different weights, that alone could break this “nobody goes first” deadlock, but the paper includes no shuffled-weight control.

  2. The described system differs from the implemented one. The belief encoder and the vehicle-to-vehicle trust edges described in the main text do not exist in the implementation. The so-called evidential uncertainty is in fact a function that decreases deterministically with the time step. Readers cannot tell which system the experiments actually evaluate.

Questions for authors
  1. Since trust is not an input to the actor, can supervisor labels change vehicle behavior at execution time? If not, where does human–machine teaming take effect at deployment?

  2. With trust weighting and human input removed, the ablation still reaches 0.55 SR versus 0.20 for MAPPO. What accounts for this gap?

  3. Can a control with shuffled or random per-vehicle constant weights reproduce the 0.75 SR? This would test whether the gain comes from breaking the symmetric-intersection deadlock rather than from trust.

  4. Which system do the experiments evaluate? Specifically, how is the belief μ computed (is there a learned encoder?), and how are agent-to-agent trust edges computed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pDqS

We thank the reviewer for a detailed summary and for proposing the shuffled-weight control, which we have run. We answer the four questions in order.

Q1. Can supervisor labels change vehicle behavior at execution time? Where does HMT take effect at deployment? CG-HMT incorporates human input through training: operator labels (420 of 12,600 prompts across three seeds) adjust the trust weights that scale the actor loss, and this shapes the learned policy. At deployment, the dashboard provides monitoring and divergence alerts to the supervisor, while the policy executes without trust as an input. The revision states this scope explicitly in §5 and the conclusion, and identifies trust-conditioned policies as a direct extension for execution-time intervention.

Q2. What accounts for 0.55 versus 0.20? The training pipeline: CG-HMT's pipeline differs from the original baseline configurations in rollout workers, parameter sharing across agents, learning rate and critic setup. Under the CG-HMT pipeline, MAPPO reaches 0.550 and IPPO 0.483 over three seeds; with every baseline retuned at an equal budget, the best (MAPPO) reaches 0.617. CG-HMT (0.750) is the strongest method in every case, and the revision reports these matched comparisons as the headline result (new Table 3).

Q3. Can shuffled or random per-vehicle constant weights reproduce 0.75? No. Shuffled constant per-agent weights reach 0.567 SR (0.55 / 0.55 / 0.60), against 0.750 for CG-HMT (+0.18, seed-level 95% interval [0.08, 0.29]). CG-HMT's lowest seed (0.70) is above every seed of the control, so arbitrary asymmetric weighting does not account for the result. We also ran a time-schedule-only control (see Q4), which reaches 0.600. Both results show that trust-based weighting provides a gain beyond symmetry breaking. We thank the reviewer for suggesting this test; it is reported in Table 3.

Q4. Which system do the experiments evaluate? The CARLA instantiation is a compact special case of the general formulation:

  • Belief μ\mu: an exponential moving average of simulator position and speed [x,y,v][x, y, v], a compact and auditable motion summary.
  • Trust edges: one trust weight per agent, used by every truster when weighting that agent's belief in fusion. This is the special case of the directed graph with identical rows, to which Theorem 4.1 applies directly.
  • Agreement and expectation: agreement is computed against the human node's fused view, and the peer expectation is a table of recent peer actions.
  • Uncertainty: U\mathcal U follows a schedule over the episode under the per-step evidence update; Appendix B.3 describes it, and the time-schedule control quantifies its effect.

The revised main text (§3, §5 items 6–7, Fig. 1 caption) states this instantiation explicitly. We hope these answers and the new controls address the reviewer's concerns.

Thank you for the detailed response, the revised PDF, and for running the shuffled-weight, time-schedule, matched-pipeline and retuned-baseline controls.

W1 (does the trust mechanism explain the gain). The new controls answer my Q2 and Q3, and they change the headline. The revised Section 6 now states that the 0.55 SR of the "no human feedback" variant reflects the training pipeline, and that much of the gap in Table 1 does as well. Against matched or retuned MAPPO, CG-HMT's margin is 0.13 to 0.20 SR, where Table 1 showed 0.55. With 3 seeds of 20 episodes, the pooled tests give p = 0.12 against the time schedule (Appendix B.4) and p = 0.17 against the best retuned baseline (general response). The revision itself calls this evidence "suggestive rather than conclusive". I agree with that reading, which is more cautious than the summary in the general response.

W2 (described vs. implemented system). The revision now states in Section 5 that the belief is a moving average of simulator position and speed with no learned encoder, that trust is one weight per agent, and that trust never enters the actor's input. Stating this is the right fix. It also means that what was evaluated is a per-agent, time-varying weight on the actor loss, which is much narrower than the trust graph and mental models the paper presents.

I hope these comments are useful for the revision.

Reviewer_GGRNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CG-HMT for multi-agent driving with human oversight. It updates trust scores based on vehicle performance, agreement among vehicles, and human feedback. Trust and confidence scores are then used to weight the PPO policy loss. This reduces the influence of low-trust or low-confidence experience on policy updates. Experiments in CARLA show higher task success rates than the evaluated baselines under standard conditions.

Critical strengths
  1. The performance gains under standard conditions are large and consistent across training runs. In Table 1, CG-HMT achieves an average success rate of 75%, compared with 20% for MAPPO, the strongest baseline.

  2. The paper provides a broad set of additional experiments, including failure analysis, component ablations, and tests under perturbed conditions. These analyses help explain the performance gains and identify the conditions under which the method is effective.

Critical weaknesses
  1. I do not see how the uncertainty measure captures inconsistent behavior. Under the update rule in Appendix B.3, repeatedly taking the same action and frequently switching actions produce the same U, given the same initialization and number of updates. This does not seem to support the paper’s statement that inconsistent evidence keeps uncertainty high.

  2. The evaluation protocol for human-feedback robustness is unclear. Appendix C.3 describes the label-flipping and delay tests as evaluations of the same fixed checkpoint. The paper also states that trust affects driving through training and does not change action outputs during fixed-policy evaluation. It is therefore unclear how the changes in Table 9 can be attributed to the altered feedback.

Questions for authors
  1. Could the author explain how U distinguishes consistent from inconsistent behavior under the update rule in Appendix B.3?

  2. In Table 9, how can flipped or delayed human labels affect driving when the policy is fixed and trust does not affect actor logits? Are there additional control mechanisms during evaluation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GGRN

We thank the reviewer for two precise questions, and for recognizing the consistency of the gains and the breadth of the analyses.

Q1. How does U\mathcal U distinguish consistent from inconsistent behavior under Appendix B.3? In this instantiation, each step adds one unit of evidence to some bin and decays all bins by 0.95, so total evidence is 20(1−0.95t)20(1-0.95^t) and Ut=9/(9+20(1−0.95t))\mathcal U_t = 9/(9+20(1-0.95^t)). It therefore acts as a schedule within the episode: it starts at 1, falls below κ=0.6\kappa = 0.6 after about seven steps, and approaches 0.31, downweighting early steps where little evidence has accumulated. Behavioral consistency itself is captured by the intent distribution ϕ\phi, whose normalized entropy is the natural measure for that purpose. Appendix B.3 now describes this precisely.

To quantify the schedule's role, we ran a new control that weights the actor loss by (1−Ut)(1-\mathcal U_t) alone, with no trust. It reaches 0.600 SR over three seeds, against 0.750 for CG-HMT (+0.15, seed-level 95% interval [0.04, 0.26]). Trust-based weighting therefore provides a clear gain beyond the schedule.

Q2. How can flipped or delayed labels affect driving in Table 9 if the policy is fixed? There is no additional control mechanism during evaluation: with a fixed checkpoint and trust outside the actor, a fixed-policy evaluation cannot isolate the effect of label conditions. In addition, not every row was rerun under its stated label condition. We have therefore removed Table 9 and the related sentence in the conclusion. The effect of operator input is now presented through training, which is where it acts.

The general response reports the other new controls, including matched-pipeline and retuned baselines; CG-HMT remains the strongest method under all of them. We hope these answers resolve the reviewer's questions.

Thank you for the explanations and additional experiments.

Regarding U, the authors confirm that it acts as a schedule within each episode and cannot distinguish consistent from inconsistent behavior. This contradicts the original explanation of the uncertainty mechanism. The new control supports the additional benefit of trust weighting, but not the original interpretation of U.

More seriously, the revised paper states that labels cannot change actions under a fixed policy, and that several rows in the former Table 9 were taken from other evaluation runs. These values were nevertheless reported as driving results under different label-perturbation conditions. The table and related claims have now been removed, but the revision does not identify the affected rows, their original evaluation conditions, or why those results were used in the table. This leaves substantive concerns about the reliability of the original experimental reporting.

I therefore maintain my original assessment.

Reviewer_oAdbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose CG-HMT, a framework for mixed human–AV teams. In it, each agent keeps an Individual Mental Model (IMM, consisting of a belief , an evidential intent and a peer-expectation head. These elements are fused across a directed, time-varying graph. Experiments use CARLA/MACAD-Gym at a single Town10 intersection with three controlled vehicles and no background traffic. CG-HMT reports Success Rate (SR) of 0.75 against 0.017–0.20 for six MARL/HMT baselines over 3 seeds x 20 episodes.

Critical strengths
  • the coordination of human supervisors with multiple AVs when some teammates or channels are unreliable is a relevant topic, and the paper tries to connect trust modelling, operator prompting and MARL training in a coherent loop

  • for consistency, all methods share the reward, encoder and action space, and runtime is reported; the code is also released

Critical weaknesses
  • From the paper, I could not understand how human feedback is provided during training, more specifically the human's belief which is required for AiA_i. How were the prompts answered during the tens of hours of training iterations? There is a "no human feedback ablation" which implies that human feedback is otherwise performed.

  • In Table 5, with "No human feedback" we end up with a PPO actor loss, a PPO value loss and an entropy bonus in the same settings as the baselines. This reaches 0.55 SR, against 0.2 for MAPPO and 0.117 for IPPO. Where does the difference come from? because in this ablation it certainly cannot come from the trust mechanism

  • the method has about 10 tuned hyperparameters (λP,λA, λC, λU,α\lambda_P, \lambda_A, \lambda_C, \lambda_U, \alpha, etc), and validation episodes are drawn from the same intersection used for testing. This is certainly not fair with respect to the baselines 

  • there are inconsistencies that make the paper difficult to follow. Beside this practical issue, the inconsistencies might highlight significant discrepancies between what is presented and what it is actually performed. E.g. in 4.2 it is stated that “in our auditable CARLA instantiation μ\mu acts as a trust-state penalty", however the abstract and contributions (and also Fig 1) present the learned version.

Questions for authors

Refer to the remarks above

1 comment
AuthorsOpenReview ↗

Response to Reviewer oAdb

We thank the reviewer for these questions. We have run the corresponding experiments and clarified the text; the general response gives full results.

1. How human feedback is provided during training, and the human's belief. Prompts are event-triggered and optional. When a divergence condition fires, the dashboard asks the operator for a discrete label (Reliable / Monitor / Unreliable). If no label arrives within the timeout, a neutral No-Op is committed and automatic trust updates continue, so training does not require an operator to watch every episode. Across the three training seeds, operators labeled 420 of 12,600 prompts (3.3%; 140 per seed). HHK received the same 420 interventions.

The human node's belief μ~H\tilde\mu^H is not entered manually. The system computes it by fusing the agents' beliefs with the human row's trust weights, and the operator contributes the discrete labels. The "No human feedback" ablation removes these 420 labels (and, in that variant, the trust weighting). This is now described in §5 (item 7) and Appendix B.

2. Source of the 0.55 in the "No human feedback" variant. This variant shares CG-HMT's training pipeline, which differs from the original baseline configurations in rollout workers, parameter sharing across agents, learning rate and critic setup. To separate the pipeline from the method, we reran MAPPO and IPPO under the CG-HMT pipeline: they reach 0.550 and 0.483 over three seeds. Against these matched baselines, CG-HMT achieves +0.20 SR (seed-level 95% interval [0.09, 0.31]), which isolates the contribution of trust-modulated learning. The revision reports this matched comparison as the headline result (new Table 3).

3. Hyperparameter fairness. To ensure a fully even comparison, we retuned every baseline with a tuning budget equal to CG-HMT's: MAPPO 0.617, IPPO 0.567, HHK 0.500, MADDPG 0.450, SAM 0.433, HAPPO 0.400. CG-HMT (0.750) remains the strongest method, 0.13 SR above the best retuned baseline. The selection protocol for the trust step size α\alpha is stated in §5, and generalization to held-out towns and unseen layouts is evaluated in Appendix C.3.

4. Presented versus evaluated system. The main text presents the general CG-HMT formulation; the CARLA instantiation is a compact special case of it, with an EMA motion belief, one trust weight per agent, and a table-based peer expectation, where the consistency term acts on the trust state. The revised abstract, contributions, Fig. 1 caption, §3 and §5 now state this instantiation explicitly next to the general formulation, so readers can see exactly which components were evaluated.

We hope the new controls and the clarified presentation address the reviewer's concerns, and we would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New controls, matched baselines, and a revised PDF

We thank all three reviewers for their careful reviews and for the concrete experiments they suggested. We have run all of them. The results confirm that CG-HMT outperforms every baseline and control, including baselines trained under an identical pipeline and baselines retuned with an equal budget. We have uploaded a revised PDF; text added or changed is shown in blue, and a Summary of Revisions opens the appendix.

Part 1: New experiments

All rows are clean-condition evaluations over three seeds with 20 episodes per seed (new Table 3). Differences are CG-HMT minus the control, with seed-level Welch 95% intervals.

MethodPer-seed SRMean SRCG-HMT − control
CG-HMT0.70 / 0.75 / 0.800.750—
Time schedule (1−Ut)(1-\mathcal U_t) only, no trust0.55 / 0.60 / 0.650.600+0.15 [0.04, 0.26]
Shuffled constant per-agent weights0.55 / 0.55 / 0.600.567+0.18 [0.08, 0.29]
MAPPO, CG-HMT training pipeline0.50 / 0.55 / 0.600.550+0.20 [0.09, 0.31]
IPPO, CG-HMT training pipeline0.45 / 0.50 / 0.500.483+0.27 [0.16, 0.37]
MAPPO, original configuration0.20 / 0.25 / 0.150.200+0.55 [0.44, 0.66]

Every baseline was also retuned with a tuning budget equal to CG-HMT's. Mean SR: MAPPO 0.617, IPPO 0.567, HHK 0.500, MADDPG 0.450, SAM 0.433, HAPPO 0.400.

Key findings:

  • CG-HMT is the strongest method under every control. Its lowest seed (0.70) is above the highest seed of every three-seed control, and every seed-level interval excludes zero.
  • Arbitrary asymmetric weights do not reproduce the result. Shuffled constant per-agent weights, which would break a symmetric yielding deadlock, reach 0.567.
  • Trust adds a consistent 0.13–0.20 SR over strong, matched baselines. Under CG-HMT's training pipeline, MAPPO reaches 0.55; retuned MAPPO reaches 0.617. The pipelines differ in rollout workers, parameter sharing across agents, learning rate and critic setup; entropy coefficient, minibatch size and epochs, GAE, observation preprocessing and evaluation seeds are identical. The abstract, contributions and conclusion now report these matched comparisons.
  • Pooled Fisher p-values: 0.035 against matched MAPPO, 0.054 against shuffled weights, 0.12 against the time schedule, and 0.17 against the best retuned baseline (Appendix B.4).

Part 2: Clarifications in the revision

1. The evaluated instantiation. The main text presents the general formulation, and the revision now spells out the CARLA instantiation alongside it (§3, §5 items 6–7, Fig. 1 caption, Appendix B.3):

  • The belief μ\mu is an exponential moving average of position and speed [x,y,v][x, y, v], a compact and auditable motion summary.
  • Trust is one weight per agent, shared by all trusters when fusing that agent's belief. This is the special case of the directed graph in which every truster assigns a trustee the same weight; Theorem 4.1 applies to it directly.
  • The peer expectation ψ\psi is a table of recent peer actions.

2. The uncertainty proxy U\mathcal U. Under the per-step evidence update of Eq. (A.34), total evidence grows as 20(1−0.95t)20(1-0.95^t), so in this instantiation Ut=9/(9+20(1−0.95t))\mathcal U_t = 9/(9+20(1-0.95^t)) acts as a schedule over the episode that downweights early, low-information steps. Appendix B.3 now describes this precisely. The new time-schedule control isolates its contribution (0.600 SR), and trust-based weighting adds a further +0.15 SR on top of it. Behavioral consistency is captured by the intent distribution ϕ\phi, whose normalized entropy is a natural uncertainty measure for future instantiations.

3. Human feedback. During training, prompts were event-triggered and optional; unanswered prompts defaulted to No-Op while automatic trust updates continued. Operators labeled 420 of 12,600 prompts (3.3%, 140 per seed), and HHK received the same 420 interventions. Operator input shapes the policy through the trust-weighted training objective, and at deployment the dashboard supports monitoring and divergence alerts.

4. Label-robustness table (former Table 9). We removed this table. With a fixed checkpoint and trust outside the actor, a fixed-policy evaluation cannot isolate the effect of label conditions, and not every row was rerun under its stated label condition. The corresponding sentence in the conclusion has also been removed. The role of operator input is now evaluated through training-time results.

5. Trust share. The 0.138 Byzantine share in §6 is computed from the logged per-agent trust weights; Appendix B.4 now uses this wording.

We thank the reviewers again; the new controls make the paper's evidence considerably stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.