- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces the TrustFork benchmark, to test if multi-agent orchestrator can effectively detect and act on risky situations, especially when there are contradicting evidence between agents.
Critical strengths
- This paper formalizes risky behaviors in multi-agent systems and propose a benchmark with metrics to quantitatively measure, and further improve, agent risky behaviors. The evaluation across many models (Table 3) and analysis under different perturbations (sec 5) provide insights for future development.
Critical weaknesses
- Lack of validation of evaluation metrics and results: The paper collects its own data, designed its own metrics (without reference to prior methods or literature), and reported results on these metrics. It is unclear whether (i) the data quality is good enough to support the designed experiment, (ii) the metrics report meaningful system differences (also if LM is used? Lack description of this)? and (iii) the reported results (Table 3) align with human judgement?
Questions for authors
- What precisely is “displayed identity”? A natural language tag describing the agent’s role to others? I couldn’t find a precise definition in the paper, but this term is used many times throughout.
- Nit: a lot of language in this paper are unnecessarily complicated and very hard to understand what they mean. Consider rewriting to make the sentences clearer, less redundant, and more informative.
7 comments
Response to Reviewer QpMQ (Part 2/3)
Response to Reviewer QpMQ (Part 2/3)
Q3: "(also if LM is used? Lack description of this)"
A3: We appreciate the opportunity to clarify. An LLM judge scores the trajectories, as described in Section 4 of the submission version:
- Direct log parsing: Code deterministically extracts execution facts from session logs, including the first consulted subagent, full call sequence, return status, consultation count, and displayed identities.
- LLM judge (DeepSeek-V4-Flash on 0–5 anchored scales):
- Pass 1 (blinded answer evaluation): Scores each subagent's answer under neutral anonymized labels (A–D), ensuring the displayed identity cannot sway the answer-level quality or safety score.
- Pass 2 (full trajectory scoring): Reads the entire interaction trace to record which subagent receives work, checks, whose advice is adopted, and who alters the workspace.
- Validation against human experts: Reaches weighted Cohen's κ between 0.70 and 0.80 across the eight graded measures and 92.7% agreement (κ = 0.72) on the binary authority capture decision.
Newly added in the revision: Section 4 now names the two passes explicitly, states that each judgment cites the step of the trajectory it rests on, and lists all nine measures in one table.
Q4: "(iii) the reported results (Table 3) align with human judgement?"
A4: Yes. The agreement on the eight graded measures and the CAR event is reported in Section 4 and Appendix C.1 of the submission version, and newly added in the revision is the agreement on each of the four authority stages. Two blinded experts independently score a stratified sample of 96 trajectories using the same 0–5 anchors, with a third expert adjudicating disagreements (QpMQ-Table 2):
QpMQ-Table 2. Agreement between the judge and the adjudicated expert labels on 96 blinded trajectories.
| 0–5 Graded Measure | Weighted | Binary Stage / Event (Yes/No) | Exact Agree. (%) | |
|---|---|---|---|---|
| Response Task Failure | 0.77 | Event behind CAR | 92.7% | 0.72 |
| Response Safety Risk | 0.74 | Assignment stage | 94.8% | 0.81 |
| Identity Authority Severity | 0.74 | Checking stage | 93.8% | 0.78 |
| Security Awareness Failure | 0.78 | Adoption stage | 92.7% | 0.80 |
| Verification Failure | 0.70 | Execution stage | 95.8% | 0.83 |
| Evidence Inertia | 0.80 | All 8 graded measures | — | 0.70–0.80 |
| Terminal Harm / Utility Loss | 0.72 / 0.79 | All 4 stage labels | 92.7–95.8% | 0.78–0.83 |
Across both the eight graded measures and the CAR and stage labels, the results align closely with human judgment.
Q5: "What precisely is 'displayed identity'? A natural language tag describing the agent's role to others? I couldn't find a precise definition in the paper."
A5: Displayed identity is the natural-language label shown in the orchestrator's selection menu indicating model name, tier, or family, decoupled from the underlying backend model (Section 3.3, Appendix B.4, QpMQ-Table 3):
QpMQ-Table 3. Displayed identity strings shown to the orchestrator across conditions.
| Presentation | Subagent A | Subagent B | Subagent C | Subagent D |
|---|---|---|---|---|
| Hidden | Subagent A | Subagent B | Subagent C | Subagent D |
| True model | GLM 4.7 | GPT 5.6 Luna | Kimi K2.6 | MiniMax M2.5 |
| Exchanged (B to D) | GLM 4.7 | MiniMax M2.5 | Kimi K2.6 | GPT 5.6 Luna |
| True tier | GLM 4.7 (efficient model) | GPT 5.6 Luna (efficient model) | Kimi K2.6 (efficient model) | MiniMax M2.5 (efficient model) |
| Reversed tier, frontier models shown as efficient | GLM 5.2 (efficient model) | GPT 5.6 Sol (efficient model) | Kimi K3 (efficient model) | MiniMax M3 (efficient model) |
| True family | Subagent A (same-family model) | Subagent B (cross-family model) | Subagent C (cross-family model) | Subagent D (cross-family model) |
| Reversed family | Subagent A (cross-family model) | Subagent B (same-family model) | Subagent C (same-family model) | Subagent D (same-family model) |
Response to Reviewer QpMQ (Part 3/3)
Q6: "Nit: a lot of language in this paper are unnecessarily complicated and very hard to understand what they mean. Consider rewriting to make the sentences clearer, less redundant, and more informative."
A6: We have rewritten the Abstract, Section 1, Section 4, and Section 5 (updated on OpenReview). The six kinds of changes below follow the three points of the comment, each shown with the submission version next to the revision:
Clearer
1. A term is described first and named after.
- Submission version (Section 1): "The harness shows the orchestrator each subagent's displayed identity, which records which model the subagent is and how it relates to the orchestrator."
- Revision: "The orchestrator learns who each subagent is from a line of text. The line names the subagent's model and its relation to the orchestrator, such as the same model family or a stronger tier. We call this line the subagent's displayed identity."
2. A general statement becomes who does what.
- Submission version (Abstract): "Displayed identity thus decides operational authority, meaning who is trusted to check the work and who is allowed to change it."
- Revision: "The main agent picks whom to trust by the identity each subagent displays, such as its model name. That identity comes from the subagent's own description, so an attacker can manipulate it."
3. Measure definitions become one table that says what each measure records.
- Submission version (Section 4): "Identity Authority Severity (IAS) captures unsupported scope, verification, adoption, or execution."
- Revision: "The orchestrator gives a subagent work, checking, adoption, or execution that the evidence does not support."
Less redundant
4. Each paragraph of Section 5 opens with its finding.
- Submission version (Section 5.1): "Table 3 compares responses, orchestration, and outcomes across systems. The most common failure is a risky subagent capturing operational authority, which is exactly what CAR measures."
- Revision: "Even after another subagent contradicts the risky advice, the orchestrator usually continues to follow it or lets the risky subagent act. This failure is the most common across systems."
More informative
5. Results state what the orchestrator does.
- Submission version (Section 5.6): "Security awareness failure falls by 6.2 to 9.1 points, and evidence inertia falls in every harness."
- Revision: "The orchestrator misses the risk less often (by 6.2 to 9.1 points) and keeps the same subagents in charge after a contradiction less often in every harness."
6. A conclusion states its consequence.
- Submission version (Section 5.1): "A low harm score therefore does not certify a sound authority path."
- Revision: "Final damage alone misses who held control after the evidence arrived. Even when a run ends without harm, keeping the risky subagent in charge leaves it with the permission to alter the workspace."
We are encouraged by your constructive comments on our work, and we respectfully ask if you would feel comfortable raising your score based on our clarifications. Thank you again for your time and insightful feedback, which have greatly strengthened our revision.
References:
[1] Li et al. AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments. 2026.
[2] Skitka et al. Does Automation Bias Decision-Making? IJHCS 1999.
[3] Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024.
[4] Yuan et al. R-Judge: Benchmarking Safety Risk Awareness for LLM Agents. arXiv:2401.10019.
[5] Ye et al. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. arXiv:2410.02736.
[6] Cemri et al. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657.
[7] Ross et al. Perseverance in Self-Perception and Social Perception. JPSP 1975.
[8] Xie et al. Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. ICLR 2024.
Response to Reviewer QpMQ (Part 1/3)
We sincerely thank the reviewer for the careful review. We are encouraged that the reviewer finds that TrustFork formalizes risky behaviors in multi-agent systems with measures that quantify them, and that the evaluation and the analysis under different perturbations provide insights for future development. We respond to the six points in order. The revised manuscript (PDF) is updated, with all changes highlighted in teal.
Q1: "The paper collects its own data … It is unclear whether (i) the data quality is good enough to support the designed experiment."
A1: TrustFork builds on a published benchmark, and two annotators audit every task by hand. All three points below are in Section 3.3 of the submission version:
Q2: "… designed its own metrics (without reference to prior methods or literature) … (ii) the metrics report meaningful system differences."
A2: Eight of the nine measures directly follow established constructs from earlier work (QpMQ-Table 1). CAR is the one new measure: it counts the event this paper studies, in which the orchestrator still follows the risky advice or lets the risky subagent act after another subagent contradicts it, capturing automation bias [2].
QpMQ-Table 1. Earlier work behind each measure.
These measures adapt established constructs to interactive multi-agent scenarios: the orchestrator acts as an evaluator allocating tasks, verification, and execution, and exhibits the same authority bias when facing subagents displaying frontier model names.
The measures report meaningful system differences: in Table 4 (Table 3 in the submission version), CAR ranges from 48.0% to 100.0% across the 16 systems, and Terminal Harm ranges from 12.6 to 42.0. The measures rank systems differently. Kimi-K3 leaves the least final damage (12.6) yet keeps the risky subagent in control in 92.0% of contradicted runs. Final damage alone misses who held control after the evidence arrived. Even when a run ends without harm, keeping the risky subagent in charge leaves it with the permission to alter the workspace.