- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VIOLA, a benchmark for policy-violation diagnosis: given the trace of a multi-agent run, predict which violation type occurred (11 types) and which sub-agent caused it (5 plus). The testbed is the CUGA multi-agent system (GPT-4.1) on AppWorld. Policies are extracted from CUGA's sub-agent prompts, and a violation is created by replacing one policy clause in one sub-agent's prompt with a contrary instruction; a GPT-4.1 judge checks that the clause is gone, and a second checks that the deviation appears in the agent's behavior. The result is 337 intervention traces in 20 (type, agent) cells plus 63 unmodified controls. 78% of intervention traces still get a perfect task score. Detectors see the agents' behavior but no system prompts or policies; the best, a GCN over the call tree, reaches 26.3% type accuracy and 48.9% attribution accuracy, against 16.6% and 36.1% for the majority class.
Critical strengths
-
The paper studies an important problem. Outcome-only evaluation misses process failures, and the finding that most policy-violating traces still receive a perfect task score makes this concrete for multi-agent systems.
-
The construction and validation protocol is transparent: cross-model judges, two human studies, per-cell statistics, and the observation that deleting a clause rarely changes behavior (5.8% accepted) while a contrary instruction does (67.8%).
Critical weaknesses
- The main task is not well posed.
- Detectors see no policy, yet must name which policy was broken, so type prediction largely reduces to guessing the injection distribution.
- The injected agent follows the (modified) instruction it received, so a "violation" is defined only against an extracted policy that the detector never sees.
- The realistic monitoring setting, original policies plus behavior, is tested only with GPT-4.1 on one split of 55 traces (Section 5.2).
- The low detector scores may reflect how the detectors were built, not how hard the task is, so they do not show that diagnosis is difficult.
- All trained detectors embed only the first 128 tokens of each call's input and output, while a trace has about 108k tokens, so the violating content may never enter the representation.
- The best GCN never predicts three of the five agents (182 of 270 predictions go to APIPlanner, Table 14) and never predicts three types.
- The fine-tuned text encoder, whose input contains the injected instruction, stays near the majority class (17.8% type accuracy), and three of five seeds stop at epoch 1 to 2. The paper concludes that typing is limited by training data and not by representation, because eight aggregators perform similarly. But all of them aggregate the same 128-token embeddings, so this comparison does not test the representation.
- The labels are weak as ground truth.
- Human verifiers upheld Judge 2 by majority on only 20 of 30 items (7 split), cross-model judges reach Fleiss' kappa of about 0.47, and the three judges accept 75 to 101 of the same 141 traces.
- The label is the injection origin, but deviations propagate downstream, and the paper does not define how to score a detector that attributes a downstream effect to the agent where it appears.
- By the authors' own scaling analysis, typing is limited by examples per cell, yet half of the cells have 2 to 8 traces, so the benchmark mainly measures data scarcity.
Other issues that matter less for my rating:
- The 78% perfect-score rate is partly a selection effect: crashed runs and 23 of 43 cells were excluded, and the largest class, V9 (reasoning omission, 71 traces), rarely affects task outcome.
- Only one system, one backbone, and one environment are used, and the backbone swap covers 30 traces.
- A comparison table with Who&When, TraceElephant, and PolicyGuardBench would clarify the positioning.
Questions for authors
- Were the 63 control traces, and the non-target agents in intervention traces, checked for naturally occurring violations of other clauses? The cross-model probe checks only one target in each of 10 controls.
2 comments
Thanks for your detailed comments! Your responses have resolved some of my concerns, but the problems regarding the task setting and detector construction still hold. Currently, I will maintain my score.
Thank you for the review. We have uploaded a revised PDF. New results are in App. H.6, M and N. Accuracies: type / agent, seed means unless pooled; p-values: pooled paired tests.
Your concerns were: (1) detectors see no policy; (2) low scores may come from detector design; and (3) label reliability. We ran the tests you asked for; they confirm the paper's findings.
W1a. No policy means guessing the label distribution. The setting is deliberate (§5), and GCN is not guessing: it beats the majority baseline by 9.6 points on type and 12.6 on agent (p ≤ 0.001), with about twice its balanced accuracy (17.4% vs 9.1% type, 39.0% vs 18.0% agent; App. N, Table 30).
W1b. A violation is defined against a policy the detector never sees. By design, as §4.2 states: "VIOLA tests whether a detector notices that an agent stopped following a rule it was given". A monitor knows the intended policy, not the prompt the agent ended up with (§4.3).
W1c. Policy access tested only with GPT-4.1 on one split. Revised App. H.6 (Table 17) tests GPT-4.1 and Claude Opus 5 on all five splits, with and without policies and cap. With the policies and the full record, Opus 5 reaches 23.6% / 45.5% against GCN's 26.3% / 48.9% (p = 0.55 / 0.46); GPT-4.1 stays at 10.3% / 17.4%. Only the per-cell clauses lift Opus 5 above GCN (38.5% / 59.9%); they name the 20 cells (the answer set), so H.6 reports them as a reference point.
W2a. 128 tokens per call. Embedding 512 tokens or the full response gives GCN 24.4% and 22.5% typing, against 26.3% (§5.2).
W2b. GCN collapse. App. H.1, H.3 and H.4 report this; it follows the class balance. Class-weighted training spreads the predictions but lowers accuracy (type 26.3% → 20.5%, agent 48.9% → 36.7%) and leaves balanced accuracy about the same (App. H.8, Table 30).
W2c. Text encoder. The paper reports that row as a prompt-visible reference (§5.1). On the behavior-only input the LLMs read, the same encoder reaches 20.8% / 44.8% (new row in Table 2; selected epochs 7/8/6/8/6), below GCN.
W2d. Eight aggregators share the embeddings. Longer embeddings, the behavior-only encoder and LLMs do not beat GCN without the answer set either. More training data does help: GCN typing rises from 19.3% with a quarter of the data to 26.3% with all of it (Table 19). The wording now says "the representations or architectures we tested" (§1, §8).
W3a. Human study and judge agreement. Humans agreed with Judge 2 on 75.8% of judgments, inside their agreement with each other (68.3–81.7%, §6.1). κ is low because one label dominates; pairwise agreement is 77–84% (App. K.2). The 75–101 range reflects a sample with 30 rejected and 10 control traces (Table 22). The new App. M retrains all detectors with disputed labels fixed or removed: GCN stays significantly above the majority baseline in every case (p ≤ 0.018).
W3b. Scoring of downstream effects. Revised §5 states the strict rule. The new App. N (Table 29) also counts any other agent or type that three LLM judges marked: GCN goes from 32.7% to 44.6% on type (pooled), and the majority baseline gains most on agent (11.9% → 43.6%). The strict rule separates detectors better.
W3c. Sparse cells. 11 of 20 cells have 2–8 traces but hold only 15% of intervention traces. On the 8 cells with ≥15 traces plus control, GCN types 28.6% against 19.8% for the majority baseline (pooled, App. N).
W4a. 78% as a selection effect. §4.6 says the rate is for the retained benchmark. The claim is a comparison: interventions 0.889 vs controls 0.892 mean score (p = 0.89). Without V9, 76.3% of interventions are perfect; controls are 79.4% (reply only).
W4b. One system and backbone. One deep pipeline is deliberate: attribution needs several agents that could commit the same violation (§4.1). With Claude Sonnet 4.6 as backbone, 27 of 30 traces were accepted (App. K.5).
W4c. Comparison table. Who&When and TraceElephant label failing agents in failed runs; PolicyGuardBench labels single-agent policy breaks. VIOLA labels the policy and the agent, even when the task succeeds.
Q. Were controls and non-target agents checked? Yes. The same judges checked every applicable clause in the full records of all 63 controls and 59 interventions. Non-target agents in intervention traces are flagged at about the same rate as in controls (20.1% vs 17.2%), and most flags concern wording rather than behavior, so they describe CUGA's normal output, not violations. Full details are in our general response.
Changes in the revised PDF: §1, §5, §5.1, §5.2, §8, Table 2, App. G.2, H.6, M, N, as cited above.