acceptodds
Under review as a conference paper at ICLR 2027

VIOLA: A Controlled Benchmark for Policy-Violation Diagnosis in a Multi-Agent LLM System

Abstract

Agent systems are increasingly deployed under explicit policies that say not only what they must accomplish but how. Task completion does not show that those policies were followed, and in a multi-agent pipeline a useful diagnosis must also name the requirement that was broken and the component responsible. Traces with such annotations do not occur naturally, and adding them by hand to runs of about 100k tokens is unreliable even for experts. We introduce VIOLA, a controlled benchmark for policy-violation diagnosis in the CUGA multi-agent system on AppWorld. A taxonomy of five categories and 12 violation types guides targeted interventions: one policy clause in one sub-agent's prompt is replaced with a contrary instruction, and two LLM-judge stages verify that the clause is gone and that the deviation appears in behavior. The result is 337 verified intervention traces over 11 violation types and five sub-agents, each labeled with its target type and agent, plus 63 controls. Judges from two other model families and seven human annotators characterize label reliability, and 78% of the intervention traces still receive a perfect task score, so outcome-based evaluation would miss them. We benchmark graph, sequence, and LLM detectors that see the agents' behavior but not their system prompts. A GCN attributes the violation to the right sub-agent in 48.9% of test traces and identifies its type in 26.3%, against 36.1% and 16.6% for the majority class. VIOLA supplies the annotated traces, the construction protocol, and the baselines needed to develop trace-based detectors, a first step toward automatic policy-compliance monitoring for multi-agent systems.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, multi-agent systems, policy compliance, benchmark, failure attribution, agent evaluation, execution traces, LLM-as-a-judge, graph neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_HyiSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VIOLA, a benchmark for policy-violation diagnosis: given the trace of a multi-agent run, predict which violation type occurred (11 types) and which sub-agent caused it (5 plus). The testbed is the CUGA multi-agent system (GPT-4.1) on AppWorld. Policies are extracted from CUGA's sub-agent prompts, and a violation is created by replacing one policy clause in one sub-agent's prompt with a contrary instruction; a GPT-4.1 judge checks that the clause is gone, and a second checks that the deviation appears in the agent's behavior. The result is 337 intervention traces in 20 (type, agent) cells plus 63 unmodified controls. 78% of intervention traces still get a perfect task score. Detectors see the agents' behavior but no system prompts or policies; the best, a GCN over the call tree, reaches 26.3% type accuracy and 48.9% attribution accuracy, against 16.6% and 36.1% for the majority class.

Critical strengths
  1. The paper studies an important problem. Outcome-only evaluation misses process failures, and the finding that most policy-violating traces still receive a perfect task score makes this concrete for multi-agent systems.

  2. The construction and validation protocol is transparent: cross-model judges, two human studies, per-cell statistics, and the observation that deleting a clause rarely changes behavior (5.8% accepted) while a contrary instruction does (67.8%).

Critical weaknesses
  1. The main task is not well posed.
  • Detectors see no policy, yet must name which policy was broken, so type prediction largely reduces to guessing the injection distribution.
  • The injected agent follows the (modified) instruction it received, so a "violation" is defined only against an extracted policy that the detector never sees.
  • The realistic monitoring setting, original policies plus behavior, is tested only with GPT-4.1 on one split of 55 traces (Section 5.2).
  1. The low detector scores may reflect how the detectors were built, not how hard the task is, so they do not show that diagnosis is difficult.
  • All trained detectors embed only the first 128 tokens of each call's input and output, while a trace has about 108k tokens, so the violating content may never enter the representation.
  • The best GCN never predicts three of the five agents (182 of 270 predictions go to APIPlanner, Table 14) and never predicts three types.
  • The fine-tuned text encoder, whose input contains the injected instruction, stays near the majority class (17.8% type accuracy), and three of five seeds stop at epoch 1 to 2. The paper concludes that typing is limited by training data and not by representation, because eight aggregators perform similarly. But all of them aggregate the same 128-token embeddings, so this comparison does not test the representation.
  1. The labels are weak as ground truth.
  • Human verifiers upheld Judge 2 by majority on only 20 of 30 items (7 split), cross-model judges reach Fleiss' kappa of about 0.47, and the three judges accept 75 to 101 of the same 141 traces.
  • The label is the injection origin, but deviations propagate downstream, and the paper does not define how to score a detector that attributes a downstream effect to the agent where it appears.
  • By the authors' own scaling analysis, typing is limited by examples per cell, yet half of the cells have 2 to 8 traces, so the benchmark mainly measures data scarcity.

Other issues that matter less for my rating:

  • The 78% perfect-score rate is partly a selection effect: crashed runs and 23 of 43 cells were excluded, and the largest class, V9 (reasoning omission, 71 traces), rarely affects task outcome.
  • Only one system, one backbone, and one environment are used, and the backbone swap covers 30 traces.
  • A comparison table with Who&When, TraceElephant, and PolicyGuardBench would clarify the positioning.
Questions for authors
  1. Were the 63 control traces, and the non-target agents in intervention traces, checked for naturally occurring violations of other clauses? The cross-model probe checks only one target in each of 10 controls.
2 comments

Thank you for the review. We have uploaded a revised PDF. New results are in App. H.6, M and N. Accuracies: type / agent, seed means unless pooled; p-values: pooled paired tests.

Your concerns were: (1) detectors see no policy; (2) low scores may come from detector design; and (3) label reliability. We ran the tests you asked for; they confirm the paper's findings.

W1a. No policy means guessing the label distribution. The setting is deliberate (§5), and GCN is not guessing: it beats the majority baseline by 9.6 points on type and 12.6 on agent (p ≤ 0.001), with about twice its balanced accuracy (17.4% vs 9.1% type, 39.0% vs 18.0% agent; App. N, Table 30).

W1b. A violation is defined against a policy the detector never sees. By design, as §4.2 states: "VIOLA tests whether a detector notices that an agent stopped following a rule it was given". A monitor knows the intended policy, not the prompt the agent ended up with (§4.3).

W1c. Policy access tested only with GPT-4.1 on one split. Revised App. H.6 (Table 17) tests GPT-4.1 and Claude Opus 5 on all five splits, with and without policies and cap. With the policies and the full record, Opus 5 reaches 23.6% / 45.5% against GCN's 26.3% / 48.9% (p = 0.55 / 0.46); GPT-4.1 stays at 10.3% / 17.4%. Only the per-cell clauses lift Opus 5 above GCN (38.5% / 59.9%); they name the 20 cells (the answer set), so H.6 reports them as a reference point.

W2a. 128 tokens per call. Embedding 512 tokens or the full response gives GCN 24.4% and 22.5% typing, against 26.3% (§5.2).

W2b. GCN collapse. App. H.1, H.3 and H.4 report this; it follows the class balance. Class-weighted training spreads the predictions but lowers accuracy (type 26.3% → 20.5%, agent 48.9% → 36.7%) and leaves balanced accuracy about the same (App. H.8, Table 30).

W2c. Text encoder. The paper reports that row as a prompt-visible reference (§5.1). On the behavior-only input the LLMs read, the same encoder reaches 20.8% / 44.8% (new row in Table 2; selected epochs 7/8/6/8/6), below GCN.

W2d. Eight aggregators share the embeddings. Longer embeddings, the behavior-only encoder and LLMs do not beat GCN without the answer set either. More training data does help: GCN typing rises from 19.3% with a quarter of the data to 26.3% with all of it (Table 19). The wording now says "the representations or architectures we tested" (§1, §8).

W3a. Human study and judge agreement. Humans agreed with Judge 2 on 75.8% of judgments, inside their agreement with each other (68.3–81.7%, §6.1). κ is low because one label dominates; pairwise agreement is 77–84% (App. K.2). The 75–101 range reflects a sample with 30 rejected and 10 control traces (Table 22). The new App. M retrains all detectors with disputed labels fixed or removed: GCN stays significantly above the majority baseline in every case (p ≤ 0.018).

W3b. Scoring of downstream effects. Revised §5 states the strict rule. The new App. N (Table 29) also counts any other agent or type that three LLM judges marked: GCN goes from 32.7% to 44.6% on type (pooled), and the majority baseline gains most on agent (11.9% → 43.6%). The strict rule separates detectors better.

W3c. Sparse cells. 11 of 20 cells have 2–8 traces but hold only 15% of intervention traces. On the 8 cells with ≥15 traces plus control, GCN types 28.6% against 19.8% for the majority baseline (pooled, App. N).

W4a. 78% as a selection effect. §4.6 says the rate is for the retained benchmark. The claim is a comparison: interventions 0.889 vs controls 0.892 mean score (p = 0.89). Without V9, 76.3% of interventions are perfect; controls are 79.4% (reply only).

W4b. One system and backbone. One deep pipeline is deliberate: attribution needs several agents that could commit the same violation (§4.1). With Claude Sonnet 4.6 as backbone, 27 of 30 traces were accepted (App. K.5).

W4c. Comparison table. Who&When and TraceElephant label failing agents in failed runs; PolicyGuardBench labels single-agent policy breaks. VIOLA labels the policy and the agent, even when the task succeeds.

Q. Were controls and non-target agents checked? Yes. The same judges checked every applicable clause in the full records of all 63 controls and 59 interventions. Non-target agents in intervention traces are flagged at about the same rate as in controls (20.1% vs 17.2%), and most flags concern wording rather than behavior, so they describe CUGA's normal output, not violations. Full details are in our general response.

Changes in the revised PDF: §1, §5, §5.1, §5.2, §8, Table 2, App. G.2, H.6, M, N, as cited above.

Thanks for your detailed comments! Your responses have resolved some of my concerns, but the problems regarding the task setting and detector construction still hold. Currently, I will maintain my score.

Reviewer_fsDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this work studies policy violations that can occur even when a multi-agent system completes its task. The authors modify one clause in a CUGA sub-agent's prompt and use LLM checks to verify the targeted behavior. The dataset contains 337 intervention traces and 63 unmodified controls from 64 AppWorld tasks, covering 11 violation types. Experiments evaluate violation-type identification and source-agent attribution, with cross-model and human checks of labels. 78% of intervention traces retain perfect task scores, showing why task completion alone is an inadequate compliance measure.

Critical strengths
  1. this work identifies a problem worth evaluating separately. The 263/337 retained intervention traces with perfect task scores demonstrate violations that a task-success metric would miss. Predicting the type and source agent is also more useful for investigation than an anomaly score alone. 2. The construction is reasonably transparent, including the intervened clauses, behavioral validation, unsuccessful combinations, and human disagreements. Task-disjoint splits also keep the same task out of both training and testing.
Critical weaknesses
  1. The target needs a clearer distinction between recovering the intervention source and identifying violations actually present in a trace. The 63 controls have unmodified prompts, but may still contain spontaneous violations; intervention traces may contain additional violations. In Human Study 2, 20 of 30 items received majority support, seven tied, and three were overturned. Only one of the latter was confirmed incorrect; two lacked user-intent information. This small sample cannot establish an overall error rate, but does show unresolved ambiguity. The confirmed error was corrected in the release while the main experiments used the original label, so its effect should also be reported.

  2. The claim that data rather than representation is the main bottleneck is not established. Main LLM experiments truncate inputs to 60,000 characters. Appendix H.6 both supplies the original policies and removes truncation, changing policy access and trace coverage together. The scaling curve shows that more data helps, but does not rule out limitations from locating evidence in long traces or missing information.

Questions for authors
  1. Were controls checked for all applicable violation types? How is a prediction scored if it identifies a real violation other than the injected target? Please also report sensitivity to correcting the known error and excluding unresolved disputed items.

  2. What is the benefit of supplying original policies under the same trace coverage and token budget? This would help distinguish data limitations from information-access limitations. Without this control, I suggest narrowing the bottleneck claim.

1 comment

Thank you for the review. We have uploaded a revised PDF. Changes are cited below by section; the new results are in App. H.6, M and N. Accuracies are type / agent, seed means over the five splits unless marked pooled.

Your questions were: (1) does the label capture the injected change or every violation present, and are the controls clean; (2) do the disputed Study-2 items affect the results; and (3) is it policy access, the 60k cap, or data that limits the detectors. We ran the tests you asked for; they confirm the paper's findings.

1. Injected change vs. violations present (Q1, Q2). The label is the type and agent of the injected change: VIOLA works like mutation testing (§4.2), and "The label marks its origin, which is also what a diagnostic tool has to find" (§7). Controls are runs "with every prompt unmodified" (§4.5); §1 now says "or no intervention".

Q2: how is another violation scored? As an error; the revised §5 states: "A prediction is correct only if it matches this label: a violation other than the injected one, or a sub-agent other than the injected one, counts as an error." The new App. N (Table 29) shows this rule does not drive the results: when any other type or agent marked by the judges below also counts, GCN goes from 32.7% to 44.6% on type and 51.5% to 70.3% on agent (pooled). It stays far above the majority baseline, and on type every graph and sequence detector stays above both LLMs under either rule.

Q1: were controls checked for all violation types? Yes. Three LLM judges checked every applicable clause of the 20 benchmark cells (all 11 types) in the full records of all 63 controls and 59 interventions. Their flags occur at about the same rate in unmodified and changed runs (17.2% vs 19.3%) and mostly concern wording rather than behavior, so they describe CUGA's normal output, not violations. Full details are in our general response.

2. Study 2 and the confirmed error (Q3). App. L.2 does not use these items to estimate an error rate ("we do not extrapolate a population error rate from it"), and Judge 2 agrees with the annotators (75.8%) as often as they agree with each other (68.3–81.7%). The new App. M (Table 28) retrains every detector on the same five splits with item 10 relabeled as a control, with all 10 disputed items removed, and with the 7 ties and 2 intent-dependent items removed. In every case GCN stays significantly above the majority baseline on type and agent (pooled paired test, p ≤ 0.018), within one standard deviation of Table 2: 23.6% / 48.5% with item 10 corrected, 25.3% / 46.9% with all disputed items removed.

3. Policy access vs. trace coverage and token budget (Q4). The revised App. H.6 (Table 17) separates the two factors for GPT-4.1 and Claude Opus 5 on all 270 test traces:

  • Same trace coverage (60k cap): the original policies raise Opus 5 attribution from 33.3% to 42.6% and do not help GPT-4.1.
  • Coverage alone: removing the cap helps neither model; the judge's cited evidence survives the cap in 319 of the 324 traces where it can be located.
  • Same token budget: adding the policies raises Opus 5 prompts from 66.7k to 100.6k tokens. Giving it even more trace tokens instead (no policy, uncapped: 143.3k) does not help (34.5% vs 33.3% agent), so the gain comes from the policy content, not prompt length. The per-cell clauses fit the 60k budget (67.6k tokens) and lift Opus 5 to 38.5% / 59.9%, but they name the 20 cells, which is the answer set, so H.6 reports them as a reference point, not a detector.
  • Overall: with the original policies and the full record, Opus 5 reaches 23.6% / 45.5% against GCN's 26.3% / 48.9% (p = 0.55 / 0.46); GPT-4.1 stays at 10.3% / 17.4%.

4. Data vs. representation. The revised text states the claim precisely: "rather than in the representations or architectures we tested" (§1), "as training data is added" (§7), "In our ablations" (§8). Embedding 512 tokens or the full response gives GCN 24.4% and 22.5% typing against 26.3% (§5.2), and removing the LLM cap does not help (H.6). Training data does move typing: 19.3% → 21.5% → 24.8% → 26.3% at 25/50/75/100% of the data (Table 19).

Changes in the revised PDF: §1, §5, §5.2, §7, §8, Table 2, App. H.6, M, N, as cited above.

Comments on the paper

2 comments

Thank you for the review. We have uploaded a revised PDF. Changes are cited below by section; the new results are in App. H.6, M and N. Accuracies are type / agent, seed means over the five splits unless marked pooled.

You asked about (1) the injected label, (2) Human Study 2, (3) policy access vs. the 60k cap, and (4) data as the bottleneck. We ran the tests you asked for; they confirm the paper's findings.

1. Is the injected label the right answer? Yes. The label is the injected change and the agent that received it, which is what a diagnostic tool must find (§7). The revised §5 states the scoring rule: "A prediction is correct only if it matches this label: a violation other than the injected one, or a sub-agent other than the injected one, counts as an error." Controls are defined by construction: the same tasks run with every prompt unmodified (§4.5); §1 and §4.3 now use this wording.

Were controls and interventions checked against all policies? Yes. Three LLM judges checked every applicable clause in the full records of all 63 controls and 59 interventions. Their flags occur at about the same rate in unmodified and changed runs (17.2% vs 19.3%) and mostly concern wording rather than behavior, so they describe CUGA's normal output, not violations; each intervention label marks a change Judge 2 verified (§4.4). Full details are in our general response.

How should other answers be scored? The new App. N (Table 29) re-scores 101 test predictions (pooled), also counting any other type or agent the judges marked. On type, GCN goes from 32.7% to 44.6%, and every graph and sequence detector stays above both LLMs. On agent, the majority baseline gains most (11.9% → 43.6%), so the strict rule separates detectors better.

2. Human Study 2. App. L.2 does not use these 30 items to estimate an error rate, and the disputed items are ones on which the annotators also disagreed with each other: Judge 2's agreement with them (75.8%) is within their agreement with each other (68.3–81.7%). To test whether these items matter, the new App. M (Table 28) retrains every detector on the same five splits with item 10 relabeled as a control, with all 10 disputed items removed, and with the 7 ties and 2 intent-dependent items removed. In every case GCN stays significantly above the majority baseline on type and agent (pooled paired test, p ≤ 0.018), within one standard deviation of Table 2.

  • Ties and context-dependent items: the third variant removes exactly these 9 items, and the results hold, so the conclusions do not depend on how they are resolved.
  • Verified subset: the second variant removes every item that the annotators did not confirm.
  • Corrected label: §5 uses the label as first judged (1 of 400 traces, App. L.2); with it corrected, GCN scores 23.6% / 48.5% (App. M).

3. Policy access vs. the 60k cap. The revised App. H.6 (Table 17) crosses policy access with the cap for GPT-4.1 and Claude Opus 5 on all 270 test traces:

  • Removing the cap helps neither model, and the judge's cited evidence survives the cap in 319 of the 324 traces where it can be located.
  • With the original policies and the full record, Opus 5 reaches 23.6% / 45.5% against GCN's 26.3% / 48.9% (p = 0.55 / 0.46); GPT-4.1 stays at 10.3% / 17.4%. These results replace the single-split run.
  • Only the per-cell clauses lift Opus 5 above GCN (38.5% / 59.9%). They name the 20 benchmark cells, which is the answer set, so H.6 reports them as a reference point, not a detector.

4. Is data the bottleneck? The claim rests on several tests, not on scaling alone: longer embeddings give GCN 24.4% and 22.5% typing against 26.3% (§5.2); a text encoder on the behavior-only input reaches 20.8% (Table 2); removing the LLM cap does not help (H.6). Training data does move typing: 19.3% with a quarter of the data to 26.3% with all of it (Table 19), and per-cell training count predicts recall (ρ = 0.61, p = 0.004). The revised text states the claim precisely (§1, §8).

Changes in the revised PDF: §1, §4.3, §5, §5.2, §7, §8, Table 2, App. H.6, M, N, as cited above.

AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their careful reading and concrete questions. We have uploaded a revised PDF; the new results are in App. H.6, M and N. We ran the requested tests, and each supports the paper's findings. Accuracies are type / agent, mean over the five task-disjoint splits.

New experiments

  1. LLMs with the original policies, all five splits, including Claude Opus 5 (App. H.6, Table 17). Opus 5 reaches 23.6 / 45.5 and GPT-4.1 10.3 / 17.4, against GCN's 26.3 / 48.9; removing the 60k cap helps neither model. Only the per-cell clauses lift Opus 5 above GCN (38.5 / 59.9); they name the 20 cells, which is the answer set, so we report them as a reference point, not a detector.
  2. Label sensitivity (App. M). With the confirmed label error corrected, or all disputed Study-2 items removed, GCN stays significantly above the majority baseline on type and agent (p ≤ 0.018).
  3. Text encoder on behavior only (Table 2): 20.8 / 44.8, below GCN.
  4. Longer embeddings (§5.2): 512-token and full-response embeddings give GCN 24.4% and 22.5% typing, against 26.3%.
  5. Scoring checks (App. N): when any other type or agent marked by three LLM judges also counts as correct, the graph and sequence detectors stay above both LLMs on type.

Were the controls and interventions checked for other violations? All three reviewers asked this. Three LLM judges (GPT-4.1, Claude Opus 5.5, Gemini 3.1 Pro) checked every applicable clause of the 20 benchmark cells (all 11 types) in the full records of all 63 controls and 59 interventions. They match the human majority on 22 of 23 Study-2 items (App. N).

  • The check has a background rate. Most flags fall on clauses that ask for specific wording, such as stating a check as a separate step, noting trade-offs, or copying the user's request verbatim. These are not violations per se; they reflect how CUGA phrases a step with its original prompts. The check flags non-injected clauses in intervention traces at about the same rate as in controls (19.3% vs 17.2%), so it does not separate unmodified runs from changed ones.
  • The remaining flags are uncertain calls. On all other clauses, 9 controls have a flag (1.7% of trace–clause pairs), and in 7 of these one of the three judges finds no violation. Judging compliance from a trace is hard: human annotators agree with each other on 68.3–81.7% of such judgments (§6.1), and judges from three model families reach Fleiss' κ 0.47 (App. K.2), so a 2-to-1 call is within normal disagreement. The two unanimous flags concern query strategy (which search parameter to use, how many result pages to read).
  • What this means for the labels. A control is the same task with every prompt unmodified (§4.5). Each intervention label marks a change from this behavior that Judge 2 verified in the trace (§4.4).

Changes in the revised PDF: scoring rule (§5); control wording (§1, §4.3); policy-aware results (Table 2, App. H.6); behavior-only text encoder (Table 2, §5.1, App. G.2); longer embeddings (§5.2); label sensitivity (App. M); scoring checks (App. N); narrower bottleneck wording (§1, §7, §8). Each reviewer's thread answers their points one by one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.