acceptodds
Under review as a conference paper at ICLR 2027

A Sober Look at Agentic Misalignment in Automated Workflows

Abstract

We study a class of emergent misalignment in multi-agent systems (MAS), with a focus on automated workflows, which we refer to as agentic misalignment. Although these systems can solve complex tasks, their agents can act inconsistently with assigned roles and intended human goals. We formally define these behaviors and analyze them within a Bayesian framework, showing how similar priors and likelihoods can leave roles insufficiently separated. To address this issue, we propose Agentic Evidence Attribution (AEA), an alignment framework that uses context-specific evidence to improve role adherence. AEA reasons over agent action traces and provides structured evidence to identify misaligned behavior through evidence injection. To better understand the role of evidence, we study two instantiations of AEA, self-reflection using the workflow model and weak-to-strong generalization using a small trained evidence model. Across eight task benchmarks and two attribution datasets, we show that specialized failure attribution improves workflow alignment to different degrees. Our results show that evidence-based alignment is necessary and can improve agent collaboration and reliability in automated workflows.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Automated workflows, LLM Agents, Misalignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rxoiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses the problem of emergent misalignment in multi-agent systems, focusing on automated workflows. The authors triy to formally define this problem and behaviour using Bayesian framework. The proposed alignment framework (AEA) uses context specific evidence to improve role adherence by reasoning over agentic traces and priding structured evidence to identify and fix misaligned behavior. Two instantiations of AEA are studied, self-reflection based and weak-to-strong generalization using a small trained evidence model. The results show that evidence-based alignment improves agent collaboration and reliability.

Critical strengths
  1. The paper studies an important and timely problem. More and more LLM systems are built as automatically generated multi-agent workflows. When one of these fails, it is often hard to tell which agent went wrong. A single agent that drifts from its role can sink the whole task. The paper’s approach is practical: it pinpoints the agent at fault and repairs only that agent, instead of rerunning or rebuilding the whole system. Attempting a formal treatment of role-specific behavior is also a worthwhile direction, even though it isn’t yet tied to the experiments.

  2. The experimental settings are fair in the sense that every fixing method gets exactly one rerun. The Gemma critic sees the same full trace and gives the same kind of output and no method has access to ground truth. Under these rules, AEA-4B beats both retrying and the critic on all 2,500 HLE questions for both strong models. It is also interesting that a small trained 4B critic helps frontier models, while outperforming the same model critiquing.

  3. The author also addressed a potential concern: is the gain just from using a different model? The authors train the same Gemma model with their recipe and compare it with the untrained version.

Critical weaknesses

1.The theorems in the paper are standard results, and none of their quantities is measured later in the results. Essentially the theorems describe adding evidence during a decision, while AEA restarts the workflow after reading the whole trace. All the compared critics see the same trace, so the theory can’t explain why they perform differently. Untrained Gemma finds the right agent about as often as AEA-4B, which suggests the gain may come from better output or better prompts rather than from attribution. Therefore the evidence supports a narrower claim than what the paper currently suggests.

  1. There is little direct evidence for the role idea. Role adherence is measured in only one workflow, by an LLM judge, with no error bars. The examples in the paper are fixes to math and physics mistakes, not the role failures used as motivation.

  2. Statistical robustness of the reported numbers is uncertain. Most benchmarks are tiny (AIME - 30 problems, Physics - 34 problems, Chemistry - 41 problems), so many per-cell differences in Table 1 amount to one or two problems averaged over three seeds.

Questions for authors
  1. How many tasks per seed are in each table, by my calculation for 3 seeds an average can only take certain values which is not consistent with the current reported numbers.
  2. How much of the gain comes from naming the right agent? For example, what happens if the new prompt goes to a random agent, or if you count only runs where both critics give valid output?
  3. Can you compare against AgenTracer, and explain how AEA differs from it and from COPPER?
  4. Can you add a baseline that uses the same compute as AEA?
4 comments
AuthorsOpenReview ↗

Response to Reviewer rxoi (Part 1/2)

We thank Reviewer rxoi for recognizing the importance of the problem, the practical design that pinpoints the agent at fault and repairs only that agent, and the fair setting in which every method gets one restart and every critic sees the same trace. We respond to each weakness and each question below.

W1: Theory, restarts and attribution.

  1. Role of the theorems. Each theorem states a property that evidence needs in order to separate roles or lower action error, and AEA is designed to supply such evidence, so the theorems serve as design principles for AEA. We build on standard results so that each principle is stated precisely. Roles with similar priors and likelihoods keep close posteriors (Theorem 3.2), so AEA changes what an agent conditions on. Feedback recomputed from an agent's own context leaves the Bayes-optimal error unchanged (Corollary 4.2), so AEA reads the turns of every agent. The premise behind these principles, an agent acting outside its assigned role, can be counted directly in the first runs of our workflows (Sec. 5.6), and the revised Sec. 4.3 states how each principle is instantiated in AEA.
  2. Restart versus evidence during a decision. AEA adds its evidence before each decision of the identified agent in the restarted run. The completed first run precedes all of these decisions (Sec. 4.3), and the revised system prompt carries the evidence into each of them.
  3. Critics that see the same trace. The analysis concerns the information the evidence carries about the role-optimal action, not the trace the critic reads. Two critics that read the same trace can still write evidence of different quality, since the quality of the extraction function determines which signals reach the agent (Sec. 4.1). The table below shows this under the protocol of Table 17, where every evidence model reads the same trace and returns the same schema.
Valid evidenceOrchestrator-workerSequentialTotal
Gemma critic51/6052/60103/120
AEA59/6059/60118/120

AEA returns valid evidence more often in both workflows, and its 12.5-point margin over the critic has a paired 95% CI of [6.7, 19.2]. The revised App. K adds this split as Table R4.

  1. Attribution versus output and prompt quality. AEA writes the attribution and the replacement prompt as one output. On attribution, it names the responsible agent on 37 of the 44 SWE-bench traces of TraceElephant, which share no task source or agent system with training. The Gemma critic names it on 35, and the three general-purpose models of Table R3 at most 34. On the complete output, AEA returns valid evidence in more runs than the critic, as the table above shows.

W2: Direct evidence for the role idea.

  1. Role failures. Role deviations are frequent and can be counted directly from the trace, as the table below shows for the first runs of both workflows.
WorkflowInstructionDeviationFirst runs95% CI
Orchestrator-workerorchestrator: concise planfull boxed solution30/60[37.7, 62.3]%
Sequentialreporter: minimum explanationwrites until the token limit12/60[11.8, 31.8]%
  1. One workflow and an LLM judge. These counts come from the two workflow structures of Tables 16 and 17, and they need no judge, because a boxed solution and a token-limit stop are read directly from the trace.
  2. Error bars. The table gives the 95% Wilson interval of each count, and the revised Sec. 5.6 reports both counts with their intervals.
  3. Examples. The revision adds a case study of an orchestrator that solves the problem instead of planning (App. J.3), in which this deviation propagates to the final answer.

W3: Statistical robustness. The comparisons we add are paired on the same runs or repeated on the same traces. On the 120 correction runs of Table 17, AEA returns valid evidence 12.5 points more often than the Gemma critic, with a paired 95% CI of [6.7, 19.2]. On TraceElephant, each general-purpose model was queried three times under the protocol of Table 14, and AEA stays ahead of each of them in every query.

Correct agentAll 85 traces44 SWE-bench traces
AEA6037
Gemma critic5935
GPT-5.557 / 58 / 5831 / 32 / 32
Claude Sonnet 4.655 / 57 / 5732 / 34 / 34
Gemini 3.1 Pro44 / 48 / 4322 / 24 / 22

AEA and the Gemma critic use greedy decoding.

AuthorsOpenReview ↗

Response to Reviewer rxoi (Part 2/2)

Q1: Tasks per seed.

  1. Tasks in each table. Each benchmark uses one fixed task list, as listed below.
BenchmarkTasksTables
HumanEval1641, 4
DataBench2571, 4
Chemistry411, 4
Physics341
AIME24301, 4
AIME25301
AIME26301, 2
HLE2,5002
TraceElephant85 traces14
AIME24/25, two workflows120 runs16, 17
  1. Attribution counts. The W3 table reports attribution as trace counts, with one count per query for each general-purpose model, and the revised App. K lists them all (Table R3).

Q2: How much of the gain comes from naming the right agent. AEA both names the responsible agent and writes the replacement system prompt for it. It names the responsible agent on 60 of the 85 TraceElephant traces against 59 for the Gemma critic (Table R3). In Table 17, it returns valid evidence in 118 of 120 runs against 103 for the critic, a paired difference of 12.5 points with a 95% CI of [6.7, 19.2]. The revised Sec. 5.5 reports this comparison.

Q3: AgenTracer and COPPER. The revision compares the designs of both methods with AEA in Sec. 2 and App. G (Table R2).

  1. Comparison with AgenTracer. Both methods attribute a failure to an agent and use the attribution to correct the workflow. AgenTracer trains its tracer on agent and step labels and feeds its reasoning back to failed runs over three further rounds. AEA also writes the replacement system prompt of the identified agent and applies it in a single restart. Table 17 compares AEA with the Gemma critic, which reads the same trace and returns the same schema, since the AgenTracer-8B checkpoint is not public. Under the one-restart protocol, AEA returns valid evidence in 118 of 120 runs. The critic returns it in 103.
  2. Differences from COPPER. COPPER writes a reflection for every agent from each failed trial and its task reward, over up to five trials. AEA names one agent and needs no task reward, which no method receives in Table 17.

Q4: A baseline with the same compute. The Gemma critic uses about the same compute as AEA, and Naive Retry runs the same restart without an evidence call.

  1. The evidence step. The Gemma critic matches the evidence step of AEA. It reads the same trace, returns the same evidence schema and spends about the same number of tokens (433.9K against 430.8K over 120 runs, Table 17), and both methods restart the workflow once. At this matched cost, AEA returns valid evidence in 118 of the 120 runs and the critic in 103.

  2. Total compute. Naive Retry matches the cost of the rest of the episode, since it runs the same full restart and makes no evidence call. Beyond this shared restart, AEA adds one evidence call, as the per-task split below shows, with CaptainAgent counted after its shared team-building step.

Tokens per taskFirst runEvidence callRestartTotal
Setting of Table 1721.9K3.6K22.1K47.6K
CaptainAgent, GPT-5.512.7K3.1K12.1K27.9K

The evidence call takes under 8% of an AEA episode in the setting of Table 17 and about 11% with CaptainAgent and GPT-5.5, and the revised App. E.4 reports these per-task costs.

We hope these answers address your questions, and we would be happy to provide any further clarification.

Dear Reviewer rxoi,

Thank you again for your comments. As the discussion phase ends in one day (November 18), we would like to ask whether our responses have addressed your concerns. If there are any remaining questions, we would be happy to respond.

If you find our responses satisfactory, we would appreciate your consideration in updating the evaluation.

Sincerely,
Authors

Thanks for the clarifications and additional results. However, I am still unclear on how much the gains are due to correct attribution vs better prompts or valid outputs, so i am maintaining my original score.

Reviewer_Hr64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "agentic misalignment," meaning agents in orchestrator-generated multi-agent workflows deviating from their assigned roles, and models it as Bayesian inference over latent role types, with posterior-stability, Fano-type, and linear-Gaussian contraction results. Its method, Agentic Evidence Attribution (AEA), reads a completed trace, names the responsible agent and step, rewrites that agent's system prompt, and restarts the workflow once. The evidence comes either from the workflow model itself (Self-Reflection) or from AEA-4B (Weak-to-Strong), a Qwen3-4B model trained with GRPO on 1,531 human-annotated traces from GAIA, AssistantBench, LiveBench, and the Who&When Algorithm-Generated split. Across HumanEval, DataBench, SciBench, AIME24/25/26, and HLE, the paper reports that AEA-4B improves accuracy, beats Naive Retry and a Gemma critic under a matched one-restart budget, and gives the best failure-attribution accuracy on Who&When and TraceElephant.

Critical strengths

The case studies are honest. They show feedback that criticizes a correct statement yet still yields the right answer (Chemistry), and Self-Reflection regressions.

Critical weaknesses

The paper states that AEA-4B is trained on the Algorithm-Generated split of Who&When (126 traces, App. K) and evaluated on the Hand-Crafted split, which is "fully held out." To my knowledge, the Who&When paper reports 126 Algorithm-Generated and 58 Hand-Crafted failure logs. Could you explain what exactly the number of evaluation traces; per-seed counts for every row;, whether the baseline rows were re-run or taken from prior work, and on which split.

The theoretical results are textbook. They are a TV-perturbation bound on Bayes posteriors, Fano's inequality, the statement that zero conditional mutual information leaves the Bayes posterior unchanged, and a conjugate Gaussian update. None of them is tested or instantiated.

The method is close to AgenTracer (Zhang et al., 2025a), which trains a failure tracer and uses its feedback for correction, and to COPPER (Bo et al., 2024), which trains a reflector that produces role-specific reflections. Neither is compared experimentally. A head-to-head comparison against AgenTracer under the same one-restart protocol is the most important missing baseline.

Questions for authors

How many traces does the Table 3 evaluation contain? Please give per-seed correct counts for every row. Why are 32 of 37 entries exact multiples of 1/126, the size of your Algorithm-Generated training split? Were the baseline rows re-run on Hand-Crafted, or taken from another source?

Do the training traces from GAIA and AssistantBench share task queries with the Who&When Hand-Crafted split or with TraceElephant? Please report query-level exact and near-duplicate overlap, and attribution accuracy on the non-overlapping subset.

Do workflow agents have web access? If so, are dataset mirrors (HF, GitHub) and answer sites blocked during both the original run and the restart?

On what data were the reward weights, the number of epochs, the AEA prompt, and the checkpoint chosen? Were AIME26 and HLE run only after all choices were frozen?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Hr64 (Part 1/2)

We thank Reviewer Hr64 for recognizing that our case studies are honest, and we appreciate the careful reading and detailed questions. We respond to each weakness and each question below.

W1: Evaluation split and baseline sources. Table 3 reports the Hand-Crafted split of Who&When. The baseline rows are not taken from prior work. Agent and Step Accuracy are exact-match rates for the annotated agent name and error-step index, as in Sec. 5.5, under the All at Once and Step by Step protocols of Zhang et al. (2025b).

W2: Role of the theory. We would like to clarify that the theorems serve as design principles for AEA. They build on standard results to state precisely what evidence needs in order to separate roles and lower action error. AEA is designed to supply such evidence. Roles that receive similar priors and likelihoods keep close posteriors (Theorem 3.2), which is why AEA changes the evidence an agent conditions on. Fano's inequality (Theorem 4.1) ties the remaining uncertainty about the role-optimal action to a lower bound on action error, so this evidence must carry information about that action. Feedback recomputed from an agent's own context leaves the Bayes-optimal error unchanged (Corollary 4.2), which is why AEA reads the turns of every agent, including those after the error. Evidence adds precision only along the directions it observes (Theorem 4.3), so AEA writes a role-specific revision. The revised Sec. 4.3 maps these results to three design choices of AEA. The role deviation modeled in our analysis can also be observed directly. In 30 of 60 first runs of our orchestrator-worker workflow, the orchestrator writes a full boxed solution instead of the concise plan it is asked for (Sec. 5.6).

W3: AgenTracer and COPPER. AgenTracer and COPPER are the methods closest to AEA, and the revision compares the designs of the three methods in Sec. 2 and App. G (Table R2). AgenTracer showed that counterfactual replay and fault injection can label agentic failures automatically. COPPER showed that counterfactual rewards for each agent's reflection can train a shared reflector to write role-specific reflections. Under a one-restart protocol, the key difference from AgenTracer is that AEA also writes the replacement system prompt of the identified agent. AgenTracer feeds the reasoning of its tracer back to failed runs over three further rounds, while AEA restarts the workflow once with only that prompt replaced. Because the AgenTracer-8B checkpoint is not public, Table 17 compares AEA with the Gemma critic under the same one-restart protocol. The critic reads the same trace and returns the same schema, so only the evidence model differs. The table below splits this one-restart comparison by workflow.

Valid evidenceOrchestrator-workerSequentialTotal
Gemma critic51/6052/60103/120
AEA59/6059/60118/120

AEA returns valid evidence more often in both workflows, and the paired 95% CI of its 12.5-point margin over the critic is [6.7, 19.2]. The revised App. K adds this split as Table R4. COPPER writes its reflections from each failed trial and its task reward, so a one-restart comparison would either give COPPER a signal that no other method receives or remove the signal its design relies on. AEA needs no reward at test time, while COPPER is built for the complementary setting in which the environment reports a reward after each trial.

AuthorsOpenReview ↗

Response to Reviewer Hr64 (Part 2/2)

Q1: Table 3 evaluation and baseline rows.

  1. Split and protocol. Table 3 reports the Hand-Crafted split of Who&When under the All at Once and Step by Step protocols of Zhang et al. (2025b). Table R3 adds the attribution counts of AEA, the Gemma critic and the three general-purpose models of Table 14 on all 85 TraceElephant traces and on their 44 SWE-bench traces. Under the protocol of Table 14, AEA names the responsible agent more often than each of the three general-purpose models in each of their queries.
  2. Baseline rows. The baseline rows are not taken from prior work.

Q2: Query overlap.

  1. Accuracy on traces with no shared task source or agent system. The 44 SWE-bench traces of TraceElephant are SWE-agent runs on 44 distinct SWE-bench tasks from 10 repositories, and no training source of AEA contains a SWE-bench task or a SWE-agent trace. On them, AEA names the responsible agent on 37 under the protocol of Table 14, more than any baseline below in any query. On all 85 TraceElephant traces, it names the responsible agent on 60 (Table 14).
  2. Shared queries. The training traces from GAIA and AssistantBench come from the same benchmarks as the questions of the Hand-Crafted split and of the 41 GAIA and AssistantBench traces of TraceElephant.
  3. Exact and near-duplicate overlap. The 44 SWE-bench traces come from a task source that no training source shares, so this subset needs no exact or near-duplicate threshold.
Correct agentAll 85 traces44 SWE-bench traces
AEA6037
Gemma critic5935
GPT-5.557 / 58 / 5831 / 32 / 32
Claude Sonnet 4.655 / 57 / 5732 / 34 / 34
Gemini 3.1 Pro44 / 48 / 4322 / 24 / 22

AEA and the Gemma critic use greedy decoding. Under the protocol of Table 14, each general-purpose model was queried three times, and its first query gives the value in that table. The revised Sec. 5.1 and App. K add this subset and the table (Table R3).

Q3: Web access.

  1. Web access. No. In every run reported in Tables 1 and 2, HLE included, the workflow agents have no network access, in both the first run and the restart. The revised Sec. 5.2 states this setting.
  2. Mirrors and answer sites. Because the agents have no network access, dataset mirrors such as Hugging Face and GitHub and answer sites are out of reach in both runs.

Q4: Selection and freezing.

  1. Data used for each choice. The reward weights, the number of epochs, the AEA prompt and the checkpoint selection rule were fixed before training, as part of the training recipe of App. D. Training uses no validation split, and the reported model is the final checkpoint. The revised App. D states both points.
  2. AIME26 and HLE. Yes. AIME26 and HLE were evaluated only after the evidence model, its prompt and its checkpoint were fixed.

We hope these answers address your questions, and we would be happy to provide any further clarification.

Dear Reviewer Hr64,

Thank you again for your comments. As the discussion phase ends in one day (November 18), we would like to ask whether our responses have addressed your concerns. If there are any remaining questions, we would be happy to respond.

If you find our responses satisfactory, we would appreciate your consideration in updating the evaluation.

Sincerely,
Authors

Reviewer_CPz4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies Agentic misalignment in multi-agent workflows, specifically, agents can act inconsistently with assigned roles and intended human goals when solving complex problems. To study these behaviors, the authors use Bayesian inference to model the multi-agent collaboration process. Then the authors propose an alignment framework called Agentic Evidence Attribution (AEA), which can identify the effect of each step and revise the agent’s prompt. The authors evaluate their methods on eight benchmarks to verify their method. The experiment results show that the proposed method can improve the accuracy on these benchmarks.

Critical strengths
  1. The research topic of how to improve multi-agent workflow reliability and task accuracy is meaningful and interesting.
  2. The authors verify their method on eight benchmarks. Who&When and TraceElephant benchmarks are also used for failure attribution.
  3. The authors also investigate the coordination behaviors of the proposed method. Experiments in Table 4 show that AEA improves the role action accuracy and reduces functional overlap.
Critical weaknesses
  1. The paper does not make comparisons with other relevant methods in agentic failure attribution, for example, AgenTracer and COPPER.
  2. There is a gap between theoretical analysis and practical application. In Section 3.1, the authors claim that little separation between role posteriors causes the agents to have similar role beliefs (Agentic Misalignment). However, the proposed method does not explicitly estimate the role posterior.
  3. Limited implementation details are provided in the main paper. Section 4.3 provides only a high-level description of the two instantiations. How the AEA revises the prompt remains unclear.
  4. The proposed method consumes more tokens than the traditional workflow. According to Appendix E.4, the initial GPT-4o workflow only uses approximately 11,400 tokens, while the proposed method uses more than 23,500 tokens.
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CPz4 (Part 1/2)

We thank Reviewer CPz4 for finding the topic of multi-agent workflow reliability meaningful and interesting, and for recognizing our evaluation on eight benchmarks, the failure attribution on Who&When and TraceElephant, and the study of coordination behavior. We answer each weakness below and then the presentation concern.

W1: Comparison with AgenTracer and COPPER. AgenTracer and COPPER are closely related to AEA. Following your suggestion, the revision adds a design comparison with both methods in Sec. 2 and App. G (Table R2). AgenTracer showed that counterfactual replay and fault injection can label agentic failures automatically, and COPPER showed that counterfactual rewards can train role-specific reflections. The table below gives the comparison.

AEAAgenTracerCOPPER
Outputagent, step, revised promptagent, step, reasoninga reflection per agent
Inputfull tracefailed tracefailed trial, task reward
Correctionone restartthree more roundsup to five trials
Rewardagent, rating, correction validity, reasoningagent, stepcounterfactual gain
Labelshuman-verifiedcounterfactual replay, fault injectioncounterfactual rollouts
BackboneQwen3-4BQwen3-8BLongChat-7B

Under our one-restart protocol, the key difference from AgenTracer is that AEA also writes the replacement system prompt of the identified agent. Because the AgenTracer-8B checkpoint is not public, Table 17 compares AEA under this protocol with the Gemma critic, which reads the same trace and returns the same schema, so only the evidence model differs. The table below splits this comparison by workflow.

Valid evidenceOrchestrator-workerSequentialTotal
Gemma critic51/6052/60103/120
AEA59/6059/60118/120

AEA returns valid evidence more often in both workflows, with a paired margin of 12.5 points over the critic (95% CI [6.7, 19.2]), while spending slightly fewer evidence tokens. The revised App. K adds this split as Table R4. On failure attribution, AEA names the responsible agent on 60 of the 85 TraceElephant traces and on 37 of the 44 SWE-bench traces among them (Table R3). Under the protocol of Table 14, this is more than GPT-5.5, Claude Sonnet 4.6 and Gemini 3.1 Pro reach in any of three queries per model. Compared with COPPER, AEA corrects a workflow from its completed trace, with no reward at test time. COPPER is built for the complementary setting in which the environment reports a reward after each trial. It writes its reflections from each failed trial and this reward, a signal withheld from every method in Table 17.

W2: Connection between the analysis and AEA. We would like to clarify that AEA does not need to estimate the role posterior, because in our analysis the posterior changes only through the evidence an agent conditions on. Acting through this evidence also lets AEA correct workflows built on proprietary models, since it needs only their prompts and traces. Theorem 3.2 shows that roles with similar priors and likelihoods keep close posteriors, so separating them requires changing what the agents condition on. Corollary 4.2 shows that feedback recomputed from an agent's own context leaves the Bayes-optimal action error unchanged, so evidence that lowers this error must carry information about the role-optimal action that is absent from that context. AEA is designed to supply this evidence. It reads the turns of every agent, including those after the error, and places its evidence in the system prompt of the identified agent, so every decision of that role in the restarted run conditions on it. The revised Sec. 4.3 states this mapping. The role deviations modeled in our analysis are also frequent in practice. In 30 of 60 first runs of our orchestrator-worker workflow, the orchestrator writes a full boxed solution instead of the concise plan it is asked for (Sec. 5.6).

AuthorsOpenReview ↗

Response to Reviewer CPz4 (Part 2/2)

W3: How AEA revises the prompt. AEA revises the prompt by writing a complete replacement system prompt for the agent it identifies, and the workflow restarts once with this prompt. The revision adds this procedure to Sec. 4.3 of the main paper as Algorithm 1. The details below follow the setting of Table 17, and the revised App. H.4 gives the full prompt.

  1. The workflow runs once with its original prompts, and the complete trace is recorded.
  2. The evidence model reads the task, the role prompts and every turn of every agent. Each turn is truncated to at most 2,000 characters, and to fewer when a trace has more than nine turns, by keeping its beginning and end. It never sees the reference answer or any correctness signal.
  3. It returns structured evidence with a collaboration rating, the responsible agent, the step of that agent's critical mistake, a reason and a complete replacement system prompt for that agent. The evidence is accepted only if it can be parsed, names a workflow agent, points to one of that agent's turns and contains a non-empty prompt.
  4. The replacement prompt takes the place of the identified agent's system prompt. All other prompts and the workflow structure stay unchanged, and the whole workflow restarts once with no turns carried over. If the evidence is not accepted, the task restarts once with the original prompts and is scored like every other task.

Algorithm 1 covers both instantiations of Sec. 4.3. In the Weak-to-Strong instantiation, the evidence model above writes this evidence. In the Self-Reflection instantiation, the workflow model itself writes it.

W4: Token cost. The App. E.4 count you cite covers a complete AEA episode, which adds the evidence call and the restart to the first run. Under our one-restart protocol, every correction method receives this restart, Naive Retry included. The table below splits the tokens of an AEA episode per task in two settings, with CaptainAgent counted after its shared team-building step.

Tokens per taskFirst runEvidence callRestartTotal
Setting of Table 1721.9K3.6K22.1K47.6K
CaptainAgent, GPT-5.512.7K3.1K12.1K27.9K

In both settings, the restart accounts for most of the added tokens, and the evidence call takes under 8% of the episode in the setting of Table 17 and about 11% with CaptainAgent and GPT-5.5. In the setting of Table 17, the evidence of AEA costs slightly fewer tokens than that of the Gemma critic, 430.8K against 433.9K over 120 runs. Under the protocol of Table 14, the table below compares the cost of attributing the 85 TraceElephant traces, with one entry per query for each general-purpose model.

AttributionCorrect agentOutput tokens
AEA6010.6K
GPT-5.557 / 58 / 5815.7K / 15.6K / 15.8K
Claude Sonnet 4.655 / 57 / 5730.5K / 30.2K / 30.1K
Gemini 3.1 Pro44 / 48 / 4316.8K / 16.5K / 16.6K

AEA names the responsible agent more often than every query of the three general-purpose models, with 10.6K output tokens against 15.6K to 30.5K. The revised App. E.4 reports these costs.

Presentation. Thank you for this feedback. To make the analysis easier to follow, we have revised Secs. 3 and 4. The revision adds a short overview of the problem formulation before the analysis (Sec. 3.1), a notation table (Table R1 in App. C.7), a paragraph that maps the results to three design choices of AEA (Sec. 4.3), and Algorithm 1.

If any other part of the paper remains unclear, we would be grateful for any pointers and will revise it further. We hope these answers address your concerns.

Dear Reviewer CPz4,

Thank you again for your comments. As the discussion phase ends in one day (November 18), we would like to ask whether our responses have addressed your concerns. If there are any remaining questions, we would be happy to respond.

If you find our responses satisfactory, we would appreciate your consideration in updating the evaluation.

Sincerely,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

Thank you for your insightful and constructive feedback. We appreciate your recognition of the importance of reliable multi-agent workflows (Reviewers CPz4, rxoi) and of the practical design that pinpoints the agent at fault and repairs only that agent (Reviewer rxoi). You also noted the honest case studies (Reviewer Hr64) and the failure-attribution evaluation on Who&When and TraceElephant (Reviewer CPz4). To answer your questions, we report the results summarized below.

AnalysisResult
Attribution on 44 SWE-bench traces outside trainingAEA 37, general-purpose models at most 34
Repeated queries of three general-purpose modelsAEA ahead of each model in every query
Valid evidence from the same trace and schemaAEA 118/120, Gemma critic 103/120
Output tokens for attribution on 85 tracesAEA 10.6K, general-purpose models 15.6K to 30.5K
Evidence call within an AEA episodeunder 8% of tokens in the setting of Table 17, about 11% with CaptainAgent and GPT-5.5

Attribution without overlap with training. Under the protocol of Table 14, AEA names the responsible agent on 37 of the 44 SWE-bench traces of TraceElephant, which share neither a task source nor an agent system with any training source of AEA. GPT-5.5, Claude Sonnet 4.6 and Gemini 3.1 Pro, each queried three times under the same protocol, name it on at most 34 of these traces. On all 85 traces, AEA reaches 60 and these models at most 58 (Table R3).

Evidence that the workflow can use. AEA returns valid evidence in 118 of 120 runs, against 103 for the Gemma critic, which reads the same trace and returns the same schema (Table 17). It leads in both workflow structures, 59 against 51 and 59 against 52, and the paired 95% CI of its 12.5-point margin is [6.7, 19.2]. Beyond the restart that every correction method receives under our one-restart protocol, AEA adds one evidence call of 3,590 tokens, slightly fewer than the critic spends.

Role of the analysis. The theorems identify the kind of evidence that separates roles and lowers action error, and AEA is built to supply it. Evidence that lowers the Bayes-optimal error must carry information absent from the agent's own context (Corollary 4.2), so AEA reads the turns of every agent and writes its evidence into the system prompt of the agent it names. The revised Sec. 4.3 maps these results to three design choices of AEA. The deviations from an assigned role that our analysis models are frequent in practice. Read directly from the traces, they occur in 30 of 60 first runs of the orchestrator. The reporter deviates in 12 of 60 first runs (Sec. 5.6).

AgenTracer and COPPER. These are the methods closest to AEA, and the revision compares the designs of the three methods in Sec. 2 and App. G (Table R2). AEA writes a replacement prompt for the agent it names and applies it in one restart, while AgenTracer feeds the reasoning of its tracer back over three further rounds. Under the same one-restart protocol, Table 17 compares AEA with the Gemma critic, since the AgenTracer-8B checkpoint is not public. The critic reads the same trace and returns the same schema, so the only difference is the evidence model. COPPER writes its reflections from each failed trial and its task reward, a signal withheld from every method in Table 17.

Revised paper. We have uploaded a revised paper in which added text is marked in dark red. Secs. 2 to 4 add the comparison with AgenTracer and COPPER, an overview of the analysis, a paragraph that maps the results to three design choices of AEA, and the correction procedure as Algorithm 1, with a notation table in App. C.7 (Table R1). Sec. 5 introduces the 44 SWE-bench traces that share no task source or agent system with training (Sec. 5.1) and states that the workflow agents have no network access in any run reported in Tables 1 and 2 (Sec. 5.2). It also adds the comparison of valid evidence (Sec. 5.5, with the split by workflow in Table R4 of App. K) and the role-deviation counts (Sec. 5.6). The appendix states that the training recipe was fixed before training (App. D) and adds per-task costs (App. E.4), the shared attribution prompt (App. H.4), a role-deviation case study (App. J.3) and the repeated TraceElephant queries (App. K). Table numbers from the submission are unchanged, and the added tables are numbered from R1.

We hope these results address your concerns, and we would be happy to provide any further clarification.

Thank you again for your time and consideration.

Sincerely,
Authors of Submission 28852

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.