acceptodds
Under review as a conference paper at ICLR 2027

APEX: Adaptive Progressive Examination of Agentic Workflows

Abstract

Large Language Model (LLM)-based agentic workflows have shown substantial gains over single agents on complex tasks, but they are harder to investigate when they underperform, since a weakness can lie in any aspect of any of their units: the agents, the communication strategy among them, the memory they share, and the tools through which they act. Existing failure attribution analyzes the trajectory of a run that has already failed and returns a single responsible agent and step, so nothing is learned until a failure has occurred, failures are attributed to agents alone, and the aspect in which the responsible unit is weak remains unidentified. We formulate unit-level weakness localization, which identifies the weak units of a workflow and the aspects in which they are weak before they fail in deployment. We present (1) APEX, which adaptively and progressively explores a workflow's execution trajectories, running probes that target the assertions of a dynamically generated rubric tree about which it is statistically least confident; (2) AXE, an execution engine that records what every unit was given and changed, and can perturb any unit in isolation; and (3) AXEBench, a labeled benchmark of perturbed workflows generated by AXE. On AXEBench, APEX correctly localizes 90.5% of injected weaknesses in agents, against 57.1% for the state of the art, and 51.5% across all units of the workflow, against 14.5%, as prior methods can only target agents; it identifies the weak aspect in 79.0% of the weaknesses it localizes. Guided by APEX, an LLM instructed to repair the workflow recovers more than twice as much of the lost performance as it does without guidance, and human experts validate both the verdicts APEX relies on and the benchmark it is evaluated on. Code and data are publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agentic workflows, multi-agent systems, weakness localization, failure attribution, active evaluation, LLM-as-a-judge, fault injection, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eh4dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

APEX examines an agentic workflow to locate weaknesses in its agents, communication channels, memory, and tools, without requiring an initial failed trajectory. It builds assertions about each unit, tests them through successive rounds of probes, and uses the resulting judgments and execution traces to rank the suspected units and identify their weak aspects. The paper also introduces the AXE execution engine and the AXEBench fault-injection benchmark. Experiments cover coding and question answering, with ablations, human validation, and a downstream repair study.

Critical strengths
  1. APEX distinguishes an agent producing poor work from its work being lost through a channel, memory, or tool. That distinction matters for repair: changing an agent's instructions will not fix a tool that discards its edits. AXE's records of context and state changes give the diagnosis evidence beyond agent messages.

  2. The agent-only comparison provides evidence that examination helps even when prior methods can name the correct unit. On the main set, APEX reaches 90.5%, compared with 57.1% for all-at-once and 71.4% for its ten-execution version. The latter is a useful control for the extra workflow executions.

  3. The evaluation includes human checks of benchmark labels and both judges, alongside a second coding workflow, question answering, and downstream repair. The appendix reports where humans disagree with the judges, rather than presenting agreement alone. This helps qualify the localization and aspect results.

Critical weaknesses
  1. The main baselines in Table 1 can only name agents, even when given ten executions. The paper discloses this restriction, but it means the non-agent results are not compared with a simple LLM that reads multiple execution traces and can name any unit. APEX-random and APEX-rule are useful ablations, but neither provides that comparison. This leaves the benefit of the full APEX procedure over simpler diagnosis less clear for channel, memory, and tool faults.

  2. The paper and supplementary code use different formulas to decide which assertions need more testing. Equation (6) uses H(α/(α+β)), the entropy of the Beta mean. RubricNode.uncertainty in apex/models.py instead uses H(Pr(p > 0.5 | evidence)), the entropy of the probability that an assertion holds more often than it fails. These formulas can give different rankings, so it matters which one was used in the experiments.

  3. The experiments establish fault localization, but not whether APEX can judge that the evidence is insufficient to diagnose a weak unit. All localization tests contain an injected fault, and Appendix C requires the localizer to name at least one weak unit. The healthy runs used for benchmark construction and repair do not test this decision. This leaves a boundary of pre-deployment use untested, without changing the reported accuracy on faulted workflows.

Questions for authors
  1. How does APEX compare with a version of all-at-once that receives the workflow specification and can name any agent, channel, memory, or tool, using the same model and ten executions? Please give results for each unit type.

  2. Which uncertainty formula was used for probe selection in the experiments: H(α/(α+β)) in Equation (6), or H(Pr(p > 0.5 | evidence)) in RubricNode.uncertainty in the supplementary file apex/models.py? Please explain why the paper and code differ.

  3. Is APEX intended only to locate faults in workflows already known to be faulty, or can it also conclude that there is not enough evidence to identify any unit as weak? This distinction matters because Appendix C instructs the localizer to name at least one weak unit.

Reviewer_9UqDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes APEX, a unit-level weakness localization. Unlike traditional agent failure attribution, APEX adpative find which units (agents, channels, shared memory, tools) are weak, and in what aspect, before any failure is observed. The paper also introduces AXE, an execution engine that records what each unit received and changed, and can inject a fault into any single unit. AXEBench uses AXE to build 6,332 perturbed workflow instances from MultiAgentBench and HumanEval tasks. On 165 AXEBench instances, APEX ranks the faulty unit first in 51.5% of cases, against 14.5% for the best Who&When baseline.

Critical strengths
  1. It points out a interesting direction on agent debugging. Previous failure attribution asks who broke one failed run. This paper asks which part of a system is weak across the runs it would produce. AXE extends failure attribution beyond agents into unit-level weakness such as channels, shared memory, tools.

  2. The experiments controls are well chosen. Both models are fixed across methods. Together with human validations of both judegs let me believes the experiments is robust and well controlled.

Critical weaknesses
  1. Stronger baselines are missing In the experiments, the paper only compared with Who&When's three prompting baselines. Stronger baseline are missing, such as AgenTracer, DoVer. I'd like to see more comparision with strong baseline so I can be more convinced.

  2. The localizer encodes the fault model

The localizer prompt (L819–843) contains one-to-one mapping rules from failures to different components, such as:

  • complete agent output that leaves the solution unchanged → TOOL;
  • missing or truncated entries → MEMORY;
  • one pair of units not seeing each other's work → CHANNEL;
  • hollow output → agent.

However, the judge prompts encode the same prompts (L813–814), and the baselines get none of them. With scores alone (APEX-rule), memory accuracy falls from 51.2% to zero. I starts to doubt that the gain compared to baseline may come from this fault-specific knowledge rather than from examination.

  1. The experiments test recovery of known faults instead of discover new failures The paper's claim is that APEX finds weaknesses before they surface. However as what I said in the previous point, every evaluated weakness is one of four injected faults in llm's prompt. No natural weakness is reported.

  2. Please redraw figure 1,2 as it seems like they are ai-generated.

Questions for authors

Please address my concern in weakness.

Reviewer_pFUSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes APEX, which examines executable agentic workflows before deployment to identify weak components and explain how they fail. It builds a component-specific checklist and adaptively chooses new tests based on earlier results. The paper also introduces an execution engine and a benchmark with injected faults. Adaptive testing improves localization over random test selection, but tool failures remain difficult to identify, and the statistical interpretation of the uncertainty scores is not fully justified.

Critical strengths
  • The method looks beyond agent mistakes and can name a faulty communication channel, memory, or tool, together with the behavior that failed. This makes its output more useful for deciding what to repair than a pointer to an erroneous agent step.

  • The comparison with random test selection keeps the execution budget fixed, helping isolate the benefit of adaptive testing. The repair study also examines whether the diagnoses help improve the workflow rather than only matching a fault label.

Critical weaknesses
  • Equation 6 gives a high uncertainty score to an assertion that repeatedly passes and fails equally often, even when this pattern is already well established. This measures inconsistent behavior rather than necessarily a lack of knowledge. Why this score is an appropriate proxy for the information a new test would provide?

  • Table 1 compares overall accuracy with baselines that can only name agents, so they cannot succeed on channel, memory, or tool faults. The agent-only and random-probe comparisons are useful, but the overall gain does not establish an advantage over a localizer allowed to inspect the same executions and name any component.

  • Table 1 and Section 6.2 show that APEX still struggles to identify faulty tools and often blames the agent using them. Distinguishing the component that causes a failure from the component that displays its effects is central to the proposed task, so the evidence for reliable tool-level diagnosis remains weak.

Questions for authors

Please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.