- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
APEX examines an agentic workflow to locate weaknesses in its agents, communication channels, memory, and tools, without requiring an initial failed trajectory. It builds assertions about each unit, tests them through successive rounds of probes, and uses the resulting judgments and execution traces to rank the suspected units and identify their weak aspects. The paper also introduces the AXE execution engine and the AXEBench fault-injection benchmark. Experiments cover coding and question answering, with ablations, human validation, and a downstream repair study.
Critical strengths
-
APEX distinguishes an agent producing poor work from its work being lost through a channel, memory, or tool. That distinction matters for repair: changing an agent's instructions will not fix a tool that discards its edits. AXE's records of context and state changes give the diagnosis evidence beyond agent messages.
-
The agent-only comparison provides evidence that examination helps even when prior methods can name the correct unit. On the main set, APEX reaches 90.5%, compared with 57.1% for all-at-once and 71.4% for its ten-execution version. The latter is a useful control for the extra workflow executions.
-
The evaluation includes human checks of benchmark labels and both judges, alongside a second coding workflow, question answering, and downstream repair. The appendix reports where humans disagree with the judges, rather than presenting agreement alone. This helps qualify the localization and aspect results.
Critical weaknesses
-
The main baselines in Table 1 can only name agents, even when given ten executions. The paper discloses this restriction, but it means the non-agent results are not compared with a simple LLM that reads multiple execution traces and can name any unit. APEX-random and APEX-rule are useful ablations, but neither provides that comparison. This leaves the benefit of the full APEX procedure over simpler diagnosis less clear for channel, memory, and tool faults.
-
The paper and supplementary code use different formulas to decide which assertions need more testing. Equation (6) uses H(α/(α+β)), the entropy of the Beta mean. RubricNode.uncertainty in apex/models.py instead uses H(Pr(p > 0.5 | evidence)), the entropy of the probability that an assertion holds more often than it fails. These formulas can give different rankings, so it matters which one was used in the experiments.
-
The experiments establish fault localization, but not whether APEX can judge that the evidence is insufficient to diagnose a weak unit. All localization tests contain an injected fault, and Appendix C requires the localizer to name at least one weak unit. The healthy runs used for benchmark construction and repair do not test this decision. This leaves a boundary of pre-deployment use untested, without changing the reported accuracy on faulted workflows.
Questions for authors
-
How does APEX compare with a version of all-at-once that receives the workflow specification and can name any agent, channel, memory, or tool, using the same model and ten executions? Please give results for each unit type.
-
Which uncertainty formula was used for probe selection in the experiments: H(α/(α+β)) in Equation (6), or H(Pr(p > 0.5 | evidence)) in RubricNode.uncertainty in the supplementary file apex/models.py? Please explain why the paper and code differ.
-
Is APEX intended only to locate faults in workflows already known to be faulty, or can it also conclude that there is not enough evidence to identify any unit as weak? This distinction matters because Appendix C instructs the localizer to name at least one weak unit.