- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HALT fits HMMs to agent action traces for online failure attribution and workflow operator discovery, reporting improvements on two attribution benchmarks and four workflow benchmarks.
Critical strengths
- The lightweight attribution method requires no LLM calls at inference and reports competitive accuracy while observing only partial traces, offering potential practical value.
- Evaluation across multiple benchmarks and two workflow backbones provides useful evidence that the approach extends beyond a single task.
Critical weaknesses
- The method ignores tool arguments, so two runs look identical when they use the same tools in the same order, even if different arguments cause one to fail. For agents with only a few tools, such as a bash tool that accepts arbitrary commands or scripts, discarding arguments can remove most of the information needed to understand their actions and identify errors.
- The core idea uses standard HMM methods, and the experiments do not show whether HMMs outperform simpler methods trained on the same data.
Questions for authors
- How well does the method detect errors caused by wrong tool arguments, and does including arguments improve accuracy?
- How does HALT compare with simpler sequence models and LLM baselines given the same training data?
- Does HMM-based operator discovery outperform asking an LLM to extract operators directly from the same traces under the same search budget?