acceptodds
Under review as a conference paper at ICLR 2027

Inferring Hidden Agentic Logic from Traces for Online Fault Attribution and Workflow Optimization

Abstract

Agent execution consists not only of explicit actions at each step, but also of implicit structural shifts in hidden states. Yet the behavioral logic that governs these states and their transitions remains largely opaque. We introduce HALT, a framework for recovering this hidden structure and its cross-run topology from execution traces using hidden Markov models (HMMs). HALT models observable atomic actions as emissions from latent workflow states, while state transitions capture the underlying logical structure of agent behavior. Fitted on historical traces, these HMMs provide a unified view for understanding, evaluating, and improving agents. We evaluate HALT on two downstream applications. For failure attribution, HALT trains separate HMMs on successful and failed traces, and attributes each step by contrasting its likelihood under the two models. This enables real-time, step-level failure localization without requiring access to the complete trace, and naturally supports early stopping. On Who&When, HALT achieves 37.0% step-level attribution accuracy, outperforming a fine-tuned LLM-based attributor that reads the entire trace (32.1%), while requiring no LLM calls and observing only about 27% of the trace. For workflow optimization, existing methods typically rely on manually specified building blocks to construct and refine workflows during search. HALT instead extracts reusable workflow operators from HMMs fitted to historical successful traces, turning latent structural patterns into automatically discovered building blocks for optimization. Workflows found by searching over these HMM-derived operators outperform both hand-designed and automatically searched baselines across all four tasks. On MBPP Pro with GLM-5.1, HALT reaches 86.2%, compared with 73.6% for the best baseline, while also incurring lower total cost. Together, these results position HALT as a lightweight structural primitive for inferring hidden agentic logic from traces and leveraging that structure for both online failure attribution and workflow optimization.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Trajectory Analysis, Failure Attribution, Workflow Optimization, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_poocOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HALT fits HMMs to agent action traces for online failure attribution and workflow operator discovery, reporting improvements on two attribution benchmarks and four workflow benchmarks.

Critical strengths
  • The lightweight attribution method requires no LLM calls at inference and reports competitive accuracy while observing only partial traces, offering potential practical value.
  • Evaluation across multiple benchmarks and two workflow backbones provides useful evidence that the approach extends beyond a single task.
Critical weaknesses
  • The method ignores tool arguments, so two runs look identical when they use the same tools in the same order, even if different arguments cause one to fail. For agents with only a few tools, such as a bash tool that accepts arbitrary commands or scripts, discarding arguments can remove most of the information needed to understand their actions and identify errors.
  • The core idea uses standard HMM methods, and the experiments do not show whether HMMs outperform simpler methods trained on the same data.
Questions for authors
  • How well does the method detect errors caused by wrong tool arguments, and does including arguments improve accuracy?
  • How does HALT compare with simpler sequence models and LLM baselines given the same training data?
  • Does HMM-based operator discovery outperform asking an LLM to extract operators directly from the same traces under the same search budget?
Reviewer_owFWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes HALT, which models agent execution traces with hidden Markov models: each step's atomic action, such as a tool call, is the observation, latent states represent workflow phases, and parameters are estimated with Baum-Welch on historical traces. For failure attribution, HALT fits two HMMs on same-domain successful and failed traces and scores each incoming step by the difference of its k-step conditional log-likelihoods under the two models, raising an alarm once the score crosses a threshold; the procedure uses no LLM calls and reads only the action sequence. For workflow optimization, HALT extracts loops, branches, and ensembles from the transition graph of an HMM fitted on successful traces, has an LLM compile each motif into an operator offline, and runs AFlow-style MCTS search over these automatically induced operators. Experiments cover Who&When and TracerTraj-public for attribution, and HumanEval Pro, MBPP Pro, MATH Lv5, and SimpleQA Verified for optimization, with GLM-5.1 and DeepSeek-V4-Pro as backbones. On Who&When HALT reaches 37.0% step-level and 70.1% agent-level accuracy versus 32.1% and 63.8% for AgenTracer, a fine-tuned attributor that reads the whole trace, while observing about 27% of the trace at zero token cost (Table 3). On all four optimization tasks HALT outperforms AutoGen, ADAS, and AFlow, reaching 86.2% versus 73.6% for the strongest baseline on MBPP Pro with GLM-5.1.

Critical strengths

The headline result is interesting, and it is the main evidence behind my judgment: with zero LLM calls and only the action sequence, HALT performs online failure attribution and still beats a fine-tuned attributor that reads the entire trace. AgenTracer needs the whole trace, 22.24 steps and 3,148 tokens on average, before concluding, while HALT raises its alarm after 5.96 steps on average (Table 3), with step-level accuracy of 42.86% versus 37.30% and agent-level 69.84% versus 63.73% on the Algorithm-Generated subset. If this result holds, the transition structure rather than the surface trace content carries most of the attribution signal. Whether it holds depends on whether the signal is trivial statistics, which is what Weakness 1 questions.

Critical weaknesses
  1. The source of the signal is not isolated. The HMMs may be learning trivial statistics such as position bias: the Algorithm-Generated subset of Who&When averages only 8.7 steps per trace (Table 6), early steps are naturally more suspicious on such short traces, and HALT's advantage on this subset cannot be separated from that bias. The paper lacks simple controls such as unigram/n-gram likelihood ratios that use only marginal statistics, and it never shows the learned transition matrix, so I cannot tell whether the latent states are meaningful workflow phases or positional patterns. The alarm threshold is fixed per dataset at τh=1.2 on Who&When and 0.8 on TracerTraj, and the paper does not explain how these values were chosen, so tuning on the test set cannot be ruled out.

  2. The workflow optimization half is an unequal comparison. The HMM training data are exactly the correct trajectories obtained by running the AFlow framework on same-domain training problems (Appendix F), and none of the baselines, AutoGen, ADAS, or AFlow, receives these traces. The induced operators are also built on AFlow's own Test, ScEnsemble, and MdEnsemble classes (Appendix D), under the same names and form. Under this setup the gains may come from the induced structure or simply from the extra successful-trace data, and the current experiments cannot tell the two apart.

  3. The evaluation raises leakage concerns, and several key comparisons are weakly supported. HumanEval Pro and MBPP Pro are self-invoking extensions built on ordinary HumanEval and MBPP, and the HumanEval and MBPP HMMs are trained on exactly those upstream problems (Appendix F, Table 7), so the stated instance-level isolation, that evaluation instances never appear in the training traces, is not enough. Table 4 reports three-run averages without variance, and with DeepSeek-V4-Pro on HumanEval Pro HALT leads the strongest baseline by only 1.3 points (80.9 versus 79.6), so stability cannot be judged. Figure 4 counts only the cost of running the final workflow on the test set, excluding trace collection and MCTS search cost, so the abstract's "lower total cost" claim is inaccurate.

Questions for authors
  1. [Clarification] The alarm thresholds are fixed per dataset at τh=1.2 and 0.8. On which data were these values determined, and did the test traces play any role in tuning?
  2. [Clarification] Can you show the learned transition matrix and add controls such as unigram/n-gram likelihood ratios that use only marginal statistics, to rule out explanations based on position bias?
  3. [Clarification] In the optimization experiments, if the baselines also received the same successful traces (or HALT did not receive them), how much of the conclusion remains?
  4. [Suggestion] The HumanEval and MBPP HMM training data overlap with the upstream problems of HumanEval Pro and MBPP Pro. Please retrain on unrelated domains or enforce stricter source isolation, and report the change in results.
  5. [Suggestion] Report variance or significance for Table 4. Extend Figure 4 to include trace collection and search cost, or state in the abstract that total cost covers only the final workflow's test-set run.
Reviewer_p4JuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

HALT models agent execution traces with HMM, each trace step is mapping to an action atom, 14-28 atoms per dataset, and a 5 state hmm is fit. The paper maily use LLM for two tasks:

  1. online failure attribution: Searate HMMs are fit on failted and successful same domain traces. Each step is scored by the log ratio of its predictive probability under the two models given the previous 3 steps. and the first step that exceeds a per dataset threshold is flagged; agent - level attribution restricts the window to the acting agent's own steps. On who&when, HALT reports 70.1% agent level and 37% step level accuracy, vs 63.8%/32.1% for the fine-tuned agetracer, with no llm calls and about 27% of each trace observed. It is best in 8 of 9 columns across the who& when and tracerTraj.
  2. workflow optimisation: An HMM is fit on successful Aflow runs from a related training task, its transition graph is pruned, and recurring patterns are extracted. Deepseek v4pro complies each pattern into an Aflow style operator, and A flow tree search builds workflows from the operators. Halt beats autogen, ADAS and AFlow in all 8 mode task cells on humaneval pro, MBPP pro, MATH lv5 and SimleQA verified at lower or comparable test time cost.
Critical strengths
  1. The method has little cost: Existing methods either have an LLM read the whole trace. Halt is just a 5 state HMM: INference costs per step amd uses no LLM tokens (Table 3), and the paper reports that HALT beats the fine tuned agenTracer on Who&When while observing only about 27% of each trace on average.
  2. The two applications are useful.- the same fitted HMM is used for failure attribution and for workflow optimisation, which gives the paper a coherent framing. And the evaluation is broad, it covers two attribution benchmarks and four workflow benchmarks.
  3. The mining operators fro the transition graph is creative: HALT turns recurring high-probability patterns in the learned transition graph into candidate workflow operators. This connects trace modelling to workflow search in a way prior work has not.
Critical weaknesses
  1. Numbers don't add up: The token counts in Table 3 for ProMAs and MASC equal all at oneces' tokens * fraction of steps observved to within one token. Will observed data be that perfectly measured?

  2. There are no simple baselines, so we cannot tell what drives the wins. The mdoel sees only action types, not content. The paper never compares against a position only predictor, a majority agent guess, or a plain n/-gram model on the same atoms. The alarm threshold waas set with no stated protocol, and thus several of the wins may be due to 1-2 traces.

  3. The operator extraction step is underspecified, the extraction function are never deefined, and it is unclear how a markov chain yields rallel ensemble patterns. and the workflow optimisation resued Aflow own hand wirtten components and are tained on A flow's runs. They may overlap with the base problems of the test items, and A2flow and somple direect call baselines are missing. Cost counts only the final rest run.

Questions for authors
  1. For each baseline numbers, was it copied from another paper or rerun, and over how many runs?
  2. simple baselines: how do a random operator, a position only predictor, a majority agent guess, a visible markov model on the same atoms, and full history filtering perform? without these , it is unclear whether the hidden states contribute anything.
  3. do the sucess and failure training traces come from the same harness and model? if not, how do you ruled out the score detecting the srouce ratehr than the failure?
  4. fairness of the workflow comparison: which optimiser llm and serach budget did AFAS, Aflow and HALT each use? what is halt's end to end cost, including trajectory generator and operator sythesis? how does it compare to A2flow, a direct CoT call amd operators proposed by an llm without the HMM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.