acceptodds
Under review as a conference paper at ICLR 2027

From Eval Rollout To Process Supervision: Measuring And Curating LLM Agent Trajectories At Scale

Abstract

Coarse pass/fail evaluation can rank long-horizon LLM agents, but it provides little insight into how they solve tasks or why they fail. Two agents with the same outcome may exhibit fundamentally different processes: one failure may result from a minor omission near task completion, whereas another may stem from unfocused exploration, incorrect localization, or ineffective recovery. Likewise, successful trajectories are not equally valuable for training. A clean and well-grounded solution should not be treated as equivalent to one that succeeds through blind retries, editing before inspection, evaluator misuse, mechanical brute force, or excessive resource consumption. This raises a central question: which process-level signals should be used to identify trajectories with the greatest training value? Addressing this question is challenging because long-horizon coding tasks rarely admit a unique correct trajectory, while their final outcomes are often only partially verifiable. We introduce FGA, a fine-grained framework for evaluating LLM-agent trajectories beyond outcome verification. FGA combines automatically observable behavioral signals—such as tool-use sequences, read–edit order, retry dynamics, verification behavior, and resource consumption—with context-sensitive semantic process patterns, including grounded exploration, effective recovery, redundant churn, false verification, and premature convergence. The resulting evaluator provides interpretable assessments at both the step and trajectory levels, distinguishing clean success from wasteful or unreliable success, and productive failure from uninformative failure. We further study whether these process assessments can improve trajectory selection and weighting for supervised fine-tuning and reinforcement learning relative to outcome-only, rule-based, and generic LLM-judge baselines. By shifting evaluation from final correctness alone to evidence-grounded process quality, FGA aims to support agents that are not only more successful, but also more efficient, rigorous, and robust.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evaluation, Benchmarking, Datasets & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_u1t9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FGA, a framework for measuring the process quality of LLM-agent trajectories beyond final task outcomes. It normalizes heterogeneous logs into a common trajectory representation and organizes 58 rule-based and semantic measurements into seven constructs. The empirical study examines detector reliability, metric redundancy, cross-source behavior, and process differences across approximately 126,000 rollout records. A downstream SFT experiment reports 14.4% success on SWE-bench Verified using Gold-v1, compared with 9.8% using a randomly selected subset, while explicitly acknowledging that the selection recipe combines outcome and process filtering.

Critical strengths
  1. The paper addresses an important evaluation problem: trajectories with identical final outcomes can differ substantially in execution quality and potential supervision value.
  2. The framework offers practical utility for systematic and auditable trajectory analysis.
Critical weaknesses
  1. The validation is insufficient to establish that the proposed measurements reliably capture process quality.\textbf{The validation is insufficient to establish that the proposed measurements reliably capture process quality.} Human validation covers only four rule-metric families, with 50 cases each, and only Deadlock/Polling passes the specified precision–recall thresholds. The 14-metric semantic reliability study uses just 20 trajectories and three Claude raters; agreement among related model judges does not establish agreement with human judgments. Consequently, findings such as the 23.0% detector-trigger rate among successful trajectories remain difficult to interpret as meaningful quality differences. The paper acknowledges this distinction, but a measurement-focused contribution still requires stronger independent validation of its central constructs.
  2. The downstream experiment does not establish the additional training value of process-based selection.\textbf{The downstream experiment does not establish the additional training value of process-based selection.} Gold-v1 filters for successful trajectories as well as process criteria, whereas Random samples from the broader trainable universe. Without a success-only baseline, the improvement could primarily reflect outcome filtering. Gold-v1 also contains 6.0% more dataset tokens, and the actual training exposure is not reported separately across conditions. Furthermore, the single training seed leaves optimization variability unmeasured, despite the reported significance across evaluation tasks. The central question of whether process information improves supervision after controlling for outcomes remains unanswered.
  3. The additional scientific and practical value over existing trajectory-analysis frameworks remains insufficiently established.\textbf{The additional scientific and practical value over existing trajectory-analysis frameworks remains insufficiently established.} ACT*ONOMY [1] and AgentLens [2] already support behavioral profiling and process-level trajectory assessment. FGA contributes a broader measurement audit, but does not clearly demonstrate which additional diagnostic insights or optimization decisions this enables. In particular, Gold-v1 predates the current audit, leaving the connection between the measurement findings and downstream data-selection decisions untested.
  4. The measurement pipeline is insufficiently specified for independent reproduction.\textbf{The measurement pipeline is insufficiently specified for independent reproduction.} Table 7 primarily provides brief metric descriptions, while complete semantic-judge prompts, scoring anchors, and normalization procedures are not specified. Several analyses also use historical exports and rubric variants without a complete mapping to the current measurement implementation. Appendix M explicitly acknowledges that the relationship between the tightened-rubric snapshot and the baseline reliability sweep is not fully documented. Given the demonstrated sensitivity to evaluation protocols, these omissions impede reproduction and comparison of the reported measurements.

[1] Gao J, Sun K, Huang J, et al. How to Interpret Agent Behavior[J]. arXiv preprint arXiv:2605.13625, 2026.

[2] Sahoo P, Mittal G, Li X, et al. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation[J]. arXiv preprint arXiv:2605.12925, 2026.

Questions for authors
  1. How were the 20 trajectories in the judge-reliability study and the 50 cases per rule family selected?
  2. Which semantic judges can access the final verifier result or external evaluation score, and which are explicitly blinded to this information?
Reviewer_6zyiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In this paper, authors proposed FGA, introducing seven dimensions of agent trajectory quality and measure them to distinguish the hidden information such as wasted work, repeated errors, verification, and use of feedback. It addresses the limitation of outcome-only evaluation since the process of agent can differ even in the same success/failures. The authors proposed 7 construct concepts (Task Progress, Execution Validity, Recovery, Verification, Efficiency, Integrity, Interaction Quality), and 58 executable measurements including 31 deterministic rule and 27 semantic measurements. The experimental results show that automatic computation metrics are useful, and several measurements reveal behavioral differences that outcome success failed to capture.

Critical strengths
  • This paper conducts extensive experiments and reports the results thoroughly and transparently. The experimental design is fine-grained, and the validity of the measurements is treated with great care.
Critical weaknesses
  • Most of metrics are not verified. Only deadlock/polling passes the human gates, only single seed used in downstream experiments, epochs/exposure is not reported in Table 3, etc. The paper is written like analysis report, rather than a research paper, so main contributions is not clearly shown in the paper.

  • Most of texts and figures seem to be generated by AI, and still need human refinement. A lot of texts in figures are overlapped and not clearly deliver the contents.

Questions for authors
  • How does FGA differ from prior reference-free trajectory evaluation [1, 2] beyond scale?

[1] Kim, Takyoung, et al. "Pipa: A unified evaluation protocol for diagnosing interactive planning agents." arXiv e-prints (2025): arXiv-2505.

[2] Kim, Wonjoong, et al. "Beyond the final answer: Evaluating the reasoning trajectories of tool-augmented agents." arXiv preprint arXiv:2510.02837 (2025).

Reviewer_vGnXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FGA, a framework for evaluating LLM-agent trajectories beyond final outcomes. FGA normalizes logs from ten sources and computes 58 process measurements across seven dimensions using deterministic rules and LLM judges. The authors analyze about 126K trajectories. A historical trajectory-selection recipe reaches 14.4% on SWE-bench Verified, compared with 9.8% for a trajectory-count-matched random subset, but this comparison does not isolate the value of the process criteria.

Critical strengths

The paper asks an important question. Final outcome labels hide meaningful differences in recovery, verification, and wasted work. This matters for both agent evaluation and training-data selection.

FGA retains event-level evidence for each measurement and tests the reliability of some measurements. The negative detector and judge results are useful because they identify signals that should not yet be trusted.

Critical weaknesses

The main Section 4 findings rely on measurements that were not directly validated. Self-Eval Calibration and Stage-Wise Error Type do not appear in the reliability tables. The 79 self-evaluation conflicts and late-stage error trends come from one Claude judge. The separate judge study uses only 20 trajectories and has no human reference labels. These results need human validation before they can be interpreted as behavioral rates.

Gold-v1 does not isolate the value of process quality. Gold-v1 contains only passed trajectories, while Random is sampled from the full pool. The paper does not report the passed fraction in Random. Gold-v1 also contains 6% more tokens, training exposure is not reported by arm, and the experiment uses one seed. A passed-only, token- and length-matched random baseline is needed to answer the paper’s central question.

Questions for authors

Can the authors provide human validation for Self-Eval Calibration and Stage-Wise Error Type? Which judge configurations produced the different Evidence-to-Action-Gap ICC values in Tables 12, 13, and 41?

What fraction of Random and Full passed the task verifier? Can the authors provide a passed-only baseline matched to Gold-v1 by task, tokens, and trajectory length? How were the Gold-v1 thresholds selected?

What distinguishes FGA from ProcBench, P2T, SWE-Prime, and AgentLens? Will the authors release the rules, judge prompts, adapters, and validation annotations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.