- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces FGA, a framework for measuring the process quality of LLM-agent trajectories beyond final task outcomes. It normalizes heterogeneous logs into a common trajectory representation and organizes 58 rule-based and semantic measurements into seven constructs. The empirical study examines detector reliability, metric redundancy, cross-source behavior, and process differences across approximately 126,000 rollout records. A downstream SFT experiment reports 14.4% success on SWE-bench Verified using Gold-v1, compared with 9.8% using a randomly selected subset, while explicitly acknowledging that the selection recipe combines outcome and process filtering.
Critical strengths
- The paper addresses an important evaluation problem: trajectories with identical final outcomes can differ substantially in execution quality and potential supervision value.
- The framework offers practical utility for systematic and auditable trajectory analysis.
Critical weaknesses
- Human validation covers only four rule-metric families, with 50 cases each, and only Deadlock/Polling passes the specified precision–recall thresholds. The 14-metric semantic reliability study uses just 20 trajectories and three Claude raters; agreement among related model judges does not establish agreement with human judgments. Consequently, findings such as the 23.0% detector-trigger rate among successful trajectories remain difficult to interpret as meaningful quality differences. The paper acknowledges this distinction, but a measurement-focused contribution still requires stronger independent validation of its central constructs.
- Gold-v1 filters for successful trajectories as well as process criteria, whereas Random samples from the broader trainable universe. Without a success-only baseline, the improvement could primarily reflect outcome filtering. Gold-v1 also contains 6.0% more dataset tokens, and the actual training exposure is not reported separately across conditions. Furthermore, the single training seed leaves optimization variability unmeasured, despite the reported significance across evaluation tasks. The central question of whether process information improves supervision after controlling for outcomes remains unanswered.
- ACT*ONOMY [1] and AgentLens [2] already support behavioral profiling and process-level trajectory assessment. FGA contributes a broader measurement audit, but does not clearly demonstrate which additional diagnostic insights or optimization decisions this enables. In particular, Gold-v1 predates the current audit, leaving the connection between the measurement findings and downstream data-selection decisions untested.
- Table 7 primarily provides brief metric descriptions, while complete semantic-judge prompts, scoring anchors, and normalization procedures are not specified. Several analyses also use historical exports and rubric variants without a complete mapping to the current measurement implementation. Appendix M explicitly acknowledges that the relationship between the tightened-rubric snapshot and the baseline reliability sweep is not fully documented. Given the demonstrated sensitivity to evaluation protocols, these omissions impede reproduction and comparison of the reported measurements.
[1] Gao J, Sun K, Huang J, et al. How to Interpret Agent Behavior[J]. arXiv preprint arXiv:2605.13625, 2026.
[2] Sahoo P, Mittal G, Li X, et al. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation[J]. arXiv preprint arXiv:2605.12925, 2026.
Questions for authors
- How were the 20 trajectories in the judge-reliability study and the 50 cases per rule family selected?
- Which semantic judges can access the final verifier result or external evaluation score, and which are explicitly blinded to this information?