- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Insights Generator (IG), a multi-agent system designed for corpus-level trace diagnostics of LLM agents. By decoupling hypothesis generation from corpus-scale statistical validation and leveraging a stateful Python data layer, IG overcomes the limitations of single-trajectory debugging and production scale. Experiments demonstrate that IG outperforms existing competing systems in blind LLM-judge evaluations, and when human experts or automated coding agents utilize IG insight reports to refine agent scaffolds, they achieve significant and stable performance gains on downstream benchmarks.
Critical strengths
- The idea of the paper is very clear and valuable. This paper studies the problem of corpus-level trace diagnostics for LLM agents and this task is valuable for current agent research.
- The design of the agent is clear. By decoupling hypothesis generation from corpus-scale statistical validation, it addresses key industry pain points that traditional single-trajectory debugging and manual inspection fail to resolve, such as detecting macro population patterns and silent failures.
- The experiments are relatively comprehensive, covering three benchmarks, and the results are also good.
Critical weaknesses
- Insufficient contribution in methodology: From an academic research perspective, the core components of the system (IG) primarily represent a combination and application of common design patterns and multi-agent workflows in existing LLM engineering. It does not introduce novel foundational learning algorithms, architectural breakthroughs, or generalizable theoretical proofs. Overall, it leans more toward a engineering work rather than a traditional research paper, and I would suggest that it is better suited for an industry track.
- Insufficient baselines. The current baseline comparison lacks rigor and breadth. The evaluation primarily relies on self-implemented variants or minor modifications of existing agent frameworks. Furthermore, the evaluation does not include lightweight or traditional heuristic baselines (e.g., rule-based clustering or statistical text analysis). Without these comparisons, it remains unclear whether the high cost and complexity of the proposed multi-agent architecture (e.g., heavy LLM calls, sandbox environments, and dynamic schema caches) provide genuine algorithmic advantages over simpler, more resource-efficient alternatives, or if the performance gains are merely driven by compute and token budgets.
- Minor issue: The paper formatting and visual presentation leave room for improvement. Key diagrams, such as the system architecture (Figure 1) and evaluation framework (Figure 2), are overly simple. Visual polish and clearer structural organization would significantly enhance the readability of the manuscript.
Questions for authors
Please explain why CC is not performing well in this task?
2 comments
Response to Reviewer svwo Questions
Q1 - Why CC does not perform well
Please explain why CC is not performing well in this task?
Response: We read "CC" as CC Subagents, our strongest baseline. Its reports score well under the judge, and practitioners improved the SSB scaffold with them, though by less than with IG's reports (Table 2). We also cover Single-Agent Coding, the plain Claude Code baseline, in case that is what you meant.
- CC Subagents. Its reports are close to IG's in quality. It runs IG's own Scout and Investigator roles as Claude Code subagents, and both the judges and the expert raters score its reports close to IG's (Tables 1 and 13). Downstream, IG's reports help more. In the human study, IG's report led to nearly twice the SSB improvement (Table 2), and the patcher loop points the same way on both benchmarks (Table 3). The reports differ in kind. IG's SSB report finds a harness bug with a direct fix (the scaffold runs only the first code block in a reply), while CC's strengths are causal analysis and negative findings, which tell a practitioner what not to change. We think this comes from the architecture. CC's subagents each run in an isolated context and return a short summary, so findings are only cross-checked in the orchestrator's reasoning, while IG's Investigators re-test each finding against the whole corpus through a shared trace table.
- Single-Agent Coding. One Claude Code session has to read, test and write up the whole corpus alone, and it ranks last or next to last on every benchmark (win rate 16.7% on SSB, 12.5% on HLE and 0% on APEX-Agents, against 70.3%, 81.8% and 88.9% for IG; rubric composite 0.67 against 1.41, Table 1). Its reports have fewer findings (6, against 7–10 for IG) and far fewer corpus-wide counts, and when it loses, the judges cite weaker quantification and depth. Its citations are real (all 84 traces it cites on APEX-Agents exist), but an entailment check finds they support its findings less often (0.72 and 0.50 under Opus 4.6 and GPT-5.4, against 0.80 and 0.73 for IG). A single context has to hold both the reading and the testing, so the agent goes deep on a few traces and rarely measures how often a pattern recurs, while IG's Scouts propose hypotheses from sampled traces and Investigators test each one against the full corpus. A natural follow-up is to rerun this comparison to measure how much of the gap a stronger model in the latest Claude Code harness closes.
Response to Reviewer svwo Weaknesses
Dear Reviewer svwo,
Thank you for your review and for the suggestions on baselines and figures, both of which we've acted on in the revision.
W1 - Methodological contribution
Response: We believe understanding how agents fail is a research problem, not only an industry one. Agents are now a research object across ML, from benchmarks (SWE-bench, Jimenez et al., 2024; HLE, Phan et al., 2025) to RL environments (SWE-Gym, Pan et al., 2025) and automated agent design (ADAS, Hu et al., 2025; VeRO, Ursekar et al., 2026), and improving any of them starts with knowing its failure modes. IG doesn't depend on the application, and we apply it unchanged to three unrelated benchmarks.
The paper makes two contributions: a method for generating insights, and a way to evaluate any such method. Existing benchmarks score trace analysis per trajectory (TRAIL, Who&When) or score insights against planted ground truth (InsightBench). Corpus-level trace analysis has neither, so we evaluate reports four ways, crossing who evaluates (an LLM judge or human experts) with what is measured (report quality or downstream impact). To our knowledge, no prior work measures trace-analysis reports by how much they help someone improve the agent.
On the method, IG proposes hypotheses on sampled traces and validates them over the full corpus. The pattern has precedent in describing differences between datasets (D5, Zhong et al., 2023, and VibeCheck, Dunlap et al., 2025), which the revision now cites. Our contribution is bringing it to agent corpora, where other systems work per trajectory (Trace2Skill), recursively over snippets (HALO) or by clustering transcripts (Docent). Dispatching subagents raises IG's pairwise win rate from 29.9% to 65.6%, and typing them as Scouts and Investigators raises it to 76.0% (Table 1). IG also keeps its lead on two open-weight backbones (Appendix A.10) and over simpler statistical baselines (W2 below). The revised introduction and Related Work (Sections 1 and 2) reflect this.
W2 - Lightweight and traditional baselines
Response: We built four lightweight baselines from the same traces IG reads, on SSB, HLE and APEX-Agents, and describe them fully in Appendix A.9 (Table 9).
The statistical methods do find real patterns, and on SSB all of them rediscover IG's top finding. However, unlike IG, they do not provide an explanation of why a pattern causes failure or a suggested fix, which is a shortcoming documented by the judge.
W3 - Figures
Response: We've redesigned both figures in the revision. Figure 1 now shows how the Scout and Investigator subagents work over a shared trace table, from sampled hypotheses to corpus-wide validation, and Figure 2 lays out the four evaluation settings as a grid of who evaluates (LLM judge or human experts) against what is measured (report quality or downstream impact).