acceptodds
Under review as a conference paper at ICLR 2027

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Abstract

Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent failure diagnosis, execution trace analysis, multi-agent systems, corpus-level diagnostics, agent evaluation, harness optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_svwoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Insights Generator (IG), a multi-agent system designed for corpus-level trace diagnostics of LLM agents. By decoupling hypothesis generation from corpus-scale statistical validation and leveraging a stateful Python data layer, IG overcomes the limitations of single-trajectory debugging and production scale. Experiments demonstrate that IG outperforms existing competing systems in blind LLM-judge evaluations, and when human experts or automated coding agents utilize IG insight reports to refine agent scaffolds, they achieve significant and stable performance gains on downstream benchmarks.

Critical strengths
  1. The idea of the paper is very clear and valuable. This paper studies the problem of corpus-level trace diagnostics for LLM agents and this task is valuable for current agent research.
  2. The design of the agent is clear. By decoupling hypothesis generation from corpus-scale statistical validation, it addresses key industry pain points that traditional single-trajectory debugging and manual inspection fail to resolve, such as detecting macro population patterns and silent failures.
  3. The experiments are relatively comprehensive, covering three benchmarks, and the results are also good.
Critical weaknesses
  1. Insufficient contribution in methodology: From an academic research perspective, the core components of the system (IG) primarily represent a combination and application of common design patterns and multi-agent workflows in existing LLM engineering. It does not introduce novel foundational learning algorithms, architectural breakthroughs, or generalizable theoretical proofs. Overall, it leans more toward a engineering work rather than a traditional research paper, and I would suggest that it is better suited for an industry track.
  2. Insufficient baselines. The current baseline comparison lacks rigor and breadth. The evaluation primarily relies on self-implemented variants or minor modifications of existing agent frameworks. Furthermore, the evaluation does not include lightweight or traditional heuristic baselines (e.g., rule-based clustering or statistical text analysis). Without these comparisons, it remains unclear whether the high cost and complexity of the proposed multi-agent architecture (e.g., heavy LLM calls, sandbox environments, and dynamic schema caches) provide genuine algorithmic advantages over simpler, more resource-efficient alternatives, or if the performance gains are merely driven by compute and token budgets.
  3. Minor issue: The paper formatting and visual presentation leave room for improvement. Key diagrams, such as the system architecture (Figure 1) and evaluation framework (Figure 2), are overly simple. Visual polish and clearer structural organization would significantly enhance the readability of the manuscript.
Questions for authors

Please explain why CC is not performing well in this task?

2 comments
AuthorsOpenReview ↗

Response to Reviewer svwo Weaknesses

Dear Reviewer svwo,
Thank you for your review and for the suggestions on baselines and figures, both of which we've acted on in the revision.

W1 - Methodological contribution

The core components… primarily represent a combination and application of common design patterns and multi-agent workflows… better suited for an industry track.

Response: We believe understanding how agents fail is a research problem, not only an industry one. Agents are now a research object across ML, from benchmarks (SWE-bench, Jimenez et al., 2024; HLE, Phan et al., 2025) to RL environments (SWE-Gym, Pan et al., 2025) and automated agent design (ADAS, Hu et al., 2025; VeRO, Ursekar et al., 2026), and improving any of them starts with knowing its failure modes. IG doesn't depend on the application, and we apply it unchanged to three unrelated benchmarks.
The paper makes two contributions: a method for generating insights, and a way to evaluate any such method. Existing benchmarks score trace analysis per trajectory (TRAIL, Who&When) or score insights against planted ground truth (InsightBench). Corpus-level trace analysis has neither, so we evaluate reports four ways, crossing who evaluates (an LLM judge or human experts) with what is measured (report quality or downstream impact). To our knowledge, no prior work measures trace-analysis reports by how much they help someone improve the agent.
On the method, IG proposes hypotheses on sampled traces and validates them over the full corpus. The pattern has precedent in describing differences between datasets (D5, Zhong et al., 2023, and VibeCheck, Dunlap et al., 2025), which the revision now cites. Our contribution is bringing it to agent corpora, where other systems work per trajectory (Trace2Skill), recursively over snippets (HALO) or by clustering transcripts (Docent). Dispatching subagents raises IG's pairwise win rate from 29.9% to 65.6%, and typing them as Scouts and Investigators raises it to 76.0% (Table 1). IG also keeps its lead on two open-weight backbones (Appendix A.10) and over simpler statistical baselines (W2 below). The revised introduction and Related Work (Sections 1 and 2) reflect this.

W2 - Lightweight and traditional baselines

The evaluation does not include lightweight or traditional heuristic baselines.

Response: We built four lightweight baselines from the same traces IG reads, on SSB, HLE and APEX-Agents, and describe them fully in Appendix A.9 (Table 9).

  • Methods. Methods (1) and (2) use no LLM. Method (1) applies statistical fault localization (Liblit et al., 2005) to trace events, ranking tool calls, error messages and n-grams by how much more often they occur in failing traces. Method (2) runs significance tests on 40-60 hand-derived trace features per benchmark, finds subgroups where failures concentrate and clusters similar traces. Methods (3) and (4) each write a report with a single LLM call. Method (3) writes from 20 traces retrieved by BM25 keyword search on the failure terms that method (1) found, and method (4) writes up the outputs of methods (1) and (2).
  • Judging. Each baseline report was judged pairwise against every existing report under Opus 4.6 and GPT-5.4, with the same prompts and citation audit.
  • Results. None of the four wins a round against IG (0 of 48), and they score far below IG on the rubric too. Win rates are averaged over the two judges, and rubric scores (0-3, Opus 4.6) are against each benchmark's frozen answer key.
SystemLLM useCost per reportWin rate, SSBWin rate, HLEWin rate, APEX-AgentsRubric, SSBRubric, APEX-Agents
IGMulti-agent\17SSB,17 SSB, \\71 HLE, \$174 APEX78.685.794.21.331.65
CC SubagentsMulti-agent$3875.078.688.51.321.26
(4) Statistics + LLM write-upOne call$0.13-0.2523.219.640.40.620.56
(3) Retrieved traces + LLM write-upOne call$0.23-0.2814.316.126.90.500.59
(2) Combined statisticsNone\0.11SSB,0.11 SSB, \\0.04 HLE, \$0.05 APEX7.17.13.80.090.25
(1) LiblitNoneNegligible0.00.03.80.020.09

The statistical methods do find real patterns, and on SSB all of them rediscover IG's top finding. However, unlike IG, they do not provide an explanation of why a pattern causes failure or a suggested fix, which is a shortcoming documented by the judge.

W3 - Figures

Key diagrams… are overly simple.

Response: We've redesigned both figures in the revision. Figure 1 now shows how the Scout and Investigator subagents work over a shared trace table, from sampled hypotheses to corpus-wide validation, and Figure 2 lays out the four evaluation settings as a grid of who evaluates (LLM judge or human experts) against what is measured (report quality or downstream impact).

AuthorsOpenReview ↗

Response to Reviewer svwo Questions

Q1 - Why CC does not perform well

Please explain why CC is not performing well in this task?

Response: We read "CC" as CC Subagents, our strongest baseline. Its reports score well under the judge, and practitioners improved the SSB scaffold with them, though by less than with IG's reports (Table 2). We also cover Single-Agent Coding, the plain Claude Code baseline, in case that is what you meant.

  • CC Subagents. Its reports are close to IG's in quality. It runs IG's own Scout and Investigator roles as Claude Code subagents, and both the judges and the expert raters score its reports close to IG's (Tables 1 and 13). Downstream, IG's reports help more. In the human study, IG's report led to nearly twice the SSB improvement (Table 2), and the patcher loop points the same way on both benchmarks (Table 3). The reports differ in kind. IG's SSB report finds a harness bug with a direct fix (the scaffold runs only the first code block in a reply), while CC's strengths are causal analysis and negative findings, which tell a practitioner what not to change. We think this comes from the architecture. CC's subagents each run in an isolated context and return a short summary, so findings are only cross-checked in the orchestrator's reasoning, while IG's Investigators re-test each finding against the whole corpus through a shared trace table.
  • Single-Agent Coding. One Claude Code session has to read, test and write up the whole corpus alone, and it ranks last or next to last on every benchmark (win rate 16.7% on SSB, 12.5% on HLE and 0% on APEX-Agents, against 70.3%, 81.8% and 88.9% for IG; rubric composite 0.67 against 1.41, Table 1). Its reports have fewer findings (6, against 7–10 for IG) and far fewer corpus-wide counts, and when it loses, the judges cite weaker quantification and depth. Its citations are real (all 84 traces it cites on APEX-Agents exist), but an entailment check finds they support its findings less often (0.72 and 0.50 under Opus 4.6 and GPT-5.4, against 0.80 and 0.73 for IG). A single context has to hold both the reading and the testing, so the agent goes deep on a few traces and rarely measures how often a pattern recurs, while IG's Scouts propose hypotheses from sampled traces and Investigators test each one against the full corpus. A natural follow-up is to rerun this comparison to measure how much of the gap a stronger model in the latest Claude Code harness closes.
Reviewer_SnAhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Insights Generator (IG), a multi-agent system that decouples hypothesis generation from hypothesis verification. IG processes trajectories through a structured Python data layer, avoiding the need to feed entire lengthy raw trajectories into the LLM’s context.The paper develops a four-part evaluation framework covering both report quality and downstream practical utility: LLM-judge-based pairwise comparisons and rubric-based scoring, expert ratings of individual findings, intervention experiments in which human engineers read reports and improve agent scaffolds, and automated patching experiments.Experiments are conducted on three benchmarks: SSB, HLE, and APEX-Agents. In the LLM-judge evaluation, IG achieves an average pairwise win rate of 80.5%, outperforming all baselines. Human engineers using IG reports improve the SSB task pass rate by 30.4 percentage points, compared with 16.2 percentage points for the strongest baseline, CC-Subagents. In automated patching experiments, IG achieves the largest held-out gains on the more challenging APEX-Agents benchmark.

Critical strengths

1.The paper presents a well-motivated system for a practically important problem. Identifying aggregate behavioral patterns, particularly silent failures, across a trajectory corpus addresses an important need in agent development. By separating hypothesis generation from verification and processing trajectories through a structured Python data layer, IG offers a reasonable approach to reducing unsupported conclusions and handling long trajectories. This gives the paper a clear systems contribution and practical relevance.

2.The evaluation connects report quality to downstream utility. Human-engineer interventions and automated patching experiments examine whether the generated insights help improve agent scaffolds, supported by a comprehensive set of ablations. This evaluation design provides more direct evidence of practical value and is an important contribution that could inform subsequent work in this area.

Critical weaknesses

1.The central evidence for downstream utility is not yet sufficiently robust. The human intervention studies include only six participants per arm on SSB and eight per arm on APEX-Agents. Although the SSB result is statistically significant (p = 0.016), the APEX-Agents result (p = 0.053) does not meet the conventional 0.05 threshold. These small samples limit confidence in the consistency and generalizability of the gains. The absence of a statistical retrieval baseline that does not use LLM-generated insights also leaves the incremental value over simpler methods, and the dependence on LLM-judge preferences, insufficiently established. This weakens confidence in the paper’s central utility claim.

  1. Cost and reliability limit the system’s usefulness in resource-constrained settings. An average run costs approximately $44 and can reach $174 on APEX-Agents, while IG’s downstream performance advantage disappears under common budget constraints of $10–50. The system primarily uses Claude-Opus-4.6, without evidence that its architectural benefits persist with weaker, less expensive open-source backbones. Citation validity also declines under constrained budgets, while the system relies mainly on post hoc auditing and lacks an internal mechanism to reject or flag low-confidence conclusions. Together, these issues substantially weaken its practical deployment value and materially affect my recommendation.
Questions for authors
  1. Given the small sample sizes in the human intervention studies, have you considered expanding these studies or supplementing them with evaluations using agents that simulate human participants?
  2. The system relies heavily on Claude-Opus-4.6. Could you provide results using open-source backbones, such as Qwen or Llama? To what extent do the observed gains come from the IG architecture versus the capabilities of the underlying model?
  3. IG loses its performance advantage under budget constraints of $10–50. What approaches could make the system cost-effective in practical settings with limited resources?
  4. The validity of trajectory citations declines substantially under constrained budgets. What mechanisms could be incorporated into IG to detect and suppress low-confidence findings before they appear in the final report?
  5. Why were the Trace2Skill and Single-Agent-Coding baselines excluded from the APEX-Agents experiments?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SnAh Weaknesses

Dear Reviewer SnAh,
Thank you for the detailed and constructive review. Your questions on baselines, backbones and low-confidence findings each became a new experiment and appendix in the revision.

W1 - Downstream utility, statistical baselines, LLM-judge dependence

The human intervention studies include only six participants per arm on SSB and eight per arm on APEX-Agents… The absence of a statistical retrieval baseline… and the dependence on LLM-judge preferences, insufficiently established.

Response:

  • Small Human Studies. With limited human participants, we use the automated patcher loop to add intervention runs beyond the human study, across two complete SSB trials and one APEX-Agents trial (Table 3, all arms in Table 14). IG's reports lead to the strongest results, first under validation selection in the second SSB trial (0.879) and with the highest best-test score on APEX-Agents (42 of 144).
  • A Baseline without LLM-Generated Insights. We built four baselines from the same traces IG reads, on SSB, HLE and APEX-Agents, and describe them fully in Appendix A.9.
    • Methods. Methods (1) and (2) use no LLM. Method (1) applies statistical fault localization (Liblit et al., 2005) to trace events, ranking tool calls, error messages and n-grams by how much more often they occur in failing traces. Method (2) runs significance tests on 40-60 hand-derived trace features per benchmark, finds subgroups where failures concentrate and clusters similar traces. Methods (3) and (4) each write a report with a single LLM call. Method (3) writes from 20 traces retrieved by BM25 keyword search on the failure terms that method (1) found, and method (4) writes up the outputs of methods (1) and (2).
    • Judging. Each baseline report was judged pairwise against every existing report under Opus 4.6 and GPT-5.4, with the same prompts and citation audit.
    • Results. None of the four baselines wins a round against IG (0 of 48), and all four score far below IG on the rubric. The single-call write-ups (3 and 4) do better than the no-LLM methods (1 and 2). Full results are in the table below and Table 9 (win rates averaged over the two judges, rubric scores against each benchmark's frozen answer key, 0-3). Table 1 now includes the strongest baseline (statistics + LLM write-up), at a rubric composite of 0.60 against 1.41 for IG.
SystemLLM useCost per reportWin rate, SSBWin rate, HLEWin rate, APEX-AgentsRubric, SSBRubric, APEX-Agents
IGMulti-agent\17SSB,17 SSB, \\71 HLE, \$174 APEX78.685.794.21.331.65
CC SubagentsMulti-agent$3875.078.688.51.321.26
(4) Statistics + LLM write-upOne call$0.13-0.2523.219.640.40.620.56
(3) Retrieved traces + LLM write-upOne call$0.23-0.2814.316.126.90.500.59
(2) Combined statisticsNone\0.11SSB,0.11 SSB, \\0.04 HLE, \$0.05 APEX7.17.13.80.090.25
(1) LiblitNoneNegligible0.00.03.80.020.09
  • LLM Judge Results. The judges rank the new baselines low for substantive reasons. Methods (3) and (4) are written by the same LLM that writes IG’s reports, so a judge that simply preferred LLM-written prose would rank them close to IG. Instead, both judge families place all four baselines below every agent-based system except Single-Agent Coding, and their rationales cite missing mechanisms and fixes rather than style. The two judges also agree with each other (Spearman 0.99 on SSB, 0.95 on HLE and 0.77 on APEX-Agents, Table 18). The judge prompts and rubrics are in Appendix A.7, and we’ll release every stored verdict.

W2 - Cost, open-source backbones, low-confidence findings

IG’s downstream performance advantage disappears under common budget constraints of $10–50… without evidence that its architectural benefits persist with… open-source backbones… lacks an internal mechanism to reject or flag low-confidence conclusions.

Response:

  • Cost and Open-Source Backbones. IG runs well on cheap open-weight backbones. On GLM-5.3 Flash it cuts cost by about 50% on SSB and 90% on APEX-Agents compared with Opus 4.6 system's. Details are in Q2 and Q3 below.
  • Low-Confidence Findings. An entailment check costing under $1 per report flags one to three findings per budget-limited report (Q4 below).
AuthorsOpenReview ↗

Response to Reviewer SnAh Questions 1-2

Q1 - Larger studies or agents that simulate participants

Have you considered expanding these studies or supplementing them with evaluations using agents that simulate human participants?

Response: Yes. The automated patcher loop (Section 4.4) is this kind of evaluation. A Claude Code patcher takes the practitioner's place, improving the scaffold from a report over several rounds.
On SSB, every arm improves on the unmodified scaffold, and IG is at the top under validation selection in both patcher trials, tied at 0.80 in the paper's runs and first at 0.879 in a second trial (Table 3, all arms in Table 14). On APEX-Agents, IG reaches the highest best-test score (42 of 144). Reports from a cheaper model also hold up downstream (Q2 below).

Q2 - Open-source backbones

Could you provide results using open-source backbones, such as Qwen or Llama? To what extent do the observed gains come from the IG architecture versus the capabilities of the underlying model?

Response: We ran IG and the four Table 1 baselines (CC Subagents, Trace2Skill, Single-Agent Coding and RLM) on two open-weight models with a 1M-token context, DeepSeek V4 Flash and GLM-5.3 Flash, on all three benchmarks. Every report was judged with the paper's pairwise judges against the other systems on its backbone and the four statistical baselines. The table compares IG with the two strongest baselines on each backbone. Each cell is the win rate (%) over 16 games, averaged over the Opus 4.6 and GPT-5.4 judges, and rows are sorted by the average over the three benchmarks. The full results are in Table 10 (Appendix A.10), and Table 1 in the revision also scores IG on each backbone against the same four Opus 4.6 baselines.

BackboneSystemSSBHLEAPEX-AgentsAverage
GLM-5.3 FlashIG97949796.0
Opus 4.6IG84919188.7
DeepSeek V4 FlashIG91948188.7
Opus 4.6CC Subagents88888888.0
GLM-5.3 FlashTrace2Skill78887881.3
DeepSeek V4 FlashTrace2Skill84787880.0
GLM-5.3 FlashCC Subagents81758480.0
DeepSeek V4 FlashCC Subagents63849179.3
Opus 4.6Trace2Skill72847577.0
  • Architecture versus Backbone. On every backbone, IG ranks first or ties for first in all but two cells, DeepSeek V4 Flash on APEX-Agents and Opus 4.6 on SSB. In pairwise games, no baseline wins both rounds against IG on the same backbone.
  • Backbone Strength. Cheaper models can produce the strongest reports. IG on GLM-5.3 Flash wins every pairwise game against the Opus 4.6 systems on all three benchmarks under both judges, IG on Opus 4.6 included. GLM-5.3 Flash lifts the baselines run on it too, so the backbone matters, and IG still leads on that backbone. IG on DeepSeek V4 Flash lands in the middle of the Opus pool.
  • Downstream Results. GLM-5.3 Flash reports are also useful downstream. We ran the SSB patcher loop with IG reports from each open model in place of Opus 4.6 reports. Under validation selection, IG reports from GLM-5.3 Flash reach 0.828, between our two Opus 4.6 trials (0.80 and 0.879), and from DeepSeek V4 Flash 0.798. On APEX-Agents, over four patcher rounds, IG reports from GLM-5.3 Flash reach 34 of 144 test tasks at their best round, above the unmodified scaffold (30.8), and DeepSeek V4 Flash reports reach 27.

Note: We tried Qwen and Llama first, but neither could run IG reliably. Qwen3.8-2.4T-A95B's open release has a 262k-token context, which IG's orchestrator exceeded, and Llama 4 Maverick doesn't reliably call tools (Appendix A.10).

AuthorsOpenReview ↗

Response to Reviewer SnAh Questions 3-5

Q3 - Cost-effective approaches

What approaches could make the system cost-effective in practical settings with limited resources?

Response: Two approaches make IG cheaper without giving up quality.

  • Cheaper Backbone. Open-weight backbones cut IG's cost by about 50% on SSB and 90% on APEX-Agents, where traces are longest. For example, an APEX-Agents report costs \19onGLM−5.3Flashagainst19 on GLM-5.3 Flash against \\174 on Opus 4.6 (Appendix A.10). The cheaper reports keep their quality (Q2 above has both backbones in full):
    • LLM judge. GLM-5.3 Flash reports are judged better than every Opus 4.6 system's, under both judges.
    • Patcher loop. On SSB they land between our two Opus 4.6 trials under validation selection, and on APEX-Agents they improve on the unmodified scaffold (34 against 30.8 of 144).
  • Checking Capped Reports. Instead of rerunning a budget-limited report with more budget, a reader can check it. The entailment check in Q4 below costs under $1 per report and flags one to three findings per capped report.

Q4 - Suppressing low-confidence findings

What mechanisms could be incorporated into IG to detect and suppress low-confidence findings before they appear in the final report?

Response: We ran this as an experiment. An LLM entailment check (Opus 4.6) reads up to three of a finding's cited traces and rates the finding as supported, partly supported or not supported. We applied it to IG's APEX-Agents reports at each budget tier from Table 18 and removed every finding rated not supported (Table 11, Appendix A.11). Win rates are over the 10-system APEX-Agents pool (20 games), averaged over the Opus 4.6 and GPT-5.4 judges. The uncapped report is the pool's own IG report, so its row is over the other 9 systems (18 games).

IG budget capFindingsNot supported (removed)Win rate, unfilteredWin rate, filtered
$107122.512.5
$259242.520.0
$509352.545.0
Uncapped10191.783.3

The check catches what you describe at low cost, under $1 per report. Removing flagged findings lowers the judged win rate at every tier, and the judges prefer the unfiltered report in 13 of 16 head-to-head rounds. We see this as an observation about how judged quality is measured rather than a drawback of filtering, since the judges reward the number of findings even when the prompt marks some as unsupported.
We'll add the check to IG, so each finding carries its support rating and unsupported findings can be flagged or dropped.

Q5 - Trace2Skill and Single-Agent Coding on APEX

Why were the Trace2Skill and Single-Agent-Coding baselines excluded from the APEX-Agents experiments?

Response: This was a gap in our original APEX-Agents evaluation, and we've now addressed it. We ran both on APEX-Agents and re-judged all 10 systems together under both judges. Under both judges, IG ranks first, Trace2Skill lands in the middle and Single-Agent Coding is last (Tables 1 and 8).
These results replace the 8-system APEX-Agents column in Tables 1, 7, and 8.

Reviewer_brAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a multi-agent system that separates hypothesis discovery on sampled traces from corpus-level investigation, using a persistent Python data layer. It evaluates diagnostic reports through model judges, expert ratings, practitioner interventions, and automated scaffold improvement.

Critical strengths

The problem is practically important, and evaluating downstream engineering outcomes is more compelling than relying exclusively on report-quality judgments. The randomized, source-blinded practitioner study is the strongest evidence: on SSB, IG recipients achieve 57.4% held-out performance versus 43.2% for CC Subagents recipients. Cross-family judge agreement provides an additional robustness check.

Critical weaknesses

The human evidence generalizes narrowly. Every participant within an arm receives the same report, so the experiment establishes the utility of particular reports rather than variability across report generations. Samples are small, the APEX comparison has p=0.053p=0.053, and expert insight ratings are essentially tied with low inter-rater reliability. Claims of consistently superior diagnostic quality should therefore be moderated.

Automated improvement is evaluated with optimistic selection. Reporting the best test score across rounds introduces selection bias; IG also receives more opportunities for selection than competing systems. The appendix acknowledges that the APEX significance disappears after multiplicity correction and that baseline variance excludes patcher stochasticity. Validation-selected checkpoints and repeated complete optimization runs are needed.

The distinctive architecture’s contribution remains uncertain. Generic subagents recover most of the judged-quality improvement. Under matched budget caps, downstream results do not clearly separate from the unmodified scaffold, while most component-ablation differences remain within single-trial noise. This weakens attribution of the gains specifically to the scout–investigator design.

Questions for authors

Do the downstream rankings persist with validation-selected checkpoints, equal total budgets, and multiple independently generated reports per intervention arm?

2 comments
AuthorsOpenReview ↗

Response to Reviewer brAu Weaknesses

Dear Reviewer brAu,
Thank you for the careful review, and in particular for pushing on checkpoint selection and how far the human results generalize.

W1 - Human evidence generalizes narrowly

Every participant within an arm receives the same report… Samples are small… expert insight ratings are essentially tied.

Response: We've stated the claim more precisely in the revision, and each point below adds new analysis.

  • One Report per Arm. Each human intervention study measures how useful one particular report was, rather than how much reports vary from run to run, and the revision makes this explicit (Section 4.2 and Limitations). We address variation between reports on the LLM-judge side, where the SSB and HLE win rates average three independently generated reports per system.
  • Addressing Small Samples. With limited human participants, we use the automated patcher loop to add intervention runs beyond the human study. Additional patcher results are in our response to W2. The revision notes the human-study sample sizes as a limitation.
  • Expert Ratings. We recomputed rater agreement across all 12 raters. Agreement between individual raters is low (Krippendorff's α = 0.10), but the 12-rater average for each insight, which Table 13 reports, is moderately reliable (intraclass correlation coefficient ICC(2,k) = 0.60). The revision replaces the two-rater kappa with these figures (Section 4.3).

W2 - Optimistic checkpoint selection

Reporting the best test score across rounds introduces selection bias… Validation-selected checkpoints… are needed.

Response: We now report the validation-selected round for every arm alongside the best test round, and treat the two benchmarks differently by how well validation predicts test.
SSB - Validation Selection. On SSB, validation and test pass rates track each other closely across rounds (Pearson r = 0.91 in the paper's loops, r = 0.93 in a second patcher trial we ran for every arm), so the validation-selected round is the main SSB result (Table 3, all arms in Table 14). IG is at the top in both trials, tied with the strongest arm at 0.80 in the paper's loops and first at 0.879 in the second trial. IG ran more rounds than the other arms, but restricted to its first three rounds (the fewest any arm ran), validation picks the same scores.

ArmPaper runs, val-selectedPaper runs, best testSecond trial, val-selectedSecond trial, best test
IG0.800.840.8790.879
Raw traces, no report--0.8280.828
RLM0.790.840.8180.838
CC Subagents0.760.810.8180.838
Trace2Skill--0.8080.828
Pure patcher0.800.800.7580.758
Unmodified scaffold0.41-0.440.41-0.44--

APEX-Agents - Best Test Round as an Upper Bound. On APEX-Agents, validation predicts test only weakly (r = 0.25), so we report the best test round as an upper bound alongside the validation-selected one. IG's reports give the best result, 42 of 144 tasks against 32-39 for the other arms, and IG reaches it in round 4, within the four rounds CC Subagents and RLM each ran.

ArmBest test (upper bound)Val-selectedRounds run
IG42287
Raw traces, no report39392
Pure patcher35353
RLM32304
CC Subagents32234
Unmodified scaffold30.830.8-

W3 - Contribution of the architecture

Generic subagents recover most of the judged-quality improvement…

Response: The comparison with generic subagents supports the typed design once the baselines are laid out.

  • Typed Subagent Clarification. Our strongest baseline, CC Subagents, already uses IG's typed roles, dispatching IG's Scout and Investigator as Claude Code subagents. Under the LLM judge, IG’s gain over a single orchestrator comes in two steps (SSB/HLE win rate, Table 1). Dispatching subagents raises it from 29.9% (Orchestrator-only) to 65.6% (Generic Subagents), and typing them as Scouts and Investigators raises it to 76.0% (full IG). Head to head, IG beats Generic Subagents in 14 of 18 SSB rounds and in both APEX-Agents rounds, and HLE is close to even.
  • Downstream Results. We also ran the component ablations through the patcher loop, on SSB in the second patcher trial and on APEX-Agents at a matched $50 cap (Table 17). Every ablation falls below full IG on both benchmarks (table below). Each ablation is a single run.
AblationSSB, val-selectedSSB, best testAPEX-Agents, tasks of 144 ($50 cap, best test)
Full IG0.8790.87934
Generic subagents0.8380.83831
Orchestrator only0.8280.84832
No LLM extraction0.8380.85919
AuthorsOpenReview ↗

Response to Reviewer brAu Questions

Q1 - Validation-selected checkpoints, equal budgets, several reports per arm

Do the downstream rankings persist with validation-selected checkpoints, equal total budgets, and multiple independently generated reports per intervention arm?

Response:

  • Validation-Selected Checkpoints. Yes on SSB, where IG is first or tied for first under validation selection in both patcher trials (0.80 and 0.879). On APEX-Agents, where validation predicts test only weakly, IG has the best upper bound (42 of 144). Both selection rules for every arm are in W2 above and Table 14.
  • Equal Budgets. At matched caps of $10-50, the analysis systems land close together downstream (Table 16), so budget-limited reports don't separate the systems. The judged quality of IG's reports rises steadily with budget (Table 18), and the uncapped report gives the largest downstream gain (42 of 144).
  • Several Reports per Arm. The SSB and HLE win rates in Table 1 already average three independently generated reports per system, and IG's lead isn't driven by one report. On HLE, every IG report outranks every CC Subagents report. Downstream, each patcher round generates a new report from fresh traces.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers: New Experiments and Paper Revisions

Dear Reviewers and Area Chair,
We thank the reviewers for their careful reading. Four concerns came up across reviews, and we ran new experiments for each. The sections below follow the table, and each reviewer's thread has the details.

ConcernNew experimentResult
1. Downstream robustness (brAu, SnAh)Validation-selected patcher rounds, and a second SSB patcher trialIG is first under validation selection on SSB in both trials, and has the best APEX-Agents upper bound
2. Simpler methods, judge dependence (SnAh, svwo)Four statistical baselines, two with no LLMNone wins a round against IG (0 of 48)
3. Backbone and cost (SnAh)IG and all baselines on DeepSeek V4 Flash and GLM-5.3 FlashIG on GLM-5.3 Flash beats every Opus 4.6 system, at 50–90% lower cost
4. Architecture (brAu, svwo)Ablations through the patcher loopFull IG outscores every ablation downstream

1. Downstream Results Beyond the Human Study

  • The automated patcher loop adds intervention runs beyond the human study, and we now report each arm's validation-selected round next to its best test round (Table 3, all arms in Table 14).
  • SSB: validation tracks test closely, so validation selection is the main result, and IG is at the top in both trials.
  • APEX-Agents: validation predicts test only weakly because the task set is small, so we report the best test round as an upper bound. IG's reports give the highest result of any arm (42 of 144, against 32–39).

2. Statistical Baselines and the LLM Judge

  • We added four baselines built from the same traces IG reads (Table 9, Appendix A.9). Two use no LLM (statistical fault localization and a combined statistical analysis), and two write up those statistics or retrieved traces with a single LLM call.
  • None wins a round against IG (0 of 48), and all four score far below IG on each benchmark's answer-key rubric.
  • Judge check: the write-ups use the same LLM as IG, so a judge that rewarded LLM prose would rank them near IG. Instead both judges rank them below every agent-based system except Single-Agent Coding and cite missing mechanisms and fixes, and the two judges agree closely (Table 8).

3. Open-Weight Backbones

  • We ran IG and the four Table 1 baselines on DeepSeek V4 Flash and GLM-5.3 Flash (Table 1, Table 10, Appendix A.10).
  • LLM judge: with the backbone held fixed, IG ranks first or ties for first in 7 of 9 backbone-benchmark cells, so its lead holds across backbones. IG on GLM-5.3 Flash also wins every pairwise game against the Opus 4.6 systems.
  • Downstream: in the patcher loop, IG reports from GLM-5.3 Flash match IG on Opus 4.6 on SSB (0.828 under validation selection, between our two Opus 4.6 trials at 0.80 and 0.879). On APEX-Agents they reach 34 of 144 at the best round, below uncapped IG on Opus 4.6 (42) but comparable to the other Opus 4.6 arms (32–35) and equal to IG on Opus 4.6 at a \25or25 or \\50 cap. DeepSeek V4 Flash reports reach 0.798 on SSB but fall below the unmodified scaffold on APEX-Agents (27 against 30.8).
  • Cost: IG on GLM-5.3 Flash cuts cost by about 50% on SSB and 90% on APEX-Agents (USD 19 against USD 174 per report), so IG's quality doesn't require a frontier-model budget.

4. Architecture Ablations

  • The judged gain comes in two steps. Dispatching subagents raises the SSB/HLE LLM judge win rate from 29.9% to 65.6%, and typing them as Scouts and Investigators raises it to 76.0% (Table 1).
  • Downstream, full IG outscores every ablation in the patcher loop on both benchmarks (Table 17).

Changes to the Paper

Our revision adds:

  • Validation-selected results for every patcher arm and the second SSB trial, with claims scoped to the selection rule that supports them
  • New appendices for the statistical baselines, the open-weight backbones and their costs
  • Table 1 now includes the strongest statistical baseline and IG on each backbone
  • The 10-system APEX-Agents judge column, now with Trace2Skill and Single-Agent Coding
  • The entailment check for low-confidence findings, all-rater reliability statistics and a clearer description of the human-study design
  • A revised introduction and related work, and redesigned Figures 1 and 2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.