acceptodds
Under review as a conference paper at ICLR 2027

Who Catches What? A Trace-Based Study of Heterogeneous LLM Review and Repair

Abstract

A common safeguard for software-engineering agents is to have a second model review the first model’s work. Existing measurements of this practice, however, come largely from benchmarks and simulated red-teaming. We study it in a production deployment using ten weeks of event logs from KISS Sorcar, where an Anthropic model implemented changes in a working repository and an OpenAI model reviewed them read-only under a budget cap. The corpus contains 486 tasks and 1.1M events, including tool calls, model switches, and per-step costs. The logs show that reviewer access rarely resulted in writes: among 2,164 reviewer-attributable calls, only 3 (0.14%) performed a write, and none modified tracked source files. On the 42 tasks for which review spending can be isolated, review accounted for 9.3% of total task cost. Task narratives contain 1,943 reported reviewer findings and 1,779 corresponding fixes, but these counts are self-reported by the builder; an independent re-coding finds roughly one-fifth fewer findings. In a 60-task re-coded sample, both readers identify 5 tasks where review exposed a problem that the builder’s tests had not caught, while the second reader identifies 7. Three case studies from the full corpus illustrate such findings in detail, including a code-execution path from model output and a repair that introduced a linearizability race. These observations characterize how model-based review is used in production and what it costs, but they do not establish its causal benefit because the deployment has no comparison arm. We release the trace corpus and analysis scripts to support further study.

Primary area:
datasets and benchmarks
Keywords:
AI Code Review, Software Engineering Agents, Builder–Reviewer Architecture, Defect Detection, Production Deployment

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_SMBGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper „A Trace-based Study of Heterogeneous LLM Review and Report in a Ten-Week Deployment” describes a self-study by the author for using two LLMs within the one project, one as builder, the other as reviewer. The data describes how many tasks were addressed, which actions the reviewer took, and some data about the findings themselves, i.e., if problems were reported.

Critical strengths
  1. Empirical evidence of real-world settings regarding agentic software development is important and still lacking
Critical weaknesses
  1. The study is for a single repository with a single maintainer that did not even over the ten week time-frame follow a single consistent protocol. The data mixes in work for the paper itself, further eroding the trust in the quality of the collected data. Overall, this is too narrow and the setting is too unrealistic for generalization, diluting the value of this study.
  2. The reported data is superficial and not according to the standards that would be expected for a scientific case study. For example, it is completely unclear what the tasks are (bug fixing, feature implementation, documentation), how they are distributed over components and languages within the repository, or which kind of problems were raised for which component, whether the raised issues were actually relevant (e.g., critical bugs) or trivial (e.g., renames, minor bugs and typos). There does not seem to have been any human validation of any of the results or – indeed- guidance regarding the analysis towards interesting aspects within the findings.
Questions for authors
  1. To which settings should this study be generalizable?
  2. How is reporting how often the review agent acted sufficient for actionable advice for others?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SMBG

Thank you for asking which settings these observations can inform and what someone could do with them. We now identify the conditions for transfer explicitly: a builder writes the review brief, a reviewer can inspect the code with tools, and the record preserves only part of the state. A team with those conditions can test the failure mechanisms we describe. The observed frequencies, costs, and repair outcomes belong to one maintainer’s changing workflow. Research tasks, dependent work on the same codebase, and changes in models and review caps limit generalization (§§3.1, 5; Appendix B).

What work was actually studied? Table 1 characterizes all 486 tasks. Prompt rules identified 72 feature tasks, 41 bug/fix tasks, and 31 refactoring tasks, alongside 173 mixed and 128 unknown. File evidence linked 171 tasks to the UI and 132 to the server, and identified Python in 292 tasks and JavaScript in 165. Components and languages overlap. Removing 17 candidate study or paper tasks left 469. In total, 301 of 486 tasks had mixed or unknown purpose. We label these rules provisional and do not report a findings-by-component breakdown. In particular, the sparse audit and overlapping component labels cannot support a reliable ranking of defect types or severity by component. We now state where the classification loses that resolution (§3.1, Table 1; Appendix B).

Were the findings consequential? We separated broad labels extracted from builder summaries from independent assessment of the original allegations. In the 45-group audit, each reader judged nine allegations to be supported defects. One judged five of those major or critical and the other judged six. Both readers supported eight of the same defects and judged six of those eight corrected on code inspection. Both also saw the builder accept 11 allegations for which they still lacked enough evidence to judge the outcome. These judgments convey both practical significance and the limits of the available records (§4.4, Table 4; Appendix G).

What would we change in a similar workflow? We would retain the original brief and report, link each to the exact reviewed state and response diff, and preserve one behavioral check that distinguishes the states. The input-delivery case shows why: 13 focused tests passed with either wiring, while following an accepted message through the application loop exposed a lost input. These recommendations draw on the behavioral check and the human allegation audit. The separate classification of request endpoints was checked by two automated methods and did not undergo human validation (Appendix F). Their effect on quality across other teams remains an empirical question (§5; Appendix F.5).

Thank you for this response, but this does not even try to address the core issue: how is a self-study of a single repository something that is a sound and generalizable setting for a scientific study rather than an anecdotal result?

To be honest, since this cannot be fixed, I see no way for this paper to go forward.

AuthorsOpenReview ↗

What one deployment can establish

Thank you for pressing this point. Our earlier reply described settings where the findings might transfer, but your question was what the study actually establishes. All 486 tasks came from one maintainer’s evolving workflow. More tasks give us more observations within that workflow. They cannot tell us how often another team would see the same failure or whether switching vendors improves review quality (§5).

One claim we can test within this case is whether a builder’s acceptance of a finding is sufficient evidence to count it as a successful repair. We drew a stratified sample of 30 tasks and had two readers assess the preserved allegations independently. In 11 allegation groups, both readers saw that the builder accepted the finding, yet neither could establish from the available evidence whether it had been corrected (§4.4). The repairs may have succeeded. Counting them as successes would answer a question the records leave open. The sampling procedure matters here because individual examples were not chosen to illustrate that result.

Appendix F.5 makes a different claim that can be checked by execution. With the model adapter held fixed, changing only the hook wiring changed whether an accepted message reached the application loop. All 13 selected queue and parser tests passed with both variants (Table 7). The earlier wiring was reconstructed, so this establishes the mechanism under the stated conditions. Its prevalence elsewhere remains an open question.

Estimating how often this happens across teams would require independent deployments. Attributing better repairs to vendor diversity would also require a controlled comparison. What this study contributes is a concrete way to audit a review log for the gap between an accepted finding and a demonstrated repair.

Reviewer_GcoJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses a case-based study of a single developer to investigate the cross-vendor GenAI activities and cost in software engineering production development environment. To achieve the goal, the author used two different GenAI from different vendors: a software developer from Claude and a reviewer from OpenAI. The input consists of 486 tasks and 1.1 M events. The OpenAI reviewer intervened in 2, 164 tasks with only 3 write operations.

Critical strengths
  • Investigates cross-vendor GenAI interactions in production level software engineering tasks
  • Releases the event logs for reproducibility
Critical weaknesses

While the study has promises, it has several weakness. First, the novelty is low, as heterogenous multi-agent GenAI interaction has already been explored in SWE venues (ICSE, FSE, ASE) and NLP/AI venues (ICLR/ *CL/ ...). Second, although the study claims mimicking production level SWE tasks, the task design has limitations. For example, the input of the tasks described in page 4 consists of "maintainers own papers". Third, the tasks are designed, executed, and verified by a single person---the author. Fourth, the findings on Section 4.5 and 4.6 are not clear. In other words, what can a enthusiastic researcher in AI in SWE can learn from these? In particular, the statement from Section 4.6 "No analysis here determines how many later-round findings were caused by earlier repairs"--What are the takeaways from this Section if someone wants to design an GenAI based developer-reviewer setting?

Questions for authors

Please see the weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer GcoJ

Thank you for asking what a practitioner could take away from the later-round findings. We revised the paper around two decisions a team building this workflow actually faces.

What should a second reviewer check? In one task, the request, implementation, and revised test all assumed a message hook was needed only in one execution mode. The reviewer questioned what happened to input accepted in the other mode. We compared two versions that differed only in the hook wiring. All 13 focused queue/parser tests passed in both. In the application loop, the old wiring allowed the task to finish with accepted input still undelivered. The corrected wiring delivered it. This suggests a specific review question when a patch edits both code and tests: which user-visible behavior would still be wrong if the new test passed? We used a deterministic model adapter and a reconstructed comparison version, so the result is specific to that execution (§§4.4, 5; Appendix F.5).

What can a later round tell us? A problem first mentioned in round two may already have existed. Among the five audited groups for which origin applied, both readers attributed two to an earlier repair, judged one pre-existing, and left one undetermined. They disagreed on the fifth. For someone designing the workflow, that distinction argues for linking each allegation to the reviewed state and the intervening diff. Counting new comments by round cannot tell a team how often its fixes introduced regressions (§4.4; Appendix G).

Setting and novelty. We describe a self-study of one maintainer’s evolving workflow. The corpus includes research and paper-editing tasks. Our explicit request-based screen identified 17 candidates; excluding them left 469 tasks, 415 with a logged review request. We could not recover the membership of the original 18-task exclusion and report the screens separately. The maintainer still selected and steered the work. The revision adds separate judgments from two technical readers who assessed the preserved allegations before seeing the builder’s responses. Section 2 credits prior heterogeneous-agent systems and narrows our contribution to the observed briefs, reports, changes, and costs (§§2–3; Table 1; Appendix B).

Two details in the original presentation also deserve correction. The 2,164 observations are tool calls inside recorded inline reviewer windows across a corpus of 486 tasks. Three direct write calls targeted review-note paths; shell-mediated changes and much of delegated review fall outside that count (§4.1). We withdraw the submitted abstract’s statement that we release the traces and analysis scripts. The revision supplies manuscript-level results, and access to the underlying event logs remains unresolved. We have corrected the data-availability statement accordingly.

Reviewer_jgioOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is a dataset and analysis work, it uses 486 tasks and 1.1M events to analyze the cross-model code review. The setup uses one model as the builder and another model as the reviewer. The authors did a lot of analysis, including reviewer behavior, reported bug findings, repair outcomes, and review cost. The analysis shows that reviewer writes are rare in the recorded inline reviews. It also presents several cases where reviewer successfully catches bugs missed by builder's tests and issues introduced from earlier fixes.

Critical strengths
  1. The real-world deployment traces are valuable to the research community for future research.

  2. The authors have shown some interesting analysis.

Critical weaknesses
  1. My main concern is the lack of baselines, e.g., builder-only and same-model review. Therefore, it is hard to know how much such cross-model review can actually help.

  2. The 9.3% cost result only comes from only 42 tasks, since the sample size is not big, it is unclear how representative it is.

  3. The novelty is very limited. The idea of cross-model review itself is not new, and most results are empirical and descriptive. To be honest, it does not show much new insight about model complementarity.

  4. Can the authors/independent human reviewers verify reported findings and fixes to measure the actual bug detection and repair success rates?

Questions for authors

Refer to the above part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer jgio

Thank you for asking what evidence would distinguish a benefit from cross-model review from the benefit of spending more effort on a task. We made that distinction explicit and added two analyses.

Comparison conditions. Appendix H tests a fixed repair-selection protocol on six authored tasks. Qwen performed self-inspection or fresh same-model review, Mistral supplied cross-model review, and Qwen chose the final repair in each condition. All three conditions started with identical implementations and repair options, had equal generation caps, and used the same model for the final selection. The starting code passed 23 of 36 hidden checks. The three conditions passed 31, 32, and 33, respectively. All corrected the same 11 failures. Their differences came from introducing three, two, and one new failures. Each reviewer beat self-inspection on one task and tied on five. Head to head, each reviewer won one task and they tied on four. In the earlier three-task feasibility run, self-inspection passed 15 of 18 checks and both review arms passed 14 (Appendix G). The pilot therefore shows how the choice of reviewer changed which repairs were selected on these tasks. We cannot infer a general advantage from the one-check aggregate difference. The deployment itself lacks a budget-matched comparison. The 55 tasks without a logged request include stopped and never-started tasks and would make a poor control (§§3.1, 5).

What the humans could verify. Two readers independently assessed 45 preserved allegation groups in a stratified sample of 30 tasks. They assessed validity before seeing the builder’s response. Both identified eight of the same allegations as supported defects and judged six of those eight corrected by inspecting code. Both judged the outcome evidence insufficient for 17 of the 45 groups. They agreed on 40/45 validity labels and 40/45 outcome labels. We use the audit to identify specific corrections that survive inspection. We cannot estimate detection recall because the sample begins with allegations the reviewer raised. Missing reports and historical states also prevent a deployment-wide repair-success estimate (§§3.6, 4.4; Appendix G).

Cost. We replaced the pooled 9.3% headline with separate estimates. In 34 selected delegated tasks, 47 reviewer runs accounted for $177.84 of $1,898.22 in recorded parent costs (9.37%). Only eight of 88 inline tasks met the telemetry criteria. Their estimated share was 8.09%, ranging from 6.11% to 24.47% when we varied how spending near a model switch was assigned. The median task cost was $13.98 in the inline panel and $19.11 across all 431 tasks with a review request. Neither panel supports a deployment-wide overhead estimate (§4.2, Table 2; Appendix C).

Contribution. We agree that the architecture is established. The contribution is evidence about what can be learned from a deployment record. For example, in 11 groups both readers saw the builder accept a finding yet could not establish the outcome. Treating acceptance as success would erase precisely those cases. The separate input-delivery experiment establishes a failure mechanism under controlled conditions. Whether changing vendors was necessary to discover it remains open (§§2, 4.4–5; Appendix F.5). We also withdraw the original abstract’s release statement. Access to the underlying traces remains unresolved (Ethics and Data Availability).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.