- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper „A Trace-based Study of Heterogeneous LLM Review and Report in a Ten-Week Deployment” describes a self-study by the author for using two LLMs within the one project, one as builder, the other as reviewer. The data describes how many tasks were addressed, which actions the reviewer took, and some data about the findings themselves, i.e., if problems were reported.
Critical strengths
- Empirical evidence of real-world settings regarding agentic software development is important and still lacking
Critical weaknesses
- The study is for a single repository with a single maintainer that did not even over the ten week time-frame follow a single consistent protocol. The data mixes in work for the paper itself, further eroding the trust in the quality of the collected data. Overall, this is too narrow and the setting is too unrealistic for generalization, diluting the value of this study.
- The reported data is superficial and not according to the standards that would be expected for a scientific case study. For example, it is completely unclear what the tasks are (bug fixing, feature implementation, documentation), how they are distributed over components and languages within the repository, or which kind of problems were raised for which component, whether the raised issues were actually relevant (e.g., critical bugs) or trivial (e.g., renames, minor bugs and typos). There does not seem to have been any human validation of any of the results or – indeed- guidance regarding the analysis towards interesting aspects within the findings.
Questions for authors
- To which settings should this study be generalizable?
- How is reporting how often the review agent acted sufficient for actionable advice for others?
3 comments
Thank you for this response, but this does not even try to address the core issue: how is a self-study of a single repository something that is a sound and generalizable setting for a scientific study rather than an anecdotal result?
To be honest, since this cannot be fixed, I see no way for this paper to go forward.
What one deployment can establish
Thank you for pressing this point. Our earlier reply described settings where the findings might transfer, but your question was what the study actually establishes. All 486 tasks came from one maintainer’s evolving workflow. More tasks give us more observations within that workflow. They cannot tell us how often another team would see the same failure or whether switching vendors improves review quality (§5).
One claim we can test within this case is whether a builder’s acceptance of a finding is sufficient evidence to count it as a successful repair. We drew a stratified sample of 30 tasks and had two readers assess the preserved allegations independently. In 11 allegation groups, both readers saw that the builder accepted the finding, yet neither could establish from the available evidence whether it had been corrected (§4.4). The repairs may have succeeded. Counting them as successes would answer a question the records leave open. The sampling procedure matters here because individual examples were not chosen to illustrate that result.
Appendix F.5 makes a different claim that can be checked by execution. With the model adapter held fixed, changing only the hook wiring changed whether an accepted message reached the application loop. All 13 selected queue and parser tests passed with both variants (Table 7). The earlier wiring was reconstructed, so this establishes the mechanism under the stated conditions. Its prevalence elsewhere remains an open question.
Estimating how often this happens across teams would require independent deployments. Attributing better repairs to vendor diversity would also require a controlled comparison. What this study contributes is a concrete way to audit a review log for the gap between an accepted finding and a demonstrated repair.
Response to Reviewer SMBG
Thank you for asking which settings these observations can inform and what someone could do with them. We now identify the conditions for transfer explicitly: a builder writes the review brief, a reviewer can inspect the code with tools, and the record preserves only part of the state. A team with those conditions can test the failure mechanisms we describe. The observed frequencies, costs, and repair outcomes belong to one maintainer’s changing workflow. Research tasks, dependent work on the same codebase, and changes in models and review caps limit generalization (§§3.1, 5; Appendix B).
What work was actually studied? Table 1 characterizes all 486 tasks. Prompt rules identified 72 feature tasks, 41 bug/fix tasks, and 31 refactoring tasks, alongside 173 mixed and 128 unknown. File evidence linked 171 tasks to the UI and 132 to the server, and identified Python in 292 tasks and JavaScript in 165. Components and languages overlap. Removing 17 candidate study or paper tasks left 469. In total, 301 of 486 tasks had mixed or unknown purpose. We label these rules provisional and do not report a findings-by-component breakdown. In particular, the sparse audit and overlapping component labels cannot support a reliable ranking of defect types or severity by component. We now state where the classification loses that resolution (§3.1, Table 1; Appendix B).
Were the findings consequential? We separated broad labels extracted from builder summaries from independent assessment of the original allegations. In the 45-group audit, each reader judged nine allegations to be supported defects. One judged five of those major or critical and the other judged six. Both readers supported eight of the same defects and judged six of those eight corrected on code inspection. Both also saw the builder accept 11 allegations for which they still lacked enough evidence to judge the outcome. These judgments convey both practical significance and the limits of the available records (§4.4, Table 4; Appendix G).
What would we change in a similar workflow? We would retain the original brief and report, link each to the exact reviewed state and response diff, and preserve one behavioral check that distinguishes the states. The input-delivery case shows why: 13 focused tests passed with either wiring, while following an accepted message through the application loop exposed a lost input. These recommendations draw on the behavioral check and the human allegation audit. The separate classification of request endpoints was checked by two automated methods and did not undergo human validation (Appendix F). Their effect on quality across other teams remains an empirical question (§5; Appendix F.5).