- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ActCI-Bench, a benchmark of 1,543 CI failure episodes drawn from three existing datasets, covering actionability classification, root-cause diagnosis, and response selection. It evaluates rule-based triage, flaky-test detectors, a repair baseline, and three repository-level agents using task-specific macro-F1, false repair rate, and patch pass rate. The results show that repository-level agents achieve stronger classification performance but frequently propose unnecessary patches, while diagnosis-first prompting and richer context improve response selection.
Critical strengths
- The paper identifies an important task that language models can automate: making decisions and fixes when a software CI run fails after code changes.
- The paper defines clear multi-class classification tasks and a comprehensive range of class categories, which make the evaluation results certain, reproducible and interpretable, compared to many recent LLM-as-judge-based benchmarks whose results have uncertainty and randomness.
Critical weaknesses
- Limited practical relevance of the retrospective setting. The paper acknowledges that its richer input settings include post-failure discussions explaining the cause or solution, follow-up repair commits and patches, and therefore frames these results as retrospective, evidence-rich evaluations rather than estimates of performance when CI first fails. I acknowledge the potential difficulty of separating evidence available at failure time from subsequent records. However, diagnosing a failure, assigning responsibility, and implementing an appropriate fix at failure time is substantially more meaningful than having human resolve the CI failure and then having language model summarize the human solution. But what the paper actually does is the latter. Although the paper also evaluates restricted-context settings, the inclusion of resolution artifacts in its full-context evaluation undermines its relevance to independent CI maintenance. I therefore find the practical significance of the retrospective task limited.
- Insufficient documentation of label construction and validation. The paper does not clearly explain which evidence-to-label mappings were automated, manually assigned, or LLM-assisted. It also does not identify who verified the labels or describe annotator agreement and disagreement adjudication procedures. These omissions make it difficult to assess the reliability of the ground-truth labels and reproduce the benchmark construction process.
- Under-specified evaluation protocol. The supplied PDF does not include the actual LLM prompt templates used to benchmark the methods in the Appendix. It is unclear whether the available evidence (logs, failed-run metadata, files, history, discussions, and repair artifacts) is supplied together in a single model call, and whether the evaluated agents can freely explore the repository, execute test commands, or iteratively revise their outputs. These are possible interpretations, but the manuscript does not establish them. No accompanying dataset or source code is provided, so I cannot resolve these uncertainties through the implementation. A precise, reproducible evaluation protocol is essential for a benchmark paper, and the manuscript falls short of this requirement.
- Missing backbone LLM specifications and comparisons. The paper names the evaluated agent frameworks but does not specify their underlying language models or model versions, let alone compare models from different providers. Consequently, it is impossible to determine whether the agent comparisons use a common model backbone or whether performance differences reflect the agent frameworks, the underlying models, or both. A benchmark of LLM agents should evaluate both agent frameworks and backbone models, or clearly justify a narrower scope; this paper leaves the language model dimension totally unexamined and insufficiently documented.
I believe Weakness 2, 3 and 4 are major deficits of the paper, which casts doubt on the authenticity and integrity of this paper. Now I will first give this paper a rating of 2. If the authors can provide a straightforward answer to my questions, I will raise my rating.
Questions for authors
The authors need to provide direct answers to the questions raised in my Weaknesses 2, 3 and 4:
- When describing mapping each input evidence to the classification categories, the paper should clarify which classifications were automated, which classifications were manually assigned, and which classifications were LLM-assisted, and also clarify who checked the labels, the annotator agreement and the adjudication procedures.
- Provide the actual LLM prompt templates when benchmarking the methods. Explain how the input evidence is fed into each method. Clarify whether the three coding agents can explore freely in the repository file system, make file edits, and run tests to validate their modifications. The authors should put these information in the Appendix.
- Specify the language model that the paper evaluates. Compare the performance of different language model providers when using the same method.
4 comments
Response to Reviewer tZZu (2/2)
[Q3] What were the actual prompts, how was the evidence delivered, and what could each runner do?
[A3] The direct decision instruction was “Choose the next response for the PR author. Return actionability, cause, response, and patch_proposed.” The diagnosis-first instruction was “State the likely root cause and author responsibility before selecting a response. Return actionability, cause, response, and patch_proposed.” The adapter passed first-stage JSON to the final stage. We counted tool and model calls separately.
The input bundle ordered the evidence blocks as logs, diff, workflow, dependencies, history and handling. The bundle used chronological records, JSON null for missing fields and a 2,000-token limit per block. Each runner started from the failed commit SHA.
| Runner | Repository access and execution after the initial bundle | Termination |
|---|---|---|
| SWE-agent | Tool loop with repository reads, edits and test execution | Revise until submission or budget limit |
| Agentless | Localization and selected file reads, edits, and runner execution of selected tests | Revise and submit within budget |
| AutoCodeRover | Repository search, file reads, edits and test execution | Revise until submission or budget limit |
The triage baseline used log rules. The flaky-test baseline isolated detected flakes and otherwise used triage. The repair baseline mapped its diff to a response.
All three adapters accepted an explicit non-patch response or deferral. The parser read the four decision fields and mapped a nonempty proposed diff to its edit type. An explicit non-patch response with patch_proposed=false recorded no patch. An empty diff alone was invalid. The runner logged submission status, diff presence, edit attempts and timeouts separately. A failed run was recorded as failed, and any earlier patch_proposal still counted. AutoCodeRover proposed a patch on 44/100 eligible non-actionable episodes.
For executable patch validation, the denominator comprised 100 prespecified repairable test episodes. AutoCodeRover proposed 86 patches, applied 81, executed checks on 78 and passed on 67. Each check ran ./validate in an isolated episode container at the failed commit with a 900-second timeout. Success required exit code zero and all manifest checks passing. No patch, setup failure or timeout scored zero, and technical failures were logged separately. The resulting patch-pass rate was 67/100. The repository README links the dataset. Anonymous replication repository · ActCI-Bench dataset
[Q4] Which LLM backed each framework, and how does a fixed framework perform across providers?
[A4] The three submitted runners all used OpenAI gpt-4.1-2025-04-14 with temperature 0, top-p 1 and no fixed seed. The per-episode ceilings were 12 model calls, 32,000 total tokens and 16 tool calls. The runners differed in repository access and control flow as described above.
We also evaluated two backends within AutoCodeRover on the same 240 failure-time episodes. Both used temperature 0 and no fixed seed. The adapter mapped each provider's native tool schema to the same set of actions. We scored invalid or truncated outputs as misses. Both backends returned 240 valid, untruncated outputs.
| Provider and model snapshot | Actionability macro-F1 | Cause macro-F1 | Response macro-F1 | Patch proposals / non-actionable |
|---|---|---|---|---|
OpenAI gpt-4.1-2025-04-14 | 0.660 | 0.392 | 0.405 | 45 / 90 (50.0%) |
Anthropic claude-sonnet-4-5-20250929 | 0.670 | 0.404 | 0.423 | 43 / 90 (47.8%) |
The paired response macro-F1 difference (Claude minus GPT-4.1) was 0.018, with an interval from −0.045 to 0.087. The interval includes zero. These results leave the two backends’ ranking under AutoCodeRover unresolved.
Thanks for your rebuttal
I thank the authors for their rebuttal. The authors have addressed all my questions. I appreciate their effort. I will increase my rating from 2 to 3.
However, as I commented in the original review, I still believe using LLMs to fix errors immediately after the CI/CD fails is much more useful than using LLMs to summarize the human actions towards CI/CD failures, but what the paper does is the latter. That is why I cannot give this paper a rating of 4.
Also, next time please do not respond to your reviewers at the last minute just before the deadline, because many reviewers may neglect late responses, and reviewers need to compare the responses of all the papers under review to make final decisions.
Finally, I wish the authors good luck in the subsequent stages.
Thank you very much for your thoughtful feedback and for reconsidering your rating. We sincerely appreciate your time and understanding.
We apologize for our late response. We spent additional time conducting supplementary experiments and carefully refining our explanations to provide the most complete and accurate response possible. Nevertheless, we understand the importance of allowing reviewers sufficient time to assess rebuttals, and we will be more mindful of this in the future.
Thank you again for your constructive feedback and kind wishes.
Response to Reviewer tZZu (1/2)
[Q1] Who assigned and verified the three labels, and what do we know about agreement before adjudication?
[A1] We recorded the source of the initial label for each of the 1,543 records and each target. For GHALogs, we linked logs by run ID and head SHA. For LCA, we linked the failed build and repair diff. For IDoFT, we matched repository, revision and test identifier, and separately checked whether a failed run could be linked.
Two software-engineering reviewers with experience in CI triage and flaky-test analysis independently applied a shared three-target codebook on the first pass. The agreement sample comprised 150 episodes selected by stratified random sampling by source and reference cause. The review included 900 individual label assignments. The table reports agreement before adjudication.
Before adjudication, the reviewers disagreed on 14 actionability labels, eight causes and 12 responses. A third reviewer applied the codebook, sought independent evidence for disputed ownership and flagged cases where ownership remained unresolved. The confidence totals in the paper (1,316 high, 222 medium and 5 low) describe the strength of the evidence for individual records.
For example, in IDoFT record
EXAMPLE-IDoFT-017, the failed test matched the catalog entry. The PR left that test unchanged, and a pre-existing flakiness ticket assigned quarantine to its maintainer. The proposed and reviewed labels werenon_actionable_for_pr_author,flaky_testandmark_or_isolate_flaky. The ticket identifies the maintainer responsible for quarantine. We retained IDoFT entries lacking a run link as source records and flagged the missing linkage. Of 495 retained IDoFT records, 180 linked to a failed run and revision, 163 also had a matching failing test, and 93 also had evidence supporting the current-author assignment.[Q2] What can the benchmark say about independent decisions when a later repair can reveal the answer?
[A2] Full-context scores measure retrospective interpretation. A history-aware input also becomes retrospective when it includes a later green run. In the failure-time replay, we used completion of the failed run as the cutoff. Eligible inputs included the log available then, the diff, workflow and dependencies at the failed revision, and records already visible at that time. The eligibility rule applies to tool-retrieved evidence as well as the initial bundle, and the repository checkout was at the failed revision. We excluded 25 of the 265 test episodes because their cutoff could not be verified.
On the paired episodes, full context increased response macro-F1 by 0.135 (interval 0.082 to 0.186) and yielded five fewer patch proposals on the same 90 non-actionable cases. The 0.405 score estimates performance with failure-time evidence on the covered cohort. The 0.540 score describes reconstruction after later evidence becomes available.