acceptodds
Under review as a conference paper at ICLR 2027

ActCI-Bench: Evaluating Selective Decisions in CI Maintenance Agents

Abstract

A software agent responding to a failed continuous integration (CI) run must decide whether to modify the repository, take a non-patch action, or defer to another owner. Repair benchmarks that start from known patch tasks do not evaluate this decision. We present ActCI-Bench, an evaluation of selective CI maintenance decisions over 1,543 real failure episodes. Each episode links failure evidence to three targets: actionability for the current author, root cause, and appropriate response. The benchmark combines logs and repository context with available CI history and developer handling, and evaluates both classification quality and false repairs on non-actionable episodes. In our evaluation, 509 episodes (33.0%) are non-actionable for the current author. Repository-aware agents achieve higher macro-F1 than specialized triage baselines, yet still propose patches on 41–48% of non-actionable cases. Richer context and an explicit diagnosis-before-response workflow improve response selection in the reported retrospective setting. These findings identify response selection and restraint as distinct dimensions of coding-agent capability. Because some evaluated context includes later developer handling, full-context scores should be read as retrospective evidence-rich results rather than estimates of performance at the moment a CI run fails.

Primary area:
datasets and benchmarks
Keywords:
Distributed LLM Inference, KV Cache Sharing, Consumer-Aware Cache Management, Adaptive Precision, Communication-Efficient Serving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tZZuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ActCI-Bench, a benchmark of 1,543 CI failure episodes drawn from three existing datasets, covering actionability classification, root-cause diagnosis, and response selection. It evaluates rule-based triage, flaky-test detectors, a repair baseline, and three repository-level agents using task-specific macro-F1, false repair rate, and patch pass rate. The results show that repository-level agents achieve stronger classification performance but frequently propose unnecessary patches, while diagnosis-first prompting and richer context improve response selection.

Critical strengths
  1. The paper identifies an important task that language models can automate: making decisions and fixes when a software CI run fails after code changes.
  2. The paper defines clear multi-class classification tasks and a comprehensive range of class categories, which make the evaluation results certain, reproducible and interpretable, compared to many recent LLM-as-judge-based benchmarks whose results have uncertainty and randomness.
Critical weaknesses
  1. Limited practical relevance of the retrospective setting. The paper acknowledges that its richer input settings include post-failure discussions explaining the cause or solution, follow-up repair commits and patches, and therefore frames these results as retrospective, evidence-rich evaluations rather than estimates of performance when CI first fails. I acknowledge the potential difficulty of separating evidence available at failure time from subsequent records. However, diagnosing a failure, assigning responsibility, and implementing an appropriate fix at failure time is substantially more meaningful than having human resolve the CI failure and then having language model summarize the human solution. But what the paper actually does is the latter. Although the paper also evaluates restricted-context settings, the inclusion of resolution artifacts in its full-context evaluation undermines its relevance to independent CI maintenance. I therefore find the practical significance of the retrospective task limited.
  2. Insufficient documentation of label construction and validation. The paper does not clearly explain which evidence-to-label mappings were automated, manually assigned, or LLM-assisted. It also does not identify who verified the labels or describe annotator agreement and disagreement adjudication procedures. These omissions make it difficult to assess the reliability of the ground-truth labels and reproduce the benchmark construction process.
  3. Under-specified evaluation protocol. The supplied PDF does not include the actual LLM prompt templates used to benchmark the methods in the Appendix. It is unclear whether the available evidence (logs, failed-run metadata, files, history, discussions, and repair artifacts) is supplied together in a single model call, and whether the evaluated agents can freely explore the repository, execute test commands, or iteratively revise their outputs. These are possible interpretations, but the manuscript does not establish them. No accompanying dataset or source code is provided, so I cannot resolve these uncertainties through the implementation. A precise, reproducible evaluation protocol is essential for a benchmark paper, and the manuscript falls short of this requirement.
  4. Missing backbone LLM specifications and comparisons. The paper names the evaluated agent frameworks but does not specify their underlying language models or model versions, let alone compare models from different providers. Consequently, it is impossible to determine whether the agent comparisons use a common model backbone or whether performance differences reflect the agent frameworks, the underlying models, or both. A benchmark of LLM agents should evaluate both agent frameworks and backbone models, or clearly justify a narrower scope; this paper leaves the language model dimension totally unexamined and insufficiently documented.

I believe Weakness 2, 3 and 4 are major deficits of the paper, which casts doubt on the authenticity and integrity of this paper. Now I will first give this paper a rating of 2. If the authors can provide a straightforward answer to my questions, I will raise my rating.

Questions for authors

The authors need to provide direct answers to the questions raised in my Weaknesses 2, 3 and 4:

  1. When describing mapping each input evidence to the classification categories, the paper should clarify which classifications were automated, which classifications were manually assigned, and which classifications were LLM-assisted, and also clarify who checked the labels, the annotator agreement and the adjudication procedures.
  2. Provide the actual LLM prompt templates when benchmarking the methods. Explain how the input evidence is fed into each method. Clarify whether the three coding agents can explore freely in the repository file system, make file edits, and run tests to validate their modifications. The authors should put these information in the Appendix.
  3. Specify the language model that the paper evaluates. Compare the performance of different language model providers when using the same method.
4 comments

Response to Reviewer tZZu (1/2)

[Q1] Who assigned and verified the three labels, and what do we know about agreement before adjudication?

[A1] We recorded the source of the initial label for each of the 1,543 records and each target. For GHALogs, we linked logs by run ID and head SHA. For LCA, we linked the failed build and repair diff. For IDoFT, we matched repository, revision and test identifier, and separately checked whether a failed run could be linked.

TargetEvidence used for the initial assignmentRuleHumanLLM
Actionability for the current PR authorPR diff and evidence about who owned the failure1,1004430
Root causeFailing step and trace matched against the cause rubric1,4001430
Appropriate responseVerified resolution mapped to a response label, with human review of ambiguous cases1,2502930

Two software-engineering reviewers with experience in CI triage and flaky-test analysis independently applied a shared three-target codebook on the first pass. The agreement sample comprised 150 episodes selected by stratified random sampling by source and reference cause. The review included 900 individual label assignments. The table reports agreement before adjudication.

TargetMatching labels / episodesExact agreementCohen's κ
Actionability136 / 15090.7%0.792
Root cause142 / 15094.7%0.928
Appropriate response138 / 15092.0%0.893

Before adjudication, the reviewers disagreed on 14 actionability labels, eight causes and 12 responses. A third reviewer applied the codebook, sought independent evidence for disputed ownership and flagged cases where ownership remained unresolved. The confidence totals in the paper (1,316 high, 222 medium and 5 low) describe the strength of the evidence for individual records.

For example, in IDoFT record EXAMPLE-IDoFT-017, the failed test matched the catalog entry. The PR left that test unchanged, and a pre-existing flakiness ticket assigned quarantine to its maintainer. The proposed and reviewed labels were non_actionable_for_pr_author, flaky_test and mark_or_isolate_flaky. The ticket identifies the maintainer responsible for quarantine. We retained IDoFT entries lacking a run link as source records and flagged the missing linkage. Of 495 retained IDoFT records, 180 linked to a failed run and revision, 163 also had a matching failing test, and 93 also had evidence supporting the current-author assignment.

[Q2] What can the benchmark say about independent decisions when a later repair can reveal the answer?

[A2] Full-context scores measure retrospective interpretation. A history-aware input also becomes retrospective when it includes a later green run. In the failure-time replay, we used completion of the failed run as the cutoff. Eligible inputs included the log available then, the diff, workflow and dependencies at the failed revision, and records already visible at that time. The eligibility rule applies to tool-retrieved evidence as well as the initial bundle, and the repository checkout was at the failed revision. We excluded 25 of the 265 test episodes because their cutoff could not be verified.

AutoCodeRover input conditionEpisodesResponse macro-F1Patch proposals / non-actionable
Failure-time evidence2400.40545 / 90 (50.0%)
Full retrospective evidenceSame 2400.54040 / 90 (44.4%)

On the paired episodes, full context increased response macro-F1 by 0.135 (interval 0.082 to 0.186) and yielded five fewer patch proposals on the same 90 non-actionable cases. The 0.405 score estimates performance with failure-time evidence on the covered cohort. The 0.540 score describes reconstruction after later evidence becomes available.

Response to Reviewer tZZu (2/2)

[Q3] What were the actual prompts, how was the evidence delivered, and what could each runner do?

[A3] The direct decision instruction was “Choose the next response for the PR author. Return actionability, cause, response, and patch_proposed.” The diagnosis-first instruction was “State the likely root cause and author responsibility before selecting a response. Return actionability, cause, response, and patch_proposed.” The adapter passed first-stage JSON to the final stage. We counted tool and model calls separately.

The input bundle ordered the evidence blocks as logs, diff, workflow, dependencies, history and handling. The bundle used chronological records, JSON null for missing fields and a 2,000-token limit per block. Each runner started from the failed commit SHA.

RunnerRepository access and execution after the initial bundleTermination
SWE-agentTool loop with repository reads, edits and test executionRevise until submission or budget limit
AgentlessLocalization and selected file reads, edits, and runner execution of selected testsRevise and submit within budget
AutoCodeRoverRepository search, file reads, edits and test executionRevise until submission or budget limit

The triage baseline used log rules. The flaky-test baseline isolated detected flakes and otherwise used triage. The repair baseline mapped its diff to a response.

All three adapters accepted an explicit non-patch response or deferral. The parser read the four decision fields and mapped a nonempty proposed diff to its edit type. An explicit non-patch response with patch_proposed=false recorded no patch. An empty diff alone was invalid. The runner logged submission status, diff presence, edit attempts and timeouts separately. A failed run was recorded as failed, and any earlier patch_proposal still counted. AutoCodeRover proposed a patch on 44/100 eligible non-actionable episodes.

For executable patch validation, the denominator comprised 100 prespecified repairable test episodes. AutoCodeRover proposed 86 patches, applied 81, executed checks on 78 and passed on 67. Each check ran ./validate in an isolated episode container at the failed commit with a 900-second timeout. Success required exit code zero and all manifest checks passing. No patch, setup failure or timeout scored zero, and technical failures were logged separately. The resulting patch-pass rate was 67/100. The repository README links the dataset. Anonymous replication repository · ActCI-Bench dataset

[Q4] Which LLM backed each framework, and how does a fixed framework perform across providers?

[A4] The three submitted runners all used OpenAI gpt-4.1-2025-04-14 with temperature 0, top-p 1 and no fixed seed. The per-episode ceilings were 12 model calls, 32,000 total tokens and 16 tool calls. The runners differed in repository access and control flow as described above.

We also evaluated two backends within AutoCodeRover on the same 240 failure-time episodes. Both used temperature 0 and no fixed seed. The adapter mapped each provider's native tool schema to the same set of actions. We scored invalid or truncated outputs as misses. Both backends returned 240 valid, untruncated outputs.

Provider and model snapshotActionability macro-F1Cause macro-F1Response macro-F1Patch proposals / non-actionable
OpenAI gpt-4.1-2025-04-140.6600.3920.40545 / 90 (50.0%)
Anthropic claude-sonnet-4-5-202509290.6700.4040.42343 / 90 (47.8%)

The paired response macro-F1 difference (Claude minus GPT-4.1) was 0.018, with an interval from −0.045 to 0.087. The interval includes zero. These results leave the two backends’ ranking under AutoCodeRover unresolved.

Reviewer_tZZuOpenReview ↗

Thanks for your rebuttal

I thank the authors for their rebuttal. The authors have addressed all my questions. I appreciate their effort. I will increase my rating from 2 to 3.

However, as I commented in the original review, I still believe using LLMs to fix errors immediately after the CI/CD fails is much more useful than using LLMs to summarize the human actions towards CI/CD failures, but what the paper does is the latter. That is why I cannot give this paper a rating of 4.

Also, next time please do not respond to your reviewers at the last minute just before the deadline, because many reviewers may neglect late responses, and reviewers need to compare the responses of all the papers under review to make final decisions.

Finally, I wish the authors good luck in the subsequent stages.

Thank you very much for your thoughtful feedback and for reconsidering your rating. We sincerely appreciate your time and understanding.

We apologize for our late response. We spent additional time conducting supplementary experiments and carefully refining our explanations to provide the most complete and accurate response possible. Nevertheless, we understand the importance of allowing reviewers sufficient time to assess rebuttals, and we will be more mindful of this in the future.

Thank you again for your constructive feedback and kind wishes.

Reviewer_KeywOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ActCI-Bench. It treats a red CI run as a decision rather than a repair task. 1,543 real failure episodes each carry three labels: actionability for the current PR author, root cause, and appropriate response. Baselines cover lexical triage, five flakiness detectors, a repair baseline, and the repository agents SWE-agent, Agentless and AutoCodeRover. Metrics are macro-F1 on the three labels, a false-repair rate on the 509 non-actionable episodes, and patch pass rate on repairable ones.

Critical strengths

The construction designed in the paper is careful. Splits are repository-disjoint (1,057 / 221 / 265), which removes the repository-memorisation shortcut. The flaky-test source IDoFT is capped at 500 episodes to match the largest non-flaky class, because otherwise the source itself becomes a shortcut. Per-label confidence is recorded (1,316 high, 222 medium, 5 low). The evidence ladder states what each rung adds. Best of all, the retrospective caveat sits in the abstract: full context includes later developer handling, so its scores are presented as a diagnostic rather than as deployable performance. Most benchmark papers bury that.

Critical weaknesses
  1. The response task is the root-cause task renamed. Root-cause counts are 500, 500, 260, 169, 105, 8, 1. Response counts are 500, 500, 260, 169, 104, 8, 2. Four classes match exactly and the rest differ by one episode. Response macro-F1 trails root-cause macro-F1 by 0.02 to 0.03 for every baseline, which is what a near-deterministic cause-to-response map produces. That makes the stated reason for scoring them jointly, namely that an agent might diagnose flakiness and still edit a test, untestable on this data. Either report how often an agent's own cause and response predictions disagree, or redesign the response set so it cross-cuts causes: rerun_ci should be plausible for a flaky cause and for an environment failure.

  2. The Table 5 ranking is not supported by the measurements. The test split holds 265 episodes across seven causes, two of which have 8 and 1 episodes in the entire benchmark. Macro-F1 weights classes equally, so a single rare-class episode moves the metric by roughly 0.14, while the gap between AutoCodeRover (0.69), Agentless (0.67) and SWE-agent (0.65) is 0.02. No seeds, standard deviations or intervals appear anywhere. The backbone model behind the repository agents is never named, nor are the prompts or adapters. The protocol and threats sections promise response appropriateness, patch quality and per-class metrics, and none of them are reported. The reproducibility statement points to a replication package that is not linked.

  3. The evidence provenance inside the benchmark is not pinned down. Full context includes flaky-test metadata, available only for the IDoFT episodes, which supplies most of the signal for one of the two largest classes. History-aware already includes subsequent green runs, so it is post-failure too. Only the log, diff, workflow and dependency rungs are available at failure time, which means the caveat in the abstract covers more of the ladder than it should. Table 4 should mark each evidence block as pre or post failure relative to the run, with a prospective-only ablation reported alongside.

Questions for authors
  1. Which LLM backs SWE-agent, Agentless and AutoCodeRover, with which prompts, and could they decline to patch? The 41 to 48 percent false-repair figures depend on the answer.

  2. Can you add paired bootstrap intervals over the 265 test episodes, at least three runs per baseline, and macro-F1 restricted to classes with a minimum test count?

  3. Does the full-context root-cause gain survive when flaky-test metadata is removed?

7 comments
Reviewer_KeywOpenReview ↗

Discussions from Reviewer Keyw

Correction to Weakness 1. After read the manuscript again, I find that the two label sets have identical counts is 5, not 4. Besides, the 0.02 to 0.03 gap between response and root-cause macro-F1 is for the rule-based baselines, repair and agent baselines, but not for every one.

Please explain first the relation between the root-cause and response labels (Weakness 1), I think this is the major problem.

As the rebuttal period is ending soon, please share the response. I will raise my rating to positive if the new results or explanations address my concerns.

Response to Reviewer Keyw (1/3)

Thank you for asking whether the small gap between cause and response macro-F1 reflects the relationship between the reference labels.

[Q1] How much of the response label can be recovered from the cause alone?

[A1] The response label records the warranted maintenance action. A separate actionability label identifies whether the current PR author should intervene. Five of the seven cause–response pairs you identified have identical marginal counts. Across all 1,543 records, 1,472 (95.4%) have the most frequent response for their reference cause. The remaining 71 have a response that differs from their cause’s most frequent response. For example, 480 of the 500 flaky-test records call for mark_or_isolate_flaky. The remaining 20 call for modify_test (13), rerun_ci (3), pin_or_update_dependency (2), modify_code (1), or update_workflow (1). Part 2 gives the complete joint distribution.

To measure how well this relationship generalizes, we used the 1,057 training episodes to map each cause to its most frequent response, choosing the lexicographically first response label in a tie. We compared all three predictors on the 264 of 265 test episodes for which both the reference cause and AutoCodeRover's predicted cause had a training-set mapping.

Response predictor on the same 264 test episodesCorrect / eligibleResponse macro-F1
Training lookup using the reference cause248 / 2640.72
Training lookup using AutoCodeRover's predicted cause194 / 2640.50
AutoCodeRover's own response192 / 2640.53

Given the reference cause, the lookup predicts the response correctly in 93.9% of covered test episodes. Given its own diagnosis, AutoCodeRover's response prediction produces two fewer exact matches than the lookup, with 0.03 higher macro-F1. Exact matches weight each episode equally, while macro-F1 weights each response class equally. Of the 16 episodes where the lookup is wrong even with the reference cause, AutoCodeRover selects the correct response in five.

[Q2] Does an agent ever diagnose correctly and still patch? Could the agents decline to patch?

[A2] Yes. For example, AutoCodeRover proposed a patch in 27 of the 82 non-actionable test episodes for which it diagnosed the cause correctly. The table separates three events. The first column requires a mapping only for the predicted cause, which covers all 265 predictions for each agent. A departure from that mapping can be correct.

AgentResponse differs from lookup on predicted cause / eligibleWrong response / correct diagnosesPatch proposed / correctly diagnosed non-actionable cases
SWE-agent22 / 26519 / 20034 / 85
Agentless23 / 26528 / 21428 / 84
AutoCodeRover26 / 26521 / 20427 / 82

Among 73 cases where AutoCodeRover correctly diagnosed flakiness and the reference called for no patch, it proposed 18 edits. None of those 18 modified a test. We will correct the example in the paper to describe what we observed.

All three submitted runners used gpt-4.1-2025-04-14 and could decline to patch. SWE-agent accepted an explicit non-patch action or deferral. Agentless could exit on either decision before patch generation. AutoCodeRover accepted either decision without a diff. The false-repair denominator was the same 100 non-actionable test episodes for each runner. The numerator counted a patch_proposal event, including one logged before a later failure. The parser recorded an explicit non-patch decision as no patch. It marked an empty diff alone as invalid and recorded a failed run separately. These are outcomes of the configured agent and adapter.

Reviewer_D73iOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

ActCI-Bench is an evidence-oriented CI benchmark that evaluates whether the current pull-request author should intervene after a continuous-integration failure, what caused the failure, and which response is appropriate. It combines 1,543 real episodes from three sources, uses repository-disjoint splits, and evaluates classification quality, false repairs, and patch success. The constructed dataset has 509 non-actionable episodes; experiments report false-repair rates of 41–48% for three repository agents, falling to 30–34% with diagnosis-first workflows. These findings concern the reported retrospective evaluation: richer contexts include subsequent developer handling, and the authors explicitly distinguish these results from performance at failure time.

Critical strengths

The evaluation addresses a useful distinction between successful repair and appropriate intervention. Separately measuring actionability, diagnosis, response selection, and false repairs can expose behavior that patch-success evaluation misses. For example, AutoCodeRover reportedly achieves a 67% patch-pass rate on repairable cases while proposing patches on 44% of non-actionable cases. The paper correctly explains that these metrics concern different populations and should be interpreted jointly, rather than subtracting their values. The evidence organization and scope qualifications are valuable. The common schema incorporates logs, changes, configuration, dependencies, and available historical evidence, while repository-disjoint splitting reduces repository-specific memorization risks. The paper consistently discloses the retrospective nature of richer inputs. Its diagnosis-first results also offer a concrete, appropriately qualified hypothesis about improving restraint across three agent frameworks.

Critical weaknesses

The benchmark retains an almost perfect source–label shortcut. IDoFT provide 495 of the 500 flaky-test examples. Downsampling helps but preserves the association between that source and flakiness. Table 1 lists only flaky-test metadata and provenance for these records, leaving unclear how their failed-run and current-author responsibility are established. The reported comparisons are insufficiently specified, and their aggregation needs clarification. The experiment configurations are under documented (e.g. backbone model/version settings, actual prompts and budgets, or the executable patch-validation procedure and eligible sample counts). These details affect interpretation of the framework comparisons and diagnosis-first gains. The evaluation gives models evidence of the resolution they are supposed to select. The labels are constructed using developer handling, repair artifacts, rerun outcomes, and later CI results. The richer evaluation conditions then provide some of these same artifacts as inputs (e.g. history-aware, full context). While the authors disclose this limitation repeatedly, the retrospective setting supports a substantially narrower contribution than the motivating broader selective-decision problem without rich contexts. The three advertised tasks may largely collapse into one root-cause classification task. There exists an almost exact matching in terms of the numbers of episodes between the different labels of root cause and response, indicating the correlation. The actionality labels also appears to be direct aggregates of root cause label categories.

Questions for authors

see weaknesses

3 comments

Response to Reviewer D73i (1/3)

Thank you for highlighting the link between data source, cause and response.

[Q1] Does IDoFT supply the answer? What establishes the failed run and the current author's responsibility?

[A1] All 495 retained IDoFT records carry the flaky-test label. They account for 495 of the benchmark's 500 flaky-test cases (99.0%). A model can still recognize the source in a repository-held-out split. A classifier that predicts each source's most frequent training cause achieves held-out root-cause macro-F1 of 0.33, with flaky-test precision of 1.00 and recall of 0.99.

A catalog entry identifies a test with a history of flakiness. Our linkage audit used the repository, failed-run ID, head SHA and fully qualified test identifier. We separately checked the evidence for ownership.

IDoFT evidence levelRecords
Retained IDoFT records495
Linked to a failed run and revision180
Of those, failing test matched to the catalog entry163
Of those, evidence also supported the current-author actionability label93

We retained records with a missing run link as source records and flagged them. Thus the corpus of 1,543 includes 315 IDoFT records without a documented failed-run/revision link. The last two rows give the evidence for test attribution and ownership. We will distinguish these source records from run-linked episodes in the dataset description.

For example, in record EXAMPLE-IDoFT-017, run EXAMPLE-RUN-017 identified the failing test ClockTest.testBoundary. The PR left that test unchanged, and a pre-existing flakiness ticket assigned quarantine to the test maintainer. The matched test and ticket supported the flaky-test cause, the mark_or_isolate_flaky response and a non-actionable label for the current PR author.

AutoCodeRover's retrospective root-cause macro-F1 is 0.513 with metadata and 0.434 with flaky-test metadata masked. The source-blinded condition scores 0.42. Field patterns may still identify the source. The one held-out flaky case from outside IDoFT is identified correctly (1/1). With only five non-IDoFT flaky positives in the full corpus, cross-source recall remains poorly estimated.

[Q2] Do the three tasks reduce to one cause prediction?

[A2] We compared training-set majority rules for each cause with AutoCodeRover's decisions on the 264 of 265 test episodes covered by the mappings.

Held-out diagnosticActionabilityResponse
Training rule applied to the reference cause, correct / eligible244 / 264248 / 264
Training rule applied to the predicted cause, macro-F10.790.50
AutoCodeRover's own output, macro-F10.690.53

The reference cause strongly predicts both labels. Six causes nevertheless contain both actionable and non-actionable episodes.

CauseActionable for the current authorNon-actionable for the current author
Flaky test30470
Dependency drift48020
Code regression2573
Test regression1636
Workflow misconfiguration1032
Environment failure17
External service issue01
Total1,034509

Ownership matters when interpreting the response label. For dependency drift, 498/500 cases carry an explicit edit response (dependency, code, test or workflow), but only 480 are actionable for the current author. For code regression, the corresponding counts are 259/260 and 257. At least 20 cases across these causes therefore pair a non-actionable label with an edit response. Counting those responses as repairs the current author should make would require episode-level ownership evidence.

AutoCodeRover also proposed a patch in 27/82 non-actionable cases where it diagnosed the cause correctly. The same-cause pairs in our reply to Keyw (Part 2) test whether agents can select different warranted responses at failure time.

Response to Reviewer D73i (2/3)

[Q3] What can the evidence ladder evaluate when later developer actions are visible?

[A3] Full-context scores measure reconstruction from the accumulated episode record. A later fix can disclose the response, and a subsequent green run can reveal that the failure was intermittent. In the failure-time replay, the cutoff is the end of the failed run.

EvidenceFailure-time treatment
Failed-run log and statusInclude what was visible when the run ended.
Diff, workflow and dependency filesUse the versions at the failed revision.
Prior runs, discussions and flaky-test annotationsInclude only entries available by the cutoff.
Subsequent green reruns, fixes and developer handlingReserve for the retrospective condition and reference construction.
Unknown availabilityWithhold the field.

This cutoff applies to both the initial evidence and the information exposed through repository tools. The paired AutoCodeRover comparison covers 240/265 test episodes. We excluded 25 for which the cutoff could not be verified. Response macro-F1 rose from 0.405 at failure time to 0.540 with full context, a difference of 0.135 (paired interval 0.082 to 0.186). False repairs fell from 45/90 to 40/90 on the same non-actionable subset. The additional context helps reconstruct the eventual decision. The failure-time result measures the earlier decision on the covered cases.

[Q4] Which configurations and denominators underlie the framework and diagnosis-first comparisons?

[A4] SWE-agent, Agentless and AutoCodeRover all used OpenAI gpt-4.1-2025-04-14 with temperature 0, top-p 1 and no fixed seed. Each run had ceilings of 12 model calls, 32,000 total tokens and 16 tool calls. All three adapters accepted an explicit non-patch response with patch_proposed=false. The diagnosis-first workflow added a stage asking the agent to state the likely root cause and author responsibility before selecting a response. It passed the first-stage JSON to the final stage within the same overall budget, evidence condition and tool permissions.

Original test outcomeDenominatorResult
Default-workflow false repairEligible non-actionable episodes41–48% across the three agents
Diagnosis-first false repairSame non-actionable episodes30–34% across the three agents
AutoCodeRover default false repair100 eligible non-actionable episodes44/100 (44%)
AutoCodeRover default patch pass100 prespecified repairable episodes67/100 (67%)

For patch validation, AutoCodeRover proposed 86 patches on the 100 repairable cases, applied 81, executed the checks on 78 and passed on 67. The runner used an isolated episode container at the failed commit and ran ./validate with a 900-second timeout. Passing required exit code zero and all manifest checks to succeed. Missing patches, setup failures and timeouts scored zero, with technical failures logged separately. The conditional pass rate among executed checks is 67/78. The reported end-to-end rate is 67/100. For classification, we pooled episodes within each run and macro-averaged over two actionability, seven cause or seven response labels, scoring an absent label's F1 as zero.

For the paired workflow comparison, bb counts episodes where a false repair disappeared and cc counts those where a false repair was introduced. On the same NN non-actionable episodes, the reduction is 100(b−c)/N100(b-c)/N percentage points.

Original comparisonbbccNNReduction, percentage points [95% interval]
SWE-agent21710014 [4.0, 24.0]
Agentless19810011 [1.0, 21.0]
AutoCodeRover18510013 [4.0, 22.0]

The full corpus contains 509 non-actionable cases (33.0% of 1,543). The original test comparison uses 100 eligible non-actionable cases. The separate failure-time replay in Part 3 uses 90.

Response to Reviewer D73i (3/3)

[Q5] Does asking for a diagnosis prevent unnecessary edits while preserving successful repairs?

[A5] On the same 240 failure-time replay episodes, we measured three outcomes. Each uses a fixed denominator across all workflows.

  • N=90N=90 are non-actionable for the current PR author. We counted patch proposals on these episodes as false repairs.
  • P=144P=144 have a reference response calling for a repository patch. We counted episodes where the agent proposed a patch.
  • V=90V=90 of the reference-patch cases were eligible for executable validation. We counted episodes where a proposed patch passed, keeping episodes with no patch in the denominator.

The reference response and the current author's responsibility are separate labels. These totals do not identify the overlap of NN and PP. The middle column measures proposal coverage on reference-patch cases. An estimate of current-author repair recall would also require identifying which of those edits the current author should make.

AgentWorkflowFalse repairs / NNPatches proposed / PPValidated repairs / VV
SWE-agentDirect46/90111/14449/90
SWE-agentNeutral two-stage43/90109/14448/90
SWE-agentDiagnosis first36/90106/14447/90
AgentlessDirect40/90108/14445/90
AgentlessNeutral two-stage39/90107/14446/90
AgentlessDiagnosis first31/90105/14445/90
AutoCodeRoverDirect45/90116/14452/90
AutoCodeRoverNeutral two-stage41/90113/14452/90
AutoCodeRoverDiagnosis first33/90109/14450/90

All three agents proposed fewer patches on non-actionable cases under diagnosis first than under direct prompting. Passing repairs changed from 49 to 47 for SWE-agent, 45 to 45 for Agentless and 52 to 50 for AutoCodeRover. For AutoCodeRover, 43 episodes passed under both workflows, nine under direct prompting alone and seven under diagnosis first alone. The paired change was −2.22 percentage points (95% interval −11.11 to +6.67). That interval admits both a loss and a gain in successful repairs, so preservation of repair performance remains unresolved.

[Q6] How much of a change can be attributed to the explicit diagnosis instruction?

[A6] We included a neutral two-stage arm to separate the effect of adding a stage from the effect of its instructions. The neutral first stage allowed inspection and notes. The diagnosis-first stage explicitly requested the likely root cause and the current author's responsibility. Both staged workflows passed their first-stage output to the final decision stage. The evidence, tool permissions and overall limits were the same. The comparison with the neutral arm therefore evaluates the two requirements together.

Mean per replay episode for AutoCodeRoverModel callsInput / output tokensTool calls
Direct6.418,200 / 2,1505.2
Neutral two-stage7.119,650 / 2,3805.6
Diagnosis first7.219,800 / 2,4705.5

Relative to the neutral arm, diagnosis first produced eight fewer false repairs (33 vs. 41 of 90, −8.89 percentage points), four fewer proposals on reference-patch cases (109 vs. 113 of 144, −2.78 points), and two fewer passing repairs (50 vs. 52 of 90, −2.22 points). These are observed differences on this cohort. The table reports actual usage under the common ceilings of 12 model calls, 32,000 total tokens and 16 tool calls per run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.