- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-CC, a framework designed to evaluate whether coding agents adhere to the contribution guidelines of software repositories. The authors argue that passing functional tests is insufficient to confirm that a contribution meets a project's requirements; therefore, they expand the scope of evaluation to include commit messages, pull request descriptions, testing procedures, documentation, code style, and the disclosure of AI usage. The authors extracted 1,759 atomic rules from the developer documentation of 12 repositories. After filtering for scope and mandatory status, they implemented 823 of these rules as executable checkers. These checkers analyze both the final artifacts and the execution traces, distinguishing between whether a rule is applicable and whether it has been satisfied. Experiments were conducted on 500 tasks from the SWE-bench Verified benchmark, involving a combination of four models, two agent frameworks, and two methods of providing rules, resulting in 8,000 total runs. In the Native setting, agents are required to retrieve the rules themselves, whereas in the Consolidated setting, they are provided with a curated file containing the relevant local rules. The primary results indicate an average violation rate of approximately 43.1% under the Native setting; providing a consolidated set of rules improves compliance by an average of about 8.7 percentage points. Furthermore, the authors validate the quality of the checkers and the stability of the observed trends through manual audits and experiments using a refined subset of the checkers.
Critical strengths
- By incorporating the contribution process into the evaluation, this work identifies critical failures that would remain invisible if only the final patch were examined. The paper distinguishes between solving a problem and completing a contribution that adheres to project requirements. Requirements such as running mandatory tests, utilizing specified tools, or providing necessary disclosures depend on the execution process or the submission materials—aspects that cannot be assessed by looking solely at the final code. This design is empirically supported: Figure 1 illustrates cases where a contribution passes functional tests yet violates project rules, while Figure 5 reveals that, under the authors' evaluation criteria, 50.3% of the violations occurring in functionally successful runs are identified through the execution trace. This demonstrates that auditing the execution trace provides insights beyond what can be gleaned from inspecting the final artifacts. Consequently, this approach expands the scope of failure types covered in the evaluation of coding agents and offers an actionable framework for analyzing real-world contribution workflows.
- The origins, applicability conditions, and implementation mechanisms of the rules are transparent, and the core findings have been subjected to meaningful robustness checks. Rather than presenting a set of opaque, automatically generated scores, the authors document the original text of each rule, its source, its context, and any exceptions; they also decouple the assessment of a rule's applicability from the assessment of its compliance. Furthermore, Appendix B.5.5 delineates the scope of responsibility—distinguishing between newly created artifacts, modified lines, and functions containing changes—which helps prevent the misattribution of pre-existing repository issues to the coding agents. Even more compellingly, the authors examine the sensitivity of their results to the use of approximate checkers. Table 8 shows that when the analysis is restricted to 294 exact checkers, providing the rules still improves compliance rates across all model and framework combinations, with variations in compliance rates remaining within 4.7 percentage points. This provides additional validation for the two fundamental observations: that clear rule violations occur, and that providing these rules improves performance.
Critical weaknesses
W1. Issues identified during the checker audit have not been incorporated into the scoring system; the error rates in actual operation remain insufficiently calibrated.
According to Appendix B.6.2, the weighted average acceptance rate for a sample of 150 checkers is 87.2% (or 83.7% if requiring consensus from two reviewers). The same section explicitly states that no checkers were corrected or removed following the audit. Consequently, checkers that were not accepted, or whose correctness could not be verified, remain part of the final evaluation.
However, the remaining proportion cannot be directly interpreted as an operational error rate, as the audit unit is the individual checker, and Appendix D.2 allows reviewers to mark a checker as rejected simply because its correctness could not be determined. Nevertheless, this implies that the current audit results are insufficient to quantify the extent to which these checkers cause false positives, false negatives, or shifts in model rankings during actual operation. The paper notes that the remaining issues primarily stem from overly broad applicability conditions—a type of error that simultaneously affects which rules are included in the scoring and which behaviors are flagged as violations.
While the precise checker comparison in Table 8 serves as useful mitigating evidence, the label precise refers only to the rule-evaluation logic; it does not guarantee the correctness of the applicability conditions or the implementation itself. The validation of individual Django checkers against real-world commits (pp. 18–19) is also helpful, though its scope is not representative of the entire corpus.
For a benchmark paper whose core contribution is the measurement of reliability, the authors must clarify the impact of these audited issues on actual operational performance and provide re-calculated scores after correcting or excluding the problematic checkers. While the current evidence confirms the existence of violations, it offers only limited support for the reported violation rates or for fine-grained comparisons between models.
W2. Compliance rates depend on the rules triggered by the agent and the available evidence; therefore, scores from different runs may not carry the same meaning.
Appendix C.4 clearly illustrates this issue. A simple commit lacking a pull request description might trigger 6 rules, all of which are passed, resulting in a 100% compliance rate. In contrast, a more comprehensive contribution might trigger 24 rules, passing 23 of them, which yields a 96% compliance rate. While reporting the trigger rate helps shed light on these discrepancies, it does not directly resolve the issue where runs with incomplete contributions might inadvertently achieve higher compliance scores.
Appendix B.5.2 further notes that nine checkers select only violating constructs, meaning compliant behaviors are labeled as not applicable. This contradicts the principle, outlined in Appendix D.3.5, that targets should be selected based on the applicability conditions of a rule, rather than exclusively targeting violations. This also creates an asymmetry in how pass and fail outcomes are factored into the denominator.
There are also significant category-based differences regarding the absence of evidence. Appendix C.3 indicates that only about 27% of triggered rules in the code quality category are scored, compared to approximately 64% in the testing category. Furthermore, 304 rules were never triggered across all 8,000 runs. While this does not imply that unscored rules should be counted as violations, it does mean that the observed compliance rate represents only a subset, one determined by the specific task, agent behavior, and the availability of evidence.
Consequently, the model variations presented in Table 3, as well as the transition from Native to Consolidated results, may reflect changes in both rule-following capability and the specific set of rules being scored. The paper should explicitly define its aggregation methodology and demonstrate—through comparisons based on a common set of scorable rules, stratification by contribution completeness, or other sensitivity analyses—that the primary conclusions are not driven by shifts in the scope of the evaluation.
W3. The conversion of natural language specifications into mandatory obligations involves unverified judgments, which are not covered by the current human review process.
According to Appendix D.1.3, criterion M4 allows a rule to be classified as a must if it specifies particular objects, constraints, or formats. Criterion M5 also permits classifying rules as must if they lack explicitly mandatory language but offer only a single way to achieve compliance—a classification that takes precedence over D1 which covers expressions of preference. Page 28 further clarifies that even when weaker language (such as avoid) is used, a rule may still be categorized as a must if it targets specific terms.
Whether a rule is specific or has an easily identifiable method of compliance is not inherently equivalent to whether a project treats it as a mandatory contribution obligation. The authors do document the context and exceptions where deviations are permitted, and the appendix provides examples of how similar rules are categorized differently—measures that are certainly commendable. However, the paper does not report how many rules entered the final corpus via M4 or M5 which are based on non-mandatory language. nor does it verify whether these classification judgments align with the project's own requirements.
This issue is particularly significant because Appendix D.2 explicitly instructs human reviewers that they need not assess whether the rules have been correctly classified. Consequently, the 87.2% acceptance rate by the checkers cannot be taken as validation that all 823 rules should indeed be treated as mandatory constraints.
Furthermore, Appendix B.5.3 includes 18 rules that require confirmation or endorsement by human contributors. While these could serve as metrics for the compliance of the overall contribution process, interpreting any failures here as a deficiency in an autonomous agent’s ability to follow rules, without further qualification, is problematic. The lack of a review of these classifications, combined with the absence of relevant ablation studies, means the final scores conflate project obligations, the authors' own normative interpretations, and conditions involving human participation that were not actually present in the experiments.
W4. The comparison between Native and Consolidated formats sheds light on the effectiveness of how rules are presented, but it is insufficient to definitively attribute remaining violations to reasoning bottlenecks.
These two configurations differ in both how rules are retrieved and how they are presented. The Native approach requires navigating scattered project documentation, whereas the Consolidated approach provides extracted, organized, atomic rules. Additionally, Appendix D.4.1 notes that Native rule files are exempt from the output truncation limits typically applied by the mini-SWE-agent. Thus, this comparison conflates several variables, including retrieval costs, text structure, and the manner in which information is delivered.
Moreover, Table 5 does not support the claim that all Consolidated runs had access to the full rule context. The average file-opening rate is 91.6%, with the figure for Kimi K2.5 under the mini-SWE-agent framework being only 42.6%. The 99.6% content delivery rate requires clarification regarding its denominator; it cannot be taken as evidence that every run received the complete set of rules.
Even when rules are successfully included in the context, any remaining violations could stem from execution failures, task prioritization issues, incomplete contributions, checker misclassifications, or the fact that certain rules inherently require human intervention. The current experiment does not sufficiently distinguish these sources from errors in rule comprehension or reasoning.
Consequently, while the paper can reasonably claim that consolidating and providing rules improves compliance performance, the assertions that the Consolidated condition isolates intrinsic reasoning capabilities and that remaining failures point to a reasoning bottleneck, require more direct, rule-specific or trajectory-specific evidence.
Questions for authors
-
What specific issues are associated with the checkers that were not accepted by the reviewers? How many of these represent confirmed implementation errors, and how many remain indeterminate? How frequently were these checkers triggered across the 8,000 runs? Do the primary violation rate, model ranking, and the utility provided by the rules remain stable after these checkers are corrected or excluded?
-
Does Table 3 employ an average of per-run ratios, an aggregation by rule, or some other method? How exactly are withheld rules factored into the denominator? If the comparison is restricted to rules that are both applicable and scorable for the same task in both settings, what is the magnitude of the utility provided by the Consolidated approach? Furthermore, how do runs with incomplete contributions affect these results?
-
Among the 823 rules, how many are classified as must primarily based on criteria M4 or M5? Has this classification been verified through an independent review? How do the overall results—as well as the results for the AI contribution specification categories—change if these rules which are open to interpretation and the 18 rules requiring human confirmation are excluded?
-
For runs where it is confirmed that the text of a given rule was included in the context, what proportion of violations can be definitively attributed—via manual trace analysis—to errors in rule interpretation or reasoning? Do the conclusions still hold if one excludes cases involving unopened rule files, incomplete execution, contested checkers, or requirements for human intervention?
-
A simple average of the compliance rates for the eight Native combinations in Table 3 yields a violation rate of approximately 43.1%. In contrast, Figure 4 and the main text report an average of 7.7 violations and 22.6 triggered rules per functionally successful sample—a rate of 34.1%. Please clarify the sample scope and aggregation methods used for both figures; specifically, is the 43.1% figure mentioned in the abstract intended to describe patches that are functionally correct? While both sets of figures could be valid under different statistical definitions, the current presentation is potentially confusing.
3 comments
Response to Reviewer hG2d (2/2): Q3, Q4
Q3. Rules classified must via M4/M5; independent review; exclusions.
- Count: 530 of the 823 rules are classified as "must" via M4 (411) or M5 (119).
- Review: the classification was reviewed manually and independently, and the released corpus records each rule's route and the source sentence it came from, so every label can be checked.
| Rules | n | Native | Consolidated | Δ |
|---|---|---|---|---|
| All (Table 3) | 823 | 56.95 | 65.70 | +8.75 |
| −M4/M5 | 293 | 48.30 | 58.70 | +10.40 |
| −M4/M5 −18 HITL | 275 | 53.25 | 64.55 | +11.30 |
- Overall: removing the M4/M5 rules lowers compliance in both settings. This indicates that agents fail even more often on explicitly stated rules, so the M4/M5 rules do not inflate our violations. Δ grows in both rows, up to +11.30, and is positive in all 8 cells.
- AI-contribution category: excluding both groups raises its compliance (Native 17.95% → 33.80%, Consolidated 28.65% → 50.10%), but it still has the lowest compliance of the 8 rule categories in both settings, and Δ grows from +10.70 to +16.30.
- Conclusion: our findings do not depend on these rules. Without them, overall compliance is lower than in Table 3, and the AI-contribution category is still the weakest category.
Q4. Violations with the rule text in context.
Thank you for this question. We will clarify how we interpret these violations in the revised paper.
- Rule in context (Consolidated): the agent gets all rules in a file. For each violation, we check not only that the agent opened the file, but that the violated rule's own sentence appeared in its view (in any command output) before its last action. In 89.4% of Consolidated violations, the agent had seen the rule it then broke; almost all of the other violations come from runs that never opened the file.
- Manual traces: for this rebuttal, we randomly selected 96 of these violations (12 per model × scaffold) and read each trajectory by hand. In all of them, the agent had opened the file and seen the rule. Only 5 show the agent stating its own reading of the rule (e.g. deciding it does not apply), so only these can be attributed to interpretation. In most others, the agent simply skips the step the rule requires, or does it wrongly. For example, one GPT-5.6 Luna run reads Matplotlib's rule "Do not commit changes to your local main branch" at the start, but later committed on
main. For these, the trajectory shows that the agent did not act on the rule. - Exclusions: excluding all four together, Consolidated compliance is 70.40%, so agents that have the rules still miss nearly a third of them. Δ is +9.90 (Table 3: +8.75), positive in all 8 cells, so the conclusions hold.
Gentle follow-up regarding our response
Dear Reviewer hG2d,
Thank you once again for your time in reviewing our paper. As the discussion period closes in two days, we would be very grateful to know whether our response and revisions have addressed your concerns.
We would be delighted to discuss any remaining issues or provide further details if needed.
Best regards,
-Authors
Response to Reviewer hG2d (1/2): Q1, Q2, Q5
We thank the reviewer for the precise questions and for noting the transparency of rule origins and applicability. The gain from providing the rules holds under every check below.
Q1. The 23 checkers not accepted by both raters.
Q2. Aggregation; withheld rules; common scorable set; incomplete contributions.
Q5. Is 43.1% meant to describe functionally correct patches?
We apologise for the confusing presentation, Figure 4 is where we analyse functionally correct patches in depth.