acceptodds
Under review as a conference paper at ICLR 2027

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

Abstract

Autonomous coding agents now resolve a substantial share of real-world GitHub issues. However, passing functional tests differs fundamentally from producing a high-quality contribution acceptable for merging. Mature open-source projects publish repository-specific contribution policies, spanning style, git, testing workflows, to ensure code quality and long-term maintainability. Because existing benchmarks evaluate patches solely on unit tests, agent compliance with repository governance remains unknown. In this paper, we introduce SWE-CC, a benchmark evaluating code and process compliance in autonomous software engineering. We develop a semi-automated pipeline that converts developer documentation across 12 open-source repositories into 823 machine-checkable atomic policies. SWE-CC introduces two features: (1) lightweight, deterministic checker functions that represent each policy, and (2) a comprehensive auditing mechanism that inspects both agent runtime behaviors and final deliverables. We evaluate the compliance of agent workflows in 500 end-to-end software contribution tasks extended from SWE-bench Verified. Our evaluation of four LLMs under two agent scaffolds shows that modern agents suffer from coding compliance issues: although agents produce functionally correct patches, they still violate 43.1% of applicable project policies, with nearly half of all violations occurring during intermediate execution steps. These results show that functional correctness does not guarantee real-world readiness, highlighting that future software engineering agents must reliably conform to repository governance to enable safe and trustworthy deployment.

Primary area:
datasets and benchmarks
Keywords:
policy compliance, coding agents, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hG2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-CC, a framework designed to evaluate whether coding agents adhere to the contribution guidelines of software repositories. The authors argue that passing functional tests is insufficient to confirm that a contribution meets a project's requirements; therefore, they expand the scope of evaluation to include commit messages, pull request descriptions, testing procedures, documentation, code style, and the disclosure of AI usage. The authors extracted 1,759 atomic rules from the developer documentation of 12 repositories. After filtering for scope and mandatory status, they implemented 823 of these rules as executable checkers. These checkers analyze both the final artifacts and the execution traces, distinguishing between whether a rule is applicable and whether it has been satisfied. Experiments were conducted on 500 tasks from the SWE-bench Verified benchmark, involving a combination of four models, two agent frameworks, and two methods of providing rules, resulting in 8,000 total runs. In the Native setting, agents are required to retrieve the rules themselves, whereas in the Consolidated setting, they are provided with a curated file containing the relevant local rules. The primary results indicate an average violation rate of approximately 43.1% under the Native setting; providing a consolidated set of rules improves compliance by an average of about 8.7 percentage points. Furthermore, the authors validate the quality of the checkers and the stability of the observed trends through manual audits and experiments using a refined subset of the checkers.

Critical strengths
  1. By incorporating the contribution process into the evaluation, this work identifies critical failures that would remain invisible if only the final patch were examined. The paper distinguishes between solving a problem and completing a contribution that adheres to project requirements. Requirements such as running mandatory tests, utilizing specified tools, or providing necessary disclosures depend on the execution process or the submission materials—aspects that cannot be assessed by looking solely at the final code. This design is empirically supported: Figure 1 illustrates cases where a contribution passes functional tests yet violates project rules, while Figure 5 reveals that, under the authors' evaluation criteria, 50.3% of the violations occurring in functionally successful runs are identified through the execution trace. This demonstrates that auditing the execution trace provides insights beyond what can be gleaned from inspecting the final artifacts. Consequently, this approach expands the scope of failure types covered in the evaluation of coding agents and offers an actionable framework for analyzing real-world contribution workflows.
  2. The origins, applicability conditions, and implementation mechanisms of the rules are transparent, and the core findings have been subjected to meaningful robustness checks. Rather than presenting a set of opaque, automatically generated scores, the authors document the original text of each rule, its source, its context, and any exceptions; they also decouple the assessment of a rule's applicability from the assessment of its compliance. Furthermore, Appendix B.5.5 delineates the scope of responsibility—distinguishing between newly created artifacts, modified lines, and functions containing changes—which helps prevent the misattribution of pre-existing repository issues to the coding agents. Even more compellingly, the authors examine the sensitivity of their results to the use of approximate checkers. Table 8 shows that when the analysis is restricted to 294 exact checkers, providing the rules still improves compliance rates across all model and framework combinations, with variations in compliance rates remaining within 4.7 percentage points. This provides additional validation for the two fundamental observations: that clear rule violations occur, and that providing these rules improves performance.
Critical weaknesses

W1. Issues identified during the checker audit have not been incorporated into the scoring system; the error rates in actual operation remain insufficiently calibrated.

According to Appendix B.6.2, the weighted average acceptance rate for a sample of 150 checkers is 87.2% (or 83.7% if requiring consensus from two reviewers). The same section explicitly states that no checkers were corrected or removed following the audit. Consequently, checkers that were not accepted, or whose correctness could not be verified, remain part of the final evaluation.

However, the remaining proportion cannot be directly interpreted as an operational error rate, as the audit unit is the individual checker, and Appendix D.2 allows reviewers to mark a checker as rejected simply because its correctness could not be determined. Nevertheless, this implies that the current audit results are insufficient to quantify the extent to which these checkers cause false positives, false negatives, or shifts in model rankings during actual operation. The paper notes that the remaining issues primarily stem from overly broad applicability conditions—a type of error that simultaneously affects which rules are included in the scoring and which behaviors are flagged as violations.

While the precise checker comparison in Table 8 serves as useful mitigating evidence, the label precise refers only to the rule-evaluation logic; it does not guarantee the correctness of the applicability conditions or the implementation itself. The validation of individual Django checkers against real-world commits (pp. 18–19) is also helpful, though its scope is not representative of the entire corpus.

For a benchmark paper whose core contribution is the measurement of reliability, the authors must clarify the impact of these audited issues on actual operational performance and provide re-calculated scores after correcting or excluding the problematic checkers. While the current evidence confirms the existence of violations, it offers only limited support for the reported violation rates or for fine-grained comparisons between models.

W2. Compliance rates depend on the rules triggered by the agent and the available evidence; therefore, scores from different runs may not carry the same meaning.

Appendix C.4 clearly illustrates this issue. A simple commit lacking a pull request description might trigger 6 rules, all of which are passed, resulting in a 100% compliance rate. In contrast, a more comprehensive contribution might trigger 24 rules, passing 23 of them, which yields a 96% compliance rate. While reporting the trigger rate helps shed light on these discrepancies, it does not directly resolve the issue where runs with incomplete contributions might inadvertently achieve higher compliance scores.

Appendix B.5.2 further notes that nine checkers select only violating constructs, meaning compliant behaviors are labeled as not applicable. This contradicts the principle, outlined in Appendix D.3.5, that targets should be selected based on the applicability conditions of a rule, rather than exclusively targeting violations. This also creates an asymmetry in how pass and fail outcomes are factored into the denominator.

There are also significant category-based differences regarding the absence of evidence. Appendix C.3 indicates that only about 27% of triggered rules in the code quality category are scored, compared to approximately 64% in the testing category. Furthermore, 304 rules were never triggered across all 8,000 runs. While this does not imply that unscored rules should be counted as violations, it does mean that the observed compliance rate represents only a subset, one determined by the specific task, agent behavior, and the availability of evidence.

Consequently, the model variations presented in Table 3, as well as the transition from Native to Consolidated results, may reflect changes in both rule-following capability and the specific set of rules being scored. The paper should explicitly define its aggregation methodology and demonstrate—through comparisons based on a common set of scorable rules, stratification by contribution completeness, or other sensitivity analyses—that the primary conclusions are not driven by shifts in the scope of the evaluation.

W3. The conversion of natural language specifications into mandatory obligations involves unverified judgments, which are not covered by the current human review process.

According to Appendix D.1.3, criterion M4 allows a rule to be classified as a must if it specifies particular objects, constraints, or formats. Criterion M5 also permits classifying rules as must if they lack explicitly mandatory language but offer only a single way to achieve compliance—a classification that takes precedence over D1 which covers expressions of preference. Page 28 further clarifies that even when weaker language (such as avoid) is used, a rule may still be categorized as a must if it targets specific terms.

Whether a rule is specific or has an easily identifiable method of compliance is not inherently equivalent to whether a project treats it as a mandatory contribution obligation. The authors do document the context and exceptions where deviations are permitted, and the appendix provides examples of how similar rules are categorized differently—measures that are certainly commendable. However, the paper does not report how many rules entered the final corpus via M4 or M5 which are based on non-mandatory language. nor does it verify whether these classification judgments align with the project's own requirements.

This issue is particularly significant because Appendix D.2 explicitly instructs human reviewers that they need not assess whether the rules have been correctly classified. Consequently, the 87.2% acceptance rate by the checkers cannot be taken as validation that all 823 rules should indeed be treated as mandatory constraints.

Furthermore, Appendix B.5.3 includes 18 rules that require confirmation or endorsement by human contributors. While these could serve as metrics for the compliance of the overall contribution process, interpreting any failures here as a deficiency in an autonomous agent’s ability to follow rules, without further qualification, is problematic. The lack of a review of these classifications, combined with the absence of relevant ablation studies, means the final scores conflate project obligations, the authors' own normative interpretations, and conditions involving human participation that were not actually present in the experiments.

W4. The comparison between Native and Consolidated formats sheds light on the effectiveness of how rules are presented, but it is insufficient to definitively attribute remaining violations to reasoning bottlenecks.

These two configurations differ in both how rules are retrieved and how they are presented. The Native approach requires navigating scattered project documentation, whereas the Consolidated approach provides extracted, organized, atomic rules. Additionally, Appendix D.4.1 notes that Native rule files are exempt from the output truncation limits typically applied by the mini-SWE-agent. Thus, this comparison conflates several variables, including retrieval costs, text structure, and the manner in which information is delivered.

Moreover, Table 5 does not support the claim that all Consolidated runs had access to the full rule context. The average file-opening rate is 91.6%, with the figure for Kimi K2.5 under the mini-SWE-agent framework being only 42.6%. The 99.6% content delivery rate requires clarification regarding its denominator; it cannot be taken as evidence that every run received the complete set of rules.

Even when rules are successfully included in the context, any remaining violations could stem from execution failures, task prioritization issues, incomplete contributions, checker misclassifications, or the fact that certain rules inherently require human intervention. The current experiment does not sufficiently distinguish these sources from errors in rule comprehension or reasoning.

Consequently, while the paper can reasonably claim that consolidating and providing rules improves compliance performance, the assertions that the Consolidated condition isolates intrinsic reasoning capabilities and that remaining failures point to a reasoning bottleneck, require more direct, rule-specific or trajectory-specific evidence.

Questions for authors
  1. What specific issues are associated with the checkers that were not accepted by the reviewers? How many of these represent confirmed implementation errors, and how many remain indeterminate? How frequently were these checkers triggered across the 8,000 runs? Do the primary violation rate, model ranking, and the utility provided by the rules remain stable after these checkers are corrected or excluded?

  2. Does Table 3 employ an average of per-run ratios, an aggregation by rule, or some other method? How exactly are withheld rules factored into the denominator? If the comparison is restricted to rules that are both applicable and scorable for the same task in both settings, what is the magnitude of the utility provided by the Consolidated approach? Furthermore, how do runs with incomplete contributions affect these results?

  3. Among the 823 rules, how many are classified as must primarily based on criteria M4 or M5? Has this classification been verified through an independent review? How do the overall results—as well as the results for the AI contribution specification categories—change if these rules which are open to interpretation and the 18 rules requiring human confirmation are excluded?

  4. For runs where it is confirmed that the text of a given rule was included in the context, what proportion of violations can be definitively attributed—via manual trace analysis—to errors in rule interpretation or reasoning? Do the conclusions still hold if one excludes cases involving unopened rule files, incomplete execution, contested checkers, or requirements for human intervention?

  5. A simple average of the compliance rates for the eight Native combinations in Table 3 yields a violation rate of approximately 43.1%. In contrast, Figure 4 and the main text report an average of 7.7 violations and 22.6 triggered rules per functionally successful sample—a rate of 34.1%. Please clarify the sample scope and aggregation methods used for both figures; specifically, is the 43.1% figure mentioned in the abstract intended to describe patches that are functionally correct? While both sets of figures could be valid under different statistical definitions, the current presentation is potentially confusing.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hG2d (1/2): Q1, Q2, Q5

We thank the reviewer for the precise questions and for noting the transparency of rule origins and applicability. The gain from providing the rules holds under every check below.

Q1. The 23 checkers not accepted by both raters.

  • What they are: of the 150 audited checkers, 127 were accepted by both raters. Of the other 23, the raters had different judgments on 9, both rejected 13 with a specific issue named, and both rejected 1 without naming an issue (indeterminate). Of the 13, 7 apply too broadly (to cases the rule does not cover) and the other 6 are too permissive.
  • How often: over the 8,000 runs, the 23 account for 1.9% of all rule triggers and 3.0% of all violations; the 13 with a named issue account for 0.4% of violations.
RulesNativeConsolidatedΔ
All 823 (Table 3)56.9565.70+8.75
−23 not accepted by both57.4065.90+8.50
−13 with a named issue56.9065.60+8.70
  • Stability: excluding them moves compliance by at most 0.45 pp; Δ stays at +8.50 or more and is positive in all 8 model × scaffold cells; the model ranking is unchanged in all four scaffold × setting orders. Our findings are therefore stable with or without these checkers.

Q2. Aggregation; withheld rules; common scorable set; incomplete contributions.

  • Aggregation: by cell, as defined in App. C.4. For each model × scaffold × setting, we pool all (run, rule) verdicts and compute compliance = passed / (passed + failed); the Average row is the plain mean of the cells.
  • Withheld rules: counted as triggered, but not in the denominator.
  • Same rules in both settings: scoring each task only on the rules graded in both its Native and its Consolidated run gives Δ +8.70, averaged over the 8 cells as in Table 3 (+8.75); it is positive in all 8 cells, and the model ranking is unchanged in both settings. This is consistent with our finding that the gain comes from providing the rules.
  • Incomplete contributions (17% of runs, mostly without a PR description): they comply slightly less than complete ones (55.1% vs 57.4% in Native), so they do not inflate compliance conclusion. Dropping every task with an incomplete run gives Δ +9.05 (Table 3: +8.75), positive in all 8 cells.

Q5. Is 43.1% meant to describe functionally correct patches?

  • 43.1%: all 4,000 Native runs, resolved and unresolved. For each of the 8 Native cells of Table 3, violation = failed / (passed + failed); 43.1% is the average of the 8 cells.
  • 7.7 of 22.6 (Figure 4): both numbers are averages over resolved runs only. 22.6 is how many rules apply to the run, including withheld rules, which are not graded. So 7.7 / 22.6 is not meant as a violation rate.
  • Functionally correct patches: no. 43.1% covers all Native runs, not only functionally correct patches. Computed the same way on resolved runs only, it is 42.6%, so the statement holds for functionally correct patches as well.

We apologise for the confusing presentation, Figure 4 is where we analyse functionally correct patches in depth.

AuthorsOpenReview ↗

Response to Reviewer hG2d (2/2): Q3, Q4

Q3. Rules classified must via M4/M5; independent review; exclusions.

  • Count: 530 of the 823 rules are classified as "must" via M4 (411) or M5 (119).
  • Review: the classification was reviewed manually and independently, and the released corpus records each rule's route and the source sentence it came from, so every label can be checked.
RulesnNativeConsolidatedΔ
All (Table 3)82356.9565.70+8.75
−M4/M529348.3058.70+10.40
−M4/M5 −18 HITL27553.2564.55+11.30
  • Overall: removing the M4/M5 rules lowers compliance in both settings. This indicates that agents fail even more often on explicitly stated rules, so the M4/M5 rules do not inflate our violations. Δ grows in both rows, up to +11.30, and is positive in all 8 cells.
  • AI-contribution category: excluding both groups raises its compliance (Native 17.95% → 33.80%, Consolidated 28.65% → 50.10%), but it still has the lowest compliance of the 8 rule categories in both settings, and Δ grows from +10.70 to +16.30.
  • Conclusion: our findings do not depend on these rules. Without them, overall compliance is lower than in Table 3, and the AI-contribution category is still the weakest category.

Q4. Violations with the rule text in context.

Thank you for this question. We will clarify how we interpret these violations in the revised paper.

  • Rule in context (Consolidated): the agent gets all rules in a file. For each violation, we check not only that the agent opened the file, but that the violated rule's own sentence appeared in its view (in any command output) before its last action. In 89.4% of Consolidated violations, the agent had seen the rule it then broke; almost all of the other violations come from runs that never opened the file.
  • Manual traces: for this rebuttal, we randomly selected 96 of these violations (12 per model × scaffold) and read each trajectory by hand. In all of them, the agent had opened the file and seen the rule. Only 5 show the agent stating its own reading of the rule (e.g. deciding it does not apply), so only these can be attributed to interpretation. In most others, the agent simply skips the step the rule requires, or does it wrongly. For example, one GPT-5.6 Luna run reads Matplotlib's rule "Do not commit changes to your local main branch" at the start, but later committed on main. For these, the trajectory shows that the agent did not act on the rule.
  • Exclusions: excluding all four together, Consolidated compliance is 70.40%, so agents that have the rules still miss nearly a third of them. Δ is +9.90 (Table 3: +8.75), positive in all 8 cells, so the conclusions hold.
AuthorsOpenReview ↗

Gentle follow-up regarding our response

Dear Reviewer hG2d,

Thank you once again for your time in reviewing our paper. As the discussion period closes in two days, we would be very grateful to know whether our response and revisions have addressed your concerns.

We would be delighted to discuss any remaining issues or provide further details if needed.

Best regards,

-Authors

Reviewer_izH9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SWE-CC, a benchmark for evaluating code and process compliance in autonomous software engineering. The authors focus on producing high-quality contributions acceptable for merging rather than only passing functional tests. SWE-CC introduces deterministic checker functions that represent each policy and an auditing mechanism that inspects agent runtime steps and final deliverables.

Critical strengths
  • Addresses an important problem.
  • Focuses on repository-specific policy compliance.
  • Evaluates both intermediate steps and final deliverables.
Critical weaknesses
  • The benchmark covers only 12 repositories.
  • There is no evidence linking compliance scores to actual merge decisions.
Questions for authors

Major

(Q1) Do different levels of policy importance matter when computing compliance scores?

(Q2) The 43.1% violation rate is interesting. Is there any intuition as to why agents continue to violate policies even when all policy content is available in context?

(Q3) Do compliance scores correlate with real-world acceptance of contributions?

Minor:

  • The font size in Figure 1 is too small, making the text difficult to read.
3 comments
AuthorsOpenReview ↗

Response to Reviewer izH9: Q1–Q3 and Figure 1

We thank the reviewer for the positive assessment and are glad the reviewer values evaluating both intermediate steps and final deliverables. We answer each question with new analyses.

Q1. Do different levels of policy importance matter?

  • Our choice: we do not weigh rules by importance. Instead, we treat every rule in a project's contribution guidelines as important, since the project wrote it for every contributor to follow.
  • Only required rules: we classify each rule as mandatory (must) or optional (should) from the project's own wording, keep only the mandatory ones, and drop the 166 optional ones (App. D.1.3).

Q2. Why do agents violate rules that are in context?

  • The rules really are in context: our Consolidated setting is this case, where every rule is given to the agent in a file before it starts. Compliance is still only 65.70%, and for 89.4% of the violations the trace shows that the violated rule reached the agent before its last action.
  • Intuition 1, rules read at the start are not recalled later: in a random sample of 96 violations we read by hand (12 per model × scaffold), nearly every time the agent skipped a required step, its reasoning never mentions the rule again after the first read.
  • Example: In a GPT-5.6 Luna's run, its first command printed Matplotlib's rules, including "Do not commit changes to your local main branch". After fixing the bug, it committed directly on main, and its trajectory never mentions or checks the rule again after that first read.
  • Intuition 2, agents are overwhelmed by the rule list: a project can have up to 167 rules, all given at the start. The agent must work out which rule applies at which step, and it often does not. We conducted a small pilot study to support this: sending the agent only the rules it had broken, once at the end, raised compliance from 39.0% to 69.0% (14 SymPy tasks, Gemini 2.5 Flash), without breaking any resolved patch.

Q3. Do compliance scores correlate with real-world acceptance?

  • Data: 480 real PRs from our 12 repositories that close an issue and carry an AI tool's signature (12 tools, e.g. Claude Code, Codex, Copilot and Cursor): 260 merged, 220 rejected. We ran each repository's own checkers on its PRs' diff, commits and description.
  • Filtering: merging also depends on things compliance does not cover, so we keep only PRs where compliance can matter. We drop maintainers' own PRs (33% of merged PRs), which are accepted on the author's standing. Among rejected PRs, we keep only those a maintainer closed for breaking the contribution rules (e.g. AI policy, PR template), and leave out those closed as duplicates, for wrong code, or without any comment. This leaves 175 merged and 42 rejected PRs.
  • Result: the more rules a PR follows, the more often it is merged.
PRs grouped by compliancePRsMerged
Lowest quarter (20–67%)5567%
Second quarter (67–75%)5481%
Third quarter (76–83%)5485%
Highest quarter (83–100%)5489%

Within each repository, compliance correlates with merging (r = +0.20, p = 0.02).

Minor. We enlarged the text in Figure 1. The 12 repositories are the scope of SWE-bench Verified, not a limit of SWE-CC (§3). Our pipeline applies to any repository with written contribution guidelines (§3.1), so the benchmark can be extended to more repositories.

AuthorsOpenReview ↗

Gentle follow-up regarding our response

Dear Reviewer izH9,

Thank you once again for your time in reviewing our paper. As the discussion period closes in two days, we would be very grateful to know whether our response and revisions have addressed your concerns.

We would be delighted to discuss any remaining issues or provide further details if needed.

Best regards,

-Authors

Thank you for addressing my comments and providing further context. I maintain my original score.

Reviewer_ayBqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that passing the unit tests is not the same as producing a pull request a maintainer would merge, and builds a way to check the rest. A model-driven pipeline reads the contributor documentation of the twelve SWE-bench Verified repositories, extracts 1,759 atomic rules, keeps the 823 that are mandatory and mechanically checkable, and compiles each into a small Python checker with a precondition and a verdict. Checkers inspect the agent's diff, commit, PR text and command history, ship with authored test cases, and 150 of them were audited blind by two raters, who agreed 94% of the time and accepted about 87%. Four models on two scaffolds run the 500 SWE-bench Verified tasks with a prompt that also asks for tests, a commit and a PR description, once with only the documentation URL and once with the extracted rules mounted as a file. Agents violate about 43% of the rules that apply to them, handing them the rule list improves this by about nine points without changing the resolve rate, half the violations are visible only in the command history, agents try to fetch the documentation in under a third of runs, and AI-disclosure rules are the least followed.

Critical strengths

The checkers are deterministic, test-backed and human-audited, which is a better oracle than the LLM judges most compliance benchmarks use. Each one has a precondition, authored cases, an ownership mode so pre-existing code is not graded, and an explicit "undetermined" verdict that is excluded from the denominator. The ranking of models survives restriction to the 294 checkers labelled exact, and the audit reports agreement and a proper interval.

Critical weaknesses
  • The conclusions are already published. RepoComplianceBench (arXiv 2607.26819) ran 106 issues from 49 repositories on four frontier agents and reported that agents almost never fetch the contribution rules on their own, that disclosure rules are the least followed, and that quoting the rules or adding verifier feedback helps only partly. SWE-Gate (arXiv 2609.04167) showed that a third of functionally passing repairs violate the reviewers' constraints. Those are SWE-CC's four findings. What is new is the scope, 823 rules covering style, testing and commit conventions rather than AI-contribution rules alone, and the deterministic checkers. But the new scope is not where the headline comes from. The paper never breaks the 43% down, so I did it from Table 17 (verdict counts per category) and Table 4 (compliance per category). AI-disclosure rules alone account for about 40% of all violations: only 37 rules, but they fire in nearly every run and agents pass 15%. PR-metadata rules add 19%. So 94 of 823 rules produce close to 58% of the violations, 18 of the disclosure checks require something only a human can do, and outside those two families compliance runs 60 to 85%. The "half of violations happen in the trajectory" result has the same shape: 73 checkers, a few of which (run the suite, run pre-commit) fire every time. Strip out the rules the earlier paper already covered and the ones an agent cannot satisfy, and what remains is a better oracle confirming a known result. Table 1 also marks RepoComplianceBench as not auditing agent behaviour, which its abstract contradicts. A table of violations by category and by the twenty worst rules, the headline with and without the disclosure family and the human-only checks, and a finding the earlier benchmarks could not have produced, would change my view.

  • Some of the 43% is checker error and the paper does not say how much. The audit found about 13% of checkers do not encode their rule, mostly because the precondition fires on more cases than the rule covers, which creates spurious violations. The authors left all of them in and never report results on the 127 checkers both raters accepted. No number has a confidence interval; every cell is one run, and a one-point gain for Kimi K2.5 is counted as part of a "consistent increase." Table 15 shows GPT-5.6 Luna, the model the abstract names, resolving 79% of tasks where the public leaderboard has 93% with the same scaffold, which the authors attribute to using the wrong model class. The abstract's 43% is also the native-condition average only, and the 44.7% pinned on GPT-5.6 Luna is its OpenHands number.

Questions for authors
  • What are the top twenty rules by violation count, what share of all violations do they account for, and how many distinct rules does a typical run violate? (Clarification)
  • What is compliance restricted to the 127 doubly-accepted checkers, and excluding the 18 human-in-the-loop checks? (Clarification)
  • How is a documentation-retrieval attempt detected, and is Kimi K2.5's 3.6% attempt rate a detector gap or a prompt-following failure? (Clarification)
  • How robust are the trajectory checkers to alternate invocations of the same tool (python -m pytest, tox, runtests.py, pre-commit versus calling black directly), and what fraction of trajectory violations are "did not run the suite or linter" rules? (Clarification)
  • Table 1 says RepoComplianceBench does not audit agent behaviour, but its abstract says it judges the trajectory against the repository's rules. Please correct this and soften the "first" claim. (Suggestion)
4 comments
AuthorsOpenReview ↗

Response to Reviewer ayBq (1/3): Novelty, Table 1, results without AI rules

We thank the reviewer for the detailed review and for re-deriving the category breakdown, and we are glad the reviewer finds our deterministic, audited checkers a better oracle than LLM judges. We address each question below.

W1/Q5. Overlap with RepoComplianceBench (RCB) and SWE-Gate; Table 1.

Thank you for pointing out the error about RCB. We will correct it, together with all other changes from this discussion, in a revised paper that we will upload. SWE-CC's novelty lies in two points, and the two papers are contemporaneous work:

  1. More comprehensive compliance auditing: SWE-CC audits every written contribution rule of a project (823 rules across 8 categories: style, tests, docs, Git, PR metadata, AI use, code quality and specialized changes) on both the final deliverables and the agent's trajectory, with deterministic checkers and no LLM judge. Prior work covers one slice: RCB covers four types of AI-contribution rule and grades them with an LLM judge; SWE-Gate checks code-level constraints on the final patch only (Table 1).
  2. Independence from task difficulty. Every rule applies to every task in its repository, whereas RCB and SWE-Gate test one rule per task. SWE-CC can therefore separate the difficulty of a rule from the difficulty of a task (finding below).
  3. Contemporaneous work. Both papers are arXiv-only and appeared 58 and 22 days before the deadline, i.e. contemporaneous under the ICLR 2027 guide; we cite them and acknowledge the overlap in the retrieval and disclosure findings.

Table 1: RCB does audit the trajectory, so we will mark it as supported under "agent behavior", fix our related-work sentence and narrow the "first" claims.

Why does independence matter?

  • The problem with one rule per task: in RCB and SWE-Gate, each rule is tested on its own task. When an agent fails, the cause could be the rule or that particular task (e.g. a hard task where the agent never reaches the PR step), and this design cannot separate the two.
  • Our design: every rule is checked on every task of its repository (e.g. Django's AI-disclosure rule on all 231 Django tasks), by every model and scaffold. If agents break a rule on easy and hard tasks alike, the cause is the rule, not the task.
  • Result: compliance depends mostly on the rule, not on the task or the model. So SWE-CC shows which rules agents fail, independently of the tasks chosen.
Share of pass/fail variation explained byAll rulesWithout AI and human-only rules
Rule72%65%
Task5%6%
Model<1%<1%
Scaffold<1%<1%

W1. Headline without the AI and human-only rules; by category.

  • By category: Table 4 gives the compliance rate per category, and Table 17 gives the triggered and graded counts.
  • Headline without these rules (compliance rate, %, as in Table 3):
Rules keptnNativeConsolidatedΔ
All rules (Table 3)82356.9565.70+8.75
Without AI-contribution and human-only rules78567.1074.70+7.60
Without AI-contribution, PR-metadata and human-only rules72872.8079.60+6.80
  • Result: without the rules RCB already covers (AI contribution) and those an agent cannot satisfy (human-only), agents still break about a third of the remaining rules in Native (67.10% compliance), and providing the rules still raises compliance in all 8 cells (+7.60). Even after also removing PR-metadata rules, 89% of Native runs break at least one rule. So our findings hold beyond the rules RCB studied.
  • Why we keep them: these rules are part of the projects' written contribution guidelines.
AuthorsOpenReview ↗

Response to Reviewer ayBq (2/3): Q1–Q4

Q1. Top-20 rules; distinct rules per run.

  • Top 20 rules: they account for 54.1% of the 31,072 Native violations.
  • Why so concentrated: SWE-bench Verified takes 61% of its tasks from Django and SymPy, and these 20 rules apply to every PR in these two projects, so they are checked on almost every run. 7 of the 20 are Git, style, code-quality or test rules.
#RuleCategoryNative violations
1Django: disclose AI tools usedAI1,804
2Django: AI agent self-reports possible violationsAI1,796
3Django: check AI-assisted work against the checklistAI1,464
4Django: document new features and behaviour changesPR metadata1,416
5Django: format with blackStyle1,273
6Django: commit message starts "Fixed #ticket"PR metadata1,013
7Django: lines of at most 88 charactersStyle687
8SymPy: no mostly AI-generated code without human oversightAI586
9SymPy: complete the PR templatePR metadata583
10SymPy: mark which code is AI-generatedAI583
11SymPy: release-notes entry in the PRPR metadata582
12SymPy: release-notes block between the markersPR metadata582
13SymPy: disclose how AI was usedAI582
14SymPy: work on a new branchGit578
15SymPy: no merge, add, commit or rebase on masterGit572
16SymPy: no commits on masterGit570
17SymPy: pass bin/test qualityCode quality565
18SymPy: run bin/test and bin/doctestTests539
19SymPy: no AI-written PR descriptionAI511
20SymPy: no AI speaking for the contributorAI511
  • Distinct rules per run: a typical Native run violates 7 rules (half of all runs violate 5–9). Figure 4 shows the distribution for resolved runs.

Q2. Compliance on the 127 doubly-accepted checkers, and without the 18 human-in-the-loop (HITL) checks.

RulesNative triggeringNative complianceConsolidated triggeringConsolidated compliance
All 823 (Table 3)24.6556.9526.1065.70
823 − 18 HITL23.9059.9025.4068.85
127 accepted by both raters30.4562.4033.2071.15
  • Without HITL: compliance rises by about 3 pp in both settings, as expected for rules an agent cannot pass.
  • The 127 are a random sample: they come from the 150 checkers drawn at random, stratified by project, for the rater audit (App. B.6).
  • Why the 127 differ: a small subset does not hold the same mix of rules as the full set, and compliance depends mostly on the rule (W1/Q5), so the average of our subset shifts. For example, the table shows that the 127 trigger more often. The gap is not checker error, as removing the 23 checkers not accepted by both raters from the full set moves Native compliance only from 56.95 to 57.40.

Q3. How retrieval attempts are detected; Kimi's 3.6%.

  • What counts as an attempt: a fetch (curl, wget, urllib, requests, or the OpenHands browser) of a page from which at least one of our rules was extracted. A retrieval also requires the fetch to return the rule text (at least 1,500 characters of policy prose).
  • How we detect it: an agent can only read a web page by running a command, and every command and its output is stored in the trajectory. So every attempt leaves a trace. The detector scans every command of every run for a fetch and checks its target against the pages the rules came from.
  • Same for all models: the prompt, tools and detector are identical for all four models.
  • Manual check: we randomly selected 60 Kimi runs marked "no attempt" (30 per scaffold). The label is correct in all 60: none fetches a page that contains rules. Kimi's 3.6% is therefore a prompt-following failure, not a detector gap.

Q4. Robustness to alternate invocations; share of "did not run the suite or linter" rules.

  • Accepted by design: each checker is written to accept every invocation the project accepts and searches the full command log, so wrappers such as cd … && or bash -c still match.
  • Example: Django's checklist checker (DJANGO-C059) counts tests/runtests.py, python -m pytest, pytest, tox, manage.py test or django-admin test as a test run, and black, flake8, isort, ruff or pre-commit as a style check.
  • One runner only when the project names it: e.g. SymPy's bin/test and pytest's tox -e linting.
  • "Did not run" rules: 18.0% of trajectory violations (9.0% of all violations). Without them, trajectory violations are still 45.4% of all violations (50.3% in Figure 5).
AuthorsOpenReview ↗

Gentle follow-up regarding our response

Dear Reviewer ayBq,

Thank you once again for your time in reviewing our paper. As the discussion period closes in two days, we would be very grateful to know whether our response and revisions have addressed your concerns.

We would be delighted to discuss any remaining issues or provide further details if needed.

Best regards,

-Authors

AuthorsOpenReview ↗

Response to Reviewer ayBq (3/3): Checker error, confidence intervals, headline numbers

W2. Checker error; confidence intervals; headline numbers; GPT-5.6 Luna's resolve rate.

  • Checker error: the 13% are the checkers not accepted by both raters in the audit (23 of 150). Removing them moves Native compliance only from 56.95% to 57.40%, and the gain stays positive in all 8 model × scaffold cells. Results on the 127 doubly-accepted checkers are in Q2 above. We will mark the 23 in the release and report scores with and without them.
  • Confidence intervals: we computed 95% intervals with a paired task-level bootstrap (2,000 resamples of the 500 tasks). Native violation is 43.1% [42.0, 44.1], Consolidated 34.3% [33.6, 35.0], and the gain +8.75 pp [8.0, 9.5]. Each task is a separate agent run, so these intervals also reflect the variation between runs. The gain is positive in all 8 cells, and its interval excludes zero in every cell. We agree that the gain for Kimi K2.5 on mini-SWE-agent is small (+1.0 [0.1, 2.0]) and will reword "consistent increase" accordingly. The resolve rate does not change in any cell.
  • Headline numbers: both are Native numbers. We use Native as the headline because it is the realistic setting: the agent must find the guidelines itself, as a new contributor would. We will name this setting in the abstract. When the policies are provided (Consolidated), the violation rate is 34.3% (Table 3). For GPT-5.6 Luna, the introduction will report both scaffolds: 44.7% (OpenHands) and 48.2% (mini-SWE-agent).
  • GPT-5.6 Luna's resolve rate: our runs are not standard SWE-bench runs. Besides the fix, the task prompt asks for tests, documentation, compliance with the project's policies, a git commit and a pull-request description (App. D), so public resolve rates serve only as a sanity check. Against the Vals AI results on the same scaffold (App. C), three of the four models are within 5 points, two of them above. We run every model under one shared, unmodified configuration so that models are compared under identical conditions. Our compliance findings do not depend on resolve rate: on resolved runs only, the Native violation rate is 42.6%, against 43.1% on all runs.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response: Summary of the Discussion Period

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their time and constructive feedback. Reviewers recognized that SWE-CC addresses an important problem (izH9), offers an actionable framework for auditing real-world contribution workflows beyond the final patch (hG2d), relies on deterministic checkers superior to typical LLM judges (ayBq), and is supported by meaningful robustness checks (hG2d).

Below is a summary of the primary concerns raised during the review process and how we resolved them:

  • Ablations: Reviewers ayBq and hG2d had initial concerns regarding the robustness of our results to checker quality, rule selection, and scoring choices. We addressed them by re-scoring all runs under every requested exclusion (disputed checkers, AI-contribution/human-only rules, and incomplete runs), and the performance gain from providing rules remains positive across all 8 model × scaffold configurations in every case (95% CI: [8.0, 9.5] pp).
  • Data Analysis: Reviewers ayBq, izH9 and hG2d requested a deeper analysis of agent violations, which we provided with new analyses on our existing runs. The rules agents violate most are process steps beyond the code fix (e.g., AI disclosure, changelog entries, linting), and violation depends primarily on the rule itself rather than the task or model, an effect that single-rule-per-task setups cannot capture. Agents also fail to apply rules they have seen: 89.4% of violations occur after the rule has already entered the agent's context. Finally, across 480 real-world AI-assisted pull requests, PRs with higher compliance are merged more often (67% vs. 89% between the lowest and highest compliance quarters).
  • Related Work: Reviewer ayBq had initial concerns regarding related work, but was satisfied with our clarifications on experimental setting differences, our novelty justification, and the contemporaneous nature of the baseline (released within 2 months), ultimately raising their score.

We sincerely thank the Area Chair and all reviewers again for helping us improve the paper.

Best regards,

Authors of Paper 23614

AuthorsOpenReview ↗

Thank you for your response and updated rating

We thank you very much for your willingness to increase the rating of our paper. We appreciate your recognition that our paper is contemporaneous with the mentioned work, and your agreement with our justification.

Reviewer_ayBqOpenReview ↗

Weak Reject

I have two critical concerns outstanding due to which I am editing my score to a weak reject:

Reject Reason 1: Policies and tasks come from different eras. The policies are extracted from each project’s latest documentation (August 2026), but the tasks are historical SWE-bench Verified snapshots. The paper never checks whether a policy can be satisfied at a task’s base commit. Evidence in the paper: several rules are violated almost always and barely move when the rule is handed to the agent. Pylint’s copilot-instructions rule stays at 98.8%, pytest’s pre-commit rule goes from 99.3% to 100%, and scikit-learn’s changelog-fragment rule goes from 99.2% to 99.6%. Evidence in the rebuttal: Django’s “format with black” and “lines of at most 88 characters” rules are in the top 20. To my knowledge Django adopted black in early 2022, after many of these tasks’ base commits.

Reject Reason 2: There is no human baseline. The checkers are never run on the merged human pull requests for the same instances. Without that, we cannot tell whether contributions the maintainers accepted would pass, and so what a 43.1% violation rate means.

AuthorsOpenReview ↗

Response to Reviewer ayBq: Policy era and human baseline

We address both concerns below.

R1. Policies and tasks come from different eras.

  • Checkers and documentation are consistent: each checker is built from the same documentation the agent receives at test time, either given in full (Consolidated) or pointed to (Native). Every verdict therefore measures whether the agent followed the documentation it had. This matches practice: a contributor who fixes a long-standing bug today must follow the project's current guidelines, however old the code is.
  • A near-constant violation rate does not mean a rule cannot be met: for example, for pytest's pre-commit rule, the configuration file exists at the base commit of all 19 tasks, yet agents almost never run it. Agents skip process steps even when the rule is in their context: in 89.4% of Consolidated violations, the violated rule had already reached the agent's context. This is one of our main findings.

R2. Human baseline.

  • Why not the SWE-bench PRs: they were merged in 2016–2023, before many of today's rules existed. Scoring them against 2026 rules would not show how accepted contributions comply with the current rules.
  • What we did: we collected real PRs from the same 12 repositories, opened in 2025–2026, the same period as the guidelines. We kept PRs that close an issue and carry an AI tool's signature (e.g. Claude Code, Codex, Copilot). We left out maintainers' own PRs, which are accepted on the author's standing. Among rejected PRs, we kept those a maintainer closed for breaking the contribution rules (e.g. AI policy, PR template). This leaves 175 merged and 42 rejected PRs. We ran each repository's checkers on their diff, commits and description. Real PRs have no trajectory, so we use the 710 rules that read only these.
  • Accepted contributions pass most rules: merged PRs comply with 77.1% of them.
  • The fewer rules a PR violates, the more often it is merged:
PRs grouped by compliancePRsMerged
Lowest quarter (20–67%)5567%
Second quarter (67–75%)5481%
Third quarter (76–83%)5485%
Highest quarter (83–100%)5489%

Within each repository, compliance correlates with merging.

  • What this means for our agents: on the same 710 rules, an average Native run complies with 64.3%, which falls in the lowest quarter. With the rules provided (71.1%), it moves to the second quarter.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.