- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines approval histories for GitHub pull requests written by agents. It observes that many were merged without a recorded approving review and that, in some approved cases, the merged commit was different from the one reviewed. The authors then study selective prediction of logged human decisions and describe a ledger plus a check at commit time intended to associate approval with a specific artifact, policy, and execution route.
Critical strengths
- The analysis treats an unrecorded approval separately from approval of an earlier version and compares the merge head with the reviewed commit. Among 3,880 retrieved approved merges, 489 had moved past the last approved commit (Table 1). The code-only subset and repository-level resampling examine possible alternative explanations.
- Alongside AUROC, the study gives certified coverage and explains that the available sample cannot establish a certificate in some settings (Section 4, Table 2). This makes the negative finding more precise than a general statement about the safety of automation.
Critical weaknesses
- A missing approving review in the records does not establish that nobody examined the change or that repository rules were breached. Section 3 observes that many agent pull requests opened by developers are merged by their own authors. Before interpreting the 83.4% value as a failure of approval safeguards, the analysis should account for the branch rules and reviewer roles that actually applied. As presented, the figure includes workflows where a single owner may legitimately approve the change by merging it.
- Section 5 describes the guarded-commit design without a working implementation or outcome measurements. Its success depends on assumptions about credential restrictions, hashing the exact artifact, equivalence of canonical context, and repeat handling for external actions. An executable forge integration with edits made after approval could measure how often it blocks stale actions, causes mistaken blocks, handles races, and burdens operators.
Questions for authors
- For the 489 approved merges whose heads changed, how many repositories required approval of the current head under their rules at merge time? What did their branch settings permit or prevent?
- Can you show an operational commit-time check when an agent changes a reviewed artifact, including a test of the interval between checking the artifact and taking the external action?
Q1. Most stale merges were permitted by the repository's rules rather than violations of them. Under current default-branch rules, 69.9% of stale merges are in repositories that require an approval, without necessarily requiring approval of the latest head, and 88% (431 of 489) are in repositories with neither a dismiss-stale nor a last-push-approval rule. Head-binding rules work where they exist (0 of 53 stale under a dismiss-stale ruleset that predates the merge, and 3 of 232 under last-push approval), but they apply to 23.9% of approved merges (Section 3, Appendix A). Merge-time settings are not public, so we read the current rules of the default branch with ruleset creation and update dates. Bypass permissions are not visible with read access. This is why the contract makes head binding a condition of every irreversible action rather than a per-repository option.
Q2. The adapter closes the check-to-action interval by binding the merge to the approved head. It passes the approved head into the merge call (REST sha, GraphQL expectedHeadOid), so a push between check and merge makes the merge fail. In 5,000 randomized interleavings on our test forge this gave 0 stale merges, against 955 with separate check and merge calls (0 of 300 against 53 on bare Git). On GitHub itself, the strong composition and a weaker configuration without last-push approval or the expected head each gave no stale merge in 30 races, because GitHub refused every merge that a push preceded or overlapped. For the publish action outside a forge, the adapter sends the approved content itself, so a later edit cannot reach the endpoint. After an agent edits a reviewed artifact, the adapter blocks and returns it to review (Section 5).
The 83.4% is a missing record, which the submission already explains. The submission states this mechanism (Section 3: "GitHub does not let authors approve their own PRs, so a developer who merges their own agent's PR alone cannot have an approval on record") and does not call these merges rule violations. The revised abstract calls the 83.4% a missing record concentrated in single-owner repositories (52.2% above 1,000 stars), and the automation test excludes these workflows (reviewed subset, Section 4). We hope that the branch-rule analysis and the operational test answer your questions, and we would be grateful if you would reconsider your score.