acceptodds
Under review as a conference paper at ICLR 2027

Guarded Commits: Transactional Human Approvals for LLM Workflows

Abstract

Large language model (LLM) workflows often require human approval before an irreversible action, such as merging a pull request that a coding agent opened. Many agent frameworks record the approval apart from the component that performs the action. Unless the code-hosting platform is configured to enforce it, nothing checks that the approval covers what is merged. We ask whether this matters and whether approval could be automated instead. In 30,796 agent-authored pull requests, 83.4% of merges carry no approving review, and 12.6% of 3,880 approved merges went in at a code version other than the one approved. We then use Learn-then-Test to bound, with stated confidence, the share of auto-approved items that a human rejected. On 8,077 reviewed agent pull requests and 17,007 writing-assistant decisions, no method that sees only the proposed change and its origin certifies any automation at a 5% bound on the full held-out sets at our sample size, and LLM judges certify at most 3.1% on a stricter subset. This holds for trained predictors and for frontier LLM judges, the best of which reaches an area under the ROC curve (AUROC) of 0.77. The reviewers' own pre-decision signals, which partly encode the decision, raise AUROC to 0.88. These results motivate guarded commits, which store each approval with its evidence in an append-only ledger. The only component that can perform the action checks the approval against the exact artifact, policy version, and executed steps. Automated decisions also need a recorded risk certificate, and reused decisions must point to the human decision they reuse.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM workflows, human approval, human-in-the-loop, AI coding agents, code review, approval integrity, risk control, Learn-then-Test, selective prediction, LLM-as-a-judge, guarded commits

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p9ALOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines approval histories for GitHub pull requests written by agents. It observes that many were merged without a recorded approving review and that, in some approved cases, the merged commit was different from the one reviewed. The authors then study selective prediction of logged human decisions and describe a ledger plus a check at commit time intended to associate approval with a specific artifact, policy, and execution route.

Critical strengths
  1. The analysis treats an unrecorded approval separately from approval of an earlier version and compares the merge head with the reviewed commit. Among 3,880 retrieved approved merges, 489 had moved past the last approved commit (Table 1). The code-only subset and repository-level resampling examine possible alternative explanations.
  2. Alongside AUROC, the study gives certified coverage and explains that the available sample cannot establish a certificate in some settings (Section 4, Table 2). This makes the negative finding more precise than a general statement about the safety of automation.
Critical weaknesses
  1. A missing approving review in the records does not establish that nobody examined the change or that repository rules were breached. Section 3 observes that many agent pull requests opened by developers are merged by their own authors. Before interpreting the 83.4% value as a failure of approval safeguards, the analysis should account for the branch rules and reviewer roles that actually applied. As presented, the figure includes workflows where a single owner may legitimately approve the change by merging it.
  2. Section 5 describes the guarded-commit design without a working implementation or outcome measurements. Its success depends on assumptions about credential restrictions, hashing the exact artifact, equivalence of canonical context, and repeat handling for external actions. An executable forge integration with edits made after approval could measure how often it blocks stale actions, causes mistaken blocks, handles races, and burdens operators.
Questions for authors
  1. For the 489 approved merges whose heads changed, how many repositories required approval of the current head under their rules at merge time? What did their branch settings permit or prevent?
  2. Can you show an operational commit-time check when an agent changes a reviewed artifact, including a test of the interval between checking the artifact and taking the external action?
1 comment

Q1. Most stale merges were permitted by the repository's rules rather than violations of them. Under current default-branch rules, 69.9% of stale merges are in repositories that require an approval, without necessarily requiring approval of the latest head, and 88% (431 of 489) are in repositories with neither a dismiss-stale nor a last-push-approval rule. Head-binding rules work where they exist (0 of 53 stale under a dismiss-stale ruleset that predates the merge, and 3 of 232 under last-push approval), but they apply to 23.9% of approved merges (Section 3, Appendix A). Merge-time settings are not public, so we read the current rules of the default branch with ruleset creation and update dates. Bypass permissions are not visible with read access. This is why the contract makes head binding a condition of every irreversible action rather than a per-repository option.

Q2. The adapter closes the check-to-action interval by binding the merge to the approved head. It passes the approved head into the merge call (REST sha, GraphQL expectedHeadOid), so a push between check and merge makes the merge fail. In 5,000 randomized interleavings on our test forge this gave 0 stale merges, against 955 with separate check and merge calls (0 of 300 against 53 on bare Git). On GitHub itself, the strong composition and a weaker configuration without last-push approval or the expected head each gave no stale merge in 30 races, because GitHub refused every merge that a push preceded or overlapped. For the publish action outside a forge, the adapter sends the approved content itself, so a later edit cannot reach the endpoint. After an agent edits a reviewed artifact, the adapter blocks and returns it to review (Section 5).

The 83.4% is a missing record, which the submission already explains. The submission states this mechanism (Section 3: "GitHub does not let authors approve their own PRs, so a developer who merges their own agent's PR alone cannot have an approval on record") and does not call these merges rule violations. The revised abstract calls the 83.4% a missing record concentrated in single-owner repositories (52.2% above 1,000 stars), and the automation test excludes these workflows (reviewed subset, Section 4). We hope that the branch-rule analysis and the operational test answer your questions, and we would be grateful if you would reconsider your score.

Reviewer_DUmLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work makes the observation that human approvals of agent-generated pull requests are frequently unrecorded or do not correspond to the artifact that is ultimately merged. It also studies whether trained predictors or LLM judges could replace the human approval step with a certified bound on the false-approve rate. It proposes a potential solution where guarded-commit contract that records each approval with its evidence and permits the irreversible action only after a verification that the approval covers the actual action (e.g. merging) being completed.

Critical strengths

This work exposes an important problem which is that human approvals of agent actions are frequently not recorded, or do not match what actually gets merged. This is very relevant since coding agents are increasingly opening PRs on their own and the same approval step guards other irreversible actions such as deployments.

Critical weaknesses
  • The analysis does not take into account the criticality of the PRs. It also does not account for the traffic (i.e. importance) of the repos.
    • Not all the PRs are equal in their impact, and this is not always correlated with their size (e.g. number of lines). Many high impact fixes are only a few lines in length.
    • Many of the high traffic repos already have a mature review process and also implement branch protection. The dataset is dominated by single-author repos with likely no mature review process. The analysis might be skewed by this.
    • Some other information (e.g. automated CI tests) are also unaccounted in the analysis.
  • The main contribution, which is the guarded commit design is not evaluated.
    • The authors could potentially run retrospective tests on the existing dataset to see if things improve if the mechanism was implemented.
    • What would be more difficult to show would be to test the mechanism against simple branch protections. It would be nice to show the potential overheads to human labor, and whether if it effectively reduces bad PRs.
Questions for authors
  • How would the conclusions change if we only admit repos that are more significant (e.g. with more than 1k stars)? This would remove the effect of the single-person repos.
  • Would requiring the CI tests to pass change the outcomes of the analysis?
1 comment

Q1. Single-owner repositories drive the missing approvals, not the stale rate or the automation result. They drive the no-approval and self-merge rates. Above 1,000 stars (642 repositories), merges without approval fall from 83.4% to 52.2% and self-merges from 82.3% to 49.4%, while the stale rate is 12.1% (11.3% above 10,000), against 12.6% overall. The automation test already excluded these workflows (Appendix B: "The two largest single-owner repositories (about 40% of all PRs) contribute no reviewed PRs"). Above 1,000 stars, no artifact predictor or judge certifies any auto-approval at α = 5%, and at most 1.0% at α = 10%, also when retrained on those repositories.

Q2. Adding CI does not change the result. T2 with a CI signal reaches AUROC 0.680 (from 0.670) and certifies 0.1% at α = 5% and 3.3% at α = 10%, against 0.1% and 3.5% for T2 alone on the same 100 partitions, drawn separately from those used in Table 2. GitHub no longer returns check runs for these PRs (133 of 7,628 fetched reviewed PRs return any), so the signal comes from CI bot comments posted before the decision. PRs with passing CI are still rejected 23.7% of the time (32.4% overall), and CI alone reaches AUROC 0.52.

Stale approvals are more frequent on critical PRs, and size is a poor proxy. The stale rates are 22.4% for security-sensitive paths, 16.7% for CI or build configuration, and 15.1% for dependency manifests, against 7–9% for tests, documentation, and other code. No artifact predictor or judge certifies any auto-approval at α = 5% in any class. As you note, small changes matter too, since 72 of the 115 substantive post-approval changes are under 50 lines.

The retrospective test you suggest is in Section 5. It gives 489 blocks, all of them stale merges, 368–489 second reviews depending on exemptions, and a median added delay of 0.2–0.7 hours each. The submission had already reported the first-order cost ("Enforcing condition 3 on the audited merges would have sent 489 of the 3,880 approved merges (12.6%) back to review", Section 5). Appendix F shows where branch protection suffices and what it lacks. We would be grateful if you would take these results into account in your score.

Reviewer_Fwp5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether human approvals in LLM-agent workflows reliably authorize the artifacts eventually acted upon, and whether such approvals can be replaced by automated decisions. An analysis of 30,796 decided agent-authored pull requests finds that 83.4% of merged PRs have no recorded approving review, while 12.6% of retrieved approved merges were performed on commits different from the last approved version. The authors further evaluate automated approval decisions using Learn-then-Test risk control on agent-generated PRs and writing suggestions. The considered artifact-based predictors and LLM judges generally cannot certify useful auto-approval coverage at a 5% false-approval bound with available calibration data. Motivated by these observations, the paper proposes guarded commits, a transaction-oriented design binding approvals to artifacts, policies, execution paths and recorded decisions.

Critical strengths

S1. The paper presents a practical empirical investigation. The audit highlights a meaningful integrity issue that approval may be recorded separately from the action it authorizes. Public agent-generated PRs provide evidence on missing and stale recorded approvals, and approval-to-artifact binding is important for trustworthy agent systems.

S2. Risk-controlled evaluation is performed, besides accuracy results. The paper asks whether automatic approval can meet a false-approval bound, analyzes certification limits and sample sizes, and distinguishes lack of certification from proof that safe automation is impossible. These lead to interesting downstream directions.

Critical weaknesses

W1. Principal system remains specified but unimplemented.

  • Section 5 (pp. 7–8) proposes a class of trusted manager, plan validator, append-only ledger and action adapter, but states that no deployed system is evaluated (p. 7). The intended guarantees under concurrency, crashes, retries, policy changes and adversarial agents remain untested.

  • credential confinement and trusted-component correctness are, as is, assumptions. GitHub branch protection, Gerrit version-tied approval, and human-approval workflows in agent SDKs cover overlapping functionality; This part could be clarified in a more modularized method with clarified roles and algorithmic or agentic pipelines. A prototype with fault injection and feature-by-feature comparisons would also help establish what this integrated design newly guarantees.

W2. Evidence does not fully justify broad claims about oversight or automation.

The 83.4% without an approving review (Section 3, pp. 3–4) does not imply absence of human inspection; developer-run agents can have PRs merged by their owners. A changed commit after formal approval need not indicate a substantively dangerous modification; the paper does include a useful code-only analysis. Section 4 (pp. 4–6) predicts observed merges/closures, which reflect priorities and governance as well as quality. The conclusions should remain limited to recorded decisions and the observed calibration setting. The generality should be better characterized with a more principled, methodology level clarification. I want to clarify this does not mean the paper needs to be significantly rewritten as a theoretical/algorithmic paper; but a proper combination of generalizable lessons or principles that are verified over wider test cases and the benefit be demonstrated over more downstream tasks.

Questions for authors

Q1 (W1, Section 5, pp. 7–8). Could the authors provide a prototype or simulation demonstrating exact artifact binding and action semantics under concurrent changes, crashes, retries and external calls?

Q2 (W1, Sections 2 and 5, pp. 2 and 8). Which guarantees exceed a composition of branch protection, signed artifacts, approval state and transactional execution? A feature comparison and failure demonstrations would clarify novelty.

Q3 (W2, Section 3, Table 1, p. 3; Section 4, p. 4): Can absent formal reviews be distinguished from absent human oversight? Would conclusions change using independent artifact correctness/risk judgments instead of merge/close outcomes?

1 comment

W1/Q1. The revision implements the contract and tests each case you list (Section 5). A racing push gave 0 stale merges in 5,000 trials against 955 without head binding, a decision-derived idempotency key gave exactly one effect in 280 crash-and-retry trials, and policy changes, ledger tampering, and calls without the credential were blocked in all 52 fault configurations. The component roles you ask about are those of Section 5 of the submission ("Trust and transactions"): transaction manager, plan validator, ACT adapter, and ledger, each with the checks listed in the contract box. Credential confinement is now enforced rather than assumed. The adapter runs in a separate process that alone holds the credentials, and a sandboxed adversarial agent recovered no credential in 20 of 20 attempts, while under SDK-style approval it did in 20 of 20 (Appendix F, Table 5). The guarantee still assumes the correctness of the four trusted components and their isolation. Exactly-once effects require an endpoint that honors the stable idempotency key, and deployment across hosts also requires authenticated transport.

Q2. A composition of existing mechanisms matches guarded commits on merges but not on policy changes, automation, or publishing. Section 2 of the submission already states that existing mechanisms each cover part of the contract, and that "None combines all three: binding the approval to the exact artifact, checking that the executed path included the required approval, and requiring a certificate for automated decisions. Moreover, forge rules do not apply to actions outside the forge, such as publishing or submitting." Appendix F now gives the property-by-property table and a test of the strongest composition (general response). It matches guarded commits on merge attacks under a fixed policy and fails on policy changes, bot approvals, and publishing. The policy failure also occurred on GitHub itself, in 10 of 10 trials. Inside a fully configured forge the composition does bind approval to the artifact, but it is rarely configured (23.9% of approved merges), and where a dismiss-stale ruleset predates the merge it works (0 of 53 stale).

W2/Q3. The submission already limits both claims to records, and the stale changes are often substantive. Section 3: "That developer may well have read the diff, but there is no record of what they inspected." Section 4: "Given the recorded pre-decision context, we ask whether these methods can reproduce the recorded human outcomes at useful certified coverage." The revision repeats the first point in the abstract and introduction. To test whether stale changes matter, an LLM labeled the 298 retrievable post-approval changes (of 299) against a rubric fixed in advance. Of these, 115 (38.6%, Wilson 95% interval 33.2–44.2%) change logic, behavior, dependencies, CI, or security-relevant code, and 72 of these are under 50 lines. By these labels, 115 of 3,880 approved merges (3.0%) merged substantive code without a recorded approval for that version. The labeling subset excludes later base-branch merges and requires the approved commit to remain in history. An author labeled a random 60 of these changes without seeing the LLM labels, and the two agree on whether a change is substantive for 56 of 60 (Cohen's kappa 0.87) and on the category for 53 of 60. All four disagreements are changes the author judged substantive and the LLM did not (three loosened or rewrote tests), so the LLM's 38.6% is, if anything, an undercount (author 50.0%, LLM 43.3% on the same 60). For the automation labels, the result holds on the strict subset, with superseded closes removed (T0–T2 at most 0.2% at α = 5%), and on the writing workload, whose labels are single human choices. On generality, we test two decision workloads (agent PRs and writing suggestions) and one action outside a forge (publishing). Other actions such as payments may differ, and Section 6 says so. We hope that the implementation and these analyses answer your main concerns, and we would be grateful if you would reconsider your score.

Comments on the paper

2 comments

The revision implements the guarded-commit contract and evaluates it with fault injection and a replay of all 3,880 retrieved approved agent merges, in response to the requests for operational evidence. It also adds the requested analyses of repository size, CI, branch rules, PR criticality, and the content of post-approval changes (Sections 3–5, Appendices A, E, F). Every change is marked in blue in the revised PDF. The rest of the text is as submitted.

The prototype implements the contract and passes the reported fault and attack tests under the stated trust assumptions (Section 5). It consists of a hash-chained append-only ledger with prefix pinning, the transaction manager, a plan validator that checks the executed-path witness, and an ACT adapter that is the only component given the credentials. It drives an in-process forge, a bare-Git forge, and a publish endpoint outside any forge. All 52 fault configurations gave the expected outcome in every trial: pushes after approval, policy changes within a transaction, seven kinds of ledger tampering, four kinds of invalid reuse, uncertified automation, four ways of skipping the barrier, and calls without the credential. With a push racing the merge on the in-process forge, passing the approved head to the merge call, which GitHub's merge API supports, gave 0 stale merges in 5,000 randomized trials, against 955 without it (0 of 300 against 53 on bare Git). These are controlled-backend results, not observed GitHub race failures. After a crash and retry, an idempotency key derived from the decision record gave exactly one effect in 280 trials at an endpoint that honors that key, while a fresh key per attempt duplicated the publication after all 70 crashes that followed the call. The commit-time check takes 0.1–0.3 ms (median) with up to 10,000 ledger entries.

The adapter runs in its own operating-system process and alone holds the credentials. The agent runs in a separate sandboxed process with none. A scripted adversarial agent tried to read the credential, call the endpoints directly, publish a different draft after approval, replay or forge decision records, reuse an approval for another draft, and submit an uncertified automated decision. All of these attacks were blocked in every trial (200 per ordinary attack and 20 credential-theft attempts; Table 5 also reports 400 race trials). Under SDK-style approval, where the agent holds the credential and a human approves the tool call, the same agent read the credential and published a different draft after approval in every trial.

We also built the strongest configuration available today: required approval, dismiss-stale and last-push rules, a merge proxy in its own process that holds the credential and passes the expected head, and a publish proxy behind a human approval. It blocks the same merge attacks as guarded commits, including the race. It fails in four cases, each in all 200 trials: a policy relaxed between approval and merge lets an unapproved head merge (guarded commits pin the policy version), a bot approval is accepted without a risk certificate, and an edited draft, or a different draft handed to the proxy, is published (Appendix F, Table 5). We repeated the merge attacks on a GitHub repository with real rulesets and a second account as the agent. Guarded commits blocked all 70 trials, and the composition again merged an unapproved head in all 10 trials in which an administrator relaxed the ruleset after approval. This was an injected administrator-side policy change, not an agent privilege escalation.

Enforcement blocks exactly the 489 stale merges, and its cost is mostly benign re-reviews (Section 5). The replay checks that the implementation enforces the audit's definition and estimates the resulting review burden. The adapter blocks the 489 stale merges and no merge whose head was approved. Under strict head binding, 489 merges (12.6%) would require another review. Alternative policies reduce this to 428 with a base-branch exemption and 368 with an additional documentation-only exemption. The replay reuse key admits one additional PR beyond the 60 base-branch-only cases in the first-order count, explaining 428 rather than 429. The estimated median added delay is 0.2–0.7 hours per second review (90th percentile 30–39 hours), inferred from historical review timings rather than measured after deployment. Of the 298 labeled post-approval changes, 183 are formatting, comments, tests, or lockfiles. These categories illustrate potential review burden but do not establish harmlessness. The manual check found substantive test changes missed by the LLM.

The contract adds head binding outside forges and two admission rules that no existing mechanism enforces (Appendix F). For code in a forge with dismiss-stale or last-push rules enabled and the expected head passed to the merge API, existing mechanisms already bind the approval to the artifact, as Section 2 of the submission says. In the current default-branch settings, these rules cover repositories accounting for 23.9% of approved merges, and 88% of stale merges are in repositories with neither (Section 3). These figures do not reconstruct the rules at merge time. The contract makes approval-to-artifact binding a condition of each protected action. The implementation evaluates this for merges and publishing, and sending and payments are examples of further applications. The evaluated mechanisms do not enforce both admission rules in full: reuse requires an equivalent canonical context, a still-valid reviewer role, and a fresh record pointing directly to a human decision in the pinned ledger prefix (4a), and automation requires a risk certificate (4b). Existing components can be extended to supply these checks. Our contribution is the explicit contract and its uniform enforcement at the action boundary, rather than an impossibility claim about composing existing tools.

Corrections (all in the revision, and no conclusion changes). (1) The trained models' PR context was the concatenation of per-commit patches, not the net diff the paper stated. The two differ for 65% of reviewed PRs. We reran all trained models on the net diff and updated every value in Table 2 and Appendix E. T0–T2 certify at most 0.1% at α = 5% (was 0%) and 2.7% at α = 10% (was 1.9%). Five statements change. T3 under chronological calibration certifies 35.8% at α = 5% (was 0%, with partition SD 23 points and Bonferroni 1.6%). With superseded closes removed, T0–T2 certify at most 0.2% at α = 5% and 5.4% at α = 10% (was 0% and 2.0%). Exact-match reuse disagrees with the recorded outcome 21% of the time (was 64%), and learned reuse certifies 2.1% at α = 10% (was 0.6%). Exact-duplicate diffs conflict 27% of the time and near-duplicates 13.0% (was 20.3%), so we drop the agent-retry explanation. We reran the Claude and GLM judges on the net diff, with GLM-5.3 pinned (AUROC 0.751, was 0.726, and 0.716, was 0.710, both still 0% certified at α = 5%). GPT-6-Astra saw the per-commit text (it differs from the net diff for 66% of judged PRs), and the revision states this. (2) The close-reason rule matched more phrases than Appendix B listed. Restricted to the four stated reasons, the superseded share is 10.8% (was 12.1%). (3) Chronological calibration tunes on the first sixth of the time-ordered holdout, calibrates on the next third, and tests on the last half. The submission described it as halves. (4) Judges' pooled top-5% rates now average over the order of tied scores: Claude 5.1% on the net diff (was 3.7%). (5) Partition standard errors reach 2.3 points on the full holdout and 3.6 on the strict subset, not "about 2".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.