acceptodds
Under review as a conference paper at ICLR 2027

MPC-Patch-Bench: Beyond Functional Tests for LLM Repair of MPC Software

Abstract

Repairing secure multi-party computation (MPC) software requires more than producing outputs that pass existing tests. Patches must also respect intended information disclosure and numerical behavior. We introduce MPC-Patch-Bench, a repository-level benchmark containing 205 executable repair tasks from five MPC frameworks. Its construction combines MPC-specific pull-request filtering with expert-reviewed synthesis of missing problem statements and regression tests. Evaluation supplements functional tests with differential checks against plaintext references and static rules targeting potential MPC-specific violations. Across eight evaluated LLMs, Sonnet 4.6 achieves the highest functional resolution rate, 22.9%, but falls to 16.1% after the additional checks; Opus 4.6 then ranks highest at 17.1%. Across models, these checks reject from 11% to 40% of functionally passing patches. The results show that functional-test performance and performance under additional MPC-specific checks are distinct evaluation signals. The benchmark supports studying this gap without treating check-passing patches as formally secure.

Primary area:
datasets and benchmarks
Keywords:
Secure Multi-Party Computation, Large Language Models, Automated Program Repair, Code Benchmark, Privacy-Preserving Machine Learning, Software Security, Differential Testing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KvaXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new benchmark for MPC patch synthesis. The benchmark construction is done using an LLM based pipeline with some human intervention, which takes in issues from github repositories, and a range of LLMs are evaluated on these benchmarks.

Critical strengths
  • The benchmark seems reasonably broad: benchmarks are taken from multiple different repositories as sources. It also is not yet saturated by the LLMs evaluated (which includes GPT 5.4) so demonstrates that there is a gap of problems that cannot be solved by LLMs at least up to GPT 5.4.

  • Whilst MPC is not my area, to a non-expert this seeems like a valuable resource that the community would benefit from seeing public.

Critical weaknesses

1 - It's not clear that the methodology behind construction pipeline can be reused for obtaining benchmarks in other domains. Pehraps the authors could elaborate on its generality?

2 - The writing needs improving: it's fairly hard to extract from the paper on first reading what the contributions actually are (is it a benchmark suite, is it a benchmark extraction pipeline, is it the MPC verifier?). The authors should add a more succinct contributions list at the end of the introduction.

3 - The figures are also confusing and slightly contradictory. Figure 1 contains the data curation framework, and (I think) the MPC verifier, but the data curation framework in figure 1 is slightly different to the data curation framework in Figure 1, and the MPCVerifier in figure 3 is slightly dfifferent to the parts of Figure 1 that I think correspond to the MPCVerifier. I suggest the authors compress these into one diagram, so that the information isn't presented multiple times in slightly different ways.

4 - There is also some confusing use of abbreviations. For instance, under FIgure 1 it says "MPC-specific static checks (SAST)" without defining what SAST stands for. The definition for SAST happens several pages later in the paper.

5 - I would like some more clarity on the guarantees (if any) of the MPCVerifier. If there are no guarantees, then an empirical evaluation of the accuracy should be added.

6 - I would also like an empirical evaluation of how accurate the benchmark extraction pipeline, and how frequently human intervention is required

Questions for authors

Please could you respond to weaknesses 1, 5 and 6 (there is no need to respond to the others unless you wish to correct something I have misunderstood. I trust that, if the paper is accepted, you will fix these presentation issues)

2 comments

W1. Which parts of construction carry over to other domains? On the same 7,305 pull requests, our pipeline produced 205 executable tasks, compared with 42 under SWE-bench-style extraction. The difference also affected which libraries could be studied. CrypTen contributed 61 tasks after completion versus zero under the baseline, and MP-SPDZ contributed 41 versus one. The reusable steps are to screen candidate changes, send uncertain cases to a domain expert, complete missing task descriptions and regression tests, and validate each task against the original fix. Adapting this process requires a rubric for relevant changes, expert review of the intended behavior, and tests and runners appropriate to the new domain. Our evidence for coverage comes from the five MPC frameworks.

W5. What does the verifier guarantee, and how accurate is it? Passing means that a patch passes the implemented static rules and agrees with the plaintext reference on the numerical inputs tested, within the configured tolerance. The verifier provides no formal guarantee of protocol security. Experts confirmed 47/57 (82.5%) static flags on functionally passing model patches, with 10/57 (17.5%) false alarms. Numerical checks detected 15/15 (100%) independently documented CrypTen precision bugs. On 205 upstream gold patches, static rules flagged 11/205 (5.4%), while numerical checks flagged 25/205 (12.2%). All 25 numerical failures were attributed to version incompatibility rather than patch defects. Of these, 24 shared an identical error signature, and one used an API unavailable at the time of the commit. These measurements answer different questions: the static audit measures the precision of flagged violations, while the numerical evaluation measures detection on a specified set of bugs.

W6. How accurate is extraction, and how much human intervention is required? On the same 200 labeled PRs, Curator achieved 93.10% precision and 81.00% recall (F1: 86.63%), outperforming both keyword screening (70.53% precision, 67.00% recall, F1: 68.72%) and a plain LLM (83.10% precision, 59.00% recall, F1: 69.01%). For the 205 final instances, all problem statements were reviewed, and 136 were rewritten. The corresponding numbers are 138/205 for test sets and 133/205 for F2P tests. Details are provided in Table C. Of 7,305 PRs, the first stage admitted 596 automatically and routed 579 to expert review. Automatic admission applied only to this screening decision. During completion, an expert validated every synthesized instance before inclusion. Appendix D reports that 163 of the 205 final instances used statements and tests produced during completion. We did not record the exact time per instance. The total time spent by an MPC expert on the 205 instances was approximately 80 hours.

Table B. Verifier validation.

Check and evaluation setWhat to reportResult
Static rules on flagged, functionally passing model patchesConfirmed / all audited, false alarms or indeterminate47/57 confirmed (82.5%); 10/57 false alarms (17.5%)
Static rules on upstream gold patchesFlagged / evaluated, expert disposition of flags11/205 (5.4%)
Numerical checks on independently documented CrypTen precision bugsDetected / tested among cases that pass ordinary tests, with selection criteria15/15 (100%)
Numerical checks on upstream gold patchesNumerical failures / evaluated, adjudicated false alarms and unresolved findings25/205 (12.2%). All 25 were attributed to version incompatibility: 24/25 (96.0%) shared an identical error signature across unrelated early commits, while 1/25 (4.0%) called an API that did not exist at the time of the commit.

Table C. Measure extraction accuracy

MethodTPFPTNFNAccuracyPrecisionRecallF1
Keyword6728723369.50%70.53%67.00%68.72%
Plain LLM5912884173.50%83.10%59.00%69.01%
Curator816941987.50%93.10%81.00%86.63%

We measured screening on a 200-PR sample. Sampling. We sampled 100 of the 205 tasks and 100 PRs that it rejected after expert review, which are generic Python infrastructure, documentation, CI and dependency changes. Because the set includes rejected PRs, missed positives are measurable. Definition. A PR is MPC-relevant if its patch changes MPC protocol logic, such as secret-sharing arithmetic, secure comparison, truncation or share conversion. Changes to build, CI, documentation, packaging or generic Python code are not. Labeling. One author with 2 years of MPC experience labeled all 200 PRs from the PR description and diff, without seeing any method's output.

We reviewed all 205 tasks' problem statements, test sets and F2P tests. For problem statements, 69 (64 verbatim + 5 lightly edited) are from upstream, 67 are rewritten and 69 are entirely new. For the test sets, 61 (58 verbatim + 3 adapted) are from upstream, 138 are new (written later), and 6 are mixed. For F2P tests, 72 instances are entirely from upstream, while 133 instances are rewritten. We do not record the exact time per instance. The total time for the 205 instances is around 80 hours for an MPC expert.

Reviewer_5Rb1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MPC-Patch-Bench is a repository-level benchmark of 205 executable repair tasks drawn from five MPC frameworks. Construction combines a domain-specific curation agent that filters 7,305 GitHub PRs down to 1,175 MPC-relevant candidates with a human-AI completion engine that synthesizes missing problem statements and fail-to-pass tests, yielding 205 validated instances compared to 42 from a SWE-bench-style extraction on the same corpus. Evaluation supplements functional tests with MPC-specific dynamic checks against plaintext references and static analysis for potential MPC violations. Across eight evaluated LLMs, the best functional resolution rate is 22.9%, which drops to 16.1% under the additional checks, and the additional checks change the top-ranked model.

Critical strengths
  1. The construction problem is real and the solution is well motivated.

SWE-bench style extraction retains only 42 of 7,305 PRs from MPC repositories, largely because MPC maintainers favor direct bug pushes over formalized issue-PR workflows and rarely ship developer-authored tests with fixes. The human-AI completion engine that synthesizes missing artifacts while preserving cryptographic correctness through expert review is a genuine methodological contribution that the community can build on for other specialized domains.

  1. The domain choice is well justified.

MPC software has properties that ordinary repair benchmarks completely ignore: information disclosure, data obliviousness, and fixed-point numerical behavior. These are fundamental to what MPC is supposed to do, and the case for evaluating them alongside functional correctness is compelling.

Critical weaknesses
  1. The paper’s central finding rests on unvalidated static heuristics.

The paper’s headline result is that the additional checks reject 11 to 40% of functionally passing patches and change the top-ranked model. However, looking at where these rejections come from, for Sonnet 4.6 eleven of fourteen rejected patches are static analysis flags rather than numerical check failures. The ranking change between Sonnet 4.6 and Opus 4.6, which is the paper’s most prominent finding, therefore rests almost entirely on unvalidated static heuristics.

The paper is appropriately cautious about this in places, stating that static flags mark potential violations rather than confirmed vulnerabilities and that a flag identifies a pattern that warrants review rather than a confirmed problem. However these caveats do not resolve the core issue: if the static flags are used to reject functionally correct patches when computing the check-passing resolution metric, the metric’s validity depends on the flags being reliable. The paper does not report what fraction of flagged patches were confirmed as genuine violations after expert review. Without this, the ranking change between models cannot be interpreted as a meaningful difference in MPC correctness.

  1. The experimental setup does not match the paper’s repository-level framing.

The paper positions itself as a repository-level benchmark, but the evaluation is one-shot patch generation. Each model receives a single prompt containing the problem statement and relevant source files and must output complete modified files. There is no repository exploration, no tool use, no test execution feedback, and no iterative repair. This is a legitimate evaluation setting, but it is not what repository-level repair typically means, and it limits the generalizability of the results. The reported 22.9% functional resolution rate should not be interpreted as the performance of contemporary coding agents on MPC repair, since agents with repository exploration and iterative debugging capabilities would likely perform differently.

Questions for authors
  1. What fraction of the static flags raised by the SAST module were confirmed as genuine MPC violations after expert review? This is essential for interpreting the check-passing resolution rates and the ranking change between models.

  2. The evaluation uses one-shot generation rather than agentic repair with repository exploration and iterative debugging. How do you expect the results to differ under an agentic setting, and does the benchmark infrastructure support agentic evaluation?

1 comment

W1 / Q1. What fraction of static flags are genuine violations, and does expert review change the ranking? Of 57 static flags, 47 (82.5%) were confirmed as genuine violations, while 10 (17.5%) were false alarms. Sonnet had 12/47 rejected patches, compared with 7/43 for Opus, leaving their ranking unchanged. We agree that the static flags need independent validation. Figure 7 shows why that matters. Nine of Sonnet’s 12 rejections came from static rules. The other three passed functional tests but failed the numerical checks. Counting both kinds of rejection left Sonnet with 35 check-passing patches and Opus with 36 out of 205 tasks. On CrypTen-45, the functional test accepts Opus 4.6’s patch because it only checks the new Chebyshev series helper (test_chebyshev_series). The patch also rewrites MPCTensor.set to store a plaintext tensor as the encrypted value (self._tensor = enc_tensor) without secret-sharing it as the base code does, so every party holds the plaintext and input confidentiality is lost. We will report the effect of adjudication for all eight models in Figure 7 and describe the two-model ordering in light of its observed margin.

W2 / Q2. Does the infrastructure support agentic repair, and how do results change? Yes. The benchmark provides base-commit repository states, per-library Docker images, and library-specific test runners. An agent can use the checkout to locate and edit files and run tests between edits. The evaluator can then score the resulting patch on the same functional and MPC checks used in the submitted experiment. In the submitted one-shot runs, Sonnet 4.6 passed the functional tests on 47 of 205 tasks. Twelve of those 47 patches failed an additional check (25.5%). For Opus 4.6, the corresponding counts were 43 of 205 and 7 of 43 (16.3%).In the repository exploration and iterative testing, agent could run all the 205 tasks. Sonnet 4.6 passed the functional tests on 70 of 205 tasks. Three of those 70 patches failed an additional check(4.3%). For Opus 4.6, the corresponding counts were 73 of 205 and 2 of 73(2.7%) . The agent (Claude Code, CLI 2.1.322) started from the problem statement. It could read, search and edit files and run the repository's existing tests. It saw neither the gold patch, nor the hidden scoring tests, nor the upstream fix through git. Each task had one attempt, limited to $\4\ (Sonnet 4.6) or $\6\ (Opus 4.6) and 25 minutes, with no turn limit. Runs that hit a limit count as failures. The mean cost per attempted task was $\0.64 for Sonnet 4.6 and $\\0.97 for Opus 4.6. We will also report results for other models.

Table A. One-shot and agentic repair. The submitted rows below cover all 205 tasks. Each rejected patch counts once.

ModelProtocolTasksFunctional passesFailures by checkRejected / functional passesPass all checksCost / task
Sonnet 4.6One-shot, submitted20547 (22.9%)9 static, 3 numerical12/47 (25.5%)35 (17.1%)≈ $0.20
Sonnet 4.6Agentic20570 (34.1%)2 static, 1 numerical3/70 (4.3%)67 (32.7%)$0.64
Opus 4.6One-shot, submitted20543 (21.0%)4 static, 3 numerical7/43 (16.3%)36 (17.6%)≈ $0.30
Opus 4.6Agentic20573 (35.6%)1 static, 1 numerical2/73 (2.7%)71 (34.6%)$0.97

Table B. Verifier validation.

Check and evaluation setWhat to reportResult
Static rules on flagged, functionally passing model patchesConfirmed / all audited, false alarms or indeterminate47/57 confirmed (82.5%); 10/57 false alarms (17.5%)
Static rules on upstream gold patchesFlagged / evaluated, expert disposition of flags11/205 (5.4%)
Numerical checks on independently documented CrypTen precision bugsDetected / tested among cases that pass ordinary tests, with selection criteria15/15 (100%)
Numerical checks on upstream gold patchesNumerical failures / evaluated, adjudicated false alarms and unresolved findings25/205 (12.2%). All 25 were attributed to version incompatibility: 24/25 (96.0%) shared an identical error signature across unrelated early commits, while 1/25 (4.0%) called an API that did not exist at the time of the commit.
Reviewer_hyNNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a repository-level benchmark for evaluating LLM code repair on secure multi-party computation (MPC) software. The authors argue that transplanting SWE-bench fails for several reasons. The benchmark contains two components, including a data curation framework that uses an LLM agent to screen PRs followed by a human-AI completion engine that synthesizes missing problem statements and tests, and a MPC verifier that supplements the functional tests with MPC-specific static checks and differential checks against plaintext references. The experimental conclusion is that functional correctness alone overestimates LLM reliability in MPC software.

Critical strengths

This paper presents the first repository-level benchmark for evaluating LLM code repair on secure multi-party computation (MPC) software. I believe the problem is important and well-motivated, especially the authors show that the existing SWE-bench-style cannot be directly generalized. But I do have questions on this motivation; see the weaknesses and questions below.

Critical weaknesses

My major concern on this paper is that the repository-level and agent framing is kind of over-claiming. And the evaluation does not really match this framing. Specifically, all the models are only using the problem statement and the relevant source files, and then the agents are asked to complete the modified files.

So, if I understand correctly, there is no repository exploration or code execution feedback. I feel this setup is not SWE-bench-style agentic evaluation. In this sense, I feel restricting the agents' capabilities is not a reasonable setup.

Questions for authors

Based on the above weakness, can the authors answer how do the results change under an agentic workflow that the agents can locate the relevant files and can run tests iteratively. Would the verifier's rejection rate fall when models can self-correct?

1 comment

W1. What does “repository-level” mean in this evaluation? Thank you for pointing out the ambiguity. The 205 tasks require changes to real MPC repositories, and we evaluate each patch at its base commit. In the submitted experiment, we supplied the relevant pre-patch files and asked each model to produce a patch in one pass. This let us measure repair with file localization supplied. We will label this setting explicitly and distinguish the agent used to construct the benchmark from the models evaluated on it.

Q1. How do repository exploration and iterative testing affect resolution and verifier rejection? In the submitted one-shot runs, Sonnet 4.6 passed the functional tests on 47 of 205 tasks. 12 of those 47 patches failed an additional check (25.5%) . For Opus 4.6, the corresponding counts were 43 of 205 and 7 of 43 (16.3%). In the repository exploration and iterative testing, agent could run all the 205 tasks. Sonnet 4.6 passed the functional tests on 70 of 205 tasks. Three of those 70 patches failed an additional check(4.3%). For Opus 4.6, the corresponding counts were 73 of 205 and 2 of 73(2.7%). The agent (Claude Code, CLI 2.1.322) started from the problem statement. It could read, search and edit files and run the repository's existing tests. It saw neither the gold patch, nor the hidden scoring tests, nor the upstream fix through git. Each task had one attempt, limited to $\4\ (Sonnet 4.6) or $\6\ (Opus 4.6) and 25 minutes, with no turn limit. Runs that hit a limit count as failures. The mean cost per attempted task was $\0.64 for Sonnet 4.6 and $\\0.97 for Opus 4.6. We will also report results for other models.

Table A. One-shot and agentic repair. The submitted rows below cover all 205 tasks. Each rejected patch counts once.

ModelProtocolTasksFunctional passesFailures by checkRejected / functional passesPass all checksCost / task
Sonnet 4.6One-shot, submitted20547 (22.9%)9 static, 3 numerical12/47 (25.5%)35 (17.1%)≈ $0.20
Sonnet 4.6Agentic20570 (34.1%)2 static, 1 numerical3/70 (4.3%)67 (32.7%)$0.64
Opus 4.6One-shot, submitted20543 (21.0%)4 static, 3 numerical7/43 (16.3%)36 (17.6%)≈ $0.30
Opus 4.6Agentic20573 (35.6%)1 static, 1 numerical2/73 (2.7%)71 (34.6%)$0.97

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.