- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a new benchmark for MPC patch synthesis. The benchmark construction is done using an LLM based pipeline with some human intervention, which takes in issues from github repositories, and a range of LLMs are evaluated on these benchmarks.
Critical strengths
-
The benchmark seems reasonably broad: benchmarks are taken from multiple different repositories as sources. It also is not yet saturated by the LLMs evaluated (which includes GPT 5.4) so demonstrates that there is a gap of problems that cannot be solved by LLMs at least up to GPT 5.4.
-
Whilst MPC is not my area, to a non-expert this seeems like a valuable resource that the community would benefit from seeing public.
Critical weaknesses
1 - It's not clear that the methodology behind construction pipeline can be reused for obtaining benchmarks in other domains. Pehraps the authors could elaborate on its generality?
2 - The writing needs improving: it's fairly hard to extract from the paper on first reading what the contributions actually are (is it a benchmark suite, is it a benchmark extraction pipeline, is it the MPC verifier?). The authors should add a more succinct contributions list at the end of the introduction.
3 - The figures are also confusing and slightly contradictory. Figure 1 contains the data curation framework, and (I think) the MPC verifier, but the data curation framework in figure 1 is slightly different to the data curation framework in Figure 1, and the MPCVerifier in figure 3 is slightly dfifferent to the parts of Figure 1 that I think correspond to the MPCVerifier. I suggest the authors compress these into one diagram, so that the information isn't presented multiple times in slightly different ways.
4 - There is also some confusing use of abbreviations. For instance, under FIgure 1 it says "MPC-specific static checks (SAST)" without defining what SAST stands for. The definition for SAST happens several pages later in the paper.
5 - I would like some more clarity on the guarantees (if any) of the MPCVerifier. If there are no guarantees, then an empirical evaluation of the accuracy should be added.
6 - I would also like an empirical evaluation of how accurate the benchmark extraction pipeline, and how frequently human intervention is required
Questions for authors
Please could you respond to weaknesses 1, 5 and 6 (there is no need to respond to the others unless you wish to correct something I have misunderstood. I trust that, if the paper is accepted, you will fix these presentation issues)
2 comments
Table C. Measure extraction accuracy
| Method | TP | FP | TN | FN | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|---|
| Keyword | 67 | 28 | 72 | 33 | 69.50% | 70.53% | 67.00% | 68.72% |
| Plain LLM | 59 | 12 | 88 | 41 | 73.50% | 83.10% | 59.00% | 69.01% |
| Curator | 81 | 6 | 94 | 19 | 87.50% | 93.10% | 81.00% | 86.63% |
We measured screening on a 200-PR sample. Sampling. We sampled 100 of the 205 tasks and 100 PRs that it rejected after expert review, which are generic Python infrastructure, documentation, CI and dependency changes. Because the set includes rejected PRs, missed positives are measurable. Definition. A PR is MPC-relevant if its patch changes MPC protocol logic, such as secret-sharing arithmetic, secure comparison, truncation or share conversion. Changes to build, CI, documentation, packaging or generic Python code are not. Labeling. One author with 2 years of MPC experience labeled all 200 PRs from the PR description and diff, without seeing any method's output.
We reviewed all 205 tasks' problem statements, test sets and F2P tests. For problem statements, 69 (64 verbatim + 5 lightly edited) are from upstream, 67 are rewritten and 69 are entirely new. For the test sets, 61 (58 verbatim + 3 adapted) are from upstream, 138 are new (written later), and 6 are mixed. For F2P tests, 72 instances are entirely from upstream, while 133 instances are rewritten. We do not record the exact time per instance. The total time for the 205 instances is around 80 hours for an MPC expert.
W1. Which parts of construction carry over to other domains? On the same 7,305 pull requests, our pipeline produced 205 executable tasks, compared with 42 under SWE-bench-style extraction. The difference also affected which libraries could be studied. CrypTen contributed 61 tasks after completion versus zero under the baseline, and MP-SPDZ contributed 41 versus one. The reusable steps are to screen candidate changes, send uncertain cases to a domain expert, complete missing task descriptions and regression tests, and validate each task against the original fix. Adapting this process requires a rubric for relevant changes, expert review of the intended behavior, and tests and runners appropriate to the new domain. Our evidence for coverage comes from the five MPC frameworks.
W5. What does the verifier guarantee, and how accurate is it? Passing means that a patch passes the implemented static rules and agrees with the plaintext reference on the numerical inputs tested, within the configured tolerance. The verifier provides no formal guarantee of protocol security. Experts confirmed 47/57 (82.5%) static flags on functionally passing model patches, with 10/57 (17.5%) false alarms. Numerical checks detected 15/15 (100%) independently documented CrypTen precision bugs. On 205 upstream gold patches, static rules flagged 11/205 (5.4%), while numerical checks flagged 25/205 (12.2%). All 25 numerical failures were attributed to version incompatibility rather than patch defects. Of these, 24 shared an identical error signature, and one used an API unavailable at the time of the commit. These measurements answer different questions: the static audit measures the precision of flagged violations, while the numerical evaluation measures detection on a specified set of bugs.
W6. How accurate is extraction, and how much human intervention is required? On the same 200 labeled PRs, Curator achieved 93.10% precision and 81.00% recall (F1: 86.63%), outperforming both keyword screening (70.53% precision, 67.00% recall, F1: 68.72%) and a plain LLM (83.10% precision, 59.00% recall, F1: 69.01%). For the 205 final instances, all problem statements were reviewed, and 136 were rewritten. The corresponding numbers are 138/205 for test sets and 133/205 for F2P tests. Details are provided in Table C. Of 7,305 PRs, the first stage admitted 596 automatically and routed 579 to expert review. Automatic admission applied only to this screening decision. During completion, an expert validated every synthesized instance before inclusion. Appendix D reports that 163 of the 205 final instances used statements and tests produced during completion. We did not record the exact time per instance. The total time spent by an MPC expert on the 205 instances was approximately 80 hours.
Table B. Verifier validation.