acceptodds
Under review as a conference paper at ICLR 2027

SCE-BENCH: FROM SYMPTOMS TO CAUSAL EXECUTIONS OF CONCURRENCY BUGS

Abstract

Concurrency bugs are hard to diagnose and understand: symptoms may not reveal their causes, and failures may depend on nondeterministic schedules. Yet existing LLM benchmarks either omit concurrency bugs or lack deterministic evaluation, limiting assessment of models' reasoning. We present SCE-Bench, an execution-grounded benchmark that tests whether models can turn symptom descriptions into machine-checkable causal executions. Each task tests this reasoning through four stages: diagnose whether a symptom is concurrency-induced, identify the mechanism by which concurrent interactions cause the failure, construct a workload, and submit an execution specification. The grader then runs each execution specification on buggy and repaired source revisions to check for the claimed failure and its absence after repair. SCE-Bench contains 66 positive tasks from disclosed kernel and userspace bugs across 12 mechanism families. Its 86 negative tasks comprise 66 repaired-source counterparts and 20 independent non-concurrency cases to test whether models distinguish actual concurrency bugs from false alarms. At pass@3, GPT-5.5 solved 9 of 66 positive tasks (13.6%), while only 10 were solved across all models. Particularly, they struggled most with the final stage of specifying an execution to trigger the bug. For the negative tasks, every model was less accurate on repaired-source controls than on independent non-concurrency cases.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Concurrency Bugs, Code Models, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ioJpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCE-Bench evaluates whether LLMs can progress from concurrency-related symptoms to executable explanations through four stages: diagnosis, mechanism identification, workload construction, and execution specification. It contains 66 positive cases, 66 repaired-source controls, and 20 independent negative cases. Grading combines categorical checks, workload contracts, and controlled execution on affected and repaired revisions without an LLM judge. Experiments with seven models over three repetitions show low end-to-end success, with execution specification being the largest conditional bottleneck and repaired-source controls proving harder than independent negatives.

Critical strengths

Execution-grounded grading connects model predictions to observable failures rather than relying on an LLM judge. Repaired-source and independent negative controls meaningfully extend evaluation beyond known-bug reproduction. The evaluation includes repeated runs, stage-level results, cost reporting, and concrete failure examples.

Critical weaknesses

The reasons for poor repaired-source diagnosis remain insufficiently explored. The contributions of concurrency-reasoning errors and execution-specification errors to T3 failures are not clearly quantified. Validation of the case-specific grading rules and adapters needs more supporting evidence.

Questions for authors
  1. Can the authors provide representative affected/repaired case pairs showing the relevant fix, inspected source, and submitted model rationale? What explains the repaired-control errors beyond their label distribution?

  2. Can T3 failures be broken down into specification/compilation failures, execution-conformance failures, and failures to produce the target harm? How do these categories relate to concurrency reasoning versus interface use?

  3. How were case-specific workload rules and execution adapters validated against valid alternatives and incorrect certificates? Please confirm where reviewers can access the promised anonymous artifact.

4 comments

Thank you for recognizing our execution-grounded grading without an LLM judge, the value of repaired-source and independent negative controls, and the evaluation's repeated runs and detailed reporting. We address the main questions below.

Q1: Thank you for requesting affected/repaired examples and explanations of diagnosis errors beyond label distributions.

We reviewed all 569 rejected repaired-control T0 rationales across 66 cases and identified three prominent, overlapping patterns:

For each pattern, we present a concrete example and a few additional cases illustrating its recurrence.

  1. Treating historical evidence as proof of a current defect. Models interpret comments, changelogs, or regression tests describing the original bug as evidence that it remains. For example, Opus 4.6 on Smithy-Go, repetition 3, cites a changelog entry as confirmation that a close/write race is “not yet” fixed. However, the supplied snapshot already contains the repair: the underlying writer is closed by the writing goroutine after its write completes. The changelog documents the repair, rather than establishing that the original race persists. Other examples: Gemini 3.1 Flash and GPT-5.4 Mini on CPython; GPT-5.4 Mini on K28.

  2. Recognizing protection without applying its consequences. Models describe synchronization correctly but still propose an execution that it excludes. For example, GPT-5.5 on Linux cgroup pressure control, repetition 1, acknowledges that the node pointer is read and its reference acquired while holding the file lock, and that removal clears the pointer under that same lock. Nevertheless, it claims removal can invalidate the pointer before reference acquisition. This overlooks the protected handoff; its additional claim about dereferencing a null pointer also misses the null check inside kernfs_get(). Other examples: Sonnet 4.6 on K28; Gemini 3.1 Flash on Playwright and LocalAI.

  3. Rejecting the original mechanism but substituting an unsupported cause. For example, GPT-5.5 on libarchive, repetition 1, correctly recognizes that the repaired constant CRC table eliminates the shared initialization race. It nevertheless selects external/configuration failure, speculating about zlib or caller synchronization without establishing either as the cause. Recognizing that an internal explanation is absent does not establish an external explanation. Other examples: GPT-5.5 on libarchive in repetitions 2 and 3; GPT-5.4 Mini on libarchive and libgit2.

Security experts also reviewed the alternative explanations and judged them incorrect. These patterns distinguish errors in interpreting evidence, applying synchronization reasoning, and drawing supported causal conclusions.

Q2: Thank you for requesting a breakdown of T3 failures across compilation and execution, and their relation to reasoning versus interface use.

We audited all rejected T3 responses and identified exactly 100 episodes whose submitted answers received failing grades for interface, grammar, or formatting violations. We agree with the reviewer that these failures introduce ambiguity between difficulty expressing an answer in ExecSpec and difficulty reasoning about the concurrency mechanism.

We therefore resumed these episodes from their saved conversation contexts and provided targeted feedback, allowing up to three additional responses per episode, bringing our total LLM evaluation expenditure to approximately US$9,097.55. For example, when an answer contained Markdown fences, the feedback requested only the required JSON object; when a declaration violated the grammar, it identified the offending token and directed the model to the supplied grammar. Other feedback pointed to the already disclosed oracle signature or identified changes to the accepted T2 workload. It did not supply the source events or harmful schedule.

The final outcomes are 26 passes, 61 semantic/contract rejections, 11 interface-validation failures, and two safeguard refusals.

Across the full updated evaluation, 435 episodes reach T3: 125 pass, 296 receive scored failures, and 14 remain unresolved (12 LLM API failures and two safeguard refusals). We identify one primary failure category per episode, so an answer with multiple defects is counted only once.

Primary failure categoryEpisodes
Incorrect source selection, coordinates, phases, or event reachability108
Incorrect ordering, holds, mediation, or visibility68
Completion-token limit50
Inconsistent runtime object identities31
Incomplete event/observation structure or coverage15
JSON/ExecSpec syntax, metadata, or workload preservation11
Disclosed language or backend-capability restriction5
Action limit5
Incorrect oracle actor roles2
Rejection awaiting further adjudication1
Total scored failures296

The three largest certificate-error categories expose different difficulties in precisely describing how a concurrency bug occurs:

  • Identifying the relevant execution events (108). The certificate must connect the mechanism to the operations, occurrences, and synchronization states that establish the bug. For example, GPT-5.5 describes the async_simple ABA sequence but selects an event from the first notify_one call where the second call's occurrence is required. The first call has already reached that source location; selecting occurrence 1 again cannot identify the later hit needed by the schedule. The general mechanism is recognizable, but the selected runtime hit cannot occupy the claimed position. This category also includes incorrect occurrences, synchronization phases, and unreachable events, reflecting errors in identifying when and where the required operations can execute.
  • Constructing a realizable harmful schedule (68). Actors must be coordinated while respecting program order, synchronization, and memory visibility. For example, GPT-5.5 pauses an eventpoll reader while it holds a mutex, then waits for another actor to free the object. That actor needs the same mutex before it can proceed. The reader cannot resume until the free occurs, while the freeing actor cannot proceed until the reader releases the mutex. The proposed control therefore prevents the operation needed to trigger the bug.
  • Tracking shared-object identity (31). The certificate must identify which accesses concern the same resource and distinguish separate objects or allocation lifetimes. For example, Gemini 3.1 Pro describes Dapper's lost-update ordering but equates the original dictionary with two separately allocated replacements. After each publication, the selected variable refers to a different dictionary, so using the same variable name does not establish the same runtime identity. The ordering is plausible, but the asserted shared identity is incorrect.

Q3: Thank you for asking how workload rules and execution adapters were validated against valid and incorrect answers, and where to access the artifact.

We validate workload rules and execution adapters using harmful reference executions, valid alternatives, and deliberately incorrect submissions.

T2 tests whether the model can construct a precise workload for the diagnosed concurrency mechanism. Its constraints exclude unnecessary PoC components, preventing the model from substituting indiscriminate concurrent activity for identifying the required resources, operations, and actor relationships.

Each task discloses its operation vocabulary, argument domains, and workload constraints. Within this bounded answer space, explicit rules grade submissions without an LLM judge. Acceptance permits valid alternatives rather than requiring an exact reference answer or the fewest possible actors.

We audited the contracts across all 66 positive cases and validate both sides of the grading boundary:

  • Valid alternatives: consistent renaming, declaration reordering, permitted actor groupings, and combinations of these variations.
  • Invalid constructions: missing or duplicated components, incorrect operations or argument bindings, and invalid call orders.
  • Consistency: generated workloads are checked against both the case-specific grader and a separately encoded answer-domain matcher.

For example, K28 accepts two- and three-actor workloads using the same three required operations. The reuse fault may run separately or after pageout on the completing actor. It cannot follow the retained-read operation on the stalled actor: that operation waits for reuse, so the actor cannot reach the call needed to unblock it. A fourth actor would be idle or introduce unnecessary operations and is outside the contract. Tests enumerate the groupings and orders of these operations against these causal requirements.

Per-case audit records and regression tests provide inspectable evidence for these acceptance boundaries, addressing both false rejection and false acceptance within the disclosed scope.

Execution-adapter validation starts with a working reference execution. For every positive case, we retain a reference answer whose affected-side execution satisfies its constraints and produces the target harm. The backends and harm oracles were verified by security experts. Runtime acceptance requires evidence of the selected source events, actor/object identities, occurrences, and execution relations before the harm oracle is evaluated; a crash or timeout alone is insufficient.

We also check alternatives and incorrect certificates at the compilation and semantic-validation layers. Tests cover equivalent source checkpoints, permitted actor groupings, alternative event orders, and consistent renaming. Negative tests alter source coordinates, occurrence counts, object bindings, and ordering or control relations. For example, the SQLite memory-VFS adapter accepts both opener-first and closer-first schedules that establish the lock cycle, while tests reject self-blocking holds and orders that contradict actor program order. An enumerated comparison checks agreement between the case's answer-domain rule and the adapter's lock-cycle predicate across event orders, holds, and mediation choices.

These static checks are complemented by retained native execution evidence. One retained batch contains 18 alternative model-submitted certificates that demonstrate harm on affected source and exclusion on repaired source. The records distinguish source exclusion from execution infeasibility. This provides inspectable evidence beyond acceptance of a single reference string; the checks concern the disclosed workload and certificate domains.

Artifact access. Please refer to our separate comment for the anonymous repository link, which provides access to the reference answers, execution results, per-case audits, and grading implementations for independent inspection.

Thank you again for your review. As today is the last day of the discussion period, we would greatly appreciate it if you could take a look at our responses and let us know if you have any remaining concerns. We would be happy to address them before the discussion period ends.

Reviewer_9YLmOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SCE-Bench, an execution-grounded benchmark for evaluating LLMs' ability to diagnose and reason about concurrency bugs. The benchmark decomposes the reasoning process into four stages: concurrency diagnosis (T0), mechanism identification (T1), workload construction (T2), and executable causal specification (T3). It includes 66 positive tasks across 12 concurrency mechanisms and 86 negative tasks, including repaired-source controls and independent non-concurrency cases. The authors evaluate seven LLMs across three repetitions. Results show that current models struggle particularly with generating executable specifications, with the best model achieving only 13.1% end-to-end success. The study also reveals difficulties in distinguishing actual concurrency bugs from unsupported diagnoses, highlighting important limitations in current LLM-based concurrency reasoning.

Critical strengths
  1. Execution-grounded evaluation with causal controls. The paper introduces an execution-based evaluation framework that verifies concurrency hypotheses through source-event constraints, execution ordering, and observed failures. Comparing affected and repaired revisions provides a meaningful causal control, making the evaluation more reliable than approaches relying on textual explanations or LLM-based judgments.
  2. Comprehensive evaluation of concurrency reasoning. The four-stage design systematically assesses diagnosis, mechanism identification, workload construction, and execution specification. Including repaired-source controls and independent negative cases is particularly valuable for evaluating diagnostic reliability. The experiments across seven models also provide useful insights into their limitations, especially at the execution-specification stage.
Critical weaknesses
  1. Limited separation between causal reasoning and interface complexity. The low T3 success rate may partly reflect difficulties with the ExecSpec interface rather than limitations in concurrency reasoning itself. As discussed in Section 4.3, many failures involve incorrect source anchors, event bindings, or execution-control declarations. These errors do not necessarily indicate incorrect causal understanding. Additional failure categorization or ablation studies would help clarify what the benchmark actually measures and strengthen the interpretation of its main findings.
  2. Insufficient evidence for benchmark correctness and reproducibility. The benchmark relies on case-specific workload contracts, execution adapters, and harm oracles across heterogeneous backends. Although the overall methodology is described, the paper provides limited independently inspectable evidence that all 66 positive tasks can be reliably reproduced and validated against repaired revisions. Given that only 10 tasks are solved by any model, per-case reference certificates and validation results would strengthen confidence that failures reflect model limitations rather than benchmark constraints.
Questions for authors

Q1. Section 4.3 reports that many T3 failures involve incorrect source anchors, event bindings, or execution-control declarations. Could the authors clarify how much of the low T3 success rate is attributable to incorrect causal reasoning versus difficulties using the ExecSpec interface?

Q2. Given the reliance on case-specific workload contracts and heterogeneous execution backends, could the authors provide more evidence that all 66 positive tasks are reliably executable and correctly validated? In particular, are reference execution certificates and affected-versus-repaired validation results available for every task?

3 comments

Thank you for recognizing the value of our execution-grounded evaluation and causal controls, and the systematic assessment of concurrency reasoning through four stages. We address the main questions below.

Q1: Thank you for asking how T3 failures reflect causal reasoning versus difficulty using ExecSpec.

We audited all rejected T3 responses and identified exactly 100 episodes whose submitted answers received failing grades for interface, grammar, or formatting violations. We agree with the reviewer that these failures introduce ambiguity between difficulty expressing an answer in ExecSpec and difficulty reasoning about the concurrency mechanism.

We therefore resumed these episodes from their saved conversation contexts and provided targeted feedback, allowing up to three additional responses per episode, bringing our total LLM evaluation expenditure to approximately US$9,097.55. For example, when an answer contained Markdown fences, the feedback requested only the required JSON object; when a declaration violated the grammar, it identified the offending token and directed the model to the supplied grammar. Other feedback pointed to the already disclosed oracle signature or identified changes to the accepted T2 workload. It did not supply the source events or harmful schedule.

The final outcomes are 26 passes, 61 semantic/contract rejections, 11 interface-validation failures, and two safeguard refusals.

Across the full updated evaluation, 435 episodes reach T3: 125 pass, 296 receive scored failures, and 14 remain unresolved (12 LLM API failures and two safeguard refusals). We identify one primary failure category per episode, so an answer with multiple defects is counted only once.

Primary failure categoryEpisodes
Incorrect source selection, coordinates, phases, or event reachability108
Incorrect ordering, holds, mediation, or visibility68
Completion-token limit50
Inconsistent runtime object identities31
Incomplete event/observation structure or coverage15
JSON/ExecSpec syntax, metadata, or workload preservation11
Disclosed language or backend-capability restriction5
Action limit5
Incorrect oracle actor roles2
Rejection awaiting further adjudication1
Total scored failures296

The three largest certificate-error categories expose different difficulties in precisely describing how a concurrency bug occurs:

  • Identifying the relevant execution events (108). The certificate must connect the mechanism to the operations, occurrences, and synchronization states that establish the bug. For example, GPT-5.5 describes the async_simple ABA sequence but selects an event from the first notify_one call where the second call's occurrence is required. The first call has already reached that source location; selecting occurrence 1 again cannot identify the later hit needed by the schedule. The general mechanism is recognizable, but the selected runtime hit cannot occupy the claimed position. This category also includes incorrect occurrences, synchronization phases, and unreachable events, reflecting errors in identifying when and where the required operations can execute.
  • Constructing a realizable harmful schedule (68). Actors must be coordinated while respecting program order, synchronization, and memory visibility. For example, GPT-5.5 pauses an eventpoll reader while it holds a mutex, then waits for another actor to free the object. That actor needs the same mutex before it can proceed. The reader cannot resume until the free occurs, while the freeing actor cannot proceed until the reader releases the mutex. The proposed control therefore prevents the operation needed to trigger the bug.
  • Tracking shared-object identity (31). The certificate must identify which accesses concern the same resource and distinguish separate objects or allocation lifetimes. For example, Gemini 3.1 Pro describes Dapper's lost-update ordering but equates the original dictionary with two separately allocated replacements. After each publication, the selected variable refers to a different dictionary, so using the same variable name does not establish the same runtime identity. The ordering is plausible, but the asserted shared identity is incorrect.

Q2: Thank you for requesting per-case evidence of executability and affected/repaired validation for all 66 positive tasks.

For every positive case, we retain a reference answer whose execution on the affected revision satisfies the specified execution constraints and produces the target harm. Thus, benchmark solvability is established independently of whether any evaluated model solves the task. The execution backends and harm oracles were verified by security experts; acceptance requires both execution conformance and the specified harm, not merely a crash or timeout.

As an additional safeguard against errors in grading or bug attribution, we check the same certificate against the repaired revision. If it remains executable and reproduces the target harm on repaired code, the grader reports CAUSAL_REPAIR_CONTROL_FAILED and prevents a pass. This flags a discrepancy requiring investigation. However, a repair can also remove a selected source statement or make the specified execution infeasible, so the certificate is not always executable on the repaired revision. The following outcomes satisfy the repaired-side check:

  • Conforming execution without harm: the certificate is realized, but the harm oracle is false.
  • Source/contract exclusion: the repaired source no longer supports compilation of the certificate, such as when a selected statement is removed.
  • Validated execution infeasibility: the backend reports a recognized exclusion supported by certificate-bound evidence. For the kernel controller, this includes checking the observed event prefix, actor/object bindings, and controller failure state.

A task passes only when the affected execution demonstrates the specified harm and the repaired-side control satisfies one of the accepted outcomes above. It provides an additional consistency check beyond the grader validation performed by multiple security experts.

Please refer to our separate comment for the anonymous repository link, which provides access to the reference answers, execution results, per-case audits, and grading implementations for independent inspection.

Thank you again for your review. As today is the last day of the discussion period, we would greatly appreciate it if you could take a look at our responses and let us know if you have any remaining concerns. We would be happy to address them before the discussion period ends.

Reviewer_bT8mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCE-Bench evaluates whether LLMs can translate concurrency symptoms into executable explanations through four stages: diagnosis, mechanism identification, workload construction, and execution specification. It contains 66 positive cases across 12 mechanism families and 86 negative cases, including 66 repaired-source counterparts and 20 independent negatives. Positive tasks advance through the stages upon passing, while negative tasks are evaluated only at T0. Grading combines explicit answer contracts with execution checks on affected and repaired revisions, without an LLM judge. The evaluation covers seven models and three repetitions. GPT-5.5 achieves the highest observed end-to-end success rate of 13.1% among scored positive episodes, while only 10 of 66 positive tasks are solved across all models. T3 has the largest proportional loss for every model, and all models achieve lower diagnostic accuracy on repaired-source controls than on independent negatives.

Critical strengths
  1. Meaningful evaluation of source-grounded diagnosis. Pairing the same symptom with affected and repaired source tests whether models base their diagnoses on the supplied code. Together with independent negatives, this design measures false concurrency diagnoses that benchmarks containing only confirmed vulnerabilities cannot adequately assess.
  2. Executable validation of model explanations. The framework connects model-generated workloads, source events, object bindings, and execution constraints to observable program behavior. Checking execution conformance before evaluating target harm provides a concrete basis for assessing explanations beyond textual plausibility. The inclusion of real kernel and userspace bugs strengthens the benchmark’s practical relevance.
Critical weaknesses
  1. Insufficient characterization of T3 failures. T3 produces the largest proportional loss for every model, but the analysis provides only selected examples and partial counts. Without a systematic breakdown by failure stage and error type, it remains difficult to determine where submissions most frequently fail and which difficulties primarily account for the low success rates.
  2. Unclear grading rules for repaired-source executions. The paper does not clearly explain how grading handles cases where a repair makes the submitted execution constraints infeasible, or how such cases are distinguished from execution-control failures. This ambiguity limits independent assessment of the repaired-source control.
Questions for authors
  1. Could you provide a quantitative breakdown of scored T3 failures by failure stage and error type, with explicit classification criteria?
  2. If a repair makes the submitted execution constraints infeasible, how is this outcome distinguished from execution-control failure and scored?
3 comments

Thank you for recognizing the value of source-grounded diagnosis through affected/repaired controls, and executable validation of model explanations on real kernel and userspace bugs. We address the main questions below.

Q1: Thank you for requesting a quantitative T3 failure breakdown by stage and error type, with explicit classification criteria.

We audited every episode that reached T3, using the submitted certificates, disclosed contracts, and source-level review evidence.

The initial audit identified 100 episodes whose answers received failing grades for grammar, interface, or formatting issues. Other reviewers also raised the concern that such failures can obscure the distinction between interface use and concurrency reasoning. We therefore resumed these episodes from their saved conversation contexts, allowing up to three additional responses to targeted format and interface feedback. The feedback identified violations of the disclosed grammar and interfaces without supplying source events or a harmful schedule. This additional evaluation brought our cumulative LLM expenditure to approximately US$9,097.55. The final outcomes of these 100 episodes are 26 passes, 61 semantic/contract rejections, 11 interface-validation failures, and two safeguard refusals.

In the updated results, 435 episodes reach T3: 125 pass, 296 receive scored failures, and 14 remain unresolved (12 LLM API failures and two safeguard refusals).

We identify one primary failure category per episode, so an answer with multiple defects is counted only once.

Primary failure categoryEpisodes
Incorrect source selection, coordinates, phases, or event reachability108
Incorrect ordering, holds, mediation, or visibility68
Completion-token limit50
Inconsistent runtime object identities31
Incomplete event/observation structure or coverage15
JSON/ExecSpec syntax, metadata, or workload preservation11
Disclosed language or backend-capability restriction5
Action limit5
Incorrect oracle actor roles2
Rejection awaiting further adjudication1
Total scored failures296

The three largest certificate-error categories expose different difficulties in precisely describing how a concurrency bug occurs:

  • Identifying the relevant execution events (108). The certificate must connect the mechanism to the operations, occurrences, and synchronization states that establish the bug. For example, GPT-5.5 describes the async_simple ABA sequence but selects an event from the first notify_one call where the second call's occurrence is required. The first call has already reached that source location; selecting occurrence 1 again cannot identify the later hit needed by the schedule. The general mechanism is recognizable, but the selected runtime hit cannot occupy the claimed position. This category also includes incorrect occurrences, synchronization phases, and unreachable events, reflecting errors in identifying when and where the required operations can execute.
  • Constructing a realizable harmful schedule (68). Actors must be coordinated while respecting program order, synchronization, and memory visibility. For example, GPT-5.5 pauses an eventpoll reader while it holds a mutex, then waits for another actor to free the object. That actor needs the same mutex before it can proceed. The reader cannot resume until the free occurs, while the freeing actor cannot proceed until the reader releases the mutex. The proposed control therefore prevents the operation needed to trigger the bug.
  • Tracking shared-object identity (31). The certificate must identify which accesses concern the same resource and distinguish separate objects or allocation lifetimes. For example, Gemini 3.1 Pro describes Dapper's lost-update ordering but equates the original dictionary with two separately allocated replacements. After each publication, the selected variable refers to a different dictionary, so using the same variable name does not establish the same runtime identity. The ordering is plausible, but the asserted shared identity is incorrect.

Q2: Thank you for asking how repair-induced infeasibility is distinguished from execution-control failure and scored.

The primary evidence is execution on the affected revision; the repaired-side check is an additional consistency check. We have made this distinction clear in the revised version of the paper.

Before execution, case-specific checks assess the submitted source events and execution relations against the task's accepted answer domain. The compiler then resolves the selected source anchors against the pinned source. The case-specific execution backends and harm oracles were verified by multiple security experts. During execution, the evaluator checks that runtime evidence matches the specified events, actor identities, shared objects, and ordering or other supported relations. Only after establishing conformance does it evaluate the target harm—for example, an incorrect value, a validated wait cycle, or a matching sanitizer diagnostic. Together, execution conformance and the observed target harm provide direct evidence of the targeted concurrency bug.

As an additional safeguard against errors in grading or bug attribution, we check the same certificate against the repaired revision. If it remains executable and reproduces the target harm on repaired code, the grader reports CAUSAL_REPAIR_CONTROL_FAILED and prevents a pass. This flags a discrepancy requiring investigation. However, a repair can also remove a selected source statement or make the specified execution infeasible, so the certificate is not always executable on the repaired revision. The following outcomes satisfy the repaired-side check:

  • Conforming execution without harm: the certificate is realized, but the harm oracle is false.
  • Source/contract exclusion: the repaired source no longer supports compilation of the certificate, such as when a selected statement is removed.
  • Validated execution infeasibility: the backend reports a recognized exclusion supported by certificate-bound evidence. For the kernel controller, this includes checking the observed event prefix, actor/object bindings, and controller failure state.

A task passes only when the affected execution demonstrates the specified harm and the repaired-side control satisfies one of the accepted outcomes above. It provides an additional consistency check beyond the grader validation performed by multiple security experts.

Thank you again for your review. As today is the last day of the discussion period, we would greatly appreciate it if you could take a look at our responses and let us know if you have any remaining concerns. We would be happy to address them before the discussion period ends.

Comments on the paper

1 comment

Dear Area Chair,

We respectfully ask you to support acceptance of SCE-Bench. SCE-Bench evaluates LLMs on challenging, previously reported concurrency bugs in kernel and userspace software. We invested US$9,097.55 in evaluating seven state-of-the-art models across 3,192 model–task episodes. The results reveal persistent difficulties in distinguishing affected from repaired source and precisely specifying executions that demonstrate a bug's mechanism. These findings establish the need for a benchmark that tests both whether a concurrency diagnosis is warranted and whether its explanation holds under execution.

All three reviewers recognized the value of execution-grounded grading and negative controls that test whether a diagnosis is supported by the supplied source. SCE-Bench's bounded answer interfaces make diagnosis, mechanism identification, workload construction, and execution specification separately gradable without an LLM judge. The revision strengthens the evidence for what this evaluation measures and how its grading is validated.

Reviewers 9YLm and ioJp raised an important concern: low execution-specification success could reflect interface difficulty rather than incorrect concurrency reasoning. We audited the rejected T3 responses and provided targeted feedback in 100 episodes with grammar, formatting, or interface violations, allowing up to three additional responses from the saved context. The feedback identified disclosed requirements without supplying source events or a harmful schedule. These episodes' final outcomes include 26 passes and 11 remaining interface-validation failures. The revised paper incorporates feedback into the evaluation protocol and reports the final results.

The systematic breakdown requested by bT8m appears in §4.3 and Appendix D. Among 296 scored T3 failures, the largest certificate-error categories concern source/event selection (108), execution relations (68), and runtime object identity (31). The analysis connects these counts to concrete errors. For example, GPT-5.5 pauses a Linux eventpoll reader while it holds a mutex, waiting for another actor to free an object; that actor needs the same mutex before it can proceed. The proposed schedule prevents the operation required to trigger the bug. We distinguish such defects from interface restrictions and unfinished, budget-exhausted episodes, without treating every rejection as proof of a conceptual misunderstanding.

The revision also addresses benchmark correctness and repaired-side grading (§3.2 and Appendix C). Every one of the 66 positive tasks has a retained reference answer whose affected-side execution conforms to its constraints and produces the target harm. Multiple security experts verified the backends and harm oracles; tests exercise valid alternatives and deliberately incorrect submissions. Affected-side execution supplies the primary evidence. The repaired-side check adds assurance by requiring no harm, source/contract exclusion, or validated execution infeasibility; an unexplained execution failure does not suffice. The anonymous artifact, linked in our separate comment, provides reference answers, execution records, per-case audits, and grading implementations.

For ioJp's question about repaired-source errors, we reviewed all 569 incorrect rationales. Appendix D documents recurring failures to distinguish historical evidence from current defects, apply recognized protections, and justify alternative causes. For example, GPT-5.5 acknowledges that cgroup pointer lookup and reference acquisition share a lock with removal, yet proposes removal between those protected operations. This explains why the negative controls matter: recognizing a familiar bug pattern does not establish that the supplied source still permits it.

Across seven models and three repetitions, the highest end-to-end success rate is 20.7%, and 22 of 66 positive tasks are solved by any model. Every model performs worse on repaired controls than on independent negatives and loses the largest proportion of passing episodes at T3. We believe the basis for acceptance is the benchmark's ability to expose and examine these two gaps: deciding when a concurrency diagnosis is warranted and precisely describing an execution that demonstrates it. The revised paper provides both the evaluation framework and inspectable evidence to support progress on these capabilities.

Thank you for your consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.