- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SCE-Bench evaluates whether LLMs can progress from concurrency-related symptoms to executable explanations through four stages: diagnosis, mechanism identification, workload construction, and execution specification. It contains 66 positive cases, 66 repaired-source controls, and 20 independent negative cases. Grading combines categorical checks, workload contracts, and controlled execution on affected and repaired revisions without an LLM judge. Experiments with seven models over three repetitions show low end-to-end success, with execution specification being the largest conditional bottleneck and repaired-source controls proving harder than independent negatives.
Critical strengths
Execution-grounded grading connects model predictions to observable failures rather than relying on an LLM judge. Repaired-source and independent negative controls meaningfully extend evaluation beyond known-bug reproduction. The evaluation includes repeated runs, stage-level results, cost reporting, and concrete failure examples.
Critical weaknesses
The reasons for poor repaired-source diagnosis remain insufficiently explored. The contributions of concurrency-reasoning errors and execution-specification errors to T3 failures are not clearly quantified. Validation of the case-specific grading rules and adapters needs more supporting evidence.
Questions for authors
-
Can the authors provide representative affected/repaired case pairs showing the relevant fix, inspected source, and submitted model rationale? What explains the repaired-control errors beyond their label distribution?
-
Can T3 failures be broken down into specification/compilation failures, execution-conformance failures, and failures to produce the target harm? How do these categories relate to concurrency reasoning versus interface use?
-
How were case-specific workload rules and execution adapters validated against valid alternatives and incorrect certificates? Please confirm where reviewers can access the promised anonymous artifact.
4 comments
Q2: Thank you for requesting a breakdown of T3 failures across compilation and execution, and their relation to reasoning versus interface use.
We audited all rejected T3 responses and identified exactly 100 episodes whose submitted answers received failing grades for interface, grammar, or formatting violations. We agree with the reviewer that these failures introduce ambiguity between difficulty expressing an answer in ExecSpec and difficulty reasoning about the concurrency mechanism.
We therefore resumed these episodes from their saved conversation contexts and provided targeted feedback, allowing up to three additional responses per episode, bringing our total LLM evaluation expenditure to approximately US$9,097.55. For example, when an answer contained Markdown fences, the feedback requested only the required JSON object; when a declaration violated the grammar, it identified the offending token and directed the model to the supplied grammar. Other feedback pointed to the already disclosed oracle signature or identified changes to the accepted T2 workload. It did not supply the source events or harmful schedule.
The final outcomes are 26 passes, 61 semantic/contract rejections, 11 interface-validation failures, and two safeguard refusals.
Across the full updated evaluation, 435 episodes reach T3: 125 pass, 296 receive scored failures, and 14 remain unresolved (12 LLM API failures and two safeguard refusals). We identify one primary failure category per episode, so an answer with multiple defects is counted only once.
| Primary failure category | Episodes |
|---|---|
| Incorrect source selection, coordinates, phases, or event reachability | 108 |
| Incorrect ordering, holds, mediation, or visibility | 68 |
| Completion-token limit | 50 |
| Inconsistent runtime object identities | 31 |
| Incomplete event/observation structure or coverage | 15 |
| JSON/ExecSpec syntax, metadata, or workload preservation | 11 |
| Disclosed language or backend-capability restriction | 5 |
| Action limit | 5 |
| Incorrect oracle actor roles | 2 |
| Rejection awaiting further adjudication | 1 |
| Total scored failures | 296 |
The three largest certificate-error categories expose different difficulties in precisely describing how a concurrency bug occurs:
- Identifying the relevant execution events (108).
The certificate must connect the mechanism to the operations,
occurrences, and synchronization states that establish the bug.
For example, GPT-5.5 describes the async_simple ABA sequence
but selects an event from the first
notify_onecall where the second call's occurrence is required. The first call has already reached that source location; selecting occurrence 1 again cannot identify the later hit needed by the schedule. The general mechanism is recognizable, but the selected runtime hit cannot occupy the claimed position. This category also includes incorrect occurrences, synchronization phases, and unreachable events, reflecting errors in identifying when and where the required operations can execute. - Constructing a realizable harmful schedule (68). Actors must be coordinated while respecting program order, synchronization, and memory visibility. For example, GPT-5.5 pauses an eventpoll reader while it holds a mutex, then waits for another actor to free the object. That actor needs the same mutex before it can proceed. The reader cannot resume until the free occurs, while the freeing actor cannot proceed until the reader releases the mutex. The proposed control therefore prevents the operation needed to trigger the bug.
- Tracking shared-object identity (31). The certificate must identify which accesses concern the same resource and distinguish separate objects or allocation lifetimes. For example, Gemini 3.1 Pro describes Dapper's lost-update ordering but equates the original dictionary with two separately allocated replacements. After each publication, the selected variable refers to a different dictionary, so using the same variable name does not establish the same runtime identity. The ordering is plausible, but the asserted shared identity is incorrect.
Q3: Thank you for asking how workload rules and execution adapters were validated against valid and incorrect answers, and where to access the artifact.
We validate workload rules and execution adapters using harmful reference executions, valid alternatives, and deliberately incorrect submissions.
T2 tests whether the model can construct a precise workload for the diagnosed concurrency mechanism. Its constraints exclude unnecessary PoC components, preventing the model from substituting indiscriminate concurrent activity for identifying the required resources, operations, and actor relationships.
Each task discloses its operation vocabulary, argument domains, and workload constraints. Within this bounded answer space, explicit rules grade submissions without an LLM judge. Acceptance permits valid alternatives rather than requiring an exact reference answer or the fewest possible actors.
We audited the contracts across all 66 positive cases and validate both sides of the grading boundary:
- Valid alternatives: consistent renaming, declaration reordering, permitted actor groupings, and combinations of these variations.
- Invalid constructions: missing or duplicated components, incorrect operations or argument bindings, and invalid call orders.
- Consistency: generated workloads are checked against both the case-specific grader and a separately encoded answer-domain matcher.
For example, K28 accepts two- and three-actor workloads using the same three required operations. The reuse fault may run separately or after pageout on the completing actor. It cannot follow the retained-read operation on the stalled actor: that operation waits for reuse, so the actor cannot reach the call needed to unblock it. A fourth actor would be idle or introduce unnecessary operations and is outside the contract. Tests enumerate the groupings and orders of these operations against these causal requirements.
Per-case audit records and regression tests provide inspectable evidence for these acceptance boundaries, addressing both false rejection and false acceptance within the disclosed scope.
Execution-adapter validation starts with a working reference execution. For every positive case, we retain a reference answer whose affected-side execution satisfies its constraints and produces the target harm. The backends and harm oracles were verified by security experts. Runtime acceptance requires evidence of the selected source events, actor/object identities, occurrences, and execution relations before the harm oracle is evaluated; a crash or timeout alone is insufficient.
We also check alternatives and incorrect certificates at the compilation and semantic-validation layers. Tests cover equivalent source checkpoints, permitted actor groupings, alternative event orders, and consistent renaming. Negative tests alter source coordinates, occurrence counts, object bindings, and ordering or control relations. For example, the SQLite memory-VFS adapter accepts both opener-first and closer-first schedules that establish the lock cycle, while tests reject self-blocking holds and orders that contradict actor program order. An enumerated comparison checks agreement between the case's answer-domain rule and the adapter's lock-cycle predicate across event orders, holds, and mediation choices.
These static checks are complemented by retained native execution evidence. One retained batch contains 18 alternative model-submitted certificates that demonstrate harm on affected source and exclusion on repaired source. The records distinguish source exclusion from execution infeasibility. This provides inspectable evidence beyond acceptance of a single reference string; the checks concern the disclosed workload and certificate domains.
Artifact access. Please refer to our separate comment for the anonymous repository link, which provides access to the reference answers, execution results, per-case audits, and grading implementations for independent inspection.
Thank you again for your review. As today is the last day of the discussion period, we would greatly appreciate it if you could take a look at our responses and let us know if you have any remaining concerns. We would be happy to address them before the discussion period ends.
Thank you for recognizing our execution-grounded grading without an LLM judge, the value of repaired-source and independent negative controls, and the evaluation's repeated runs and detailed reporting. We address the main questions below.
Q1: Thank you for requesting affected/repaired examples and explanations of diagnosis errors beyond label distributions.
We reviewed all 569 rejected repaired-control T0 rationales across 66 cases and identified three prominent, overlapping patterns:
For each pattern, we present a concrete example and a few additional cases illustrating its recurrence.
Treating historical evidence as proof of a current defect. Models interpret comments, changelogs, or regression tests describing the original bug as evidence that it remains. For example, Opus 4.6 on Smithy-Go, repetition 3, cites a changelog entry as confirmation that a close/write race is “not yet” fixed. However, the supplied snapshot already contains the repair: the underlying writer is closed by the writing goroutine after its write completes. The changelog documents the repair, rather than establishing that the original race persists. Other examples: Gemini 3.1 Flash and GPT-5.4 Mini on CPython; GPT-5.4 Mini on K28.
Recognizing protection without applying its consequences. Models describe synchronization correctly but still propose an execution that it excludes. For example, GPT-5.5 on Linux cgroup pressure control, repetition 1, acknowledges that the node pointer is read and its reference acquired while holding the file lock, and that removal clears the pointer under that same lock. Nevertheless, it claims removal can invalidate the pointer before reference acquisition. This overlooks the protected handoff; its additional claim about dereferencing a null pointer also misses the null check inside
kernfs_get(). Other examples: Sonnet 4.6 on K28; Gemini 3.1 Flash on Playwright and LocalAI.Rejecting the original mechanism but substituting an unsupported cause. For example, GPT-5.5 on libarchive, repetition 1, correctly recognizes that the repaired constant CRC table eliminates the shared initialization race. It nevertheless selects external/configuration failure, speculating about zlib or caller synchronization without establishing either as the cause. Recognizing that an internal explanation is absent does not establish an external explanation. Other examples: GPT-5.5 on libarchive in repetitions 2 and 3; GPT-5.4 Mini on libarchive and libgit2.
Security experts also reviewed the alternative explanations and judged them incorrect. These patterns distinguish errors in interpreting evidence, applying synchronization reasoning, and drawing supported causal conclusions.