acceptodds
Under review as a conference paper at ICLR 2027

Stop Stealing, Stop Attacking: Countering Benchmark Hacking with Kernel-Enforced Contracts Outside the Agent's Reach

Abstract

Instead of solving a benchmark task, an agent can steal the answer—and attack real systems to obtain answers. OpenAI reported evaluated agents compromising Hugging Face's production infrastructure while seeking test solutions. Benchmark hacking thus threatens both the validity of evaluation scores and the security of systems outside the benchmark. Existing controls cannot distinguish protected answers from legitimate references on the same host, or unauthorized operations from permitted ones. Post hoc audits cannot undo exposure or external effects. We give each evaluation run a bidirectional contract listing which answers may not come in and which operations may not go out. Written in a declarative domain-specific language (**DSL**), the contract also states the evidence a score requires and is frozen before each run. To better prevent the agent from escaping, we choose to have the host operating-system kernel, which is outside the agent's reach, assist with the checks. We reduced successful answer theft by **81.8%** on 60 SWE-bench Verified and Pro tasks, and the agents' escape rate by **91.3%** in our attack study.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Benchmark Hacking, Agent Evaluation, Reward Hacking, LLM Agents, Runtime Enforcement, eBPF, Agent Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EyoMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses answer retrieval and unauthorized external operations during LLM-agent evaluation. It proposes frozen, per-trial bidirectional contracts specifying prohibited incoming answers, permitted outgoing operations, and the evidence required to admit evaluation scores. The system combines host-controlled routing, a file-descriptor barrier, request/response inspection, and independent evidence verification.

Experiments cover SWE-bench Verified/Pro, AgentDojo, Gitea, and controlled fault tests. In paired repair experiments, successful runs exposed to protected answers decrease from 11/240 to 2/240. In the host-path experiments, confirmed unauthorized effects decrease from 23/128 to 2/128. Additional studies examine component contributions, benign-task utility, detection errors, and runtime overhead.

Critical strengths
  1. The paper addresses an important problem through a practically relevant combination of enforcement and evidence verification.

    Bringing incoming answer exposure, outgoing unauthorized operations, and score integrity into one framework addresses limitations of destination-only filtering and retrospective auditing. Independent witnesses and conservative score-admission rules provide a useful foundation for more trustworthy agent evaluation.

  2. Several experiments meaningfully isolate component contributions.

    The separate routing and file-descriptor-barrier interventions, together with the comparison of the same Progent checker with and without the host boundary, provide clear evidence for the importance of controlling actual communication paths. Reporting benign-task utility losses and failure cases also helps contextualize the security improvements.

Critical weaknesses
  1. Content recognition remains a substantial bottleneck, limiting the evidence for the broader answer-theft prevention claim.

    Ensuring that traffic passes through an inspector does not establish that the inspector can recognize protected answers. Section 5.4 reports detection of only 14/24 paraphrased protected objects. In the local content tests in Appendix A.19, both the lexical gate and the gate with registered decoders release all 16/16 protected algorithmic-explanation responses. Semantic classification improves some results but blocks 30/48 legitimate responses in a separate reversible-representation test.

    I appreciate that the authors disclose these limitations. Nevertheless, they directly concern the answer-retrieval problem motivating the paper. The headline 81.8% reduction concerns successful runs exposed to protected answers under a particular experimental setting; it does not establish comparable protection against semantically conveyed solutions. The paper should distinguish reliable enforcement of inspection from reliable identification of prohibited content and align its principal claims with the demonstrated scope.

  2. Incomplete outcome accounting prevents a sufficiently robust assessment of the headline escape-reduction result.

    The 91.3% reduction in Table 3B is calculated from confirmed effects decreasing from 23/128 to 2/128. However, Appendix A.19 provides only symbolic bounds involving unknown-outcome counts, while Appendix A.21 explicitly states that host-path unresolved effects have no assigned count. Consequently, readers cannot assess differential outcome observability across arms or calculate a conservative bound on the security improvement.

    The Gitea experiment already reports confirmed effects, unknown effects, and known no-effect outcomes, together with conservative bounds. The host-path experiment supporting a headline result should provide similarly complete accounting. Without it, the reported reduction establishes a decrease in confirmed effects but leaves the robustness of the broader security conclusion unresolved.

    The experimental presentation also makes these distinctions difficult to track. Figures 2–3 combine results from different experimental populations and execution conditions without sufficiently specifying their sample sizes, denominators, and relationships in the captions. For example, the zero-effect results in Figure 2(c) and the nonzero F/Q results in Table 3B come from different experiments, but this distinction is not sufficiently explicit in the figure caption. The main plots also do not directly convey the uncertainty reported elsewhere.

Questions for authors
  1. Please provide the numbers of confirmed effects, unknown effects, and confirmed no-effect outcomes for every arm in Table 3B, together with conservative bounds on the between-arm differences. Does the improvement underlying the reported 91.3% reduction remain positive under an adversarial allocation of unknown outcomes?

  2. What caused the two remaining exposed successful runs in the paired repair experiment? Does the independent exposure annotation cover algorithmic explanations, semantic paraphrases, and reversible representations? Can the existing records clarify how strongly the reported 81.8% reduction depends on answers retaining substantial similarity to the original patch text?

  3. For public external services accessed during the repair tasks, which receiver-side evidence is required for strict score admission, and how is it obtained? Please distinguish evidence required by the online admission mechanism from independent receiver records used only for offline experimental labeling. This distinction matters for assessing deployment beyond controlled services.

1 comment
AuthorsOpenReview ↗

Escape reduction stays positive with unknowns counted adversarially; online admission needs no receiver proof from public services

W2 / Q1 Per arm, per 128 calls (confirmed effects / unknown effects / confirmed no-effect outcomes):

  • broker only: 23 / 3 / 102
  • broker + host route: 11 / 2 / 115
  • broker + FD barrier: 17 / 3 / 108
  • full guard F (host boundary plus our contract proxy): 2 / 2 / 124
  • proxy reference Q (a second, independently built proxy executing the same contract): 2 / 2 / 124

For arms a and b, E is confirmed effects and U is unknown effects. The all-effect difference lies in [(E_a - E_b - U_b)/128, (E_a + U_a - E_b)/128].

  • broker only - F: [19/128, 24/128] = [14.84, 18.75] points

Counting F's 2 unknowns as effects and broker only's 3 unknowns as no effect, the combined difference is still 19/128 (14.84 points). Escape effects fall by at least 82.6%, from 23 to at most 4, against 91.3% for confirmed effects. Every route and barrier contrast also stays above zero; the smallest lower bound is 3/128 (barrier with route off).

W1 / Q2 In RQ1, answer withholding lowers answer-exposed successful runs from 11 to 2 of 240, an 81.8% reduction.

  • Run 1 (Verified task): the agent read an issue-thread comment in which a maintainer described the fix in prose, with the changed condition rewritten inline.
  • Run 2 (Pro task): the agent installed a later released version of the project from the package registry.

Two raters, blind to the arm and to the guard's predicate verdicts, label whether the protected object reached the agent's observed delivery path. The 28 exposed labels: 22 verbatim or reformatted patch text, 3 semantic paraphrases, 2 algorithmic explanations, 1 reversible representation.

Of the 11 exposed successes with withholding off, 9 delivered the fix in a form the frozen predicates match, and these are exactly the 9 the guard withheld. The remaining 2 fall below that threshold; they are the 2 exposed successes left with withholding on (a prose paraphrase and a package archive).

The host boundary guarantees that every response passes through the inspector. Whether the inspector recognizes a protected answer is a separate guarantee.

Q3 Online admission always uses two things:

  1. the guard's per-transaction record; and
  2. independently captured wire bytes, authenticated with session keys exported through the trusted guard's key-log callback, so the captured bytes can be checked against what the guard recorded.

Receiver-side proof enters admission only for stateful writes the contract declares, and only on replicas we own; as the Ethics statement says, "no real third-party system is targeted". The receiver effect checks used to label outcomes in the experiments are offline labels: they measure admission and do not decide it.

For public external services, the contract forbids writes, so admission needs no receiver proof from them. A contract that would require receiver proof from a service we do not control makes the trial abstain: it is neither admitted nor counted as a violation, and it is reported separately as having insufficient evidence.

W3 The zero-effect results in Figure 2(c) and the nonzero F/Q results in Table 3B (2/128 each) come from different experiments; the revision states this where both are introduced.

The main text already states that Figure 2's Wilson ranges are nominal per-run references, not task-level intervals, and the text reports whole-task bootstrap intervals for each headline contrast. Example: withholding on minus off, valid scores 6/240, [0.42, 5.00] points.

Changes in the revision: the 81.8% figure is stated as measured under the paper's exposure annotation, and the 91.3% figure as a reduction in confirmed effects with the bounds above.

Reviewer_WCdsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies a problem with agents being granted network access during evaluation, leading to benchmark hacking (e.g., trying to retrieve the answer directly) and in worse cases (which post-hoc auditing can’t prevent), exploiting the environment in an effort to acquire the answer (e.g., the OpenAI/Hugging Face incident). They acknowledge that existing controls can decide where an agent can connect, such as through URL gating, but currently don’t monitor the content passed through the connection against per-problem rules. They propose creating a per-problem contract that declares, in a DSL, what answers can’t be delivered into the agent’s environment and what outbound operations are allowed. This enforcement is placed outside of the agent’s reach via eBPF and an independent witness over network communication helps decide whether scores should be approved, rejected, or abstained in cases such as guard tampering. This mechanism reduces cheating via direct answer stealing (11->2 out of 240) and agents more frequently go on to solve the tasks in a valid manner (121 -> 127). Furthermore, in a test setting using planted unauthorized instructions, the guard prevents any undesirable actions from going through.

Critical strengths
  • The problem of guardrailing allowed/disallowed actions during an evaluation is a realistic challenge, especially to avoid contamination and other hacking attempts from making benchmark scores harder to interpret.

  • The pre-release intercept provided by the designed guardrail has some nice evaluation and safety properties, such as allowing models to correct towards valid actions and preventing unsafe behaviors during the course of evaluation from propagating (which post-hoc auditing cannot prevent).

  • The addition of the independent witness is an interesting parallel verifier to avoid cases where the guard may be untrustworthy/tampered.

Critical weaknesses

My biggest concern is that this paper is very hard to read. Many sentences are condensed in a way that makes following them difficult, e.g. experimental evaluation sections like “Studies use selected 60 repair and 32 AgentDojo tasks plus fixed fixtures, with Claude Haiku 4.5 and Claude Sonnet 5, reset replicas, and randomized arms within task/model/repeat blocks… Admission passes policy; validity needs no established violation. Admitted= valid + contaminated + unresolved, while gaps retain launches and can intersect failed tests. Two blinded raters/adjudicator label exposure and independent tests label effects; unknowns leave valid numerators with all launches in denominators“. The figures and tables also lack many details to make them understandable, both within the visual content and the captions.

Some examples include:

  • the numerous abbreviations for the different arms that don’t have a centralized location explaining what each refers to clearly
  • Table 1 is not only very vague about what each entry means, but also is never referred to in any of the text (Table 2 is also never mentioned)
  • Figures 2 and 3 have a bunch of the abbreviations thrown into the caption with glossed over explanations, y-axis values are also not standardized and explained.

These are just a few examples that make reading difficult, and the extent of rewriting needed may be out of scope of this rebuttal period. The weaknesses about the technical content I'd highlight are:

  • For RQ1, it’s not clear the advantage of using the DSL control over simply creating an offline environment. For example, SWE-bench’s leaderboard requires no web-browsing by default. I would recommend a different evaluation setting such as a deep research task like BrowseComp [1] that requires internet access but may suffer from websites that directly leak the answer, motivating the need for the control much more. More evidence that the evaluated models need to make valid internet queries to solve would strengthen the need for this methodology. Other settings may also provide an opportunity to discuss considerations in designing contracts that a practitioner would need to make.

  • The “enforcing outside of an agent’s reach” doesn’t seem like a substantial contribution. It’s intuitive that monitoring should live outside of the agent’s reach to avoid hacking by the agent and eBPF is an interesting solution. However, it isn’t clear how the core mechanism of contribution (i) is meaningfully better than other simpler mechanisms such as just forcing guard-routing via host firewall rules (which equally are out of reach of the agent). Currently, it reads more as an engineering decision. Experiments testing different boundary implementations would help convince me of contribution (i).

[1] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv preprint arXiv:2504.12516, April 2025. https://arxiv.org/abs/2504.12516

Questions for authors

See Weaknesses above as well:

  • For RQ2, it would be helpful to qualitatively understand the false positive (e.g., too strict) and false negative (e.g., malicious payload in permitted operation) cases produced by the guard DSL, especially compared to other controls, such as contracts written in other languages/llm-as-judge based solutions (I do see some LLM-as-judge results in A.19, but it’s unclear why it isn’t a baseline throughout – consider using the llm judge over transcripts discussed in the Cursor post the authors already cite [2]). Also, can different control methods (e.g., DSL contract + LLM-as-judge) be stacked for even greater accuracy?

  • The Huggingface incident motivation in the introduction doesn’t really have a comparable evaluation in the paper, where an agent tries to explicitly get past the guard via exploits. Can a closer evaluation setting be added that directly addresses such a case?

[2] Naman Jain. Reward hacking is swamping model intelligence gains. Engineering blog post, Cursor, June 2026. https://cursor.com/blog/reward-hacking-coding-benchmarks. Published 2026-06-25.

1 comment
AuthorsOpenReview ↗

Firewall routing alone leaks handed-in sockets (Table 3A); host eBPF binds each connection to the trial, role, and phase a gateway cannot see

W3: Contribution (i) versus host firewall routing Contribution (i) is the finding that being out of reach does not give complete mediation, meaning that every outbound connection of the trial passes the checkpoint with no bypass. Linux records the creating cgroup in a socket at creation, so a socket created outside the trial and handed in keeps its outside identity, and routing rules do not see it. In Table 3A, the row "namespace route only" is host-forced routing to the guard: 0/16 direct TCP cases take effect, but 8/16 descriptor cases still do. Routing plus the launch barrier reaches 0/32. Our design already uses host default-deny rules for routing (Implementation, Section 4).

Host eBPF is the bridge between the host and the gateway. The actor (agent process tree or evaluator judge) and the phase (solving or scoring) are not in the bytes a gateway sees, and the contract forbids taking them from headers the agent writes. The host connect hook binds each connection attempt to the trial, role, and phase, and records it under the same lock as the allow decision. A trial with a fault, a stale generation, or an incomplete record is not admitted. A firewall rule that was never loaded or was flushed leaves no such trace. For example, during the score phase the judge process may write the score sink, while during the solve phase the agent's process tree may not. Both run on the same host and may open identical connections, so a gateway sees two flows it cannot tell apart, while the host hook sees two different roles in two different phases.

Three of the ten contract fields concern operations that never cross the network: protected objects, score ownership, and monitor-integrity attacks such as writing the collector's memory through /proc/<pid>/mem. The revision restates contribution (i) as a four-part host boundary: forced routing, the launch descriptor barrier, a per-trial boundary record bound to the allow decision that gates admission, and a monitor-integrity record. It also corrects Appendix A.8: connect() on a UDP socket runs the connect4/connect6 hook, and the host default-deny rules already close UDP.

W2: Offline evaluation in RQ1, and BrowseComp An offline setting is also a contract, the most restrictive one, and it measures a different capability. The Cursor audit we cite found public-web fix retrieval in 57% of 731 Opus 4.8 Max trajectories and existing-fix retrieval in 63% of that model's successful resolutions. On the RQ1 tasks, closed-book admits 104/240 valid scores with 0 contaminated, while our full guard admits 127/240 valid with 1 contaminated. Outbound tasks have no offline option: in RQ2, destination filtering (D) let 65/128 unauthorized effects through. For BrowseComp, the protected object should be defined by object, such as copies of the dataset and pages that publish question-answer pairs, rather than by string.

Q1: LLM-as-judge baseline and stacking A transcript judge reads a record the agent can influence, and it runs after the fact, so it cannot stop an effect that already happened. The contract's evidence field can list the judge's verdict as one admission condition. In RQ1, stacking it on the strict witness lowers contaminated admissions from 1 to 0, and valid scores from 122 to 115; in RQ2, stacked on the full guard (F), effects stay at 0/128 and benign completion falls from 92 to 87 of 128.

Q2: The Hugging Face incident Abuse of a permitted channel is what RQ2 and the adaptive Gitea attacks measure. Under F, confirmed effects stop at 3/64, and none of the three uses a path around the guard. Zero-days in the guard's own parser are outside the threat model.

Reviewer_KEdJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes per-task contracts that restrict incoming answer content and outgoing operations, enforced through a protected guard and host-controlled network boundary. Independent observations determine whether evaluation results have sufficient evidence for admission. Experiments on SWE-bench repair tasks, AgentDojo replicas and Gitea assess answer exposure, unauthorized effects, bypass resistance, benign utility and enforcement overhead.

Critical strengths
  • Addresses a concrete limitation of tool-level authorization: child processes and inherited descriptors can bypass the checked interface. The route × FD-barrier experiment directly tests both paths.
  • Careful outcome accounting: independent receiver evidence, explicit unknowns, task-level intervals and benign utility. The matched withholding experiment improves valid successes from 121 to 127 while reducing answer-exposed successes from 11 to 2.
Critical weaknesses
  • Answer protection remains substantially dependent on lexical predicates. The held-out study misses 24/120 protected objects; both word-based gates pass all 16 algorithmic-explanation requests (§5.4). Secure interception alone therefore provides limited protection against semantically equivalent answers.
  • The incremental benefit of the full design is unclear. F and Q both yield 2/128 effects in Table 3B; Progent+host and F both yield 3/64 on Gitea, with similar benign utility. These results strongly support the host boundary, but less clearly establish the additional value of the bespoke DSL.
Questions for authors
  • Under adaptive answer transformations, what contamination–utility tradeoff remains when predicates are frozen before testing?
  • Which measured capability requires the full DSL beyond a protected proxy or Progent+host with matched policies?
1 comment
AuthorsOpenReview ↗

The DSL carries per-trial answers and score evidence that Progent+host cannot express; F = Q shows the contract runs on any engine

W2 / Q2: Incremental benefit of the full design (F = Q, Progent+host = F), and which measured capability requires the full DSL The DSL is not an alternative to a protected proxy. Table 3B reports 2/128 effects both for our full guard (F) and for a second, independently built proxy that executes the same contract (Q), against 23/128 for broker only. The outcome therefore comes from the contract plus the host boundary, not from our guard's code, and a different engine can execute the same contract.

On Gitea, Progent runs its official checker with a matched frozen policy: the outbound operation rules of our contract, translated into Progent's policy. The policy is held fixed on purpose, so the boundary is the only variable. Confirmed receiver effects per 64 attacked episodes are 12 for Progent, 3 for Progent+host (Progent with our host boundary added), and 3 for F, with 2 unknown in each arm. Benign completion per 64 is 49, 48, and 47. Progent+host differs from F by 0/64, with unknown-effect bounds [-3.125, 3.125] points. This matches the design of the comparison, which measures the boundary rather than the DSL's expressiveness.

Why "+host" makes Progent match F Every contract decision is made on a typed action of five fields: operation, destination, object, actor, and phase. The actor is the contract role, for example the agent's process tree or the evaluator's judge process, and the phase is solve or score. Actor and phase are not in the bytes a proxy receives, and the contract forbids taking them from agent-supplied headers. They exist only on the host, where the host boundary binds every connection attempt to the trial, role, and phase before any byte reaches the guard.

What Progent+host cannot express Two kinds of contract fields:

  1. Inbound answers: what counts as an answer for this trial and must not be delivered to the agent. Progent represents privilege as "symbolic rules over tool names and arguments", and its paper states that it "does not tackle attacks that influence only the agent's textual outputs without triggering dangerous tool calls", listing output filtering as a possible extension.
  2. Evidence and dispositions: which evidence a score needs before it is admitted, and what happens when that evidence is missing.

Each field drives a result reported in the paper. The answers field drives answer withholding in RQ1: answer-exposed successes fall from 11 to 2 of 240, an 81.8% reduction, and valid scores rise from 121 to 127 of 240, while each setting admits 1 contaminated score. The evidence and dispositions fields drive strict admission in the fault tests, where 0/168 injected faults are admitted as clean. In the same fault sets the five DSL families reject 60/60 violations and abstain on 60/60 evidence gaps, while admitting 56/60 legal cases.

An appendix table lists each DSL field, the research question that uses it, and whether Progent or a generic proxy can express it. The main text states which factor each comparison isolates: Q, Progent, and Progent+host.

W1 / Q1: Answer protection depends on lexical predicates; adaptive transformations and the contamination-utility tradeoff

Two separate guarantees "Every response is inspected" and "the inspector recognizes every protected answer" are two separate guarantees. The host boundary and the full guard (F) provide the first, for any answer predicate. The answer predicates are a replaceable part of the contract's answer field, and RQ4 measures the current predicates.

Predicates frozen before testing Predicates are frozen on eight disjoint calibration tasks and tested on 24 held-out tasks. At a local response gate with seven frozen representations, the registered-decoder profile delivers 6/112 protected bodies, against 36/112 for the lexical profile; both still deliver 112/112 legal bodies. The lexical profile detects 96/120 transformed protected objects, with 9/120 legal false flags (RQ4). By transformation: unchanged 24/24, whitespace 24/24, lexical rewrite 18/24, application encoding 16/24, paraphrase 14/24.

Agent runs In agent runs whose responses are native, encoded, or paraphrased (144 trials each), strict admission keeps 92, 88, and 84 valid scores, with 0, 8, and 8 contaminated admissions (Appendix A.22).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: main revisions, new results, and key findings

We thank all three reviewers for their constructive feedback. In the revised paper and our individual responses, we address three central questions: (1) what host enforcement adds beyond network proxies, (2) whether the measured protection remains substantial under conservative outcome accounting, and (3) how stronger admission restrictions affect benchmark utility.

1. Clarifying the core contribution

Our contribution is not another network-content filter, but a host-enforced contract that links agent execution boundaries to benchmark score admission.

A network-only proxy cannot independently establish trusted host-side trial identity and execution phase, or enforce contract conditions on operations that never traverse the network. We clarify how four components address these gaps: forced routing, a launch-time descriptor barrier, trial-bound evidence for score admission, and monitor-integrity checks. Host eBPF supplies execution context alongside the network gateway rather than replacing its content inspection.

We also make the roles of Progent, the independent proxy Q, and Progent-plus-host comparisons explicit, and add an appendix mapping contract fields to their enforceability under these alternatives.

2. Strengthening the experimental evidence

Host-path protection (Table 3B; response to EyoM). The full guard F reduces confirmed attack effects by 91.3% relative to broker-only enforcement, from 23/128 to 2/128. Even when both unknown outcomes under F are counted as failures and the broker-only unknowns are not, F has 19 fewer effects per 128 calls, corresponding to a conservative descriptive reduction of 82.6%. The independent proxy Q matches F on this aggregate endpoint; our distinction from network-only approaches concerns the additional host-side contract properties they cannot independently enforce.

Closed-book comparison (response to WCds). On the same 60 RQ1 tasks (240 executions), closed-book execution yields 104 admitted scores with no observed contamination, versus 127 admitted scores with one contaminated admission under the full guard with withholding. Thus, eliminating network access reduces admitted scores by 9.58 percentage points, illustrating the utility cost of the stricter restriction.

Transcript-judge comparison (response to WCds). Adding an LLM judge to the strict RQ1 witness reduces contaminated admissions from 1 to 0, while admitted scores decrease from 122 to 115. In the separate RQ2 experiment, stacking the judge on F preserves 0/128 confirmed effects, while benign completions decrease from 92 to 87. These results quantify the cost of the additional admission condition rather than establishing a general limitation of LLM judges.

3. Improving clarity and scope

We revised the presentation to use descriptive arm names, explain Tables 1 and 2 in the main text, and make experimental populations, sample sizes, denominators, and endpoint distinctions explicit. In particular, Figure 2(c) and Table 3B are now clearly identified as different experiments.

We also clarify that inspecting every response does not imply recognizing every protected answer; the reported 81.8% is defined under our exposure annotation, while 91.3% refers specifically to confirmed-effect reduction.

The revision separates evidence used for online score admission from receiver-side checks used for offline outcome labeling. It also discusses closed-book execution, browsing-oriented contracts such as BrowseComp, and transcript-judge integration.

Finally, we corrected Appendix A.8: connected UDP sockets pass through cgroup connect hooks, while unconnected sends use sendmsg hooks; UDP routing remains restricted by host-managed default-deny rules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.