- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses answer retrieval and unauthorized external operations during LLM-agent evaluation. It proposes frozen, per-trial bidirectional contracts specifying prohibited incoming answers, permitted outgoing operations, and the evidence required to admit evaluation scores. The system combines host-controlled routing, a file-descriptor barrier, request/response inspection, and independent evidence verification.
Experiments cover SWE-bench Verified/Pro, AgentDojo, Gitea, and controlled fault tests. In paired repair experiments, successful runs exposed to protected answers decrease from 11/240 to 2/240. In the host-path experiments, confirmed unauthorized effects decrease from 23/128 to 2/128. Additional studies examine component contributions, benign-task utility, detection errors, and runtime overhead.
Critical strengths
-
The paper addresses an important problem through a practically relevant combination of enforcement and evidence verification.
Bringing incoming answer exposure, outgoing unauthorized operations, and score integrity into one framework addresses limitations of destination-only filtering and retrospective auditing. Independent witnesses and conservative score-admission rules provide a useful foundation for more trustworthy agent evaluation.
-
Several experiments meaningfully isolate component contributions.
The separate routing and file-descriptor-barrier interventions, together with the comparison of the same Progent checker with and without the host boundary, provide clear evidence for the importance of controlling actual communication paths. Reporting benign-task utility losses and failure cases also helps contextualize the security improvements.
Critical weaknesses
-
Content recognition remains a substantial bottleneck, limiting the evidence for the broader answer-theft prevention claim.
Ensuring that traffic passes through an inspector does not establish that the inspector can recognize protected answers. Section 5.4 reports detection of only 14/24 paraphrased protected objects. In the local content tests in Appendix A.19, both the lexical gate and the gate with registered decoders release all 16/16 protected algorithmic-explanation responses. Semantic classification improves some results but blocks 30/48 legitimate responses in a separate reversible-representation test.
I appreciate that the authors disclose these limitations. Nevertheless, they directly concern the answer-retrieval problem motivating the paper. The headline 81.8% reduction concerns successful runs exposed to protected answers under a particular experimental setting; it does not establish comparable protection against semantically conveyed solutions. The paper should distinguish reliable enforcement of inspection from reliable identification of prohibited content and align its principal claims with the demonstrated scope.
-
Incomplete outcome accounting prevents a sufficiently robust assessment of the headline escape-reduction result.
The 91.3% reduction in Table 3B is calculated from confirmed effects decreasing from 23/128 to 2/128. However, Appendix A.19 provides only symbolic bounds involving unknown-outcome counts, while Appendix A.21 explicitly states that host-path unresolved effects have no assigned count. Consequently, readers cannot assess differential outcome observability across arms or calculate a conservative bound on the security improvement.
The Gitea experiment already reports confirmed effects, unknown effects, and known no-effect outcomes, together with conservative bounds. The host-path experiment supporting a headline result should provide similarly complete accounting. Without it, the reported reduction establishes a decrease in confirmed effects but leaves the robustness of the broader security conclusion unresolved.
The experimental presentation also makes these distinctions difficult to track. Figures 2–3 combine results from different experimental populations and execution conditions without sufficiently specifying their sample sizes, denominators, and relationships in the captions. For example, the zero-effect results in Figure 2(c) and the nonzero F/Q results in Table 3B come from different experiments, but this distinction is not sufficiently explicit in the figure caption. The main plots also do not directly convey the uncertainty reported elsewhere.
Questions for authors
-
Please provide the numbers of confirmed effects, unknown effects, and confirmed no-effect outcomes for every arm in Table 3B, together with conservative bounds on the between-arm differences. Does the improvement underlying the reported 91.3% reduction remain positive under an adversarial allocation of unknown outcomes?
-
What caused the two remaining exposed successful runs in the paired repair experiment? Does the independent exposure annotation cover algorithmic explanations, semantic paraphrases, and reversible representations? Can the existing records clarify how strongly the reported 81.8% reduction depends on answers retaining substantial similarity to the original patch text?
-
For public external services accessed during the repair tasks, which receiver-side evidence is required for strict score admission, and how is it obtained? Please distinguish evidence required by the online admission mechanism from independent receiver records used only for offline experimental labeling. This distinction matters for assessing deployment beyond controlled services.
Escape reduction stays positive with unknowns counted adversarially; online admission needs no receiver proof from public services
W2 / Q1 Per arm, per 128 calls (confirmed effects / unknown effects / confirmed no-effect outcomes):
For arms a and b, E is confirmed effects and U is unknown effects. The all-effect difference lies in [(E_a - E_b - U_b)/128, (E_a + U_a - E_b)/128].
Counting F's 2 unknowns as effects and broker only's 3 unknowns as no effect, the combined difference is still 19/128 (14.84 points). Escape effects fall by at least 82.6%, from 23 to at most 4, against 91.3% for confirmed effects. Every route and barrier contrast also stays above zero; the smallest lower bound is 3/128 (barrier with route off).
W1 / Q2 In RQ1, answer withholding lowers answer-exposed successful runs from 11 to 2 of 240, an 81.8% reduction.
Two raters, blind to the arm and to the guard's predicate verdicts, label whether the protected object reached the agent's observed delivery path. The 28 exposed labels: 22 verbatim or reformatted patch text, 3 semantic paraphrases, 2 algorithmic explanations, 1 reversible representation.
Of the 11 exposed successes with withholding off, 9 delivered the fix in a form the frozen predicates match, and these are exactly the 9 the guard withheld. The remaining 2 fall below that threshold; they are the 2 exposed successes left with withholding on (a prose paraphrase and a package archive).
The host boundary guarantees that every response passes through the inspector. Whether the inspector recognizes a protected answer is a separate guarantee.
Q3 Online admission always uses two things:
Receiver-side proof enters admission only for stateful writes the contract declares, and only on replicas we own; as the Ethics statement says, "no real third-party system is targeted". The receiver effect checks used to label outcomes in the experiments are offline labels: they measure admission and do not decide it.
For public external services, the contract forbids writes, so admission needs no receiver proof from them. A contract that would require receiver proof from a service we do not control makes the trial abstain: it is neither admitted nor counted as a violation, and it is reported separately as having insufficient evidence.
W3 The zero-effect results in Figure 2(c) and the nonzero F/Q results in Table 3B (2/128 each) come from different experiments; the revision states this where both are introduced.
The main text already states that Figure 2's Wilson ranges are nominal per-run references, not task-level intervals, and the text reports whole-task bootstrap intervals for each headline contrast. Example: withholding on minus off, valid scores 6/240, [0.42, 5.00] points.
Changes in the revision: the 81.8% figure is stated as measured under the paper's exposure annotation, and the 91.3% figure as a reduction in confirmed effects with the bounds above.