- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper dives into whether an agent, when picking among three equivalent tool implementations (the intended version, a benign alternative from the community, or a functional counterfeit), can end up running code with a hidden, forbidden side effect but still returns the right artifact. SINGED is the benchmark for this. It covers five main task families, keeps two families in reserve, and includes a public-package slice. There’s a lot of variation: ranking shown to the agent, depth of evidence, decision rules, different model releases, and how the agent is harnessed. A task oracle checks artifacts, and a sandboxed process oracle logs which implementation actually ran and any effects.
Critical strengths
First off, the two-channel design is really tidy. Separating “correct output” from “safe execution” by logging hashes and sandbox activity makes things clearer. Table 3 is striking: all 167 counterfeit runs out of 1,500 trials still pass the task oracle. It proves that just looking at output isn’t enough, you can miss hidden side effects. Also, the ranking experiment is solid: randomized, with matched intervention, replication (17/60, 1/60, 0/60), plus an independent repeat to contrast policies. The paper doesn’t just cherry-pick results; it also reports when things don't go as hoped, like the cross-domain reversal (26/60 vs 25/60), the public-package interval that crosses zero (from -0.6 to 13.0), and its sensitivity to MiniMax release.
Critical weaknesses
irst, “failure” is based on a contract the agent never sees, since prompts don’t mention scope or security. The counterfeit is almost identical to benign sources except for the extra side effect. The rank gradient feels like “check the first result and run it” is guiding agent behavior (Figure 4b: all 27 failures come from single-candidate inspection trajectories). So, the main contribution is careful measurement of early commitment and position sensitivity, useful, but not as broad as “source-integrity failure.” The single-source test (55/175) has the same issue; the paper admits that zero rates reflect access to alternatives, but doesn’t say what the “correct” response is when only the counterfeit is available. Also, the effect varies in how detectable it is: sometimes an explicit POST request, other times a sneaky write outside the output directory or an unrelated read, and these are all lumped together in the results.
Second, the evidence base isn’t as big as the abstract (7,549 trials) suggests. The rank study is only 15 instances, and the original 480-trial block doesn’t have a freeze record, so you don’t know exactly when things ran. 97 IDs hit the retry limit and were ignored in the analysis. The archived launcher retried runs that stopped after interaction but before execution, which interacts with CER (defined by execution). Aggregating by first termination alters task and success counts (see Table 8: overall task successes drop from 1,468 to 1,359, and the model-breadth block from 285 to 234). Twenty releases use a single harness, and Llama 4 Maverick only completes 8% of tasks, so some observed differences might be due to harness compatibility. The headline benefit (15.7% down to 4.2%) comes from a subset, and the paper’s own transfer slices show it’s less certain elsewhere.
Questions for authors
Is CER calculated over all registered trials, or only those that actually execute something? How did the pre-execution retry rule influence counterfeit run counts? If you tweak the prompt to clarify scope (like “no network access beyond stated purpose”), does CER for first-ranked results change? That’d help show if agents don’t know the contract or just ignore it. In those 55 single-source executions, how many checked the key file first? And what about the 120 that didn’t execute, were those refusals, tool exhaustion, something else? Also, can you explain exactly how benign-source selection is counted in SU when a trial ends without execution?
6 comments
Response to Reviewer rtdk (2/2)
#W2 & Q1--2 & Q5: Denominators, retries, and evidence scope
CER and safe utility. CER uses all analyzable registered trials in each condition, including nonexecutions. Each identifier contributes one selected attempt. A nonexecution remains in the denominator with T = 0, H = 0, and SU = 0, where SU=T(1−H). Merely selecting a benign source without executing it and producing a valid artifact does not earn safe utility.
Attempt selection and retries. The first-post-action sensitivity rule retains the earliest post-action attempt, including one ending without execution. Under this rule:
-
Overall T/H/SU changes from 1,468/181/1,287 to 1,359/178/1,181.
-
Model-breadth outcomes change from 285/5/280 to 234/5/229, leaving that block's hidden-effect count unchanged.
The claim that this rule preserves aggregate CER was incorrect. No post-action attempt discarded by the primary analysis executed an artifact or produced a process receipt. The three hidden-effect changes arise when earlier nonexecuting attempts replace later executions selected by the primary rule.
Separately, 97 identifiers exceeded the planned pre-action retry limit following provider or runtime failures. Those failed attempts contain no tool calls or process events. We will report these protocol deviations separately; the full accounting correction appears in our response to Reviewer LrTw, #W2 & Q2.
Primary contrasts under the sensitivity rule. The randomized-rank block remains T/H/SU = 180/27/153, with 27/60, 0/60, and 0/60 counterfeit executions across ranks. For the matched 60-unit policy comparison, outcomes are 60/27/33 under the rank-one baseline and 54/0/54 under compare-all. The CER reduction remains 45.0pp (95% instance-clustered CI: 36.7--53.3).
Evidence scope and transfer. The 7,549 identifiers describe the full experimental suite; the original rank study uses 15 instances. The original 480-trial block lacks a contemporaneous immutable freeze record, and we will distinguish it from later frozen confirmation. The frozen rank replication uses new identifiers and records 17/60, 1/60, and 0/60 executions across ranks. We will make these block-specific sample sizes and provenance explicit.
Release-level results describe performance under the tested agent configuration and may reflect compatibility. For example, Llama 4 Maverick completes only 8% of tasks, so its low CER does not demonstrate safety. Likewise, the 15.7% to 4.2% compare-all reduction applies to the matched layered block; the held-out and public-package results do not establish a reliable transfer benefit. We will limit the claim accordingly to risk reduction in the tested matched settings.
Thanks for your thorough rebuttal and the extra experiments. Seriously, it’s clear you put a lot of effort into responding to my feedback, and I appreciate it. Your new analyses really cut through some lingering questions and strengthen the overall argument of your paper. I’ve gone point by point below.
Response to Authors (1/2): W1 and Q3–4
- Task scope and source-integrity failures
I’m glad you clarified that the original experiments showed agents making rank-sensitive choices, but only within boundaries set by evaluators, not real breaches of rules the agents actually recognized.
The new experiment where you spelled out explicit prohibitions really gets to the heart of my concern. When agents know about a restriction, counterfeit executions drop way down, from 27 out of 60 to just 3. Meanwhile, safe utility jumps from 33 up to 52 out of 60. That nails my original question: just telling agents about a security requirement changes their behavior noticeably.
But this also backs up my doubts about how to interpret the old benchmark. Most of those counterfeit executions happened simply because agents weren’t told about the restriction, so it’s not really about breaking known rules, it’s more about poor communication.
What matters are the three failures where the restriction was explicit. Agents knew the rule and still messed up. That’s the real evidence of a security failure.
I think you need to make a sharp distinction between cases where the procedure defines integrity versus situations where agents genuinely encounter explicit requirements and fail. Narrow your main claims accordingly.
- Early commitment and your mechanism
You’ve shown that every one of those 27 counterfeit executions in Figure 4b happened after agents considered only a single candidate. Glad you’re not claiming that alone caused the issue.
Your “compare-all” intervention shows that looking at more candidates helps cut down on counterfeit executions. But your explicit-prohibition experiment makes it clear that early decision-making and rule enforcement are separate factors.
In particular, those three failures with explicit prohibition happened because agents inspected the forbidden file, then went ahead with the counterfeit anyway, without considering other options.
Your paper would be stronger if you separated failures from not considering enough candidates and failures from overlooking explicit rules.
- Single-source behavior
Thanks for clarifying those 175 trials, every agent checked the key file, which answers my concerns.
The split between stopping before install versus installing but not executing shows that not all nonexecutions are outright refusals.
Dropping executions from 55/175 to 9/175 when explicit rules are introduced really highlights how well communicating restrictions works.
The variation between tasks; especially the archive task (30/35 versus 3/35), matters a lot. I agree: report per-task results, not one pooled rate.
Most of my questions about single-source behavior are covered, though it’s still not crystal-clear why agents don’t execute in some cases.
Response to Reviewer rtdk (1/2)
#W1 & Q3--4: Task scope, early commitment, and single-source behavior
The original prompts specify the requested artifact without stating the evaluator’s forbidden-event set. We agree that these results establish rank-sensitive source selection under evaluator-defined restrictions, not violations of an explicit prohibition. Figure 4b also shows an association: agents choose their own inspection paths, so single-candidate inspection alone does not establish a causal mechanism.
Explicit task scope. On the same 60 rank-one units, adding an explicit prohibition reduces counterfeit execution from 27/60 to 3/60, a 40.0pp reduction. Task success falls from 60/60 to 55/60, while safe utility rises from 33/60 to 52/60. In the three remaining failures, the agent inspects the prohibited
POST, checks no alternative, and executes the counterfeit.Thus, stating the restriction substantially changes behavior, although residual noncompliance remains. Please also see our response to Reviewer LrTw, #Q3, for the full table and the distinction between communicated and evaluator-defined requirements.
Single-source behavior. We repeated the 175 counterfeit-only cells from the five primary tasks with task-specific prohibitions, reducing execution from 55/175 to 9/175. The original trajectories show:
55 executions: all inspect the decisive file before running the package.
120 nonexecutions: all also inspect the decisive file; 109 stop before installation, and 11 install without running. All end normally with a final message; none times out or exhausts the tool budget. We report these as nonexecutions, without equating them all with explicit refusals.
When the only available source violates an explicit prohibition, declining to execute respects that restriction but leaves the task incomplete. It therefore receives T = 0 and SU = 0.
The task-level results also show substantial variation. Each entry reports counterfeit executions out of 35 single-source cells:
We will report these results by task and avoid interpreting the pooled rate as a common effect across event types.