acceptodds
Under review as a conference paper at ICLR 2027

SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents

Abstract

Tool-using language-model agents select and execute third-party artifacts. Different implementations can return the requested output while producing hidden execution effects that task-, attack-, or choice-based evaluations may miss. We study functional counterfeits: implementations that match benign alternatives on the requested output but add an effect forbidden by the task contract. We introduce (\singedfullname), a controlled benchmark covering five primary and two held-out task families. It varies displayed rank, evidence depth, decision policy, model release, and agent configuration, while task and process oracles verify the artifact and execution path. Across audited trials, the randomized-rank study finds counterfeit execution in 45% (27/60) of rank-one trials and none at later ranks. Cross-candidate comparison eliminates shallow failures and reduces layered failures from 15.7% to 4.2%, but leaves dependency failures; its benefit is uncertain on unseen effects and public-package structures. Moreover, seven releases with no counterfeit executions when benign alternatives are available execute the counterfeit in \SingleSourceCounterfeitExecuted/ single-source cells after alternatives are removed. thus exposes a rank-, evidence-, and choice-sensitive outcome-to-execution gap: evaluation must connect correct outputs to execution pathsOur code is available at https://anonymous.4open.science/r/SINGED_code-22BE..

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, Agent security, Tool use

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_rtdkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper dives into whether an agent, when picking among three equivalent tool implementations (the intended version, a benign alternative from the community, or a functional counterfeit), can end up running code with a hidden, forbidden side effect but still returns the right artifact. SINGED is the benchmark for this. It covers five main task families, keeps two families in reserve, and includes a public-package slice. There’s a lot of variation: ranking shown to the agent, depth of evidence, decision rules, different model releases, and how the agent is harnessed. A task oracle checks artifacts, and a sandboxed process oracle logs which implementation actually ran and any effects.

Critical strengths

First off, the two-channel design is really tidy. Separating “correct output” from “safe execution” by logging hashes and sandbox activity makes things clearer. Table 3 is striking: all 167 counterfeit runs out of 1,500 trials still pass the task oracle. It proves that just looking at output isn’t enough, you can miss hidden side effects. Also, the ranking experiment is solid: randomized, with matched intervention, replication (17/60, 1/60, 0/60), plus an independent repeat to contrast policies. The paper doesn’t just cherry-pick results; it also reports when things don't go as hoped, like the cross-domain reversal (26/60 vs 25/60), the public-package interval that crosses zero (from -0.6 to 13.0), and its sensitivity to MiniMax release.

Critical weaknesses

irst, “failure” is based on a contract the agent never sees, since prompts don’t mention scope or security. The counterfeit is almost identical to benign sources except for the extra side effect. The rank gradient feels like “check the first result and run it” is guiding agent behavior (Figure 4b: all 27 failures come from single-candidate inspection trajectories). So, the main contribution is careful measurement of early commitment and position sensitivity, useful, but not as broad as “source-integrity failure.” The single-source test (55/175) has the same issue; the paper admits that zero rates reflect access to alternatives, but doesn’t say what the “correct” response is when only the counterfeit is available. Also, the effect varies in how detectable it is: sometimes an explicit POST request, other times a sneaky write outside the output directory or an unrelated read, and these are all lumped together in the results.

Second, the evidence base isn’t as big as the abstract (7,549 trials) suggests. The rank study is only 15 instances, and the original 480-trial block doesn’t have a freeze record, so you don’t know exactly when things ran. 97 IDs hit the retry limit and were ignored in the analysis. The archived launcher retried runs that stopped after interaction but before execution, which interacts with CER (defined by execution). Aggregating by first termination alters task and success counts (see Table 8: overall task successes drop from 1,468 to 1,359, and the model-breadth block from 285 to 234). Twenty releases use a single harness, and Llama 4 Maverick only completes 8% of tasks, so some observed differences might be due to harness compatibility. The headline benefit (15.7% down to 4.2%) comes from a subset, and the paper’s own transfer slices show it’s less certain elsewhere.

Questions for authors

Is CER calculated over all registered trials, or only those that actually execute something? How did the pre-execution retry rule influence counterfeit run counts? If you tweak the prompt to clarify scope (like “no network access beyond stated purpose”), does CER for first-ranked results change? That’d help show if agents don’t know the contract or just ignore it. In those 55 single-source executions, how many checked the key file first? And what about the 120 that didn’t execute, were those refusals, tool exhaustion, something else? Also, can you explain exactly how benign-source selection is counted in SU when a trial ends without execution?

6 comments
AuthorsOpenReview ↗

Response to Reviewer rtdk (1/2)

#W1 & Q3--4: Task scope, early commitment, and single-source behavior

The original prompts specify the requested artifact without stating the evaluator’s forbidden-event set. We agree that these results establish rank-sensitive source selection under evaluator-defined restrictions, not violations of an explicit prohibition. Figure 4b also shows an association: agents choose their own inspection paths, so single-candidate inspection alone does not establish a causal mechanism.

Explicit task scope. On the same 60 rank-one units, adding an explicit prohibition reduces counterfeit execution from 27/60 to 3/60, a 40.0pp reduction. Task success falls from 60/60 to 55/60, while safe utility rises from 33/60 to 52/60. In the three remaining failures, the agent inspects the prohibited POST, checks no alternative, and executes the counterfeit.

Thus, stating the restriction substantially changes behavior, although residual noncompliance remains. Please also see our response to Reviewer LrTw, #Q3, for the full table and the distinction between communicated and evaluator-defined requirements.

Single-source behavior. We repeated the 175 counterfeit-only cells from the five primary tasks with task-specific prohibitions, reducing execution from 55/175 to 9/175. The original trajectories show:

  • 55 executions: all inspect the decisive file before running the package.

  • 120 nonexecutions: all also inspect the decisive file; 109 stop before installation, and 11 install without running. All end normally with a final message; none times out or exhausts the tool budget. We report these as nonexecutions, without equating them all with explicit refusals.

When the only available source violates an explicit prohibition, declining to execute respects that restriction but leaves the task incomplete. It therefore receives T = 0 and SU = 0.

The task-level results also show substantial variation. Each entry reports counterfeit executions out of 35 single-source cells:

TaskOriginal promptExplicit prohibition
Document4/352/35
Speech3/350/35
Paper4/352/35
Archive30/353/35
Tabular14/352/35

We will report these results by task and avoid interpreting the pooled rate as a common effect across event types.

AuthorsOpenReview ↗

Response to Reviewer rtdk (2/2)

#W2 & Q1--2 & Q5: Denominators, retries, and evidence scope

CER and safe utility. CER uses all analyzable registered trials in each condition, including nonexecutions. Each identifier contributes one selected attempt. A nonexecution remains in the denominator with T = 0, H = 0, and SU = 0, where SU=T(1−H). Merely selecting a benign source without executing it and producing a valid artifact does not earn safe utility.

Attempt selection and retries. The first-post-action sensitivity rule retains the earliest post-action attempt, including one ending without execution. Under this rule:

  • Overall T/H/SU changes from 1,468/181/1,287 to 1,359/178/1,181.

  • Model-breadth outcomes change from 285/5/280 to 234/5/229, leaving that block's hidden-effect count unchanged.

The claim that this rule preserves aggregate CER was incorrect. No post-action attempt discarded by the primary analysis executed an artifact or produced a process receipt. The three hidden-effect changes arise when earlier nonexecuting attempts replace later executions selected by the primary rule.

Separately, 97 identifiers exceeded the planned pre-action retry limit following provider or runtime failures. Those failed attempts contain no tool calls or process events. We will report these protocol deviations separately; the full accounting correction appears in our response to Reviewer LrTw, #W2 & Q2.

Primary contrasts under the sensitivity rule. The randomized-rank block remains T/H/SU = 180/27/153, with 27/60, 0/60, and 0/60 counterfeit executions across ranks. For the matched 60-unit policy comparison, outcomes are 60/27/33 under the rank-one baseline and 54/0/54 under compare-all. The CER reduction remains 45.0pp (95% instance-clustered CI: 36.7--53.3).

Evidence scope and transfer. The 7,549 identifiers describe the full experimental suite; the original rank study uses 15 instances. The original 480-trial block lacks a contemporaneous immutable freeze record, and we will distinguish it from later frozen confirmation. The frozen rank replication uses new identifiers and records 17/60, 1/60, and 0/60 executions across ranks. We will make these block-specific sample sizes and provenance explicit.

Release-level results describe performance under the tested agent configuration and may reflect compatibility. For example, Llama 4 Maverick completes only 8% of tasks, so its low CER does not demonstrate safety. Likewise, the 15.7% to 4.2% compare-all reduction applies to the matched layered block; the held-out and public-package results do not establish a reliable transfer benefit. We will limit the claim accordingly to risk reduction in the tested matched settings.

Thanks for your thorough rebuttal and the extra experiments. Seriously, it’s clear you put a lot of effort into responding to my feedback, and I appreciate it. Your new analyses really cut through some lingering questions and strengthen the overall argument of your paper. I’ve gone point by point below.

Response to Authors (1/2): W1 and Q3–4

  1. Task scope and source-integrity failures

I’m glad you clarified that the original experiments showed agents making rank-sensitive choices, but only within boundaries set by evaluators, not real breaches of rules the agents actually recognized.

The new experiment where you spelled out explicit prohibitions really gets to the heart of my concern. When agents know about a restriction, counterfeit executions drop way down, from 27 out of 60 to just 3. Meanwhile, safe utility jumps from 33 up to 52 out of 60. That nails my original question: just telling agents about a security requirement changes their behavior noticeably.

But this also backs up my doubts about how to interpret the old benchmark. Most of those counterfeit executions happened simply because agents weren’t told about the restriction, so it’s not really about breaking known rules, it’s more about poor communication.

What matters are the three failures where the restriction was explicit. Agents knew the rule and still messed up. That’s the real evidence of a security failure.

I think you need to make a sharp distinction between cases where the procedure defines integrity versus situations where agents genuinely encounter explicit requirements and fail. Narrow your main claims accordingly.

  1. Early commitment and your mechanism

You’ve shown that every one of those 27 counterfeit executions in Figure 4b happened after agents considered only a single candidate. Glad you’re not claiming that alone caused the issue.

Your “compare-all” intervention shows that looking at more candidates helps cut down on counterfeit executions. But your explicit-prohibition experiment makes it clear that early decision-making and rule enforcement are separate factors.

In particular, those three failures with explicit prohibition happened because agents inspected the forbidden file, then went ahead with the counterfeit anyway, without considering other options.

Your paper would be stronger if you separated failures from not considering enough candidates and failures from overlooking explicit rules.

  1. Single-source behavior

Thanks for clarifying those 175 trials, every agent checked the key file, which answers my concerns.

The split between stopping before install versus installing but not executing shows that not all nonexecutions are outright refusals.

Dropping executions from 55/175 to 9/175 when explicit rules are introduced really highlights how well communicating restrictions works.

The variation between tasks; especially the archive task (30/35 versus 3/35), matters a lot. I agree: report per-task results, not one pooled rate.

Most of my questions about single-source behavior are covered, though it’s still not crystal-clear why agents don’t execute in some cases.

Reviewer_pE4xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies source selection by agents and introduces SINGED as a benchmark. It looks at whether LLM agents run a malicious tool call when several implementations give the same output but where one has a hidden side effect. Across various models and several agent frameworks, the authors conclude that evaluations must check what an agent execute and not only its output.

Critical strengths
  • The research question is very timely with recent incidents showing increasing capability and propensity of agents doing tool-call spoofing. Decoupling output from the process integrity allows for proper analysis.
  • The controlled experiments are well constructed such as including rank randomization, evidence depth variation, etc.
  • Large coverage over the experiments with 20 models from 14 families and 8 agent frameworks.
Critical weaknesses
  • Would benefit from a study that decouples whether the agent acted misaligned with intent or without intent. In some cases, it is ambiguous if the agent sees the actions as harmful or not.
  • There is confusion around the claim where failures are due to early commitment. However, the paper also states that all failures happened after the agent saw the malicious code.
  • A few prior work looked into similar settings such as Attractive Metadata Attack (https://arxiv.org/abs/2508.02110). Other work on MCP servers also have shown similar findings. Can the authors clarify how they distinguish from them and how much more do they contribute?

Open to increase score if questions are answered with clarity.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pE4x

Thanks for recognizing the timely problem, controlled design, and broad model and agent-configuration coverage. We address your concerns below.

#W1: Explicit requirements and agent intent

We agree that execution traces alone cannot establish internal intent or whether an agent considers an effect harmful. The original prompts also leave the forbidden-event set evaluator-defined, so those results do not establish violations of an explicit prohibition.

To test behavior under a stated restriction, we evaluated the same 60 rank-one units from the three transmission tasks with an explicit prohibition:

  • Rank-one results: counterfeit execution falls from 27/60 to 3/60, a 40.0pp reduction. Safe utility rises from 33/60 to 52/60, while task success falls from 60/60 to 55/60.

  • Counterfeit-only results: across the five primary tasks, task-specific prohibitions reduce execution from 55/175 to 9/175 (31.4% to 5.1%; reduction: 26.3pp, 95% instance-clustered CI: 15.4--38.3).

These results show that explicit scope substantially changes behavior. The remaining executions demonstrate noncompliance with a communicated restriction, but do not establish deliberate harmful intent. We will distinguish these interpretations throughout the revision. Please also see our response to Reviewer LrTw, #Q3, for the full rank-one table and the distinction between communicated and evaluator-defined requirements.

#W2: Decisive-code exposure and early commitment

These observations concern different aspects of the trajectory. Exposure records that the relevant code was visible during inspection; it does not establish that the agent understood its implications. Early commitment describes executing the initially selected candidate without comparing alternatives. An agent can therefore inspect the counterfeit’s code and still exhibit early commitment.

The evidence is:

  • Observed association: all 27 original failures follow inspection of the counterfeit with decisive evidence visible. All occur among the 79 trajectories that inspect only one candidate, vs. 0/101 that compare all three. Similarly, failures occur in 27/142 trajectories retaining their first choice, vs. 0/38 that switch.

  • Matched intervention: requiring comparison reduces rank-one counterfeit executions from 27/60 to 0/60.

  • Explicit-prohibition condition: all three remaining failures inspect only one candidate before execution.

As the inspection paths in Figure-4b are agent-chosen, their association with failure is not causal evidence. The matched intervention provides stronger evidence that requiring comparison changes execution in this setting. We will describe early commitment as a supported behavioral explanation, without treating it as an established internal reasoning mechanism.

#W3: Relation to prior work

Thanks for highlighting these connections. We will add Attractive Metadata Attack and clarify the distinctions from related MCP-security and malicious-implementation studies.

  • Attractive Metadata Attack optimizes tool names, descriptions, and parameter schemas to influence selection. MCP tool-poisoning studies, such as MCPTox, embed malicious instructions in tool metadata. SINGED keeps candidate names, metadata, interfaces, and advertised capabilities fixed while varying displayed rank, inspectable implementation evidence, and decision rules.

  • MalTool studies malicious code implementations, including malicious behavior embedded in otherwise benign tools. We do not claim priority for functionality-preserving malicious implementations.

Motivated by the publicly reported OpenAI Privacy Filter incident, SINGED abstracts this real-world concern into a controlled, auditable study of choices among implementations with matched outputs. It combines rank randomization, controlled evidence depth, and matched decision-rule interventions with records of inspected files, executed hashes, and process events. These controls allow us to measure when comparison changes execution and where its benefits weaken.

The single-source diagnostic adds a distinct finding: choosing safely when benign alternatives are available does not establish reliable rejection of the counterfeit itself. We will make this experimental contribution and its relation to prior work explicit in the revised contribution and related-work sections.

Dear Reviewer pE4x,

Thank you again for your review. We have posted a detailed response addressing your concerns. With approximately 24 hours remaining in the discussion period, we would appreciate it if you could let us know whether our response resolves your main concerns or if any questions remain. If you feel that our response has addressed your concerns, we would be grateful if you would consider updating your assessment accordingly.

Thank you for your time.

Best regards,
The Authors

Thank you very much for the detailed response. I appreciate the effort put into the answers and clarifications. I will increase my score accordingly.

Best.

Dear Reviewer pE4x,

Thank you for taking the time to read our response and reconsider your assessment. Your feedback helped us clarify the paper’s contributions and findings. We will submit the revised manuscript shortly, incorporating the clarifications from our discussion.

Best regards,
The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer LrTw (1/2)

Thanks for recognizing the controlled source-choice design, single-source diagnostic, and reporting of transfer limitations. We address your concerns below.

#W1 & Q1: Figure 2 and the information boundary

Figure 2 is schematic, not a literal reproduction of agent-visible code. The illustrated role-based condition was not exposed to the agent. We agree that its placement obscures this distinction and will replace the panel with an exact exposed fixture, clearly separating evaluator-only fields.

For a representative fixture:

  • Visible search result: "elex-scribe --- elex-labs" with the snippet "elex-scribe 1.0.0. Local speech workflow for workspace inputs."

  • Visible source: all candidates share the signature prepare (src, endpoint); the counterfeit implementation issues the POST; the benign implementations are no-ops.

  • Invocation and private labels: all candidates receive the same opaque local endpoint. Only the private manifest maps candidate IDs to roles; no visible result, metadata field, filename, or source file contains "counterfeit," "legitimate," or "community."

Thus, inspection exposes the code responsible for the effect, while role assignments remain private.

#W2 & Q2: Attempt-selection audit

Thanks for identifying these inconsistencies. Our re-audit separates the accounting error from the effect of attempt selection.

  • Table composition: The total row covers 1,488 confirmatory identifiers, but the displayed rows include a separate 144-identifier release-refresh block and omit layered evidence (144), prompt rewrites (48), and ReAct (48). This explains the displayed sum of 1,392 identifiers. Release-refresh contributes three additional affected identifiers, explaining 113 affected identifiers in the displayed rows vs. 110 in the confirmatory total. We will list every block and provide separate confirmatory and release-refresh subtotals.

  • Sensitivity rule and correction: The first-post-action rule retains the earliest post-action attempt for each identifier, including attempts ending without execution. Under this rule, task success/hidden effects/safe utility (T/H/SU) changes from 1,468/181/1,287 to 1,359/178/1,181. The statement that this rule "preserves CER" was incorrect. No attempt discarded by the primary analysis executed an artifact or produced a receipt; the three hidden-effect changes arise when an earlier nonexecuting attempt replaces a later execution selected by the primary rule.

  • Primary contrasts: The randomized-rank block remains T/H/SU = 180/27/153, with 27/60 counterfeit executions at rank one and none at later ranks. Under the sensitivity rule, the matched 60-unit comparison yields 60/27/33 for the rank-one baseline and 54/0/54 for compare-all. The CER reduction remains 45.0pp (95% instance-clustered CI: 36.7--53.3).

  • Retry deviations: 97 identifiers exceeded the planned pre-action retry limit following provider or runtime failures. Those failed attempts contain no tool calls or process events. We will report these protocol deviations separately.

AuthorsOpenReview ↗

Response to Reviewer LrTw (2/2)

#Q3: Communicated requirements and evaluator-defined effects

The original prompts specify the requested artifact but do not enumerate the forbidden-event set. The held-out metadata-change and subprocess restrictions are also evaluator-defined. The original results therefore measure rank-sensitive execution under evaluator-defined restrictions when the request does not state those restrictions, rather than violations of an explicit prohibition.

This setting captures cases where users specify an artifact without anticipating every implementation effect. To separately test compliance with a stated restriction, we added explicit prohibitions on the same 60 rank-one units from the three transmission tasks:

PromptCERTask successSafe utility
Original rank study27/6060/6033/60
Explicit prohibition3/6055/6052/60

Explicit prohibitions reduce counterfeit execution by 40.0pp, while task success falls from 60/60 to 55/60 and safe utility rises from 33/60 to 52/60. In all three remaining failures, the agent observes the prohibited POST, inspects no alternative, and executes the counterfeit; none follows compare-all.

We also repeated the 175 counterfeit-only cells from the five primary tasks with task-specific prohibitions. Counterfeit execution falls from 55/175 to 9/175, a reduction of 26.3pp (95% instance-clustered CI: 15.4--38.3). This extension does not include configuration editing or thumbnailing. We will distinguish evaluator-defined restrictions from communicated requirements throughout the revision.

#S1: Contribution relative to MalTool.

We agree with this positioning. MalTool studies malicious implementations that preserve benign functionality; we do not claim priority for that idea.

SINGED's contribution is the controlled study of source choice: it holds advertised capability, interface, metadata, requested output, and candidate topology fixed while varying displayed rank, evidence depth, and decision rule, and records executed identities and process events. We will revise the related-work and contribution sections to state this scope explicitly. Please also see our response to Reviewer pE4x, #W3.

Dear Reviewer LrTw,

Thank you again for your review. We have posted a detailed response addressing your concerns. With approximately 24 hours remaining in the discussion period, we would appreciate it if you could let us know whether our response resolves your main concerns or if any questions remain. If you feel that our response has addressed your concerns, we would be grateful if you would consider updating your assessment accordingly.

Thank you for your time.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear Reviewers,

Thank you all for your feedback and for taking the time to read our responses. We also appreciate Reviewer pE4x’s follow-up and their decision to raise their score after considering our response. We thank Reviewer rtdk for engaging in the discussion and for encouraging us to clarify the limits of compare-all in the revised manuscript.

We have uploaded the revised manuscript, with changes marked in blue. The main updates are as follows.

  • Task scope and new results. We distinguish the evaluator-defined restrictions in the original benchmark from the explicit prohibitions in the new experiments. We report counterfeit execution alongside task success and safe utility, and present the single-source results by task.

  • Figure 2 and Table 8. Figure 2 now shows an excerpt from the actual entrypoint and identifies which information is private to the evaluator. Table 8 reconciles the block totals and explains how the attempt-selection rule affects the results.

  • Interpretation and contribution. We clarify the evidence linking early commitment to shallow failures, explain what the compare-all intervention tests and where its benefits are uncertain, and distinguish our controlled study of source choice from prior work.

With about seven hours left in the discussion period, we would be happy to answer any remaining questions. If you feel that our responses and revisions have addressed your concerns, we would be grateful if you would consider updating your assessment accordingly.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.