acceptodds
Withdrawn from ICLR 2027

Correct Labels Are Not Enough: Which Part of a Judgment Keeps a Tool Agent Safe?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agents often pass on only the verdict of an earlier check. We ask whether correct labels kept alone still stop an agent from acting on a false claim in its context. In the airline task of τ -bench, a colleague’s note falsely says that the airline cancelled a flight, which would allow a refund; we keep parts of the agent’s saved assess- ment and grade the final database. When gpt-oss-120b judged the note FALSE and planned to KEEP the booking, its whole assessment led to no forbidden cancella- tion and its two labels alone to 76%; all ten agents of our panel lose protection. Our thesis is that agents act on stated facts more than on verdicts, and weigh an- other agent’s message by who is said to send it and whether it says it was checked. One sentence of the fact restores most of the protection wherever it stands, a con- clusion does not, and the loss also appears when the claim favours refusal. The same stop instruction protects more from another agent than from the agent itself in all five agents we test, and a verdict protects more when it claims a check. Our evidence comes mainly from one airline task, where the record only implies the fact; the two strongest models we tried ignore the note. Systems should check permission against the database and pass a checked premise, not only a verdict.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reliability, say-do discrepancy, tool agent, ai safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TVsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper essentially checks the agents' downstream behaviour when given prior agents' verdict or verdict plus reasoning / "explanation"

Critical strengths

I think the paper looks at an interesting problem and runs some interesting experiments. I actually think there is a really interesting story and analogy here that is completely missed by the authors -- how humans behave when shown an AI verdict vs a verdict + explanations. There are some analogous results to how agents act here, but it's not quite the same and I think the difference can be one of the most interesting findings here.

Critical weaknesses

This paper is almost unreadable -- this is the main reason for my recommendation to reject. A paper should be a tool of communication and should bring the reader into the project, carefully explaining the choices and reasoning. If I were not obliged to try and understand this paper due to being assigned to review it, I cannot imagine myself putting the required effort. This is more stark, as this is not a complicated paper in terms of narrative or setup, so I do not see a reason not to invest more in making the paper engaging and clear to the reader.

Due to this, I am also not able to properly verify the soundness of the paper -- although I didn't find any clear issues with the design.

Questions for authors

-- Did you, honestly, read this paper top to bottom before submitting it? Do you think it is clear and tells a strong narrative of the problem, your choices, your results and their implications that would be clear to someone not familiar with the work?

-- Why did you only focus on one problem / setup? Seems easy to expand this to at least 1-2 more tasks.

3 comments
AuthorsOpenReview ↗

Reply part 1

Thank you for your valuable review and constructive suggestions. Here are our replies to your main concerns.

Weakness. the paper is almost unreadable even though its narrative and setup are not complicated, so you could not properly verify its soundness, although you found no clear issue with the design.

Thanks for your feedback. We've revised the main text. The story is: A verifier agent reads a colleague's note that falsely says the airline cancelled a flight, which would allow a refund. It judges the note FALSE and plans to KEEP the booking. If this whole assessment reaches the agent that acts, that agent rarely makes the forbidden cancellation. If only the two labels, FALSE and KEEP, reach it, the forbidden cancellation becomes much more common: 36% more on average over our ten models. Most of the protection comes back once the verdict says which claim is false and what is true instead: one sentence that states the true fact does this in 28 of 30 model-task pairs, which cover all ten models in all three tasks. In all three tasks we now test, the bare verdict is never safer than the whole assessment.

The introduction explains this with a familiar picture, a clerk who receives a form together with a slip that says only REJECTED. Section 2 explains each design choice, for example why we fix the labels and keep the false note in view, and Table 1 shows one concrete example per task. Section 6 turns the results into five rules for system builders, each with the case in which it fails (Table 4): check permission in code where possible, otherwise pass on the whole assessment from a check you trust, and if only a short field can pass, make it say which claim it rejects and what is true. We hope this also makes the design easy to check.

Question 1. whether it is clear and tells a strong narrative of the problem, our choices, our results and their implications to a reader not familiar with the work.

The revised introduction now gives that narrative in order: the problem (a correct verdict that arrives without its reason), our choices (a correct assessment held fixed while the false claim stays in view, read from the final database), our results (the bare verdict fails, and saying which claim is false and what is true restores protection) and their implications (the rules of Table 4). Its last paragraph says what is new: in the airline task the same FALSE protects far more when it quotes the false line than when it quotes a true one, and an order to stop protects more, for nine of our ten models in at least one chat role, when it appears to come from another agent (Table 3, Appendix R). We would be glad to hear whether it now reads clearly to you.

Question 2. why we focused on only one problem and setup, when it seemed easy to expand to at least one or two more tasks.

Thanks for your suggestions. We began with τ-bench airline because in its permission rules a false claim directly unlocks a forbidden write, and the final database shows whether it happened (Section 2). We've added two tasks: a retail task adapted from τ-bench and a document-sharing task. The core comparison now runs in all three and inside a LangGraph supervisor, each with five open and five closed models. In all three tasks the bare verdict is never safer than the whole assessment, and one sentence of the true fact closes most of the gap in 28 of the 30 model-task pairs. In each task, a permission rule turns on one fact that the record shows directly or by inference, such as whether the airline cancelled the flight. In our audit of the policies of τ-bench airline and retail and τ²-bench telecom, 83 of the 134 rules that decide an action (62%) rest on a fact that a tool output states or implies, the structure our three tasks share (Section 2, Appendix T).

AuthorsOpenReview ↗

Reply part 2

Your suggestion: a story we had missed, how people behave when shown an AI's verdict versus a verdict with an explanation, where the difference between people and our agents could be one of the most interesting findings.

Thank you so much for raising this interesting discussion; we did some exploration and added a new paragraph in Section 7. We find that people tend to rely too much on an AI's verdict. Nine of our ten models do the opposite with a bare one: under a header saying that the verifier checked the note, they follow a correct bare verdict in at most one retail conversation in five with our standard note (Appendices D and E). Once an explanation is added, they follow the verifier more, right or wrong, as people do. When the verifier is right, the whole assessment is followed at least as often as the bare verdict in all 30 model-task pairs. When it is wrong, the three models we tested follow a wrong whole assessment in 16–68% of conversations and a wrong bare verdict in 1–2%. There is one more parallel, with a correction that repeats the false claim, which is what quoting the false line does: in the airline task under the plain header, FALSE quoting the false line leaves all ten models fewer forbidden writes than the bare FALSE, 1–32% against 25–99% (Table 2). The comparison draws on published studies of people (further investigation and comparison with people will be in our future plan). Studies of automation and of AI-assisted decisions document over-reliance on automated and AI advice [1, 2]. An explanation raises acceptance of AI advice whether the advice is right or wrong [3, 4]. Even a bare predicted label raises people's accuracy in deception detection [5], and a correction that repeats the false claim can work better than one that does not [6].

If the rewritten paper now lets you verify its soundness, we hope you will reconsider your score.

References

[1] R. Parasuraman and D. H. Manzey. Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3):381–410, 2010.

[2] Z. Buçinca, M. B. Malaya, and K. Z. Gajos. To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–21, 2021.

[3] G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld. Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2021.

[4] S. S. Y. Kim, J. Wortman Vaughan, Q. V. Liao, T. Lombrozo, and O. Russakovsky. Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–19, 2025.

[5] V. Lai and C. Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 29–38, 2019.

[6] U. K. H. Ecker, J. L. Hogan, and S. Lewandowsky. Reminders and repetition of misinformation: Helping or hindering its retraction? Journal of Applied Research in Memory and Cognition, 6(2):185–192, 2017.

Dear Authors,

This is a rebuttal, not a revise and resubmit. I do not intend to re-read your paper, it was difficult enough the first time. I don't think the issues that I found are fixable without a full rewrite, which is out of scope within this process.

I highly suggest you give the draft to a human friend or colleague to test readability before submitting again.

Reviewer_1dD1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an agent that correctly rejects a false claim stays safe when only its verdict labels are kept or passed on.

Critical strengths

The core loss is well controlled. Every kept version comes from one saved assessment, so conditions differ only in the kept text. The loss holds for all ten models (Figure 2), without editing, and for four of five models between agents (Table 2). Eligible-task completion stays at 93–100% (Tables 44, 48), ruling out over-refusal.

The sender result is novel. The same stop instruction protects more when headed as a verifier agent’s message than as the agent’s own, for all five models (Table 4). Both are user-role messages, so chat role cannot explain it, and no account in Table 6 predicts it.

Critical weaknesses

Evidence rests mainly on one task with a templated false note, and the main text omits a key boundary. The loss is measured with the record before the note; an identical record message (a reminder plus the record) leaves gpt-oss-20b, gpt-oss-120b and GLM-4.5-Air at 73/56/47% forbidden cancellations before the note but 31/5/5% after it, with no assessment. Losses vary widely: 11–36 points outside gpt-oss (Figure 2).

The thesis, that agents act on stated facts more than on verdicts, is confounded with a simpler explanation: a bare FALSE does not say which claim it rejects. Between agents, labels quoting the claim work about as well as the stated fact for four of five models (Table 2). Quoting was not tested in the agent’s own turn, where a test with the note cut to its false sentence rules this out only for gpt-oss-20b (Table 33). Table 66 recommends claim-quoting verdicts; but the thesis does not explain them.

Questions for authors

(Clarification) Why is showing the record after untrusted input (Figure 3) absent from Section 5’s defenses and Tables 5, 66 and 67?

(Suggestion) A whole-versus-labels comparison beyond the templated note (e.g. another τ-bench domain) would address weakness 1.

3 comments
AuthorsOpenReview ↗

Reply part 1

Thank you for your valuable comments and constructive suggestions. Here are our replies to your concerns.

Weakness 1. the evidence rests mainly on one task with a templated false note; that the main text omits a key boundary, since the loss is measured with the record before the note, and the same message of a reminder plus the record, with no assessment, leaves gpt-oss-20b, gpt-oss-120b and GLM-4.5-Air at 73/56/47% forbidden cancellations before the note but 31/5/5% after it; and that losses vary widely, 11–36 points outside gpt-oss.

On one task and one note: thanks for your suggestion and the comparison now runs in three tasks, the airline task, a retail task adapted from τ-bench and a document-sharing task with its own note, policy and records, each with five open and five closed models. In the 30 model-task pairs, the bare verdict is never safer than the whole assessment. In the airline task the loss also survives six model-written rewordings of the note per task, each checked to keep every fact: in the agent's own turn the bare verdict leaves 86–97% forbidden writes against 5–85% with the model's own assessment, and the loss is positive under every rewording for all three models tested. Without any note, when the customer or a tool result states the false claim instead, the verifier's labels alone still leave 53–98% forbidden writes, and the same reply with its fact line 0–25%.

On the boundary: the main text now states it (Section 3). With the record and a reminder after the note, the bare verdict does as well as the whole assessment for all four models we tested: both leave 0–2% for gpt-oss-20b, gpt-oss-120b and GLM-4.5-Air, and 45% and 49% for Qwen3-32B. Without a verdict, neither part works alone: the record alone leaves 45–94% and the reminder alone 69–96%.

On the size of the loss: it varies by model, but its direction does not. All ten models lose protection in their own turn (sign test over models, p = 0.001). A mixed model puts the average own-turn loss at 27 points over all ten models (95% interval 12 to 40), and at 20 points (11 to 29) without the two gpt-oss models, and it predicts a loss above zero for 97% of new models like the other eight (Section 3, Appendix J). In the handoff, the bare verdict is never safer than the whole assessment in any of the 30 model-task pairs (Appendix F).

AuthorsOpenReview ↗

Reply part 2

Weakness 2. the thesis, that agents act on stated facts more than on verdicts, confounded with a simpler explanation: a bare FALSE does not say which claim it rejects. Between agents, labels that quote the claim worked about as well as the stated fact for four of five models; quoting had not been tested in the agent's own turn, where the one-line-note test ruled the explanation out only for gpt-oss-20b; and the thesis did not explain the claim-quoting verdicts we recommended.

Thank you for this suggestion. We tested it directly: it explains part of the loss, but not all of it. This is why our revised thesis is about both what a verdict rejects and what it says is true, and why it now explains the claim-quoting verdicts we recommend.

The test confirms the part your explanation predicts. We fixed the verdict at FALSE and KEEP and added one clause that quotes either the false line or a true line of the same note. In the agent's own turn, for the four open models whose bare verdict leaves room for the test (51–81%), quoting the false line cuts forbidden writes to 0–40%, while quoting a true line leaves 48–86%, within 11 points of the bare verdict. DeepSeek-V4-Flash's bare verdict leaves only 9% in its own turn, and the quote 0% (Table 2). In another agent's message under the plain header, the same holds for GLM-4.5-Air, with 80% for the bare verdict, 1% for the quote and 73% for the true-line quote. It also holds for the closed models: for all five the quote leaves 2–13%, and for the three whose bare verdict leaves room for the test the true-line quote leaves 55–88% (Appendix E). So naming the rejected claim restores much of the protection.

Three results show that naming the claim is not the whole story.

  • A bare verdict can fail even when there is nothing ambiguous to name. Cut to its false line, the note holds only one claim that FALSE can reject. This removes most of gpt-oss-120b's own-turn loss (from 65 to 7 points), as your explanation predicts. But gpt-oss-20b keeps a loss of 37 points in its own turn, and in another agent's message under the plain header, the bare verdict with the one-line note still leaves 41–98% forbidden writes for all five open models (Appendices D and E).
  • Naming the claim is not enough when denying it does not settle the rule. In retail, FALSE on "delivered 10 days ago" leaves open whether the 30-day window has passed, and the quote closes most of the gap in only 2 of the 10 retail pairs. With a false line we wrote whose denial does settle it, "delivered within the last 30 days", the quote protects five of our ten models (5–19%), while one sentence of the stated fact protects all ten, at 0–5% (Appendix G). In the conversations, GPT-5.4 under this quote stops calling the order returnable (1 of its 90 returns, against 29 of 120 conversations with the bare verdict), yet it treats the delivery date as unverified rather than past the window and still returns 75% of orders.
  • Who appears to send a message matters even when it names no claim at all: the same order to stop protects more when it appears to come from another agent than as the model's own earlier message, for nine of our ten models (Section 5).

Across tasks, the stated fact is the part that protects reliably. It closes most of the gap between the bare verdict and the whole assessment in 28 of the 30 model-task pairs, brings all five closed models to 0–1% in documents and 0–20% in retail, and in the agent's own turn clearly beats the quote for gpt-oss-20b, 1% against 30% (Appendix F). In the airline task the two do about equally well, as you noted for the handoff, because there FALSE next to the quoted line already implies the fact.

Our thesis is therefore that much of the protection lies not in the verdict itself but in what it carries: which claim is false and what is true instead. Naming the claim is one part of this, and it is what makes the claim-quoting verdicts we recommend work; the stated fact is the part that protects most widely, in 28 of the 30 model-task pairs. This also answers your last point: our recommendation now follows from the thesis. Rule 3 of Table 4 asks a short verdict to name the claim it rejects, which makes a quote work where its denial settles the rule, and to state what is true, which protects most widely.

AuthorsOpenReview ↗

Reply part 3

Clarifying question. why showing the record after the untrusted input was absent from the defenses of Section 5 and Tables 5, 66 and 67.

Thanks for pointing this out and we've added it: it is now rule 4 of Table 4, with Qwen3-32B listed as the case in which it fails, and a row of Table 73, which lists where the loss is absent. Section 2 explains why the record comes first in our design: after the note it protects more, and Appendix N gives the rates you cited. Table 70 compares defenses and their cost on eligible tasks within one run; since we compare rates only within a run, the record-after rates appear in Table 4 and Appendices D and N.

Suggestion. a whole-versus-labels comparison beyond the templated note, for example in another τ-bench domain.

Thanks for your suggestion. We've added a retail task adapted from τ-bench. Its record gives only a shipping date, so the note's delivery date is false only by inference from it; to keep every assessment correct, code writes the reason from the order record, as our protocol fixed in advance (Section 3). Against that whole assessment, the bare verdict adds 79 to 92 points of forbidden returns for each of the five open models, which return about as many orders as with no assessment at all (80–100% against 73–99%), so it undoes almost none of what the false note adds over the honest note (11–63%). All five closed models lose protection too: 62–100% with the bare verdict against 0–2% with the whole assessment. For GPT-5-mini, the false note raises returns from 5% to 81%, the whole assessment brings them to 0% and the bare verdict leaves 62%. The other four return 91–99% of orders even under the honest note, and the bare verdict leaves them at 96–100%; what protects them is the true fact that the reason carries. One sentence stating the true fact, "The parcel was delivered more than 30 days ago", brings all five to 0–20% (Appendices D and F). With a second false line that we wrote, the loss holds for all ten models: 28–100% with the bare verdict against 0% with the whole assessment (Appendix G).

the sender result is novel: the same stop instruction protects more when it is headed as a verifier agent's message than as the agent's own, for all five models, and chat role cannot explain it.

We kept it and extended it. In the airline task, the same order to stop protects more when another agent is said to send it, for all seven models of Table 3, from five developers. For the two we added, Claude Haiku 4.5 and GLM-4.5-Air, it leaves 24% and 15% under the verifier's header, against 58% and 75% when it appears to be their own earlier message. A separate test of chat roles adds GPT-5.4 and Grok 4.3, so the gap holds for nine of our ten models in at least one role (Section 5, Appendix R).

If this resolves the confound and the boundary you raised, we hope you will reconsider your score. Thanks so much

Reviewer_WM8tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines whether retaining only judgment and plan labels is sufficient to preserve safe behavior after a tool agent has correctly identified a false claim. In the τ-bench airline environment, the authors compare different versions of the same saved assessment—full assessment, reasoning, or labels only—and measure forbidden writes through the final database state. The results suggest that labels alone may lose much of the protection, while retaining premises directly relevant to the policy is more effective. The paper also examines the effects of the stated sender and claimed check in handoffs.

Critical strengths

The problem is practically important: the paper distinguishes correctly identifying a false claim from subsequently acting safely, highlighting a potential risk in context compaction and multi-agent handoffs when only a verdict is passed along. The paired interventions on the same saved assessment, evaluated through actual database changes, also make the central observation more convincing than an analysis of model text responses alone.

Critical weaknesses
  1. The writing reads more like a technical report than a focused research paper, and the central contribution is insufficiently distilled. The main text presents many experimental conditions and model results, with frequent cross-references to appendices, but does not clearly distinguish the primary contributions from supplementary validation. Readers must cross-check multiple rounds, tables, and appendices to assess the evidence and its limits. Although Section 7 discusses related work, the distinction from research on knowledge conflict, context compaction, and handoff safety remains unclear; the paper does not adequately explain which findings go beyond existing accounts.
  2. The experiments vary many factors, but their coverage remains too limited to support broad conclusions. The main evidence comes from τ-bench airline, where the true premise typically has to be inferred from fields such as “available,” whereas the false claim is stated directly. Adding models and wording variants does not substitute for validation across tasks, datasets, and agent frameworks. Appendix V’s document-sharing task offers some additional evidence, but among cases where gpt-oss-120b correctly judged the claim as FALSE/HOLD, forbidden sends increase only from 8% with the whole assessment to 12% with labels only (Table 70), providing limited support for the central claim. Handoffs are also primarily simulated through constructed messages, without replication in different real agent frameworks. The current evidence therefore does not clearly distinguish a general risk from compressing judgments from effects specific to the task, evidence representation, or message format.
Questions for authors
  1. Please clarify the paper’s contribution relative to prior work on knowledge conflict and handoff safety. Since “labels tied to the claim” in Table 2 appears to perform similarly to a stated fact, does this suggest that the key factor is explicitly linking labels to the claim, rather than retaining a premise being more important than retaining a verdict?
  2. Beyond the airline setting, is there stronger evidence that labels-only assessments substantially increase forbidden writes even when the assessment is already correct? In particular, could you explain the results on the correctly judged subset of the document-sharing task?
  3. How sensitive are the results to the handoff format and agent framework? Could you replicate the core comparison in a different, practical handoff workflow to test whether the findings depend on the current message construction?
3 comments
AuthorsOpenReview ↗

Reply part 1

Thank you for the careful reading and constructive suggestions. Here are our replies to your concerns.

Weakness 1. the paper read like a technical report rather than a focused research paper: it presented many conditions and model results with frequent cross-references, did not separate the main contributions from supplementary validation, and left unclear which findings go beyond work on knowledge conflict, context compaction and handoff safety.

As you suggest, we have revised the writing of the main text around one result and the two findings that explain it.

The result: in three tasks and for ten models, a correct verdict passed on as bare labels is never safer than the whole assessment, and on average it leaves 20% more forbidden actions in documents, 36% in the airline task and 89% in retail.

The findings: much of the protection comes from saying which claim is false and what is true instead, and in the airline task an order that names no claim protects more when it appears to come from another agent.

On which findings go beyond earlier work: each of the three lines of work explains a different way an agent comes to act on a false claim, and none of them covers our case.

  • Work on knowledge conflict sets the context against the model's memory, and finds that coherent evidence in the context tends to win. In our setting there is no such conflict: the false claim and the correct verdict are both in the context, and the question is which part of a correct judgment keeps the agent from acting on the claim.
  • Work on context compaction and handoff safety shows that information gets lost on the way: rules are dropped, or the fact that a claim was never checked. In our setting nothing is lost: the correct verdict arrives and the false claim stays in view, yet a bare verdict still fails.

Three findings go beyond these accounts. First, a correct bare verdict is never safer than the whole assessment, for all ten models in all three tasks and in a LangGraph supervisor. Second, the protection depends on what the verdict points to: in the airline task, FALSE in another agent's message protects far more when it quotes the false line than when it quotes a true line of the same note, by 42 to 93 points for all eight models whose bare verdict leaves room for the test, and one sentence that states the true fact closes most of the gap in 28 of the 30 model-task pairs. Since the verdict and the false claim are both present, lost information cannot explain this. Third, who appears to send the message matters: the same order to stop, naming no claim, protects more when it appears to come from another agent than as the model's own earlier message, for nine of our ten models, which earlier accounts do not predict (Sections 1, 5 and 7).

Weakness 2. the coverage is too limited for broad conclusions: the evidence came mainly from the airline task, where the true premise has to be inferred from fields such as "available" while the false claim is stated directly; more models and wordings do not replace other tasks, datasets and agent frameworks; the document task gave limited support; and handoffs were mostly constructed messages. So the evidence could not separate a general risk from effects of the task, the evidence representation or the message format.

Thanks for the valuable suggestion. We've added two tasks and a real agent framework, each with five open and five closed models. One task is a retail task adapted from τ-bench. The other is a document-sharing task of our own, with its own policy, records, tools and note, so the result does not rest on τ-bench alone. In the 30 model-task pairs, the bare verdict is never safer than the whole assessment, and in a LangGraph supervisor it loses protection as well.

On the inferred premise: when the record states it outright, listing each flight as "cancelled by the airline: no", the bare verdict still leaves more forbidden writes than the whole assessment for all four models we tested, by 12 to 48 points, with Holm-corrected p = 0.0004 each (Tables 14 and 15). We give the evidence beyond the airline under your second question and the message format under your third.

AuthorsOpenReview ↗

Reply part 2

Question 1. clarify the paper's contribution relative to prior work on knowledge conflict and handoff safety, and, since labels tied to the claim did about as well as a stated fact in the submitted Table 2, whether the key factor is linking the labels to the claim, rather than keeping a premise instead of a verdict.

On prior work: as we set out under your first weakness, knowledge conflict concerns context against memory, and work on handoffs and compaction concerns information lost on the way. In our setting the correct verdict arrives and the false claim stays in view, yet a bare verdict fails, and its protection depends on which line it names and on who appears to send it. For the seven models of Table 3, from five developers, the same order to stop leaves 0–42% forbidden writes under the verifier's header against 58–95% as the model's own earlier message, and a separate test of chat roles, which adds GPT-5.4 and Grok 4.3, brings the count to nine of our ten models.

On the key factor: yes, thanks for pointing this out and we now show it directly. In the airline task, we fixed every verdict at FALSE and KEEP and added one clause that quotes a line of the note. For gpt-oss-20b, gpt-oss-120b and Qwen3-32B, quoting the false line cuts forbidden writes from 70–99% to 1–40%, both in the model's own turn and in a handoff under a plain header, while quoting a true line of the same note leaves 70–97% (Table 2). GLM-4.5-Air shows the same in both places. Under the plain header, so does DeepSeek-V4-Flash, with 34% for the bare verdict, 1% for the quote and 58% for the true-line quote. So do the three closed models whose bare verdict leaves room for the test: the bare verdict leaves them 66–97%, the quote of the false line 12–13% and the quote of a true line 55–88%. So a correct verdict alone is not enough: it has to name the right claim.

In the airline and document tasks, where marking the claim false settles the rule, this link does most of the work: the quote closes most of the gap to the whole assessment in 17 of these 20 pairs, and in documents the quote and the stated fact leave 0–21% and 0–17% forbidden sends across all ten models. Retail shows what else a short verdict needs. There, FALSE on "The carrier delivered the parcel 10 days ago" does not settle the 30-day return window, since the line is also false if the parcel arrived 20 days ago. We therefore wrote a false line whose denial does settle it, "The carrier delivered the parcel within the last 30 days". Its quote lowers forbidden returns for all ten models and brings five of them, three open and two closed, to 5–19%. The other five still return 30–88%, while the stated fact leaves all ten at 0–5%. So naming the rejected claim helps all ten, but only the true fact protects them all.

Question 2. whether, beyond the airline setting, there is stronger evidence that labels-only assessments substantially increase forbidden writes even when the assessment is already correct, and explain the results on the correctly judged subset of the document-sharing task.

Yes. Outside the airline we fix every verdict at the correct labels, FALSE and KEEP (HOLD in documents), and every whole assessment carries those labels and cites the record, so each assessment is correct by construction. In retail, with the whole assessment written by code from the order record, our five open models make forbidden returns in 0–13% of conversations. With the bare verdict they return 80–100%, about as many as with no assessment at all (73–99%), while the same note without its false line leads to 11–63%, so a correct bare FALSE undoes almost none of what the false line adds. The five closed models lose protection too: 62–100% with the bare verdict against 0–2% with the whole assessment, and one sentence that states the true fact, "The parcel was delivered more than 30 days ago", brings all five to 0–20%. A second false line that we wrote, "delivered within the last 30 days", shows the same loss for all ten models: 28–100% forbidden returns with the bare verdict against 0% with the whole assessment.

In documents, the bare verdict is never safer than the whole assessment for any of the ten models, and leaves 33 to 55 points more forbidden sends for gpt-oss-20b, Qwen3-32B, GLM-4.5-Air and Gemini 2.5 Flash; the other six refuse most sends even with the bare verdict (at most 17%). In the correctly judged subset, gpt-oss-20b goes from 23% with the whole assessment to 80% with the bare verdict, and Qwen3-32B from 12% to 69%. DeepSeek-V4-Flash and GPT-5-mini refuse most of these sends with either text. The 8% and 12% you cite are gpt-oss-120b's (now 7% and 10%). Its few sends there are a different error, a misreading of the redaction rule: all 15 are redacted copies sent after it rejected the note, none claiming that approval exists, and in the handoff all 21 of its bare-verdict sends are redacted copies too.

AuthorsOpenReview ↗

Reply part 3

Question 3. how sensitive the results are to the handoff format and the agent framework, and whether the core comparison replicates in a different, practical handoff workflow, to test whether the findings depend on our message construction.

The core comparison replicates in a real framework, for all ten models. We ran a LangGraph supervisor with the library's default of passing on only the last message, the reply-only pattern that is the default or documented one in 15 of the 24 frameworks and protocols we checked (Table 10). The verifier had the policy, was always consulted first, and replied with a JSON verdict. With the verdict alone, four of our five open models make a forbidden write in 87–95% of conversations, and GPT-5.4, GPT-5-mini and Gemini 2.5 Flash in 24–79%; one added sentence of reason brings all seven to 1–14%. Claude Haiku 4.5, Grok 4.3 and DeepSeek-V4-Flash refuse most of these writes even with the verdict alone (5–22%), and the reason takes them to 0–4%. With the reason, all ten models complete 96–100% of eligible tasks.

On message format: a header that names the note gives FALSE a target, as the quote does, and takes gpt-oss-120b's bare verdict from 94% to 9%. When we split our standard header into its two parts, naming the note lowers the bare verdict's forbidden writes for all five open models, by 10 to 85 points, while a claimed check alone moves them by at most 2 points, except for gpt-oss-120b (Section 5). For gpt-oss-20b, Qwen3-32B and GLM-4.5-Air, the bare verdict leaves 66–99% under all four headers we tried (Table 3), while the whole assessment protects under both the plain and the standard header. The chat role does not rescue a bare verdict either: for six models, two of them closed, it leaves 43–100% whether it arrives as a user message, an assistant message or a tool result, against 67–99% with no message at all.

We hope the revision addresses your concerns. If it does, we would be grateful if you would reconsider your score.

References

[1] Y. Sun, H. Wang, Y. Zhu, Z. Li, Z. Zhao, and Y. Yuan. When "Must" becomes "Maybe": Erosion of workflow constraints in LLM agent handoffs. arXiv preprint arXiv:2608.24569, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.