acceptodds
Under review as a conference paper at ICLR 2027

Same Requirements, Different Decisions: How Presentation Changes Which Requests LLM Agents Allow

Abstract

LLM agents increasingly decide whether an action is allowed under an organization's policies. Policies list their requirements, yet they rarely say how these requirements combine into one decision, for example whether every applicable document must permit the action before it proceeds. The agent must therefore infer this unstated rule. We examine whether LLMs reach the same decision under every presentation of the same requirements. We compare the same requirements under six presentations that leave every correct decision unchanged, from layout and wording to the number of documents and the output requested. Across 12 open-weight and three Claude model configurations, models do not apply one rule. Layout, wording, and length move the rate of wrongly allowed requests by at most 5.4 percentage points, in both directions. By contrast, splitting one policy into two documents, or asking for supporting fields with the decision, raises that rate by 22 to 32 points, almost entirely toward wrong approvals. The errors consistently reflect the rule each presentation suggests. Reasoning models, for example, treat the two documents as alternatives. Some models require every condition in a bare list to hold and refuse requests they should allow. Claude Opus 5 makes no errors on plain-text policies. Yet Opus 5 wrongly approves a request in 32 of 72 attempts once one organization's rules are split across two of its own documents and supporting fields are requested. The same split with only the decision requested produces no wrong approval. The presentation thus supplies the decision rule that the policy leaves unstated. A test of a policy-following agent under one presentation therefore does not show how the agent will decide under another.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, policy compliance, authorization decisions, presentation sensitivity, robustness evaluation, output format, AI safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xiZLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how presentation affects language models’ authorization decisions when policy requirements remain unchanged. The authors derive reference answers from formalized policies and conduct paired comparisons across layout, vocabulary, length, document splitting, output fields, and structured representations. The results show that splitting documents and requesting five-field outputs substantially increase incorrect approvals, while listing conditions without a combination rule can lead to unnecessary denials. The paper uses single-rule tests and condition-level judgments to examine how these errors relate to policy combination, and discusses implications for authorization interface design.

Critical strengths
  1. Holding policy content fixed while varying document organization and the output interface directly tests the consistency of authorization decisions. This provides a useful complement to existing policy-following evaluations.
  2. The paired comparisons account for text length and the number of governing authorities, while reference answers are cross-checked through exhaustive enumeration and an independent Z3 encoding. These measures provide good control over experimental variables and label quality.
  3. The prompt templates, model configurations, and protocol changes are well documented. The paper also explains the data corrections, making the experiments easier to audit.
Critical weaknesses
  1. The attribution of document-splitting errors still conflates policy interpretation with rule combination. Some responses treat unmet policies as “not applicable,” while the condition-level tests introduce additional explanations. These tests therefore cannot directly rule out interpretation errors in the original task.
  2. The five-field template does not explicitly define fields such as “applicable principals” and “remaining obligations,” potentially leading models to misunderstand the output requirements. The current comparison does not distinguish errors caused by requesting supporting information from those caused by unclear field definitions. Clarifying these definitions is another plausible remedy, so the recommendation to return only the decision in deployment is too broad.
  3. The core evaluation contains only 24 template-based cases, and the public-policy extension does not test document splitting under a single authority or changes in output format. Coverage of real-world policy settings therefore remains narrow.
Questions for authors
  1. In the original five-field records, can you distinguish between cases where the model treats an unmet document as inapplicable and cases where it recognizes that the document requires denial but nevertheless approves the request? What proportion falls into each category?
  2. If all five fields are retained but their meanings are explicitly defined, does the same increase in incorrect approvals persist? This comparison would help distinguish an effect of this particular template’s semantics from an effect of requesting supporting information itself.
Reviewer_JL7nOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents are increasingly used to approve or deny actions depending on a set of policies laid out by organisations. The work investigated how the presentation of the policies and requirements would affect the LLM's false approval rate. A reference is fixed and the policy and request facts are presented in six different ways, with a shared correct decision in which the ground truth is established reference evaluator.

The work finds that surface changes such as layout, wording, and length, perturb the false-approval weight to a smaller degree, i.e. "adds noise" than structural changes, which can severely impact the false-positive rate. These include splitting a single policy across two documents or requesting supporting fields along with the accept/deny decision. In particular, it is shown that requesting supporting fields significantly increases the incidence of false approvals.

Critical strengths
  • The experimental matched-comparision design is strong.
    • The setup uses a machine-generated ground truth removes subjectivity on what the ground truth result could be.
    • Coverage of open-weight models is broad.
  • Experimental history is transparent and clearly discloses exclusions, fixes, and limits of applicability.
    • The experiment properly distinguishes between false-approval and false-denial rates, which are regarded differently in terms of safety.
    • The work goes beyond reporting accuracy changes and conducts thorough mechanism-oriented diagnostics, leading to the interesting conclusion that models can judge individual conditions correctly, yet judge them incorrectly when combined.
Critical weaknesses
  • Too much of the paper's findings are overly dependent on referencing the Appendix.
    • Appendix Table 8 in particular contains information that should feature more prominently in the paper that is needed to convey the paper's claims.
    • Lines 975-980 includes an important detail that is omitted from the main paper - that the "variant identifier differs between the one- and two-document conditions." This should be disclosed in the main body of the paper.
    • The external-validity check only covers a single sentence in the main text.
  • Although the coverage of open-weight models is broad, there are a few weaknesses regarding the selection of experimental parameters:
    • Only a single closed-frontier model was used, under different conditions from the open-source models (no seed or temperature control). Although many models were tested, the selection of models is heterogeneous - the models differ in quantization, reasoning budgets, and structured decoding mechanisms. In particular the closed-source model (Claude) rejecting entire cases and the fact that these are removed is enough to challenge the comparability of this model to the rest.
    • The effective sample size is small - the templated cases used may share linguistic/logical regularities that could limit the scope of the claims.
Questions for authors
  • (198 to 209) How sensitive are the findings to constrained decoding, quantization, reasoning budget, and prompt field order?
  • (205 to 206) Why are the Claude models run without seed controls?
    • Why was Claude the only class of closed-weight frontier model tested?
  • (211 to 215) Do the document-split effects persist when all experimental identifiers are removed from the prompt visible by the model?
  • (211 to 215) Can the main split and output-format findings be preregistered and replicated on a new set of cases generated from different templates or by independent authors?
  • How would humans interpret the unstated combination rules, i.e. are these combination rules abundantly obvious for a human faced with these rules?
  • (332 to 349) Why does the explicit conjunction sentence fail for several models?
    • Would a more operational instruction, such as computing one verdict per document and then applying a supplied Boolean expression, succeed without providing the full compiled checklist?

Numbers here refers to lines of the paper.

Reviewer_wWZpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors studied whether LLMs make same decisions when the same policy requirements are presented in different ways. Authors created synthetic policy cases where the correct decisions are known beforehand. They vary six presentation factors (layout, vocab, number of documents, output format) while keeping the correct decision consistent. They evaluate 12 open-weight and three Claude model configurations on 24 held-out cases across 8 domains. Authors presented some interesting findings, like surface-level variations have small effects but splitting policies into multiple documents increases incorrect approvals by 22-32 percentage points. They further analyze whether the errors happen because models misunderstood conditions or couldn't compose them correctly.

Critical strengths
  • Authors presented interesting findings, like policy presentation can influence LLM agents decisions even if the actual requirements remain the same. They performed fine-grained analysis, e.g., separately analyzed incorrect approvals vs incorrect denials, whether models correctly understand individual policy conditions but combine them incorrectly when making the final decision
Critical weaknesses
  • Authors used limited number of synthetic cases in main experiment, 24 held-out cases with 12 prohibited and 12 permitted requests. Also, authors didn't perform document-splitting and output-format experiments on three real public policy documents. It would be interesting to know whether similar effects are observed in real-world settings.

  • Experiments mainly involve a single allow-or-deny decision. It is unclear how well these findings apply to agents performing longer tasks with multiple tool calls.

Questions for authors
  • Would it be possible to perform document-splitting and output-format experiments on real-world policies?

  • How do you ensure that different presentations have exactly the same meaning? Did you also perform independent human validation?

  • How do you position your work against Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges (Weng et al., 2026)? What is the main difference in the findings beyond the application setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.