- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how presentation affects language models’ authorization decisions when policy requirements remain unchanged. The authors derive reference answers from formalized policies and conduct paired comparisons across layout, vocabulary, length, document splitting, output fields, and structured representations. The results show that splitting documents and requesting five-field outputs substantially increase incorrect approvals, while listing conditions without a combination rule can lead to unnecessary denials. The paper uses single-rule tests and condition-level judgments to examine how these errors relate to policy combination, and discusses implications for authorization interface design.
Critical strengths
- Holding policy content fixed while varying document organization and the output interface directly tests the consistency of authorization decisions. This provides a useful complement to existing policy-following evaluations.
- The paired comparisons account for text length and the number of governing authorities, while reference answers are cross-checked through exhaustive enumeration and an independent Z3 encoding. These measures provide good control over experimental variables and label quality.
- The prompt templates, model configurations, and protocol changes are well documented. The paper also explains the data corrections, making the experiments easier to audit.
Critical weaknesses
- The attribution of document-splitting errors still conflates policy interpretation with rule combination. Some responses treat unmet policies as “not applicable,” while the condition-level tests introduce additional explanations. These tests therefore cannot directly rule out interpretation errors in the original task.
- The five-field template does not explicitly define fields such as “applicable principals” and “remaining obligations,” potentially leading models to misunderstand the output requirements. The current comparison does not distinguish errors caused by requesting supporting information from those caused by unclear field definitions. Clarifying these definitions is another plausible remedy, so the recommendation to return only the decision in deployment is too broad.
- The core evaluation contains only 24 template-based cases, and the public-policy extension does not test document splitting under a single authority or changes in output format. Coverage of real-world policy settings therefore remains narrow.
Questions for authors
- In the original five-field records, can you distinguish between cases where the model treats an unmet document as inapplicable and cases where it recognizes that the document requires denial but nevertheless approves the request? What proportion falls into each category?
- If all five fields are retained but their meanings are explicitly defined, does the same increase in incorrect approvals persist? This comparison would help distinguish an effect of this particular template’s semantics from an effect of requesting supporting information itself.