- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether language-model decisions change with labels, source descriptions, and prior route advice. It organizes these behaviors into ten overlapping bias categories and evaluates static choices, answer evaluation and correction, and hidden-grid navigation across 59 model endpoints, collecting 381,258 response and episode records. The reported findings include label and position preferences, source-dependent ratings with mixed correction effects, and substantially lower navigation success under misleading advice. The intended contribution is a unified empirical characterization of these behaviors.
Critical strengths
The paper addresses a relevant evaluation problem. Examining sensitivity to presentation and advice across several decision settings yields useful descriptive observations about model behavior beyond isolated answer accuracy. The fixed-content attribution experiment includes a useful anonymous baseline. This helps distinguish favorable treatment of own-attributed answers from unfavorable treatment of other-attributed answers, as illustrated by the GPT-4o-mini results in Section 4.3.
Critical weaknesses
The scientific contribution is not sufficiently established. Related Work mentions prior research without explaining the added value over label-bias measurement [1], self-preference studies [2], and existing multi-bias evaluation such as CoBBLEr [3]. The ten categories combine input variations with summary measurements, but their additional explanatory or diagnostic value remains unclear. The central reliability interpretations are not adequately isolated. Under requested temperature zero, the evidence-free choice tasks permit any option, so unequal selection alone does not establish a decision failure. In navigation, agents lack global reward information and are not told that the decoy is suboptimal. The success gap therefore does not distinguish planning inertia from limited information and exploration difficulty. The Discussion acknowledges this ambiguity, but the relevant controls remain unevaluated. Limited evidence for generalization and attribution effects. The answer-evaluation experiments use six basic questions, and navigation uses two maps, limiting generalization. The supplementary code provides exact-body matching and model-level bootstrap analysis for attribution comparisons, which is helpful. However, the resulting matched-subset estimates, confidence intervals, and retained sample counts are not reported, leaving the robustness of the correction effects unclear. Incomplete reporting of robustness results. The supplement includes code for strict-validation and repair-exclusion analyses, but I could not locate the empirical results corresponding to Appendix E. The inconsistent headline counts between the abstract and main text also remain unresolved. These gaps make the reported findings difficult to verify.
Questions for authors
Which principal finding or evaluation capability is new relative to [1–3], and what additional insight does the ten-category framework provide? What evidence connects the measured label and position preferences to incorrect decisions when task-relevant evidence is available? Can no-advice and explicit decoy-warning or full-information controls distinguish planning inertia from information-access and exploration limitations? Under exact-body matching, including identical option order, how many attribution pairs remain, and do the correction effects persist with uncertainty estimates? Could the authors provide the Appendix E sensitivity results, including retained sample counts and principal effects after strict validation and repair exclusion, and reconcile the abstract and main-text statistics?
I have read the author's response. The new results partly solve my concerns. However, these new results are still insufficient to solve Weaknesses 1 and 2. Therefore, I maintain my scores.