- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper is an empirical behavioral study that examines current LLMs from a very small but very interesting perspective: task-relevant content is not necessarily decision-relevant evidence. The authors design four experiments covering everyday decisions, real-world Reddit opinions, product selection, and misinformation judgments. By keeping the task information and candidate options unchanged, they examine the influence of additional text on model behavior. The results demonstrate the insight that external text can not only change model preferences but also cause models to violate explicit user requirements or become more likely to accept false information.
Critical strengths
- Research Significance
This paper approaches the problem from a relatively narrow but interesting perspective, focusing specifically on how external information influences LLM decision-making. The topic is closely related to real-world LLM applications and has certain practical significance.
- Detailed Experimental Design and Clear Presentation of Results
The paper presents a relatively comprehensive and carefully designed set of experiments, considering different models, prompt formats, and answer orders. It examines both subjective choices without a uniquely correct answer and objective judgments with explicit correctness criteria. The four experiments also have a certain logical connection. In addition, the experimental visualizations are clear and aesthetically pleasing, using various forms such as scatter plots, bar charts, and box plots, with vivid colors that help readers intuitively understand and compare the results.
Critical weaknesses
- Limited Technical Contribution and Unclear Distinction from Prior Work
This paper does not propose a new model architecture, optimization objective, or algorithm. Its core approach consists of controlling input content and prompt formats, and even the mitigation strategies are primarily prompt-level interventions rather than a newly implemented and validated system. This is an empirical paper centered on input interventions and behavioral measurements, with almost no algorithmic contribution.
Furthermore, prior studies such as Shi et al. (2023) and Yoran et al. (2024) have already demonstrated from different perspectives that irrelevant context can interfere with LLM outputs and cause errors even when the correct answer remains unchanged. The Related Work section mainly distinguishes this paper from previous studies based on task categories, such as QA vs. decision-making and prompt injection vs. passive exposure, but does not clearly explain what substantially different insights this work provides beyond extending existing findings to another task setting.
- The Rigor of the Experimental Design Requires Further Clarification
The paper's central research question is: "Can such passive exposure steer a decision even when the added content provides no reason to change it?" However, the justification for considering the additional content decision-irrelevant is insufficient. In Sections 3.1 and 3.2, although the added content is not directly related to the specific task, this does not necessarily mean that it should not influence the model's decision. These are two different concepts.
For example, in Section 3.2, Reddit users' experiences with buying and renting homes actually provide decision-relevant considerations, such as maintenance responsibilities and housing autonomy. Therefore, it may be reasonable for models to consider these opinions and change their choices. The authors should explain why such content is considered decision-irrelevant.
Questions for authors
-
Given that prior studies have already shown how irrelevant context can influence LLM outputs and decisions, could the authors clarify what new insights this work provides beyond extending existing findings to different decision-making scenarios?
-
In Sections 3.1 and 3.2, the additional content may provide useful information for decision-making. Could the authors provide further evidence or clarification to support the claim that "the added content provides no reason to change it"?
-
Minor presentation issues: Figure 1 is relatively simple, with text and lines that are too small and thin, making it difficult to read. Additionally, the abstract does not clearly explain the research gap or distinguish this work from existing studies.
Reply to author response
Thanks for the detailed response and the new experiments. The answers to Q1, Q4, and Q5 are clear and helpful. Please include these results in the revised version.
I still have some concerns about Q2 and Q3:
I'd appreciate it if the authors could add this information. For now, I will keep my score.