- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a study that evaluates if humans and AI agents can detect that they are being manipulated by an LLM recommender assistant they are interacting with. Rather than focusing on evaluating if an LLM recommender assistant can be manipulated, this work is the first that studies whether this has any downstream effect on the users (both AI and humans). This is done in a synthetic environment that simulates a platform for hotel booking, with 6 different hotels and reviews assigned to each hotel, focusing on different quality aspects. The evaluation is done in a few settings, with different amounts of visible data (only assistant’s output, +reviews, +note that they might be manipulated). The results indicate that both humans and AI agents are manipulated in a large percentage of cases, and that the patterns are different. Overall, this is an interesting starting point in evaluating the manipulations of LLM recommenders end to end.
Critical strengths
[significance] The paper addresses a real gap in literature, focusing on the efficiency of recommender systems’ manipulations on the actual users, rather than on the recommender LLM only which is extensively covered by previous research.
[quality] The setup is clean, and ablations are extensive and make sense. The authors investigate the effects on both humans and agents in multiple settings, also simulating different types of users. Human study seems to be done carefully.
[clarity] The paper is very easy to follow, there are extensive additional information in the appendix that support the main story
Critical weaknesses
[quality] The main disadvantages of the paper are limited domain (only hotel recommendations) with small number of options (6 hotels), two types of manipulation (explicit prompt which is unrealistic and reviews which are more realistic) and too simplistic synthetic design (specifically the split between “good” and “bad” hotels, and three key features). The study is a good starting point, providing initial indicators that the manipulations work, but the contribution claims are somewhat inflated, as they seem to try to generalize the observed effect.
[quality] While the effect for these settings has been established, little has been done to investigate why certain things happen (e.g., why are there different patterns in the way humans and AI agents detect manipulation? Why are agents seemingly so much better at detecting manipulation in the auditing phase?).
Questions for authors
- In the appendix when describing the pilot and human study, I see that you asked many questions for which I didn’t find any reported numbers (e.g., the age of participants)
- In A.6.6, you mention that you kept only participants that finished the study procedures and interacted with the recommender. Could you give more details on how the quality of the human data was ensured? Did it include any checks other than (1) participants managing to choose a hotel and (2) participants having some interaction with a recommender?
- Have you done any evaluation of whether there are different patterns between users that changed their choice in the second phase vs. the ones that didn’t?
- Have you done evaluation with any stronger model (in the appendix I found only up to Sonnet-level models, I wonder if the results would be different with more capable models)? Similarly, any evaluations with open-source models?
- Did you check if the reason that the agents perform better in the auditing phase isn’t just because they are able to read more reviews than humans?
5 comments
W2, Q3 &Q5: Thank you for raising this concern and suggesting additional analysis; as these are closely related, we address them together here.
W2: Little has been done to investigate why certain things happen.
Our primary experiment was designed to estimate the effect of the assigned manipulation on downstream decisions. Identifying the mechanisms underlying human–agent differences is a distinct question that would require interventions on candidate mechanisms individually. Such interventions can establish whether a particular factor contributes under the tested conditions, but not provide a unique or complete explanation of the observed behavior. We agree, however, that exploratory correlational analyses can provide insight into possible explanations. We will clearly distinguish these analyses from causal claims.
different patterns between users that changed their choice in the second phase vs. the ones that didn’t
We conducted an exploratory analysis of review-explorer behavior. We measured engagement through distinct hotel filters, keyword searches, rating filters, and sorting actions. These interactions provide a proxy for evidence exploration, although they do not reveal which reviews participants read closely or merely skimmed.
Participants who changed their choice spent more time auditing than those who retained it (median 6.0 vs. 4.1 minutes), while overall explorer engagement was similar (median 6 vs. 5 actions). We further examined whether participants explicitly filtered reviews for their Anchoring-task choice or at least one alternative hotel. The table reports the percentage of participants in each decision-transition group who used each filter. Since some participants filtered both, the columns are not mutually exclusive.
| Group | Filtered Anchoring choice | Filtered an alternative hotel |
|---|---|---|
| Choice transition | ||
| Incorrect → Correct | 50.0% | 50.0% |
| Correct → Correct | 54.2% | 45.8% |
| Correct → Incorrect | 50.0% | 50.0% |
| Incorrect → Incorrect | 53.8% | 23.1% |
| Choice revision | ||
| Changed choice | 42.1% | 47.4% |
| Retained choice | 61.5% | 34.6% |
Participants who revised their choice were more likely to filter an alternative hotel and less likely to filter their Anchoring choice than those who retained it (47.4% vs. 34.6% and 42.1% vs. 61.5%, respectively). Among initially incorrect participants, those who corrected their decision were more likely to filter an alternative than those who remained incorrect (50.0% vs. 23.1%), while filtering of the Anchoring choice was similar (50.0% vs. 53.8%). Thus, the observed difference is associated with broader exploration of alternatives instead of more re-examination of the initial choice.
agents perform better in the auditing phase isn’t just because they are able to read more reviews than humans?
We agree that agents’ ability to access and process more evidence is a plausible contributor to their higher auditing accuracy. Using review-explorer interactions as a proxy, humans performed a median of 5–6 targeted actions, whereas agents made approximately 34–35 explorer-tool calls. We reran the manipulated agent condition with five calls, using the same 64 personas and three rounds each.
| Group | Anchoring | Auditing | Incorrect → correct | Filtered an alternative hotel |
|---|---|---|---|---|
| Humans | 0.34 | 0.44 | 28.6% | 40.9% |
| Original agents | 0.53 | 0.92 | 85.2% | 100% |
| Five-call agents | 0.53 | 0.77 | 59.3% | 79.2% |
The tool call limit substantially reduced agent correction, showing that more evidence exposure contributed to their advantage. However, it did not fully explain the difference: agents with tool call limit inspected alternative hotels more often than humans (79% versus 41%) and were less likely to abandon correct choices. Thus, both evidence access and more systematic evidence search appear to contribute.
I thank the authors for clarifications, answers to each point below:
W1: Thank you for the clarifications on the options. I understand that the framework can theoretically support a more in-depth evaluation – it’s one of the reasons why I think it’s a good starting point. However, I still stand by my opinion that the limited domain and design (not because they are synthetic, but because of their breadth) are not enough to draw general conclusions that the paper makes.
Q1: Thanks for the clarifications!
Q2: Could you please clarify what you mean by procedural adherence (or required procedures)? If I understand correctly, my assumption here was right and no further quality checks are done (e.g., a user could have technically been adversarial or doing random things as long as it managed to reach a decision and had some interaction with the assistant)?
Q4: Thank you for providing additional results for the anchoring phase with Opus 5.5 and an open-source model.
W2 and Q2, Q3: Thank you for providing some additional analyses on what’s going on in terms of patterns. Cool to see some of the experiments have also non-trivial explanations!
W1:
The main disadvantages of the paper are limited domain.Thank you for raising the question of scope. As noted among the strengths that the setup is clean, the synthetic domain is a deliberate design choice: it provides known ground truth and controls for price, location, brand familiarity, and prior experience, which are factors that would otherwise complicate the definition and causal interpretation of decision accuracy. The six hotels and three pilot-informed attributes define one controlled instantiation, not restrictions of the framework. We also want to clarify that the joint intervention models coordinated institutional or application-level steering, where a RAG-system developer or deploying institution can control both the system instructions and the information indexed for retrieval. We will make this threat model explicit in the revision.
Our broader contribution is a multi-stage framework that separates assistant-level attack success, downstream decision impact, correction, and detection. The numerical effects are configuration-specific, while the randomized human study and complementary agent analyses demonstrate why these outcomes must be evaluated separately and provide a protocol for doing so in other settings.
Q1:
In the appendix when describing the pilot and human study, I see that you asked many questions for which I didn’t find any reported numbers (e.g., the age of participants)Thank you for the question. Appendix A.3 reproduces the complete pilot questionnaire for transparency. We report the responses that informed the experimental design, including hotel-selection criteria and information-source usefulness. The optional pilot demographics do not characterize the main experimental sample and were not used in the human-study analysis. We will clarify which questionnaire items informed the design and distinguish them from profile variables used in the agent experiments.
Q2:
In A.6.6, you mention that you kept only participants that finished the study procedures and interacted with the recommender. Could you give more details on how the quality of the human data was ensured? Did it include any checks other than (1) participants managing to choose a hotel and (2) participants having some interaction with a recommender?Thank you for the question. All analyzed participants completed the required procedures and outcome measures, interacted with the assistant at least once, and experienced no session interruption or API failure. Completed records were reviewed for procedural adherence. To keep exclusions objective and outcome-independent, we did not exclude participants based on condition, hotel choice, performance, message content, or a subjective assessment of engagement. The only completed session excluded had zero assistant interactions, meaning that the participant was not exposed to the experimental intervention. We will clarify these checks in Appendix A.6.6.
Q4:
Have you done evaluation with any stronger model (in the appendix I found only up to Sonnet-level models, I wonder if the results would be different with more capable models)? Similarly, any evaluations with open-source models?Thank you for this suggestion. As the reviewer mentioned, our original model-pairing experiment covers up to Sonnet. The original model-pairing results showed substantial variation across assistant–participant pairings, without a consistent ordering across the evaluated model families. To provide more model coverage as requested, we report the Claude Opus 5.5 and Llama 3.1 8B evaluation below using the same setting as Figure 4.
These results suggest Opus is more resistant in both roles. When used as the assistant, Opus instantiated the manipulation less consistently than Llama, which may reflect stronger post-training safeguards or instruction-conflict handling. When used as the participant, Opus showed no accuracy degradation despite high assistant ASR, suggesting greater robustness in evaluating manipulated recommendations.