acceptodds
Under review as a conference paper at ICLR 2027

The System Is Manipulated. Are You? Assessing the Downstream Impact of Manipulated LLM Decision Support

Abstract

LLM alignment is dual-use with implications for advertising, propaganda, and censorship: mechanisms intended to align models with user intent can be repurposed to steer decisions by selectively presenting and reframing evidence. Yet successfully steering an LLM’s recommendations does not necessarily mean successfully steering the downstream decision-maker. This raises a more fundamental question: does LLM decision steering actually work, and can users detect when it is happening? We study such steering in a synthetic RAG decision-support task involving humans and persona-conditioned delegated agents. Our manipulated condition combines poisoned retrieval sources with system-prompt instructions favoring target options. The joint manipulation substantially reduces initial decision accuracy for both humans and delegated agents: by 56.3 percentage points among 64 human participants and by 44.4 percentage points in our main persona-conditioned agent setting. Descriptively, agent runs show more correction among initially incorrect decisions than humans after access to additional evidence (85.2% vs. 28.6%). Detection shows a different pattern: the persona-conditioned agents identify manipulation more often than humans after disclosure (72.8% vs. 46.9%), but also have a substantially higher false-positive rate (59.1% vs. 21.9%). Together, these results show that joint RAG manipulation can bias both human and delegated-agent decisions, with distinct patterns in correction and detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Retrieval-Augmented Generation (RAG), human-AI decision-making, LLM auditing, AI Governance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RpgaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a study that evaluates if humans and AI agents can detect that they are being manipulated by an LLM recommender assistant they are interacting with. Rather than focusing on evaluating if an LLM recommender assistant can be manipulated, this work is the first that studies whether this has any downstream effect on the users (both AI and humans). This is done in a synthetic environment that simulates a platform for hotel booking, with 6 different hotels and reviews assigned to each hotel, focusing on different quality aspects. The evaluation is done in a few settings, with different amounts of visible data (only assistant’s output, +reviews, +note that they might be manipulated). The results indicate that both humans and AI agents are manipulated in a large percentage of cases, and that the patterns are different. Overall, this is an interesting starting point in evaluating the manipulations of LLM recommenders end to end.

Critical strengths

[significance] The paper addresses a real gap in literature, focusing on the efficiency of recommender systems’ manipulations on the actual users, rather than on the recommender LLM only which is extensively covered by previous research.

[quality] The setup is clean, and ablations are extensive and make sense. The authors investigate the effects on both humans and agents in multiple settings, also simulating different types of users. Human study seems to be done carefully.

[clarity] The paper is very easy to follow, there are extensive additional information in the appendix that support the main story

Critical weaknesses

[quality] The main disadvantages of the paper are limited domain (only hotel recommendations) with small number of options (6 hotels), two types of manipulation (explicit prompt which is unrealistic and reviews which are more realistic) and too simplistic synthetic design (specifically the split between “good” and “bad” hotels, and three key features). The study is a good starting point, providing initial indicators that the manipulations work, but the contribution claims are somewhat inflated, as they seem to try to generalize the observed effect.

[quality] While the effect for these settings has been established, little has been done to investigate why certain things happen (e.g., why are there different patterns in the way humans and AI agents detect manipulation? Why are agents seemingly so much better at detecting manipulation in the auditing phase?).

Questions for authors
  1. In the appendix when describing the pilot and human study, I see that you asked many questions for which I didn’t find any reported numbers (e.g., the age of participants)
  2. In A.6.6, you mention that you kept only participants that finished the study procedures and interacted with the recommender. Could you give more details on how the quality of the human data was ensured? Did it include any checks other than (1) participants managing to choose a hotel and (2) participants having some interaction with a recommender?
  3. Have you done any evaluation of whether there are different patterns between users that changed their choice in the second phase vs. the ones that didn’t?
  4. Have you done evaluation with any stronger model (in the appendix I found only up to Sonnet-level models, I wonder if the results would be different with more capable models)? Similarly, any evaluations with open-source models?
  5. Did you check if the reason that the agents perform better in the auditing phase isn’t just because they are able to read more reviews than humans?
5 comments

W1: The main disadvantages of the paper are limited domain.

Thank you for raising the question of scope. As noted among the strengths that the setup is clean, the synthetic domain is a deliberate design choice: it provides known ground truth and controls for price, location, brand familiarity, and prior experience, which are factors that would otherwise complicate the definition and causal interpretation of decision accuracy. The six hotels and three pilot-informed attributes define one controlled instantiation, not restrictions of the framework. We also want to clarify that the joint intervention models coordinated institutional or application-level steering, where a RAG-system developer or deploying institution can control both the system instructions and the information indexed for retrieval. We will make this threat model explicit in the revision.

Our broader contribution is a multi-stage framework that separates assistant-level attack success, downstream decision impact, correction, and detection. The numerical effects are configuration-specific, while the randomized human study and complementary agent analyses demonstrate why these outcomes must be evaluated separately and provide a protocol for doing so in other settings.

Q1: In the appendix when describing the pilot and human study, I see that you asked many questions for which I didn’t find any reported numbers (e.g., the age of participants)

Thank you for the question. Appendix A.3 reproduces the complete pilot questionnaire for transparency. We report the responses that informed the experimental design, including hotel-selection criteria and information-source usefulness. The optional pilot demographics do not characterize the main experimental sample and were not used in the human-study analysis. We will clarify which questionnaire items informed the design and distinguish them from profile variables used in the agent experiments.

Q2: In A.6.6, you mention that you kept only participants that finished the study procedures and interacted with the recommender. Could you give more details on how the quality of the human data was ensured? Did it include any checks other than (1) participants managing to choose a hotel and (2) participants having some interaction with a recommender?

Thank you for the question. All analyzed participants completed the required procedures and outcome measures, interacted with the assistant at least once, and experienced no session interruption or API failure. Completed records were reviewed for procedural adherence. To keep exclusions objective and outcome-independent, we did not exclude participants based on condition, hotel choice, performance, message content, or a subjective assessment of engagement. The only completed session excluded had zero assistant interactions, meaning that the participant was not exposed to the experimental intervention. We will clarify these checks in Appendix A.6.6.

Q4: Have you done evaluation with any stronger model (in the appendix I found only up to Sonnet-level models, I wonder if the results would be different with more capable models)? Similarly, any evaluations with open-source models?

Thank you for this suggestion. As the reviewer mentioned, our original model-pairing experiment covers up to Sonnet. The original model-pairing results showed substantial variation across assistant–participant pairings, without a consistent ordering across the evaluated model families. To provide more model coverage as requested, we report the Claude Opus 5.5 and Llama 3.1 8B evaluation below using the same setting as Figure 4.

ParticipantAssistantNeutral acc.Manipulated acc.Accuracy loss [95% CI]Round ASR
GPT-4o-miniOpus 5.5100%95%5 pp [0, 15]55%
Opus 5.5GPT-4o-mini95%100%−5 pp [−15, 0]95%
Opus 5.5Opus 5.5100%100%0 pp [0, 0]45%
GPT-4o-miniLlama 3.1 8B100%95%5 pp [0, 15]85%
Llama 3.1 8BGPT-4o-mini95%80%15 pp [−5, 35]100%
Llama 3.1 8BLlama 3.1 8B90%75%15 pp [−5, 35]100%

These results suggest Opus is more resistant in both roles. When used as the assistant, Opus instantiated the manipulation less consistently than Llama, which may reflect stronger post-training safeguards or instruction-conflict handling. When used as the participant, Opus showed no accuracy degradation despite high assistant ASR, suggesting greater robustness in evaluating manipulated recommendations.

W2, Q3 &Q5: Thank you for raising this concern and suggesting additional analysis; as these are closely related, we address them together here.

W2: Little has been done to investigate why certain things happen.

Our primary experiment was designed to estimate the effect of the assigned manipulation on downstream decisions. Identifying the mechanisms underlying human–agent differences is a distinct question that would require interventions on candidate mechanisms individually. Such interventions can establish whether a particular factor contributes under the tested conditions, but not provide a unique or complete explanation of the observed behavior. We agree, however, that exploratory correlational analyses can provide insight into possible explanations. We will clearly distinguish these analyses from causal claims.

different patterns between users that changed their choice in the second phase vs. the ones that didn’t

We conducted an exploratory analysis of review-explorer behavior. We measured engagement through distinct hotel filters, keyword searches, rating filters, and sorting actions. These interactions provide a proxy for evidence exploration, although they do not reveal which reviews participants read closely or merely skimmed.

Participants who changed their choice spent more time auditing than those who retained it (median 6.0 vs. 4.1 minutes), while overall explorer engagement was similar (median 6 vs. 5 actions). We further examined whether participants explicitly filtered reviews for their Anchoring-task choice or at least one alternative hotel. The table reports the percentage of participants in each decision-transition group who used each filter. Since some participants filtered both, the columns are not mutually exclusive.

GroupFiltered Anchoring choiceFiltered an alternative hotel
Choice transition
Incorrect → Correct50.0%50.0%
Correct → Correct54.2%45.8%
Correct → Incorrect50.0%50.0%
Incorrect → Incorrect53.8%23.1%
Choice revision
Changed choice42.1%47.4%
Retained choice61.5%34.6%

Participants who revised their choice were more likely to filter an alternative hotel and less likely to filter their Anchoring choice than those who retained it (47.4% vs. 34.6% and 42.1% vs. 61.5%, respectively). Among initially incorrect participants, those who corrected their decision were more likely to filter an alternative than those who remained incorrect (50.0% vs. 23.1%), while filtering of the Anchoring choice was similar (50.0% vs. 53.8%). Thus, the observed difference is associated with broader exploration of alternatives instead of more re-examination of the initial choice.

agents perform better in the auditing phase isn’t just because they are able to read more reviews than humans?

We agree that agents’ ability to access and process more evidence is a plausible contributor to their higher auditing accuracy. Using review-explorer interactions as a proxy, humans performed a median of 5–6 targeted actions, whereas agents made approximately 34–35 explorer-tool calls. We reran the manipulated agent condition with five calls, using the same 64 personas and three rounds each.

GroupAnchoringAuditingIncorrect → correctFiltered an alternative hotel
Humans0.340.4428.6%40.9%
Original agents0.530.9285.2%100%
Five-call agents0.530.7759.3%79.2%

The tool call limit substantially reduced agent correction, showing that more evidence exposure contributed to their advantage. However, it did not fully explain the difference: agents with tool call limit inspected alternative hotels more often than humans (79% versus 41%) and were less likely to abandon correct choices. Thus, both evidence access and more systematic evidence search appear to contribute.

I thank the authors for clarifications, answers to each point below:

W1: Thank you for the clarifications on the options. I understand that the framework can theoretically support a more in-depth evaluation – it’s one of the reasons why I think it’s a good starting point. However, I still stand by my opinion that the limited domain and design (not because they are synthetic, but because of their breadth) are not enough to draw general conclusions that the paper makes.

Q1: Thanks for the clarifications!

Q2: Could you please clarify what you mean by procedural adherence (or required procedures)? If I understand correctly, my assumption here was right and no further quality checks are done (e.g., a user could have technically been adversarial or doing random things as long as it managed to reach a decision and had some interaction with the assistant)?

Q4: Thank you for providing additional results for the anchoring phase with Opus 5.5 and an open-source model.

W2 and Q2, Q3: Thank you for providing some additional analyses on what’s going on in terms of patterns. Cool to see some of the experiments have also non-trivial explanations!

Reviewer_SHZPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether steering an LLM decision-support system also steers the downstream decision-maker, and whether the decision-maker notices. In a synthetic hotel-recommendation task with a GPT-4o-mini RAG assistant, the manipulated condition combines a system prompt that always recommends one of a hidden target set {D,E,F}\{\mathrm{D},\mathrm{E},\mathrm{F}\} with 20 injected, positively reframed reviews per target hotel. A between-subjects human study (N=64N=64) measures accuracy before evidence access, after access to a review explorer, and post-disclosure identification. Manipulation cuts initial accuracy from 91% to 34% (56.3 pp), accuracy recovers only to 44% with evidence, and 46.9% of manipulated participants identify the manipulation against a 21.9% false-positive rate. A secondary study runs 64 persona-conditioned GPT-4o-mini agents (personas built from the matched humans' profiles, logs and debriefs): agents lose 44.4 pp initially, recover far more often (85.2% vs. 28.6%), and label 72.8% of manipulated and 59.1% of neutral sessions as manipulated.

Critical strengths

The central question is well-chosen and timely: the security literature on RAG poisoning and prompt injection overwhelmingly reports model-level attack success, while this paper measures the quantity that actually matters for harm, namely the downstream human (and agent) decision, and shows that high automated attack-success rates can coexist with near-zero decision impact.

The writing is careful and clear for a human-subjects study at an ML venue. Randomization, attrition, exclusions, compensation, IRB approval, and partial-disclosure debriefing are documented; all prompts (assistant, bias injection, review generation, persona, auditor) are given verbatim in the appendix.

Finally, the ablations in the appendix are more informative than the main results and clearly show self-criticism and limitations of the study.

Critical weaknesses
  1. The headline effects are measured under a single, hand-selected configuration, and the paper’s own ablations show the effect is not stable. The human study uses one non-frontier assistant (GPT-4o-mini) and one manipulation prompt chosen “through preliminary author-led testing of whether the assistant recommended a target hotel” (Sec. 3.3). Appendix A.1.3 then shows that with a revised prompt the GPT-4o-mini self-pair yields only a 0.05 accuracy loss (CI [0.00, 0.15]) versus 0.48 in the main matched-agent analysis, that 13 of 16 model pairings have accuracy-difference CIs that include zero, and that agent susceptibility ranges from 5 pp to 87.5 pp depending on persona design. Moreover, the channel ablation (Table 4) attributes 0.38 of the loss to the system prompt and only 0.06 to corpus poisoning, with the poisoning effect explicitly limited by the fixed-k-per-hotel retrieval design. Taken together, the paper’s framing as a study of “poisoned retrieval sources” and “RAG-system manipulation” overstates what was shown: the robust finding is that an assistant whose operator instructs it to “always recommend one of {target}” steers users of one specific model, which is a much narrower and less surprising claim. The title, abstract and introduction should be re-scoped accordingly, or the human study should be replicated with at least one additional model/prompt so the reported 56.3 pp has some external validity.

  2. The evidence base is thin for an empirical paper whose contribution is entirely empirical, and reproducibility is incomplete. N = 64 is small, with a mixed convenience/Prolific sample in which the 20 pilot participants recruited from the authors’ personal networks are retained in the analyzed sample and also supplied both the power calculation (1.00 vs 0.50 accuracy) and the survey that defined the correctness rubric and evidence-exploration stage (A.3, Sec. 4); the claim that the “protocol … remained unchanged” after the pilot is hard to reconcile with the statement that the pilot survey “informed the task design, including the evaluation rubric.” The study was not preregistered, all secondary outcomes are unadjusted, and a single author coded in-session suspicion.

Questions for authors

The main manipulation prompt was selected for assistant-side attack success. Was any prompt that failed author pretesting discarded, and how many candidates were tried? Can you report human or matched-agent results under the revised prompt of A.7.3 so readers can see whether the 56.3 pp human effect survives a prompt that is more honest but still targeted?

6 comments

W1: The headline effects are measured under a single, hand-selected configuration.

Thank you for the comment. We agree that the human study evaluates a single assistant configuration and will clarify that the 56.3-point estimate applies to this implementation. However, the model-pairing analysis was not designed as a robustness test in which every configuration should reproduce the 56.3-point human effect. It tests a central motivation of the paper: whether successful manipulation of an assistant’s recommendations reliably translates into manipulation of the downstream decision-maker. Despite high assistant-level attack success across pairings (0.55–1.00), downstream accuracy differences range from −0.05 to 0.90. This heterogeneity motivates our end-to-end framework for evaluating how manipulation affects downstream decision-makers.

The agent ablation finds a larger prompt-level effect under our specific implementation. As an illustrative sensitivity check, we replace 5 of the 20 reframed poisoned reviews per target with positive-only reviews. Under this stronger corpus intervention, accuracy decreases by 57.2 percentage points.

ConditionAccuracy95% persona-bootstrap CI
Neutral0.978[0.962, 0.991]
Poison-only (20 strategic reframed reviews)0.925[0.881, 0.962]
Prompt-only0.600[0.528, 0.672]
Poison-only (15 strategic reframed + 5 positive-only reviews)0.406[0.353, 0.462]

Our goal is to demonstrate that joint manipulation can produce downstream consequences in one operationally realistic setting, instead of comprehensively benchmark poisoning strategies and retrieval implementations. Such systematic variation is beyond the current scope. Nevertheless, a central contribution of our work is the controlled framework and experimental environment that enables such future analysis.

W2: The evidence base is thin for an empirical paper whose contribution is entirely empirical.

Thank you for the comment. We respectfully disagree that the sample is insufficient for the primary randomized comparison. The pilot-based power analysis required 10 participants per condition, whereas we recruited 32 per condition. The estimated accuracy difference is 56.3 percentage points (95% CI [33, 72]), whose lower bound remains substantively large. Excluding the pilot participants yields a similar 59.1-point difference, showing that the result is not driven by the pilot. A larger sample would improve precision but is not necessary to support our primary claim.

We will clarify the chronology: the pre-study survey informed construction of the task, rubric, and evidence-exploration stage before the experimental protocol was finalized. All experimental sessions subsequently used the same procedure, allocation, eligibility criteria, and outcome definitions.

We appreciate the reviewer’s recognition that the submission documents the human-study procedures, randomization, exclusions, ethics safeguards, complete prompts, and statistical analyses. We believe these details strongly support procedural replication. The study was not preregistered, but the primary outcome, fixed recruitment target, exclusion rule, allocation procedure, and analysis were not changed during continued recruitment. Secondary outcomes are explicitly exploratory and do not support multiplicity-adjusted confirmatory claims. Single-author coding applies only to exploratory in-session suspicion, not primary accuracy or participants’ direct post-disclosure identification; we will make this distinction more prominent.

Q1:The main manipulation prompt was selected for assistant-side attack success. Was any prompt that failed author pretesting discarded, and how many candidates were tried? Can you report human or matched-agent results under the revised prompt of A.7.3 so readers can see whether the 56.3 pp human effect survives a prompt that is more honest but still targeted?

Thank you for the question. Before the human study, we tested four prompt versions and selected one based only on whether the assistant recommended the target hotels, the other three were discarded. No human outcomes were observed or used during this selection.

We agree that not every targeted prompt or system implementation will produce the same downstream human effect, this is also not our claim. The randomized human study establishes that one operationally plausible, jointly manipulated decision-support system can produce a substantial downstream effect. The 56.3-point estimate is internally valid for that intervention, but its magnitude should not be generalized across prompts or assistants.

We did not rerun the human study with the revised A.7.3 prompt and did not claim that the human effect would persist. The existing revised-prompt GPT-4o-mini self-pair provides an agent sensitivity result: despite 0.95 round-level assistant attack success, the downstream accuracy loss is only 0.05 (95% CI [0.00,0.15][0.00,0.15]) across 20 paired personas. We are not testing whether the effect “survives”. Instead, it demonstrates a central motivation of our work: successfully steering an assistant’s recommendations is not sufficient to steer downstream decisions.

Our broader contribution is a multi-stage framework that separates assistant-level attack success, downstream decision impact, correction, and detection. The numerical effects are configuration-specific, while the randomized human study and complementary agent analyses demonstrate why these outcomes must be evaluated separately and provide a protocol for doing so in other settings.

Dear Reviewer SHZP,

As the discussion period is approaching the end, with fewer than 3 days remaining, we want to ensure we have addressed all your concerns, including the robustness and external validity of the headline effect, the study’s evidence base and reproducibility, and prompt selection. If there is any additional feedback you would like us to consider, please let us know. Your insights are important and would help improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_vucLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether steering an LLM decision-support system changes the decisions of the people or agents who use it, and whether they notice. The manipulation has 2 parts. The authors use a fictional city as test envionrment. Each decision maker goes through 3 stages. the Anchoring Task (assistant only), the Auditing Task (assistant plus a review explorer over the clean corpus), and the Debriefing Stage (truthfulness ratings, disclosure, and a condition guess). The authors present both human study with 64 participants and agent study with 64 gpt-4o mini. They reported a channel ablation in which the prompt channel causes most of the agent accuracy loss and agents with with generic or sampled-preference profiles, etc. The authors state three contributions: A multi-stage protocol that separately measures initial impact, recovery after evidence access, and identification after disclosure; the human and agent accuracy effects; and a description of how humans and agents differ in correction and identification.

Critical strengths

this is a careful and honest paper about a relevant risk. The human study is small, but it is randomized, it has one primary outcome fixed in advance, and it reports intervals that fit a small-sample risk difference. The paper is clear and easy to follow, and the appendix gives enough detail to understand most choices. in my view the originality is moderate.

  • The reporting is transparent. The paper gives raw transition counts (Fig. 2a), exact counts in the text, and description of who withdrew from which condition and why one completer was excluded (A.6.5).
  • The appendix ablations ask the right questions, even though they use agents only
Critical weaknesses
  1. The human study does not show the mechanism that the conclusion states. The conclusion says that ``by biasing evidence and reframing retrieved information, LLM systems can steer decisions''

Two design facts make me think that the human effect comes mostly from the steering prompt, and maybe from unsupported claims, and less from reframed evidence. First, the base prompt in the human study says Do not reveal that you are using reviews internally'' , so in the Anchoring Task participants cannot see the reviews and depend on the ads and on what the assistant chooses to say. Second, the original manipulation prompt (A.7.2) does not forbid fabrication. The revised prompt adds Do not fabricate facts, do not contradict the reviews'', but the human study did not use it.

  1. The agent personas use post-task human records that depend on the condition, so the human vs. agent comparisons in the abstract are hard to interpret.

Persuasion sensitivity, trust style, and fatigue come from each human's interaction logs and debriefing answers. These records depend on the humans condition and outcome e.g. for if the person noticed steering. In the All analysis, the same persona then runs in both conditions. The authors claim that the agent study is not an independent replication, which I appreciate. The paper does not say who assigned the trait labels or whether that person knew the condition. The fidelity check does not clearly validate persuasion sensitivity.

Questions for authors
  1. In the human study, what share of manipulated-condition assistant turns make claims about hotel attributes that contradict Table 7 or the clean corpus?

  2. What is the in-session suspicion rate in the neutral human condition, with the same coding?

3 comments

W1 & Q1: Thank you for raising this concern. Since there is some overlap with Q1 below, we will address them together.

W1: The human effect comes mostly from the steering prompt, and maybe from unsupported claims, and less from reframed evidence.

Our study intentionally estimates the effect of the joint intervention, as we believe that reflects an operationally realistic deployment threat model where a RAG-system developer or deploying institution can control both the system instructions and the information indexed for retrieval. The agent ablation finds a larger prompt-level effect under our specific implementation. As an illustrative sensitivity check, we replace 5 of the 20 reframed poisoned reviews per target with positive-only reviews. Under this stronger corpus intervention, accuracy decreases by 57.2 percentage points.

ConditionAccuracy95% persona-bootstrap CI
Neutral0.978[0.962, 0.991]
Poison-only (20 strategic reframed reviews)0.925[0.881, 0.962]
Prompt-only0.600[0.528, 0.672]
Poison-only (15 strategic reframed + 5 positive-only reviews)0.406[0.353, 0.462]

Our goal is to demonstrate that joint manipulation can produce downstream consequences in one operationally realistic setting, instead of comprehensively benchmark poisoning strategies and retrieval implementations. Such systematic variation is beyond the current scope. Nevertheless, a central contribution of our work is the controlled framework and experimental environment that enables such future analysis.

base prompt and revised prompt difference

We want to clarify that the instruction not to reveal the internal review used is identical in both experimental conditions and therefore cannot explain the between-condition difference. Moreover, during the Auditing Task, participants gain access to the clean review corpus while the manipulated assistant continues retrieving from the poisoned corpus, the condition difference remains substantial at that stage. This shows that the joint effect is not confined to the no-review-access setting.

Q1: In the human study, what share of manipulated-condition assistant turns make claims about hotel attributes that contradict Table 7 or the clean corpus?

We used GPT-5.4-mini to read the downstream properties of the hotel attributes and the assistant's responses, then manually verify the. Explicit contradictions occurred in 1.47% of manipulated-condition turns and 2.05% of neutral-condition turns. Therefore, we believe that the effect arises from the joint intervention, particularly targeted recommendation, selective evidence exposure, and strategic reframing, rather than from excess fabrication in the manipulated condition.

W2: The agent personas use post-task human records that depend on the condition, so the human vs. agent comparisons in the abstract are hard to interpret.

Thank you for raising the concern. Persona labels were generated by deterministic rules, and we agree that the human-derived persona analysis is descriptive rather than an independent replication. However, we also report two ablations that use no post-task human records: generic agents receive only task instructions, and sampled-preference agents receive randomly selected hotel preferences in Appendix A.1.2 and Table 5. Both retain large Anchoring effects under manipulation (accuracy: 1.00 to 0.13 and 0.98 to 0.28, respectively), showing that the initial agent effect does not depend on condition-informed personas.

Q2: What is the in-session suspicion rate in the neutral human condition, with the same coding?

Thank you for the question. Using the same coding, we identify suspicion during the Anchoring Task in 2/32 (6.3%) neutral-condition participants versus 6/32 (18.8%) manipulated-condition participants. By the end of the Auditing Task, the corresponding rates were 4/32 (12.5%) and 11/32 (34.4%). After disclosure, 7/32 (21.9%) neutral-condition participants versus 15/32 (46.9%) manipulated-condition participants identified their system as manipulated. We will add these condition-by-stage comparisons to the updated manuscript.

Dear Reviewer vucL,

As the discussion period is approaching the end, with fewer than 3 days remaining, we want to ensure we have addressed all your concerns, including the human-study mechanism, the agent-persona construction, and the two quantitative questions. If there is any additional feedback you would like us to consider, please let us know. Your insights are important and would help improve our work.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer vucL,

As the discussion period is approaching the end, with less than 1 day remaining, we want to ensure we have addressed all your concerns, including the human-study mechanism, the agent-persona construction, and the two quantitative questions. If there is any additional feedback you would like us to consider, please let us know. Your insights are important and would help improve our work.

Thank you for your time and effort in reviewing our paper.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful engagement and constructive feedback and the chairs for guidance and oversight of our submission. We are glad that the reviewers find our work addresses a timely and underexplored gap, the study is carefully conducted, and the ablations are extensive and informative.

We summarize the discussion below.

Scope

  • Configuration-specific effects and framework generalizability (SHZP, Rpga). We clarified the framework's generalizability to other domains while acknowledging that responsible extensions require domain-specific threat modeling, ethics reviews, and additional human and agent studies beyond the current scope of the paper. We are glad that Reviewer SHZP considered the concern addressed, and Reviewer Rpga recognized the framework's potential to support more in-depth evaluation.

Additional Analysis

  • Stronger corpus intervention (vucL, SHZP). We showed that evidence manipulation can become the dominant channel under a stronger intervention. We are glad that Reviewer SHZP considered the technical concerns addressed.
  • Human–agent behavioral differences (Rpga). We conduct additional experiments and showed that evidence access and systematic search together explain agents’ stronger recovery. We are glad that Reviewer Rpga found these findings non-trivial.
  • Broader model coverage (Rpga). We added Opus 5.5 and Llama 3.1 8B evaluations. We appreciate Reviewer Rpga's acknowledgment of the additional results.

Clarifications

  • Manipulation mechanism (vucL). We clarified that the instruction was identical across conditions and that contradictions were rare, indicating the effect was not driven by fabrication.
  • Agent-persona construction (vucL). We clarified how persona labels were generated and pointed to the persona-free ablations reported in the original manuscript, which show that large manipulation effects retain without post-task human information.
  • Prompt selection, sample size, and pilot methodology (SHZP). We disclosed the prompt-selection process, justified statistical power, and clarified the pilot chronology and exploratory analyses. We are glad that Reviewer SHZP considered these concerns resolved.
  • Participant quality and reporting (Rpga, vucL). We clarified objective exclusion criteria and reported condition-by-stage suspicion rates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.