- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to make LLMs adopt attacker-controlled content by changing its presentation while keeping the claim fixed. It recovers model-specific presentation preferences from SAE-based acceptance directions and claim-level attention, and uses them to initialize and optimize attack texts for evidence poisoning and recommendation manipulation.
Critical strengths
- Using internal signals to analyze presentation effects on evidence adoption is an interesting angle, supported by causal interventions.
- The experiments cover several models and two attack settings.
Critical weaknesses
- The threat model is not well supported for current systems. The paper's own results show that frontier models are rarely swayed by a single conflicting source by default, while the main experiments rely on smaller, older open models and a bare RAG prompt without any safeguards.
- The core claim that internal signals are better than output-level signals is not supported:
- The premise that outcome-level signals are coarse and noisy is stated without evidence.
- The only output-based baseline is the model's self-rating of styles; ranking styles by their measured ASR on a small development set is not compared.
- The optimizer is adopted from ContextBench (LLM-assisted EPO), but the same optimizer with an output-level objective is not compared. The ContextBench baseline is also run with less frequent LLM assistance.
- The attack gains cannot be clearly attributed to presentation preferences:
- The Acceptance Vectors are confounded. For Trust, positives are cases where the model accepts the supplied claim and negatives are cases where it rejects it; these come from different claims, and since models tend to accept true claims and reject false ones, the direction may largely capture claim truth. For Compliance/Sycophancy, negatives include inputs with the pressure text removed. As the representation is taken before generation, these directions can only encode input differences rather than acceptance itself.
- The optimization constrains only the claim span. The final texts add fabricated supporting evidence and direct instructions to the model (e.g., "immediately acknowledge"), contradicting the paper's own rule of rejecting instruction-like rewrites, and the initial style is not preserved. The gains therefore mix content and prompt-injection effects.
- The transfer to closed-source models likely relies on the same added content and instructions.
- Many design choices are not justified or validated:
- The layers for Acceptance Vectors and Information Attention are chosen without explanation and differ from each other.
- The four acceptance facets and the 120 styles are hand-defined, and how the styled samples and behavior labels are produced is not described.
- The held-out evaluation uses a split of the same behavioral corpus; there is no out-of-distribution evaluation.
- The method requires a public SAE for each model and per-model offline profiling, whose cost is not reported.
- The evaluation setting may overstate attack difficulty. The "nine sources supporting the correct answer" include unrelated retrieved passages (e.g., the example in Appendix D.2), the questions and false target answers are generated by an LLM without quality checks, the recommendation task has only three categories of ten synthetic products, and the black-box evaluation uses only 195 examples.
- No defense or detection is evaluated, although many optimized texts contain obvious manipulative cues.
Questions for authors
see above
7 comments
Response to Reviewer HmK2 (2/5)
W/Q 3.1
The Acceptance Vectors are confounded.
Thank you for raising concerns about the construction of the Acceptance Vectors and the interpretation of representations extracted before generation.
Trust is supervised by claim adoption, with factual correctness balanced across positive and negative samples. Both the acceptance and rejection groups contain factually correct and incorrect claims in matched proportions. Positive and negative labels reflect whether the model adopts the supplied claim. This construction targets acceptance versus rejection while controlling for factual correctness.
Compliance and Sycophancy characterize acceptance under pressure, rather than general claim acceptance. Positive examples are inputs in which the model adopts the target stance under explicit compliance pressure or sycophancy-related user cues. The negative pool combines two complementary controls:
- Pressure-present non-adoption examples: the pressure cues remain, but the model does not adopt the target stance.
- Pressure-removed non-adoption examples: the pressure cues are removed while retaining the same claim and question, and the model does not adopt the target stance. The two types of negatives provide complementary contrasts. Pressure-present negatives contrast resisting pressure with yielding to it, while pressure-removed negatives contrast maintaining the original stance without pressure with adopting the target stance under pressure. We combine these negatives using a 2:1 sampling design to balance the two contrasts. Together, they support extracting representations associated with acceptance under compliance or sycophancy pressure.
Pre-generation representations can encode behaviorally relevant states and causally influence subsequent responses. Extracting representations before generation does not restrict them to surface-level input differences. Representation Engineering explicitly uses representations immediately preceding a model’s prediction to read out high-level concepts [1]. More directly, refusal directions extracted from post-instruction tokens, before answer generation, can induce or suppress subsequent refusal through activation interventions [2].
Our experiments provide corresponding evidence for claim adoption: with the input text held fixed, interventions on selected acceptance features and claim-directed attention shift the token-level decision margin in the expected directions (Figure 4; Appendix D.5). Forward and reverse interventions, together with random controls, support the causal relevance of these internal states to subsequent decisions.
[1] Zou A, Phan L, Chen S, et al. Representation engineering: A top-down approach to ai transparency[J]. arXiv preprint arXiv:2310.01405, 2023.
[2] Arditi A, Obeso O, Syed A, et al. Refusal in language models is mediated by a single direction[J]. Advances in Neural Information Processing Systems, 2024, 37: 136037-136083.
Response to Reviewer HmK2 (3/5)
W/Q3.2 - 3.3
The optimization constrains only the claim span. The transfer to closed-source models likely relies on the same added content and instructions.
Thank you for raising the concern that additional supporting content and direct instructions could contribute to the reported gains. We address this by distinguishing the experiments validating presentation preferences from subsequent attack optimization, clarifying the shared baseline settings, and quantifying target-adoption directives in the optimized texts.
Our experiments support the proposed mechanism linking presentation changes to model acceptance : Our framework modifies text in two stages:
- Preference-guided template rewriting. We instantiate high-preference presentation templates by inserting the target claim into the designated placeholder. The template and its generic surrounding text are predefined independently of individual claims. This stage uses rule-based substitution without generating claim-specific supporting content.
- LLM-assisted gradient optimization. We further optimize the template-instantiated text while preserving the target claim verbatim. This stage can introduce claim-specific background, explanations, or supporting assertions, such as the scarcity statement in M2.
The preference-validation experiments precede LLM-assisted optimization. Figures 3–4 and Table 18 all use first-stage, template-instantiated texts. Figure 3 establishes score–ASR correlations of , and Table 18 confirms the behavioral effectiveness of preference rankings. Figure 4 establishes that interventions on selected acceptance features and claim-directed attention causally affect the token-level decision margin under matched template-instantiated inputs. These results demonstrate that presentation preferences improve claim adoption, without confounding effects from claim-specific supporting evidence.
Our content-editing scope and assistant-LLM budget follow established baseline settings. Existing GEO and poisoning methods permit additional supporting content. AutoGEO [1]’s rules encourage source citations, statistics, examples, and explanatory context, while PoisonedRAG [2] constructs misleading passages to induce attacker-chosen answers. These precedents support allowing supporting-content additions during attack optimization.
In our evaluation, methods share the same controlled source and competing context. Table 13 details their resource budgets, our method and ContextBench both use three assistant-LLM rewrite calls per attack text. Within this shared content-editing setting, our two-stage framework uses internal signals to profile, select, and optimize model-preferred presentations, achieving 8.67–31.87 percentage-point ASR improvements over the strongest baseline across the evaluated poisoning settings (Table 1).
Our method does not rely on command-based templates and outperforms the explicit-command baseline. Coercive Command is included among the 120 presentation strategies as a diagnostic dimension for studying how models respond to instructional content. It is excluded from template selection for our main attacks. On FEVER, the selected high-preference templates achieve higher ASR (without gradient optimization) than Coercive Command:
| Model | Selected template | Template ASR | Coercive Command ASR |
|---|---|---|---|
| Llama-3.1-8B-Instruct | Administrative Document Style | 46.33% | 33.33% |
| Qwen3-8B | Emphatic Markup | 59.00% | 42.67% |
Response to Reviewer HmK2 (1/5)
W/Q1
Thank you for raising this concern. Our evaluation follows established community settings and demonstrates effectiveness across diverse models, datasets, and frontier black-box targets. We adopt the controlled-source poisoning setup used by existing baselines (Appendix D.1), profile seven models across the Llama, Gemma, and Qwen families, and evaluate attacks on FEVER, SQuAD, and three recommendation categories. Table 3 further demonstrates effective transfer to three frontier black-box models.
W/Q2
Thank you for recommending comparisons with observed adoption outcomes in addition to model-generated ratings. We provide complementary evidence on preference prediction, style selection, and subsequent text optimization, including the additional Output ASR baseline.
Internal signals better characterize presentation preferences and identify styles that increase claim adoption. Table 4(a) shows that internal Preference Scores correlate more strongly with downstream ASR than model-generated output ratings: r=0.687–0.792, compared with 0.175–0.509.
We additionally introduce an Output ASR baseline that ranks the 120 presentation styles by their observed rejection-to-adoption rates on the same behavioral dataset used to construct our Acceptance Vectors. This complements the Output ratings baseline with a ranking derived from actual model behavior.
Using exactly the downstream evaluation data of Table 18, we compare the effectiveness of styles selected by these three rankings. For each ranking, we divide the 120 styles into consecutive groups of 20. Each value below is the group’s mean ASR divided by the model’s mean ASR across all 120 styles.
Internal-guided optimization achieves higher ASR and larger improvements over template initialization under the same optimization budget. We compare the effectiveness of attack-text optimization guided by internal versus output-based objectives on Qwen3-8B using FEVER. Each of the three methods first selects its top-40 presentation styles using its own signal—internal Preference Scores, output ratings, or observed ASR—and uses these styles to initialize attack texts.
Our method then runs EPO with an optimization objective combining the internal Preference Score and a PPL regularizer. For both output-based methods, we replace the internal Preference Score with the model’s output-rating score while retaining the PPL regularizer, and replace gradient-based candidate proposals with prompt-based proposals.
We evaluate all three methods under the same optimization budget (iterations, candidates per iteration and assistant-LLM call frequency), both with and without LLM assistance.
Under the same optimization budget, our method achieves higher ASR and larger gains over template initialization, with and without LLM assistance. These results support the advantage of our proposed internal objective over output-based objectives for guiding attack-text optimization.