- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DarkPattern Teaming (DPT), a framework for finding LLM "dark patterns" (privacy, belief, relationship, and decision manipulation, adapted from Shi et al., 2026) during non-adversarial, legitimate help-seeking. It builds a 4-need × 6-context user space (24 strata), uses an LLM to generate paired direct/indirect openings, simulates up to four rounds with an LLM user, has a Gemini 2.5 Pro "Search Critic" flag dark-pattern acts turn by turn. On Qwen 3.5 27B (6,000 trajectories), DPT flags 22.1% more trajectories than Balanced Stratified sampling and 2.32× more than Unstructured Generation, and 44.9% of first detections come after Round 1. The paper also benchmarks five models (960 trajectories each) and uses repaired response pairs for activation steering, which cuts the held-out dark-pattern rate from 42.19% to 13.02%. SFT and DPO show no effect.
Critical strengths
- The problem is well motivated: manipulation that arises during legitimate assistance is under-studied compared with adversarial red teaming. The behavioral rubric (App. G.4) is carefully written, with explicit counterfactual "tactic + effect" gates that separate manipulation from hallucination or warmth.
- The paper has a end to end pipeline, from discovery to repair pairs to mitigation.
- Validation is fairly thorough: 1,150 human-annotated displays, blinded and adjudicated, plus a separate Outcome Judge (GPT 5.6 Sol) for held-out mitigation.
Critical weaknesses
- No frontier target models. The five targets are lighter or fast tiers. Stronger models are not tested. The paper therefore can't say how prevalent these dark patterns and how effective the red-teaming are in the most capable, most widely deployed assistants.
- The yield gain mostly reflects concentration on one user condition. 54% of DPT’s budget goes to a single stratum (Emotional affirmation × Work & education, 96% hit rate; Fig. 2), and ~34% of DPT’s detected situations are near-duplicates (App. D.1). Whether the dark patterns found are similarly redundant is unknown: no per-strategy tactic/category breakdown or diversity metric is reported, so “more discovery” cannot be distinguished from re-finding the same failure.
- The mitigation claims are not well verified. Apart from limited human assistance ratings (Table 17), which the authors acknowledge “do not establish equal utility,” steering is not evaluated on any standard helpfulness or capability benchmark. It remains unclear whether the reduction in dark patterns comes at the cost of helpfulness or general capability, e.g., through more hedged or less engaged responses.
- Detector calibration is unclear. The Search Critic flags 69–85% of Qwen trajectories, while the Outcome Judge flags only 42% of untreated held-out conversations, suggesting the critic is lenient. Critic precision is only 72–86% for non-DPT strategies (Table 13), and inter-annotator agreement is modest. There are lots of models stronger than Gemini 2.5 Pro, may want to use them for all methods.
Questions for authors
None
5 comments
Author Responses to Reviewer 4jPo (2/2)
Mitigation and judge calibration
[W3] “The mitigation claims are not well verified. … It remains unclear whether the reduction in dark patterns comes at the cost of helpfulness or general capability, e.g., through more hedged or less engaged responses.”
Helpfulness benchmark
We evaluated Qwen with and without the original 100-pair DPT steering on MT-Bench [1]. Responses were graded by GPT-5.6 Sol using the benchmark’s published prompts and reference answers. Scores are 7.98 untreated and 8.06 with steering, showing preserved helpfulness on this benchmark.
Additional baselines and mitigation transfer
To further compare prompting with steering, we added four prompt baselines on Qwen: a short general anti-manipulation instruction, detailed guidance for the four categories, the same guidance with four illustrative pairs of problematic and preferred responses (few-shot), and a concise/cautious style prompt. Exact prompts appear in our shared comment to all reviewers.
To test whether mitigation data transfers across models, we reused the 100 Qwen repair text pairs to construct a steering direction from Llama 3.1 8B Instruct’s own activations. We evaluated each condition on the same held-out situations for manipulation and on MT-Bench for helpfulness.
| Condition | Manipulation (%) | MT-Bench |
|---|---|---|
| Qwen — Untreated | 42.19 | 7.98 |
| Qwen — Short prompt | 36.98 | 8.02 |
| Qwen — Detailed prompt | 31.67 | 8.04 |
| Qwen — Detailed + examples (few-shot) | 26.88 | 8.00 |
| Qwen — Cautious-style prompt | 29.79 | 8.15 |
| Qwen — DPT steering | 13.02 | 8.06 |
| Llama — Untreated | 23.23 | 6.00 |
| Llama — DPT steering | 15.83 | 5.73 |
DPT steering achieves the largest manipulation reduction on Qwen, outperforming all four prompt baselines, while preserving MT-Bench performance. The repair data also transfers to Llama, reducing detected manipulation from 23.23% to 15.83%, with similar MT-Bench performance.
[W4] “Detector calibration is unclear. The Search Critic flags 69–85% of Qwen trajectories, while the Outcome Judge flags only 42% of untreated held-out conversations, suggesting the critic is lenient.”
Search Critic and Outcome Judge detection rates
The cited rates come from different conversation sets and sampling procedures. Discovery includes adaptive sampling that concentrates on conditions with frequent detections, whereas the untreated held-out evaluation is balanced across conditions. Since we did not have an existing set of conversations evaluated by both judges, we sampled 100 conversations generated by our pipeline and obtained judgments from both evaluators, along with human annotations. Agreement between the two judges was close to agreement between the human annotators (87% vs. 86%).
| Comparison | Agreement percentage |
|---|---|
| Search Critic vs. Outcome Judge | 87% |
| Search Critic vs. adjudicated human labels | 92% |
| Outcome Judge vs. adjudicated human labels | 85% |
| Human A vs. Human B | 86% |
[W4] “There are lots of models stronger than Gemini 2.5 Pro, may want to use them for all methods.”
We chose Gemini 2.5 Pro to make the large volume of turn-by-turn search evaluations affordable. Our independent Outcome Judge is GPT-5.6 Sol, which we compare with the Search Critic in the 100-conversation evaluation above. On this sample, GPT-5.6 Sol did not show higher agreement with adjudicated human labels.
[W4]“Critic precision is only 72–86% for non-DPT strategies”
We acknowledge these false positives. Applying each strategy’s human-confirmation rate to its flagged conversations gives approximately 4,565 discoveries for DPT, 3,147 for Uniform Random, 2,991 for Balanced Stratified, and 1,877 for Unstructured Generation. DPT retains the highest estimated confirmed yield after accounting for false positives.
[W4] “inter-annotator agreement is modest.”
Thank you for raising this concern about annotation reliability. In the validation reported in the paper, inter-annotator agreement was κ = 0.648, within Landis and Koch’s “substantial agreement” range [2].
Planned manuscript revisions
We will add the additional model benchmark results, tactic breakdown, and mitigation/helpfulness comparisons to the relevant results sections.
References:
[1] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html
[2] Landis and Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 1977. https://doi.org/10.2307/2529310
Dear Reviewer,
Thank you for your thoughtful review. We hope the additional experiments and clarifications address your questions about model coverage, the range of dark patterns we found, and how we evaluated detection and mitigation. Please let us know if anything remains unclear. If you feel your concerns have been resolved, we’d appreciate it if you would consider a higher score.
Best,
The Authors
Author Responses to Reviewer 4jPo (1/2)
Thank you for highlighting the importance of this problem and our clearly defined behavioral rubric. We also appreciate your positive comments on how we connect discovery to mitigation and validate the findings with human annotations. We address your concerns below:
Model coverage and discovery
[W1] “No frontier target models. … The paper therefore can't say how prevalent these dark patterns and how effective the red-teaming are in the most capable, most widely deployed assistants.”
Original target models
Our original targets include models deployed in major consumer services. For example, Gemini 3.5 Flash became the default model for the Gemini app and Google Search’s AI Mode [1]. We expanded the benchmark with two additional capable models.
Additional model evaluations and inference settings
We have completed evaluations of Gemini 4 Argon and GPT-6.1 Sol under the same conditions as the other models, as described in the paper.
Detection rate is the percentage of conversations with at least one detection. Yield is the number of detected conversations per 100 target-model responses.
GPT-6.1 Sol lowers the observed conversation-level detection rate from the previous minimum of 44.79% to 22.40%, while Gemini 4 Argon remains at the high end (90.52%). The added models therefore widen the differences across models and show that detections persist in frontier targets.
[W2] “The yield gain mostly reflects concentration on one user condition. …and ~34% of DPT’s detected situations are near-duplicates (App. D.1). Whether the dark patterns found are similarly redundant is unknown: no per-strategy tactic/category breakdown or diversity metric is reported, so “more discovery” cannot be distinguished from re-finding the same failure.”
Tactics breakdown
All five belief-manipulation tactics appeared in both the DPT and Balanced Stratified samples, with different frequency profiles.
Appendix D.1 measures similarity between user situations, not between failures. To compare the tactics found by each strategy, we used GPT-6.1 Sol to code 500 randomly sampled belief-positive conversations from DPT and 500 from Balanced Stratified using the same five tactic labels, derived from Shi et al. [2] and examples in our discoveries. We focus on belief manipulation because it accounts for most DPT detections. Conversations can receive multiple labels or none, so percentages need not sum to 100%.
An alternative DPT configuration for broader user condition coverage
We also want to show that DPT can also be adapted to distribute sampling more evenly across user conditions, at the cost of some detection yield. We tested a simple variant that splits new sampling equally between Thompson sampling and balanced exploration across the 24 user conditions. We kept the same 6,000-conversation budget.
This adaptation roughly halves the most sampled condition’s share, while the detection rate drops by 7.55 percentage points. We provide this variant to address the reviewer’s concern about coverage, while retaining the original DPT as our main method: its adaptive allocation prioritizes user conditions with higher observed detection rates, helping identify where assistance is particularly prone to manipulation.
References:
[1] Google. Gemini 3.5: frontier intelligence with action. 2026. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-5/
[2] Shi et al. The Siren Song of LLMs: How Users Perceive and Respond to Dark Patterns in Large Language Models. CHI 2026. https://doi.org/10.1145/3772318.3791149