acceptodds
Under review as a conference paper at ICLR 2027

LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses

Abstract

Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences. We introduce LLMAdBench, a human-preference benchmark for studying advertising in LLM-generated content. The benchmark isolates a simple but practically important decision: given a user conversation, an LLM response, and a matched advertisement, where should the ad be placed? Our dataset compares pairs of responses that differ only in ad position while holding all other conditions fixed including the user query, base answer, advertisement, and disclosure condition. Human annotators evaluate each pair based on six criteria from both advertiser's and user's perspectives. The resulting benchmark contains more than 18,000 human judgments across two disclosure conditions: explicitly labeling the ad as sponsored and merging it into the response without disclosure. We use LLMAdBench to evaluate eight frontier LLMs as preference judges and find that they are not reliable substitutes for human evaluation. Even the most stable models reverse roughly one quarter of their decisions when the presentation order is swapped, agreement across models is low, and their placement preferences differ systematically from those of human annotators. Moreover, LLMAdBench contains substantial learnable signal. In particular, a Qwen3-8B model fine-tuned on the human preferences improves substantially over its base model and outperforms all zero-shot frontier judges on the held-out prediction task. Beyond model evaluation, LLMAdBench provides quantitative evidence on the advertiser-user trade-off and shows that the sponsorship disclosure systematically changes users' preference over ad placement. Together, these results position LLMAdBench as a public testbed for evaluating, diagnosing, and improving LLM systems that integrate advertising into generated content, while providing an empirical foundation for understanding how users respond to ads embedded in LLM outputs. Code and datasets are available at https://anonymous.4open.science/r/LLMADBENCH/.

Primary area:
datasets and benchmarks
Keywords:
LLM, Advertisement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g8DQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LLMAdBench evaluates preferences over four advertisement placements in LLM responses under labeled and unlabeled sponsorship conditions. It contains 258 source conversations and 18,576 dimension-level human judgments covering six evaluation criteria. The paper evaluates eight LLM judges, examines presentation-order sensitivity and aggregate placement preferences, and demonstrates improvements from adapting Qwen3-8B to the collected preferences

Critical strengths
  1. The benchmark addresses a useful evaluation problem with a structured design. Holding the conversation, base answer, and advertised entity fixed makes the task substantially more interpretable than unrestricted ad-response generation. Separating visibility, helpfulness, appropriateness, and overall preference also avoids treating advertiser and user objectives as interchangeable.

  2. The evaluation offers several complementary findings. Presentation-swap tests reveal substantial judge inconsistency, while aggregate preference analysis identifies a disclosure-dependent difference between human and model judgments. The source-instance-level training split appropriately prevents comparisons from the same conversation from appearing in both training and testing.

Critical weaknesses
  1. The reliability of the individual human targets is insufficiently established. High split-half agreement between aggregate placement-score vectors does not establish that independent annotators agree on the individual A/B/T labels used for model evaluation. The main text needs clearer information on repeated annotation, label aggregation, and annotator variability. Because the 18,576 judgments derive from only 258 source conversations, dependence among comparisons also matters when assessing the significance of leaderboard differences.

  2. The design does not fully isolate the effect of placement. Section 2.1 permits position-specific changes to advertisement wording. Such changes may directly affect naturalness, helpfulness, and preference, even when the advertised entity and core message remain unchanged. The results therefore characterize placement under a particular rewriting procedure, rather than the effect of moving identical advertising content. This distinction weakens the stronger placement-only claims.

  3. The adapted judge lacks a simple baseline that could explain much of its improvement. DPO outperforming zero-shot judges establishes that supervision helps, but does not show how much context-sensitive preference learning is required. Since the paper identifies strong aggregate patterns involving placement, disclosure, and evaluation dimension, a predictor using only these variables is an important reference. Without it, the additional value of adapting an 8B language model remains unclear.

Questions for authors
  1. How many independent annotators evaluated each comparison, and what is their agreement on the individual labels? Please also reconcile the 12 participants described in Appendix B.4 with the 30/32 annotators reported in Table 16.

  2. Do the principal placement findings hold when the advertisement text is identical across positions?

  3. How does the adapted judge compare with a placement-only baseline, and do its gains remain significant when uncertainty is computed at the source-conversation level?

Reviewer_Xz6kOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of measuring the effects of inserting advertisements into LLM responses and presents LLMAdBench, a benchmark that tests ads within fixed user–LLM conversations. It studies three questions: where to place the ad (FRONT / EARLY / LATE / END), whether to show the ad labeled as sponsored or unlabeled, and how well LLMs can predict human preferences over placements, both zero-shot and after DPO fine-tuning on human labels.

The benchmark uses 258 WildChat conversations, each with one fixed ad presented in 4 candidate positions × 2 disclosure conditions. Human graders compare variants pairwise (A/B/Tie) on six dimensions (naturalness, visibility, helpfulness, click likelihood, appropriateness, overall), for a total of 18,576 judgments. The paper evaluates (a) 8 zero-shot frontier LLMs as judges, (b) a Qwen3-8B model DPO-tuned on the human pairs, and (c) human placement and disclosure preferences.

Main result: DPO fine-tuning on the human preferences improves Qwen3-8B by 10.29 points in the labeled condition and 19.24 points in the unlabeled condition, moving it from below the zero-shot frontier judges to first on the leaderboard on the held-out prediction task.

Critical strengths

Clean, controlled design: placement × disclosure is isolated as the experimental variable and evaluated with human pairwise labels, rather than as free-form ad insertion inside generation. An honest negative result on LLM judges, supported by the right diagnostics (swap test, placement-profile comparison, and others).

Critical weaknesses

Click likelihood is judged, not measured. The benchmark is an offline stated-preference panel with no CTR, lift, conversions, or online A/B validation. Ad tech measures these with A/B tests, ghost ads, and incrementality studies; the paper never relates its proxies to observed behavior, so the “advertiser value” dimension is weak. Accuracies have no floor and no ceiling. With 3-way exact match and ~20% ties, the trivial floor is ~40%; no chance or majority baseline is reported. Pre-DPO Qwen3 model is at chance, so the DPO delta is inflated; frontier judges are only few pp above chance. Human reliability is reported only in aggregate and no human–human agreement is given, so absolute accuracies are uninterpretable and part of the “error” is label noise. Unequal comparison: the DPO judge is in-distribution with 80/20 split by conversation, while frontier judges are zero-shot with a prompt that pushes “choose Tie when uncertain.”

Questions for authors

Why is the DPO improvement in the unlabeled condition (19.24 points) roughly twice that in the labeled condition (10.29 points)?

Requested additional analysis that would affect my assessment:

Please add per-dimension (or per-cluster) accuracy. Please report random-A/B/Tie and majority-class baselines so the floor for the reported accuracies is clear; binary accuracy on non-tie rows; and a human–human agreement estimate (e.g. re-annotate a subset with a second grader).

Suggestions for future work: Position the benchmark as a surrogate for pre-deployment screening, with a plan to calibrate against online outcomes.

Reviewer_8xWyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore a task of ad placement in the context of the users interacting with LLMs, where an ad, user query/convo, and LLM response are known, and there are several options to place the ad (before/after, towards beginning/end) and if it should be disclosed as an ad or not. The authors provide a novel data set to support this investigation, and compare multiple LLMs in several configs (zero- and few-shot) to analyze the impact and performance of various approaches.

Critical strengths
  • Relevant and timely problem being investigated.
  • Novel data set shared with the community.
  • Some interesting insights surfaced in the experiments.
Critical weaknesses
  • Very limited novelty and contribution of the paper.
  • Limited generalizability and somewhat questionable practical value of the work.
  • Experiments can be improved significantly.
Questions for authors

While the paper is relatively nicely written (it is easy to follow the problem and thinking of the authors), the novelty and experiments are rather suboptimal, which are the main items impacting my recommendation. Detailed comments follow:

  • The authors explore a timely problem, but the setup is rather constrained and only a single data set is used.
  • The data is not that large, considering how large the problem setup potentially is.
  • The data set is not well explained also, there are no detailed discussions around the data, and the labeling process is also not well explained (e.g., what are the backgrounds of the labelers).
  • The authors also abuse the appendix, a large number of relevant discussions and experiments is relegated to appendix even if it is important for the main story that the authors want to say. While it is welcome to use the appendix of course, it should not be used as extra pages to house the critical information. And this is a problem throughout the paper.
  • Can you clarify what random splitting into halves is showing, and why is it done? It was not super easy to follow the motivation behind that as the authors did not spend a lot of time explaining that part.
  • "the auxiliary placement-attention item ...", unclear what is this referring to, there is no explanation clarifying this.
  • It would be good to include a non-LLM baseline as well.
  • The LLM approach is adding a lot of latency and cost to the ad placement process which needs to be extremely fast. Yet the authors do not discuss this at all, completely ignoring the subject.
  • The authors do not show any qualitative results, which would increase the quality of the work.
  • How aligned is the click-likelihood score aligned with the actual CTR? This should be discussed to confirm that the labelers are providing reasonable labels.
  • The experiments would be significantly stronger if there were real-world experiments. I understand that is not easy to provide, but given weak explanations and evaluation earlier (as commented above), this would push the paper recommendation significantly higher.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.