- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LLMAdBench evaluates preferences over four advertisement placements in LLM responses under labeled and unlabeled sponsorship conditions. It contains 258 source conversations and 18,576 dimension-level human judgments covering six evaluation criteria. The paper evaluates eight LLM judges, examines presentation-order sensitivity and aggregate placement preferences, and demonstrates improvements from adapting Qwen3-8B to the collected preferences
Critical strengths
-
The benchmark addresses a useful evaluation problem with a structured design. Holding the conversation, base answer, and advertised entity fixed makes the task substantially more interpretable than unrestricted ad-response generation. Separating visibility, helpfulness, appropriateness, and overall preference also avoids treating advertiser and user objectives as interchangeable.
-
The evaluation offers several complementary findings. Presentation-swap tests reveal substantial judge inconsistency, while aggregate preference analysis identifies a disclosure-dependent difference between human and model judgments. The source-instance-level training split appropriately prevents comparisons from the same conversation from appearing in both training and testing.
Critical weaknesses
-
The reliability of the individual human targets is insufficiently established. High split-half agreement between aggregate placement-score vectors does not establish that independent annotators agree on the individual A/B/T labels used for model evaluation. The main text needs clearer information on repeated annotation, label aggregation, and annotator variability. Because the 18,576 judgments derive from only 258 source conversations, dependence among comparisons also matters when assessing the significance of leaderboard differences.
-
The design does not fully isolate the effect of placement. Section 2.1 permits position-specific changes to advertisement wording. Such changes may directly affect naturalness, helpfulness, and preference, even when the advertised entity and core message remain unchanged. The results therefore characterize placement under a particular rewriting procedure, rather than the effect of moving identical advertising content. This distinction weakens the stronger placement-only claims.
-
The adapted judge lacks a simple baseline that could explain much of its improvement. DPO outperforming zero-shot judges establishes that supervision helps, but does not show how much context-sensitive preference learning is required. Since the paper identifies strong aggregate patterns involving placement, disclosure, and evaluation dimension, a predictor using only these variables is an important reference. Without it, the additional value of adapting an 8B language model remains unclear.
Questions for authors
-
How many independent annotators evaluated each comparison, and what is their agreement on the individual labels? Please also reconcile the 12 participants described in Appendix B.4 with the 30/32 annotators reported in Table 16.
-
Do the principal placement findings hold when the advertisement text is identical across positions?
-
How does the adapted judge compare with a placement-only baseline, and do its gains remain significant when uncertainty is computed at the source-conversation level?