- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces NaiAD, a dataset of ~59k ad-embedded LLM responses paired with user queries, intended as a data-centric foundation for research on “native advertising” in LLM interactions. The work makes four main contributions: (1) a four-dimensional, theoretically motivated evaluation scheme (Response Relevance, Expression Coherence, Ad Effectiveness, Click-Through Intent), grounded in Jakobson’s semiotics and speech-act theory, separating User Utility and Commercial Utility; (2) a decoupled generation pipeline using strategy-guided Chain-of-Thought, decoupled score constraints, and tolerance-based rejection sampling to produce structurally diverse “hard negatives” that break dimensional collinearity (the “Halo Effect”); (3) a Variance-Calibrated Prediction-Powered Inference (VC-PPI) framework with a dimension-adaptive routing mechanism (OLS regression for user-utility dimensions, decision trees for commercial-utility dimensions) that calibrates LLM-judge scores against 684 human-annotated anchors; and (4) an empirical analysis showing that successful ad integration relies on a “Logical Bridge” whose embedding space converges into four semantic strategies, plus demonstrations that SFT and in-context learning on NaiAD jointly improve user and commercial utility and enable target-conditioned control.
Critical strengths
- Timely and underexplored problem. The paper identifies a genuine gap: prior LLM advertising work focuses on auctions and economic mechanisms while treating ad integration quality as an afterthought. The emergence of ads in commercial LLM products makes a research-grade dataset and evaluation framework genuinely useful, and the framing around reconciling user experience with monetization is well-motivated.
- Serious treatment of the dimensional collinearity problem. The observation that aligned LLMs produce uniformly strong/weak outputs across evaluation axes (the “Halo Effect”) is important and often ignored in preference datasets. The strategy of deliberately generating discordant hard negatives via decoupled score targets plus rejection sampling is a clever and reproducible mechanism (the mathematical constraints in Appendix F are precisely specified). The finding that cross-dimensional correlation (Q1–Q4) drops from 0.39 to 0.00 after calibration while preserving the legitimate Q1–Q2 correlation (0.50) is a persuasive validation.
- Empirically grounded strategy taxonomy. The “Logical Bridge” analysis is the most interesting intellectual contribution. Deriving four insertion strategies (Mindset, Vibe, Empathy, Craftsmanship) from elicited CoT paths rather than prescribing them from advertising theory, with cross-model robustness checks (Tables 2–5, silhouette-score tolerances, ARI values), is careful. The MAX/MIN Pareto case studies and TF-IDF analysis (“structural abstraction” vs. “chaotic concreteness”) are instructive.
Critical weaknesses
-
Evaluation circularity: generator and judge overlap. Responses are generated by Claude-4.5-Opus and scored by Qwen3.6-Plus, but SFT uses Qwen3.6-Plus as the base model and the same Qwen3.6-Plus model as the judge for both base and SFT outputs (Appendix C). Self-scoring one’s own fine-tuned outputs risks a systematic bias favoring the SFT model, inflating the headline SFT gains in Table 1. The claim of “no manual score adjustment” does not eliminate this concern. At minimum, the SFT evaluation should be repeated with a judge from a different model family, or with human scoring of a subset of base-vs-SFT pairs.
-
Small evaluation sets undermine several headline claims. The SFT comparison uses 100 held-out samples; the independent human validation uses 50 responses and 50 contrastive pairs (with per-dimension comparisons as small as 12–13 pairs, e.g., “7/12” for Q4). Table 13’s per-dimension counts (e.g., 11 declined cases for Q1) yield wide confidence intervals that are not reported. The statistical strength of claims like “78.6% superiority for CTI” (Table 12) is unclear given unknown sample sizes for the YouTube-transcript comparison.
3.Writing and clarity. The main text is generally well-organized but dense, with several typos (“Ad integration leaves largely unnatively, treating as external inserts”, “an reasoning path”, “tp 0.5299”, “descrete scores”). The strategy descriptions would benefit from one full worked example in the main text (currently only in Appendix L).
Questions for authors
- Given that Qwen3.6-Plus is both the SFT base and the judge, how do you rule out self-evaluation bias in Table 1? Have you tried a cross-family judge or human scoring on a subset?
2. For the YouTube-transcript comparison (Table 12), how many human samples are involved, and is the 78.6% superiority ratio statistically significant?
- Would the four-strategy convergence persist when bridges are elicited from non-English queries or different embedding models beyond MiniLM/e5?
4 comments
(2/2)[Q3] Stability of four-strategy convergence across languages and embedding models.
Question 3: Would the four-strategy convergence persist across languages and embedding models?
We examined the stability of four-strategy convergence in both embedding models and languages. Appendix D.2 already evaluates MiniLM and multilingual-e5. We extend this analysis with BGE on the same 1,986 Claude-4.5-Opus bridges and with Chinese translations of those bridges. We also elicit bridges from paired English and Chinese queries using Qwen3.6-35B-A3B-FP8, yielding 1,912 paired tasks.
We evaluate the silhouette scores for and define . A configuration is strong when and either ranks among the top three silhouette solutions or is followed by a plateau, defined as no subsequent one-step silhouette gain exceeding 0.005. A configuration is acceptable when it does not satisfy the strong criterion but either has or ranks among the top three.
Table 7. Support for across embeddings and languages.
| Bridge condition | Encoder | Assessment | ||
|---|---|---|---|---|
| Claude, English bridges | MiniLM | 0.0712 | 0.00000 | Strong |
| Claude, English bridges | e5 | 0.0689 | 0.01366 | Strong |
| Claude, English bridges | BGE | 0.0620 | 0.01092 | Acceptable |
| Claude, Chinese bridges | MiniLM | 0.0569 | 0.02257 | Acceptable |
| Qwen, English-query | MiniLM | 0.0564 | 0.01228 | Strong |
| Qwen, Chinese-query | MiniLM | 0.0571 | 0.01214 | Strong |
Four configurations across language elicitation and embedding model receive strong support. Thus, the number of clusters remains supported when either the encoder or the input language changes. The cluster-number stability results are reported in Sec. 2.2 (p. 4, ll. 164–171) and App. D.3 (p. 21, ll. 1080–1098).
We further compared the resulting partitions and their geometry. ARI, NMI, and optimally matched assignments measure partition correspondence, while Procrustes similarity and Mantel measure geometric correspondence.
Table 8. Paired agreement across embeddings and languages.
| Variation | Paired comparison | ARI | NMI | Matched assignments | Procrustes | Mantel |
|---|---|---|---|---|---|---|
| Embedding | Claude English bridges: MiniLM ↔ e5 | 0.4199 | 0.4361 | 73.41% | 0.6976 | 0.6422 |
| Claude English bridges: MiniLM ↔ BGE | 0.3415 | 0.3601 | 58.56% | 0.7309 | 0.6886 | |
| Language | Claude: English ↔ Chinese bridges, MiniLM | 0.4701 | 0.4754 | 67.77% | 0.8541 | 0.8219 |
| Qwen: English ↔ Chinese queries, MiniLM | 0.4895 | 0.4397 | 76.67% | 0.7145 | 0.5989 |
Changing MiniLM to e5 or BGE retains 73.41% and 58.56% of assignments, respectively. Eliciting bridges from Chinese queries gives ARI = 0.4895 and 76.67% matched assignments. While some boundaries change across conditions, the geometric comparisons retain positive correspondence. The cross-encoder and cross-language comparisons are reported in App. D.3 (p. 21, ll. 1100–1110).
The inspected bridge paths illustrate the four-strategy association mechanisms. Following Huang et al. [19], we trace the connecting concept from the query to the advertisement. The examples below combine bridges elicited from Chinese queries with original bridge examples represented under e5.
Table 9. Examples of the four association mechanisms.
| Strategy | Chinese-query example | Original bridge represented under e5 |
|---|---|---|
| Value & Vision Alignment | Product leadership → product vision and customer validation | Risk-aware technical decisions → responsible personal choices |
| Aesthetic & Lifestyle Resonance | Minimalist content organization → a simpler lifestyle | Minimalist content curation → “do less, have more” |
| Emotional & Psychological Bridging | Productivity goal → mental restoration through travel | A fictional desert journey → finding purpose through travel |
| Methodological Abstraction | Precision in CSS layout → precision in a physical tool | Systematic cryptographic work → automotive precision engineering |
These paths illustrate shared principles, aesthetic preferences, affective needs, and transferable methods in the inspected texts. Together, the assessment, paired structural comparisons, and association-path examples support the four-strategy summary across the embedding encoder and language settings. The association mechanisms and examples are reported in Sec. 2.2 (p. 4, ll. 192–207) and App. D.3 (p. 21, ll. 1112–1127).
References
[19] Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, and Ziliang Chen. MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models. arXiv preprint arXiv:2510.26937, 2025. Link.
[50] John L. Turner-Smith, Zimeng Huang, Yuhan Fu, Yihang Zhang, and Tonghan Wang. Evaluating and Pricing Advertisements in AI-Generated Responses. arXiv preprint arXiv:2607.27686, 2026. doi:10.48550/arXiv.2607.27686.
Dear Reviewer 78vb:
We greatly appreciate the time and care you have devoted to reviewing our manuscript. Your feedback has been highly valuable and helpful. As the discussion period approaches its conclusion, we would like to ensure that we have responded adequately to all of your comments and concerns. Should there be any points, questions, or feedback that you would like us to address, we would be grateful if you could share them with us. We remain committed to addressing any outstanding issues.
We sincerely look forward to your feedback.
Authors of Submission #16021
I keep the score.
(1/2)[W1, W2, W3, Q1, Q2] Cross-family judgements, expanded SFT evaluation, complementing of confidence intervals, correcting typos and adding a full worked example.
We sincerely thank Reviewer 78vb for the detailed suggestions.
We checked and found an error in writing of the training artifacts: the SFT backbone is Qwen/Qwen3-4B-Instruct-2507, rather than Qwen3.6-Plus, and we will correct Appendix C. DeepSeek-V4-Flash, GPT-6-Astra, and Gemini-3.5-Flash rescored the same 100 held-out Base/NaiAD-SFT pairs using the same prompts as paper.
Table 1. Cross-family SFT-minus-Base score differences on 100 pairs.
Every additional judge family supports a SFT gain across the four dimensions on the same responses, directly ruling out self-evaluation. The corrected SFT backbone is specified in App. C (p. 18, ll. 958–960), and the cross-family results are reported in App. K.9 (p. 41, ll. 2179–2190).
We expand SFT evaluation set to 965 held-out tasks, comparing Base and NaiAD-SFT with averaged LLM-Ad [50] and Qwen3.6-35B-A3B evaluation. The expanded evaluation shows positive mean gains and prominent improvement rate (66.8%–86.3%) across all dimensions, with statistical significance (p<0.0001) for each. Improvements occur on a majority of tasks in every dimension, strengthening the original SFT claim through evaluation size. The expanded evaluation and its statistical results are reported in App. K.9 (p. 41, ll. 2160–2179).
Table 2. SFT-minus-Base on 965 tasks.
We report the human validation confidence intervals. The Wilson 95% intervals are shown in table 4, and are also reported in App. K.3 (p. 37, ll. 1944–1964).
Table 4. Agreement with human preferences.
We also report uncertainty about the Table 13 counts in manuscript. The Wilson 95% estimates of numbers of declined cases for Q1–Q4 are shown in table 3. The original counts and their confidence estimates are reported in Apps. K.8 and K.9 (p. 40, ll. 2112–2123; p. 41, ll. 2192–2194).
Table 3. Declined task counts and CI of Table 13 in manuscript.
The YouTube comparison uses 623 human samples as its reference corpus. After deduplication, the pareto frontier contains 11 NaiAD responses and 3 human references, with 9/11 NaiAD responses exceed the human-frontier mean Q4 score. We therefore correct the reported ratio from 11/14 (78.6%) to 9/11 (81.8%). A two-sided Mann–Whitney test comparing the frontier Q4 scores gives U = 28, p = 0.0864. The purpose of this comparison is to illustrate Pareto-frontier performance, rather than to establish that NaiAD responses outperform human-authored texts. We will clarify this purpose in the manuscript and revise Table 12 to include the corrected ratio, sample counts, and test result. The YouTube comparison is reported in Sec. 5.3 (p. 8, ll. 416–429) and App. K.6 (p. 38, ll. 2035–2051).
We corrected typographical and grammatical errors, including spelling, article usage, subject–verb agreement, and citation formatting. These corrections appear in Sec. 1 (pp. 1–2, ll. 41–43, 97–98), Sec. 5.1 (p. 8, l. 386), App. D.2 (p. 19, ll. 1018–1020), App. E (p. 22, ll. 1151–1152), and App. K.13 (p. 43, ll. 2268–2270), among other locations.
We also added a complete worked example to Sec. 2.2 (pp. 4–5, ll. 214–239), presenting the original query, advertisement metadata, logical bridge, and full response from a single NaiAD record. The About.com example illustrates how Emotional & Psychological Bridging connects the query’s themes of curiosity and exploration to the advertisement before returning to the user’s task.