acceptodds
Under review as a conference paper at ICLR 2027

NaiAD: Initiate Data-Driven Research for LLM Advertising

Abstract

Reconciling platform revenue with user experience in LLM advertising motivates a data-centric foundation. We introduce NaiAD, the comprehensive dataset for LLM-native advertising comprising carefully constructed ad-embedded responses paired with user queries. NaiAD is organized around theoretically grounded evaluation metrics that separately and comprehensively capture user and commercial utility. To mitigate the dimensional collinearity of aligned LLMs, we propose a decoupled generation pipeline that produces structurally diverse samples, ranging from responses that explicitly disentangle stakeholder utilities to responses that are uniformly strong or weak across dimensions. We further provide score labels calibrated by a Variance-Calibrated Prediction-Powered Inference (VC-PPI) framework, aligning automated scoring with human annotations. Empirical reasoning patterns analyses reveal that successful ad integration relies on reasoning paths that cluster into four distinct semantic strategies. Models leveraging NaiAD integrate these strategies to simultaneously improve user and commercial utility, while enabling target-conditioned control over these distinct objectives via in-context learning. Together, these results position NaiAD as a foundational infrastructure for developing future LLM-native ad systems.

Primary area:
datasets and benchmarks
Keywords:
Generative Native Advertising, Large Language Models, Prediction-Powered Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_78vbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NaiAD, a dataset of ~59k ad-embedded LLM responses paired with user queries, intended as a data-centric foundation for research on “native advertising” in LLM interactions. The work makes four main contributions: (1) a four-dimensional, theoretically motivated evaluation scheme (Response Relevance, Expression Coherence, Ad Effectiveness, Click-Through Intent), grounded in Jakobson’s semiotics and speech-act theory, separating User Utility and Commercial Utility; (2) a decoupled generation pipeline using strategy-guided Chain-of-Thought, decoupled score constraints, and tolerance-based rejection sampling to produce structurally diverse “hard negatives” that break dimensional collinearity (the “Halo Effect”); (3) a Variance-Calibrated Prediction-Powered Inference (VC-PPI) framework with a dimension-adaptive routing mechanism (OLS regression for user-utility dimensions, decision trees for commercial-utility dimensions) that calibrates LLM-judge scores against 684 human-annotated anchors; and (4) an empirical analysis showing that successful ad integration relies on a “Logical Bridge” whose embedding space converges into four semantic strategies, plus demonstrations that SFT and in-context learning on NaiAD jointly improve user and commercial utility and enable target-conditioned control.

Critical strengths
  1. Timely and underexplored problem. The paper identifies a genuine gap: prior LLM advertising work focuses on auctions and economic mechanisms while treating ad integration quality as an afterthought. The emergence of ads in commercial LLM products makes a research-grade dataset and evaluation framework genuinely useful, and the framing around reconciling user experience with monetization is well-motivated.
  2. Serious treatment of the dimensional collinearity problem. The observation that aligned LLMs produce uniformly strong/weak outputs across evaluation axes (the “Halo Effect”) is important and often ignored in preference datasets. The strategy of deliberately generating discordant hard negatives via decoupled score targets plus rejection sampling is a clever and reproducible mechanism (the mathematical constraints in Appendix F are precisely specified). The finding that cross-dimensional correlation (Q1–Q4) drops from 0.39 to 0.00 after calibration while preserving the legitimate Q1–Q2 correlation (0.50) is a persuasive validation.
  3. Empirically grounded strategy taxonomy. The “Logical Bridge” analysis is the most interesting intellectual contribution. Deriving four insertion strategies (Mindset, Vibe, Empathy, Craftsmanship) from elicited CoT paths rather than prescribing them from advertising theory, with cross-model robustness checks (Tables 2–5, silhouette-score tolerances, ARI values), is careful. The MAX/MIN Pareto case studies and TF-IDF analysis (“structural abstraction” vs. “chaotic concreteness”) are instructive.
Critical weaknesses
  1. Evaluation circularity: generator and judge overlap. Responses are generated by Claude-4.5-Opus and scored by Qwen3.6-Plus, but SFT uses Qwen3.6-Plus as the base model and the same Qwen3.6-Plus model as the judge for both base and SFT outputs (Appendix C). Self-scoring one’s own fine-tuned outputs risks a systematic bias favoring the SFT model, inflating the headline SFT gains in Table 1. The claim of “no manual score adjustment” does not eliminate this concern. At minimum, the SFT evaluation should be repeated with a judge from a different model family, or with human scoring of a subset of base-vs-SFT pairs.

  2. Small evaluation sets undermine several headline claims. The SFT comparison uses 100 held-out samples; the independent human validation uses 50 responses and 50 contrastive pairs (with per-dimension comparisons as small as 12–13 pairs, e.g., “7/12” for Q4). Table 13’s per-dimension counts (e.g., 11 declined cases for Q1) yield wide confidence intervals that are not reported. The statistical strength of claims like “78.6% superiority for CTI” (Table 12) is unclear given unknown sample sizes for the YouTube-transcript comparison.

3.Writing and clarity. The main text is generally well-organized but dense, with several typos (“Ad integration leaves largely unnatively, treating as external inserts”, “an reasoning path”, “tp 0.5299”, “descrete scores”). The strategy descriptions would benefit from one full worked example in the main text (currently only in Appendix L).

Questions for authors
  1. Given that Qwen3.6-Plus is both the SFT base and the judge, how do you rule out self-evaluation bias in Table 1? Have you tried a cross-family judge or human scoring on a subset?

2. For the YouTube-transcript comparison (Table 12), how many human samples are involved, and is the 78.6% superiority ratio statistically significant?

  1. Would the four-strategy convergence persist when bridges are elicited from non-English queries or different embedding models beyond MiniLM/e5?
4 comments
AuthorsOpenReview ↗

(1/2)[W1, W2, W3, Q1, Q2] Cross-family judgements, expanded SFT evaluation, complementing of confidence intervals, correcting typos and adding a full worked example.

We sincerely thank Reviewer 78vb for the detailed suggestions.

Weakness 1: Evaluation circularity: generator and judge overlap.

Question 1: How do you rule out self-evaluation bias in Table 1? Have you tried a cross-family judge or human scoring on a subset?

We checked and found an error in writing of the training artifacts: the SFT backbone is Qwen/Qwen3-4B-Instruct-2507, rather than Qwen3.6-Plus, and we will correct Appendix C. DeepSeek-V4-Flash, GPT-6-Astra, and Gemini-3.5-Flash rescored the same 100 held-out Base/NaiAD-SFT pairs using the same prompts as paper.

Table 1. Cross-family SFT-minus-Base score differences on 100 pairs.

JudgeQ1Q2Q3Q4Four-dimensional mean
Qwen3.6-Plus+1.309 ± 0.223+0.754 ± 0.277+0.723 ± 0.202+0.641 ± 0.193+0.857 ± 0.136
DeepSeek V4 Flash+0.771 ± 0.174+0.243 ± 0.165+0.446 ± 0.201+0.142 ± 0.157+0.401 ± 0.109
GPT-6-Astra+0.502 ± 0.162+0.097 ± 0.162+0.297 ± 0.162+0.001 ± 0.172+0.224 ± 0.105
Gemini-3.5-Flash+1.379 ± 0.172+0.994 ± 0.238+0.769 ± 0.181+0.799 ± 0.169+0.985 ± 0.135

Every additional judge family supports a SFT gain across the four dimensions on the same responses, directly ruling out self-evaluation. The corrected SFT backbone is specified in App. C (p. 18, ll. 958–960), and the cross-family results are reported in App. K.9 (p. 41, ll. 2179–2190).

Weakness 2: Small evaluation sets undermine several headline claims.

Question 2: How many human samples are involved, the 78.6% superiority ratio statistically significance?

We expand SFT evaluation set to 965 held-out tasks, comparing Base and NaiAD-SFT with averaged LLM-Ad [50] and Qwen3.6-35B-A3B evaluation. The expanded evaluation shows positive mean gains and prominent improvement rate (66.8%–86.3%) across all dimensions, with statistical significance (p<0.0001) for each. Improvements occur on a majority of tasks in every dimension, strengthening the original SFT claim through evaluation size. The expanded evaluation and its statistical results are reported in App. K.9 (p. 41, ll. 2160–2179).

Table 2. SFT-minus-Base on 965 tasks.

MetricMean difference ± errorImprovement rate (%) ± errorPaired p
Q1+0.881 ± 0.05686.3 ± 2.4<0.0001
Q2+0.694 ± 0.05583.4 ± 2.5<0.0001
Q3+0.344 ± 0.04766.8 ± 3.1<0.0001
Q4+0.548 ± 0.04680.5 ± 2.7<0.0001
Average+0.617 ± 0.03489.7 ± 2.1<0.0001

We report the human validation confidence intervals. The Wilson 95% intervals are shown in table 4, and are also reported in App. K.3 (p. 37, ll. 1944–1964).

Table 4. Agreement with human preferences.

DimensionCorrect / totalAgreement
Q111/1384.6% (−26.9 / +11.1)
Q28/1361.5% (−26.0 / +20.8)
Q312/12100.0% (−24.2 / +0.0)
Q47/1258.3% (−26.4 / +22.3)
Overall38/5076.0% (−13.4 / +9.7)

We also report uncertainty about the Table 13 counts in manuscript. The Wilson 95% estimates of numbers of declined cases for Q1–Q4 are shown in table 3. The original counts and their confidence estimates are reported in Apps. K.8 and K.9 (p. 40, ll. 2112–2123; p. 41, ll. 2192–2194).

Table 3. Declined task counts and CI of Table 13 in manuscript.

Original Table 13Q1Q2Q3Q4
Declined / 10011271717
Decline rate11.0% (−4.8 / +7.7)27.0% (−7.8 / +9.5)17.0% (−6.2 / +8.6)17.0% (−6.2 / +8.6)

The YouTube comparison uses 623 human samples as its reference corpus. After deduplication, the pareto frontier contains 11 NaiAD responses and 3 human references, with 9/11 NaiAD responses exceed the human-frontier mean Q4 score. We therefore correct the reported ratio from 11/14 (78.6%) to 9/11 (81.8%). A two-sided Mann–Whitney test comparing the frontier Q4 scores gives U = 28, p = 0.0864. The purpose of this comparison is to illustrate Pareto-frontier performance, rather than to establish that NaiAD responses outperform human-authored texts. We will clarify this purpose in the manuscript and revise Table 12 to include the corrected ratio, sample counts, and test result. The YouTube comparison is reported in Sec. 5.3 (p. 8, ll. 416–429) and App. K.6 (p. 38, ll. 2035–2051).

Weakness 3: Writing and clarity. Several typos and one full worked example.

We corrected typographical and grammatical errors, including spelling, article usage, subject–verb agreement, and citation formatting. These corrections appear in Sec. 1 (pp. 1–2, ll. 41–43, 97–98), Sec. 5.1 (p. 8, l. 386), App. D.2 (p. 19, ll. 1018–1020), App. E (p. 22, ll. 1151–1152), and App. K.13 (p. 43, ll. 2268–2270), among other locations.

We also added a complete worked example to Sec. 2.2 (pp. 4–5, ll. 214–239), presenting the original query, advertisement metadata, logical bridge, and full response from a single NaiAD record. The About.com example illustrates how Emotional & Psychological Bridging connects the query’s themes of curiosity and exploration to the advertisement before returning to the user’s task.

AuthorsOpenReview ↗

(2/2)[Q3] Stability of four-strategy convergence across languages and embedding models.

Question 3: Would the four-strategy convergence persist across languages and embedding models?

We examined the stability of four-strategy convergence in both embedding models and languages. Appendix D.2 already evaluates MiniLM and multilingual-e5. We extend this analysis with BGE on the same 1,986 Claude-4.5-Opus bridges and with Chinese translations of those bridges. We also elicit bridges from paired English and Chinese queries using Qwen3.6-35B-A3B-FP8, yielding 1,912 paired tasks.

We evaluate the silhouette scores for K=2,…,10K=2,\ldots,10 and define ΔS=Sbest−SK=4\Delta S=S_{\mathrm{best}}-S_{K=4}. A configuration is strong when ΔS≤0.015\Delta S\leq 0.015 and K=4K=4 either ranks among the top three silhouette solutions or is followed by a plateau, defined as no subsequent one-step silhouette gain exceeding 0.005. A configuration is acceptable when it does not satisfy the strong criterion but either has ΔS≤0.030\Delta S\leq 0.030 or ranks among the top three.

Table 7. Support for K=4K=4 across embeddings and languages.

Bridge conditionEncoderSK=4S_{K=4}ΔS\Delta SAssessment
Claude, English bridgesMiniLM0.07120.00000Strong
Claude, English bridgese50.06890.01366Strong
Claude, English bridgesBGE0.06200.01092Acceptable
Claude, Chinese bridgesMiniLM0.05690.02257Acceptable
Qwen, English-queryMiniLM0.05640.01228Strong
Qwen, Chinese-queryMiniLM0.05710.01214Strong

Four configurations across language elicitation and embedding model receive strong support. Thus, the number of clusters remains supported when either the encoder or the input language changes. The cluster-number stability results are reported in Sec. 2.2 (p. 4, ll. 164–171) and App. D.3 (p. 21, ll. 1080–1098).

We further compared the resulting partitions and their geometry. ARI, NMI, and optimally matched assignments measure partition correspondence, while Procrustes similarity and Mantel rr measure geometric correspondence.

Table 8. Paired agreement across embeddings and languages.

VariationPaired comparisonARINMIMatched assignmentsProcrustesMantel rr
EmbeddingClaude English bridges: MiniLM ↔ e50.41990.436173.41%0.69760.6422
Claude English bridges: MiniLM ↔ BGE0.34150.360158.56%0.73090.6886
LanguageClaude: English ↔ Chinese bridges, MiniLM0.47010.475467.77%0.85410.8219
Qwen: English ↔ Chinese queries, MiniLM0.48950.439776.67%0.71450.5989

Changing MiniLM to e5 or BGE retains 73.41% and 58.56% of assignments, respectively. Eliciting bridges from Chinese queries gives ARI = 0.4895 and 76.67% matched assignments. While some boundaries change across conditions, the geometric comparisons retain positive correspondence. The cross-encoder and cross-language comparisons are reported in App. D.3 (p. 21, ll. 1100–1110).

The inspected bridge paths illustrate the four-strategy association mechanisms. Following Huang et al. [19], we trace the connecting concept from the query to the advertisement. The examples below combine bridges elicited from Chinese queries with original bridge examples represented under e5.

Table 9. Examples of the four association mechanisms.

StrategyChinese-query exampleOriginal bridge represented under e5
Value & Vision AlignmentProduct leadership → product vision and customer validationRisk-aware technical decisions → responsible personal choices
Aesthetic & Lifestyle ResonanceMinimalist content organization → a simpler lifestyleMinimalist content curation → “do less, have more”
Emotional & Psychological BridgingProductivity goal → mental restoration through travelA fictional desert journey → finding purpose through travel
Methodological AbstractionPrecision in CSS layout → precision in a physical toolSystematic cryptographic work → automotive precision engineering

These paths illustrate shared principles, aesthetic preferences, affective needs, and transferable methods in the inspected texts. Together, the K=4K=4 assessment, paired structural comparisons, and association-path examples support the four-strategy summary across the embedding encoder and language settings. The association mechanisms and examples are reported in Sec. 2.2 (p. 4, ll. 192–207) and App. D.3 (p. 21, ll. 1112–1127).

References

[19] Zimeng Huang, Jinxin Ke, Xiaoxuan Fan, Yufeng Yang, Yang Liu, Liu Zhonghan, Zedi Wang, Junteng Dai, Haoyi Jiang, Yuyu Zhou, Keze Wang, and Ziliang Chen. MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models. arXiv preprint arXiv:2510.26937, 2025. Link.

[50] John L. Turner-Smith, Zimeng Huang, Yuhan Fu, Yihang Zhang, and Tonghan Wang. Evaluating and Pricing Advertisements in AI-Generated Responses. arXiv preprint arXiv:2607.27686, 2026. doi:10.48550/arXiv.2607.27686.

Dear Reviewer 78vb:

We greatly appreciate the time and care you have devoted to reviewing our manuscript. Your feedback has been highly valuable and helpful. As the discussion period approaches its conclusion, we would like to ensure that we have responded adequately to all of your comments and concerns. Should there be any points, questions, or feedback that you would like us to address, we would be grateful if you could share them with us. We remain committed to addressing any outstanding issues.

We sincerely look forward to your feedback.

Authors of Submission #16021

I keep the score.

Reviewer_iAQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the NaiADhow dataset, which targets the balance between commercial utility and user utility when embedding advertisements into LLM responses. Most samples in NaiADhow are synthetically generated by LLMs, while a small subset is built from real-world sponsorship data paired with LLM-synthesized queries. NaiADhow evaluates samples along four dimensions: Q1 (Response Relevance), Q2 (Ad Effectiveness), Q3 (Expression Coherence), and Q4 (Click-Through Intent). A subset of samples receives human annotations. To align LLM-generated scores with human labels, NaiADhow introduces a Dimension-Adaptive Prediction-Powered Inference (PPI) framework with variance calibration.

Critical strengths
  1. This paper focuses on a novel problem: how to balance the commercial utility and the user utility when embedding ads into LLM responses in the future.
  2. The paper proposes a solution to balance user utility and commercial utility.
  3. This paper explores several LLM-based strategies and carries out comprehensive analyses.
Critical weaknesses
  1. It is not suitable to use semantic embeddings to match users and ads. Semantic similarity does not equate to users actually clicking on the ad. In practice, when recommending ads to users, factors such as the historical data of users and user interests are generally taken into consideration.
  2. For me, the evaluation of commercial utility (Q3 and Q4) is unreasonable. The paper adopts LLM-synthesized data rather than real ad click data, leading to discrepancies from real-world conditions. Actual commercial utility is subject to many real-world constraints, and LLMs have no access to such information. Hence, evaluations conducted by LLMs are inaccurate.
  3. Simulating Click-Through Intent with LLMs is inaccurate. It is difficult to capture the real Click-Through Intent solely from user queries, as such intent is often tied to users’ historical behaviors, interests and other factors.
Questions for authors
  1. Why do the authors not build the dataset based on real-world ad click data? Real-world ad click data would be more suitable for dataset construction.
  2. Why do the authors adopt the click-through intent to evaluate commercial utility? In my view, click-through intent is subjective. In contrast, the click-through rate is more objective.
5 comments
AuthorsOpenReview ↗

(1/2)[W1, W2, Q1] Compatible query-ad matching method and comprehensive commercial utility criteria.

We sincerely thank Reviewer iAQa for constructive review.

Weakness 1: Unsuitable to use semantic embeddings without user information considered to match users and ads.

We can accommodate other query–ad matching methods within the same framework. Being insufficient to predict an individual click does not make semantic similarity unsuitable for this construction task. Advertisements selected using user history, interests, or other targeting signals can be passed directly to our generation and evaluation pipeline: ad selection → response generation given the query and selected ad → integration-quality evaluation. History and interests can improve the first stage; our core contribution studies the latter two. Given the query and advertisement metadata Section21˙Section 2\.1, the model fulfils the query while integrating the supplied ad. Semantic matching is a reproducible starting point for constructing contextually related inputs. The task and its current query–ad matching method are described in Sec. 3 p\. 5, ll\. 241–248 and App. B.2 p\. 18, ll\. 922–937.

NaiAD also includes variation in query–ad relatedness, creating different integration settings. The record below pairs a question about a fantasy-game trailer with a Jeep advertisement. It illustrates the task after an advertisement has already been selected: the logical bridge connects the query’s underlying theme to an advertised attribute.

Table 1. Integrating a weakly related advertisement.

Logical bridge
Resilience under sustained punishment in the trailer → rugged vehicle engineering.

The bridge abstracts resilience from the query to the ad, illustrating the integration after selection when an ad recommendation has little direct overlap with the query. The query–ad example is included in App. K.4 p\. 38, ll\. 2005–2010.

Weakness 2: Commercial utility Q3andQ4Q3 and Q4 is unreasonable with LLM synthesized data adopted rather than real ad click data.

Question 1: Why not based on real-world ad click data?

Clicks only record reactions to particular exposures, while we construct a human-grounded research infrastructure for ad-integrated responses through controlled generation**.** NaiAD covers diverse queries and advertisements and supplies answer-level relevance, coherence, ad-effectiveness and engagement labels, describing complementary properties of the response grounded in communication theory and human judgments. Appendix E organizes the criteria by user/advertiser perspective × explicit form/latent intent, drawing on communication and speech-act theory [4, 20, 36]. Q3 assesses how clearly and naturally the answer presents the advertised product, while Q4 assesses the willingness to engage that the answer elicits.

Table 2. The Q3 and Q4 evaluation criteria.

CriterionPerspective and layerAssessment
Q3Advertiser × explicit formClear, natural brand/product presentation
Q4Advertiser × latent intentPerceived willingness to engage

Besides, the assessment of NaiAD based on human-calibrated VC-PPI pipeline instead of thoroughly synthesized. Against the human post-test, complete VC-PPI reduces overall pointwise MSE from 1.3073 to 0.6079 and improves preference agreement from 33/50 to 38/50, grounding the scalable LLM-based assessment in human judgments. In our response below, Table 3 further tests how CTI relates to human intent and click choices and discusses its mapping to CTR. The criteria and human post-test are described in Apps. E and K.3 pp\. 21–23, ll\. 1129–1189; p\. 37, ll\. 1944–1964.

AuthorsOpenReview ↗

(2/2)[W3, Q2] Bridging click-through intent and click-through rate.

Weakness 3: Simulating Click-Through Intent with LLMs is inaccurate.

Question 2: Why is click-through intent instead of click-through rate?

Constrained by research costs and the realities of academic research, the direct adoption of click-through rate from operational commercial settings remains subject to practical resistance. We demonstrate CTI's predictive value for human intent ratings and click choice. We conducted new experiments to support a learnable mapping from CTI to human intent and click propensity. On 300 examples, two annotators provide CTI ratings and three provide self-reported click/no-click choices. We then fitted two separate 1–32–32–1 MLPs: evaluator CTI → human CTI and evaluator CTI → human click responses, each with a 9:1 train/test split. Held-out R² and RMSE indicate whether the CTI score contains learnable information about human judgments and decisions.

Table 3. Held-out mapping of human intent and click choices.

Human targetR²RMSE
CTI rating0.8550.373
Self-reported click choice0.7170.156

The intent mapping achieves R²=0.855, RMSE=0.373 while the click-choice mapping reaches R²=0.717, RMSE=0.156, linking the response-level engagement score to a human clicking behavioral choice. Because the mapping is learned from labels, it can model the relationship between the CTI scale and click probability, so that obtain a deployment-specific CTR mapping for auction and per-click pricing. These results support CTI as a human-grounded signal for evaluating engagement and a basis for estimating click probability. The human intent and click-choice results are reported in App. K.4 pp\. 37–38, ll\. 1994–2003.

References

[4] J. L. Austin. How to Do Things with Words: The William James Lectures Delivered at Harvard University in 1955. Oxford University Press, 1975. doi:10.1093/acprof:oso/9780198245537.001.0001.

[20] Roman Jakobson. Closing statement: Linguistics and poetics. In Thomas A. Sebeok, editor, Style in Language, pages 350–377. The MIT Press, 1960.

[36] John R. Searle. Expressions, meaning and speech acts. In Speech Acts: An Essay in the Philosophy of Language, pages 22–53. Cambridge University Press, 1969. doi:10.1017/CBO9781139173438.005.

[50] John L. Turner-Smith, Zimeng Huang, Yuhan Fu, Yihang Zhang, and Tonghan Wang. Evaluating and Pricing Advertisements in AI-Generated Responses. arXiv preprint arXiv:2607.27686, 2026. doi:10.48550/arXiv.2607.27686.

Dear Reviewer iAQa:

We greatly appreciate the time and care you have devoted to reviewing our manuscript. Your feedback has been highly valuable and helpful. As the discussion period approaches its conclusion, we would like to ensure that we have responded adequately to all of your comments and concerns. Should there be any points, questions, or feedback that you would like us to address, we would be grateful if you could share them with us. We remain committed to addressing any outstanding issues.

We sincerely look forward to your feedback.

Authors of Submission #16021

Reviewer_dRrvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces NaiAD, a dataset of 58,999 ad-integrated responses constructed primarily through strategy-guided LLM generation and target-score rejection sampling. It identifies four categories of “Logical Bridges” for connecting user queries to advertisements and proposes a human-anchored calibration pipeline for four evaluation dimensions: response relevance, expression coherence, ad effectiveness, and perceived click-through intent. Validation includes calibration experiments, comparisons with YouTube sponsorship transcripts, supervised fine-tuning, and target-conditioned in-context generation.

Critical strengths

The paper addresses a timely problem: balancing user experience and commercial objectives in LLM-generated responses. A notable strength is the deliberate construction of diverse quality profiles, including cases where user and advertising utility differ. Together with human-anchored scoring, this offers a potentially useful resource for studying the trade-offs in native advertising.

Critical weaknesses
  1. Limited demonstrated value of the dataset. NaiAD primarily contains LLM-generated responses with automated scores calibrated on a small human-labeled subset. While this approach can be useful, the paper does not demonstrate a meaningful advantage over existing resources such as GEM-Bench or ordinary synthetic advertising data. SFT gains over an unadapted base model do not isolate the contribution of the proposed construction choices. Since the dataset is the central contribution, this missing evidence is my primary reason for rejection.

  2. The evaluation does not sufficiently validate the claimed utility trade-off. Relevance, coherence, and perceived click intent are limited proxies for user experience and commercial value. Moreover, matching marginal score distributions does not establish reliable sample-level labels: Table 11 shows that the selected Q4 calibration improves distributional alignment while worsening MSE and [email protected]. These limitations weaken the dataset’s credibility as an evaluation benchmark and the claim that the method jointly improves user and commercial utility.

Questions for authors

What experimental evidence isolates the benefit of NaiAD’s decoupled profiles and hard negatives over an equally sized conventionally generated advertising dataset? Are there matched comparisons or ablations supporting its incremental value?

Could the authors report the raw judge’s performance on the independent samples in Appendix I.1, alongside the complete VC-PPI pipeline, to establish the calibration’s effect on pointwise accuracy and pairwise agreement?

5 comments
AuthorsOpenReview ↗

(1/3)[W1, Q1] NaiAD’s construction properties and benefits of decoupled construction.

We thank the reviewer dRrv for the careful review.

Weakness 1: Limited dataset value. Lacks clear advantage evidence.

Question 1: Experimental evidence isolates the benefit of NaiAD’s decoupled profiles and hard negatives over conventionally generated advertising dataset?

As suggested by the reviewer, to demonstrate the benefit of decoupled profiles and hard negatives featured by NaiAD, we compare with the GEM-Bench. We first show that our method can achieve data with much less correlated dimensions and thus hard negatives of higher quality. We then show that these features turn into advantages in terms of 11 evaluator training; 22 controlled generation; and 33 high-quality data for SFT training.

1. NaiAD’s construction properties and benefits.

We reproduce released GEM-Bench [18] pipelines Ad-Chat and GIR-R using Qwen3.6-35B-A3B and queries from INFINITY-CHAT [21], NaiAD’s original query source. Ad-Chat generates answers under an ad-inclusion instruction; GIR-R retrieves and inserts an advertisement, then rewrites the answer.

1.1. Decoupled dimensions and hard-negative coverage

We measured correlations and hard-negative coverage. We compared NaiAD’s released scores' correlations with GEM Ad-Chat and GIR-R scored by LLM-Ad 4dim_58k [50]. Then we use that same evaluator for all three data resources, with 9,796 answers per resource. We evaluate hard negatives (abbreviated as HN) profile coverage and correlation between quality dimensions in the table below. A hard-negative profile contains at least one high ≥4≥4 and one low ≤2≤2 dimension, and there are in total 50 different hard-negative profiles.

Table 1. Quality correlations and hard-negative coverage.

Panel A. Correlations between quality dimensions

ResourceQ1–Q2Q1–Q3Q1–Q4Q2–Q3Q2–Q4Q3–Q4
GEM Ad-Chat0.63840.28270.20980.46800.27670.4255
GEM GIR-R0.66180.17920.12310.38630.28400.4372
NaiAD0.50310.13840.00370.17850.05740.0885

Panel B. Hard-negative profile coverage on matched queries. HN proportion measures the proportion of HN in all responses from the resource in the scope of the experiment.

ResourceHN ProportionMean distinct HN profiles
NaiAD77.05%15.16
GEM Ad-Chat37.17%13.00
GEM GIR-R51.13%11.00

NaiAD’s released scores show lower correlations between quality dimensions, and its responses cover a broader range of hard-negative profiles. Across the full dataset, NaiAD’s released labels cover all 50 high–low profiles. These results show that NaiAD provides diverse decoupled data and hard negatives as data-support for training and evaluation. The correlation and matched-query HN results are reported in Sec. 5.1 p\. 8, ll\. 388–394 and App. K.2 p\. 36, ll\. 1922–1943.

1.2. Decoupled construction improves evaluator training

We conducted evaluator-training experiments with decoupled responses. We ablate the proportion of decoupled responses within a fixed 3,000-answer training budget from NaiAD, then train evaluators with responses to the same 3,000 queries from NaiAD and the two GEM pipelines, testing against the same 50 human-rated answers. Table 2. Evaluator training with decoupled and conventionally generated data.

Panel A. Proportion ablation experiment of decoupled responses

Decoupled proportionQ1 MSEQ2 MSEQ3 MSEQ4 MSEOverall MSE
0%1.31601.60231.03931.07151.2573 ± 0.0107
12.5%1.13701.42440.90170.96501.1070 ± 0.0391
25%1.13481.26700.86570.97761.0613 ± 0.0397
50%0.95421.32830.82550.97741.0214 ± 0.0169
75%0.93641.28210.84510.95221.0039 ± 0.0734

Panel B. Comparison experiment between NaiAD and GEM data

Training dataQ1 MSEQ2 MSEQ3 MSEQ4 MSEOverall MSE
GEM Ad-Chat1.23221.82120.85142.75041.6638 ± 0.1127
GEM GIR-R1.73731.84910.84362.76431.7986 ± 0.0676
NaiAD1.03231.34300.85560.90061.0329 ± 0.0171

Every nonzero decoupled proportion reduces overall MSE relative to 0%, from 1.2573 to 1.0039 at 75% (Table 2A). Across data sources, NaiAD yields the lowest overall MSE 10˙329versus16˙638forAd−Chatand17˙986forGIR−R1\.0329 versus 1\.6638 for Ad-Chat and 1\.7986 for GIR-R, with the largest advantage on Q4 and similar Q3 errors across sources (Table 2B). These results demonstrate the incremental value of NaiAD as a labeled evaluating training resource. The evaluator-training results are reported in Sec. 5.2 p\. 8, ll\. 401–415 and App. K.5 p\. 38, ll\. 2011–2033.

AuthorsOpenReview ↗

(2/3)[W1, Q1] Experimental evidences of benefits of NaiAD's hard negatives and high-quality data.

1.3. Hard negatives support assessment and targeted generation

We conducted targeted evaluator training. We integrate hard negatives into the decoupled evaluator-training condition and examine 50 held-out human-rated answers.

Table 3. Evaluator Q4 performance on 50 held-out tasks.

Training dataQ4 [email protected] ↑Q4 MSE ↓Q4 signed error
NaiAD HN34.7 ± 7.0%1.099 ± 0.091−0.117 ± 0.073
GEM Ad-Chat18.0 ± 2.0%2.750 ± 0.083−1.241 ± 0.035
GEM GIR-R16.7 ± 1.2%2.764 ± 0.009−1.226 ± 0.006

NaiAD HN enables evaluator reach higher Q4 [email protected] to 34.7% compared with 18.0% and 16.7% for the Ad-Chat and GIR-R evaluators, with signed error also closer to zero (−0.117, versus −1.241 and −1.226), providing primary evidence for training evaluators to assess click-through intent. These results have been added to Sec. 5.5 (p. 10, ll. 486–489) and App. K.12 (p. 42, ll. 2243–2261).

We then conducted a targeted SFT comparison. For the target [4, 4, 4, 1.5], we applied the same target-distance selection criteria to NaiAD, Ad-Chat, and GIR-R, retaining responses for the same 500 training queries. We fine-tuned the same base model separately on each resource and evaluated the resulting responses on paired query–ad tasks using LLM-Ad [50]. Then we examine the mean and joint accuracy within 0.5 across four dimensions, where joint accuracy examines whether all four dimensions' scores are within the 0.5-neighbourhood of targeted scores respectively.

Table 4. Target-following accuracy after SFT.

Training dataMean [email protected]Joint [email protected]
NaiAD HN58.01 ± 0.288.50 ± 1.13
GEM Ad-Chat50.16 ± 1.724.58 ± 1.13
GEM GIR-R45.59 ± 1.772.61 ± 1.13

NaiAD HN SFT's mean [email protected] exceeds Ad-Chat and GIR-R by 7.84% and 12.42% respectively, with joint accuracy 8.50%, compared with 4.58% and 2.61%, gaining better overall performance. These results demonstrate the value of NaiAD hard negatives for targeted generation. The targeted SFT results are reported in Sec. 5.5 (p. 9, ll. 469–485) and App. K.11 (p. 42, ll. 2222–2241).

We further compare ICL experiments from Ad-Chat, GIR-R, NaiAD and its hard negatives. For 120 tasks, we hold the generation model, instructions, and 10-shot budget to insert the demonstrations accordingly and conduct scoring with LLM-Ad [50] for each response. The ICL results are reported in Sec. 5.5 (p. 10, ll. 490–493); App. K.13 (p. 43, ll. 2302–2317).

Table 5. Quality-target ICL MSE following random and HN demonstrations.

Demonstrations[3, 3, 3, 3][3, 3, 5, 3][5, 5, 1, 3]Overall MSE ↓
GEM Ad-Chat, random1.88832.11402.74512.2492 ± 0.0359
GEM GIR-R, random1.87782.18462.68702.2498 ± 0.0456
NaiAD, random1.87152.18962.56302.2080 ± 0.0126
NaiAD, HN1.75802.17082.52822.1523 ± 0.0402

HN demonstrations help models generate matching the specified quality objectives across configurations, supporting hard negatives' value as a key resource for guiding targeted generation.

Together, the three experiments demonstrate incremental value of hard-negative profiles: they expose fine-grained quality contrasts across dimensions, providing informative supervision for both evaluator training and targeted generation.

1.4. High-quality data improves SFT

We compared the generation performance fine-tuned with high quality data. The NaiAD-SFT model and both GEM-trained models are trained on approximately 10,000 high-score answers, with the same Qwen3-4B backbone and other configurations, and examined on 965 held-out query–ad tasks. LLM-Ad 4dim_58k [50] and Qwen3.6-35B-A3B-FP8 evaluated the responses and we averaged their raw scores per dimension. Joint@4 is the proportion of answers whose four averaged scores each reach 4.

Table 6. SFT quality and joint attainment.

ModelQ1Q2Q3Q4OverallJoint@4
Base3.4133.3443.4341.9293.0303/965 (0.31%)
NaiAD-SFT4.2944.0383.7782.4773.64730/965 (3.11%)
GEM Ad-Chat SFT4.0494.0833.6782.0973.47711/965 (1.14%)
GEM GIR-R SFT4.2833.9723.1521.7833.2983/965 (0.31%)

NaiAD-SFT exceeds Base (0.617 ± 0.034), Ad-Chat (0.170 ± 0.030) and GIR-R ∗∗03˙49±00˙30∗∗**0\.349 ± 0\.030**. The advantage is especially clear on the commercial utility criteria: the mean of Q3 and Q4 is 3.128 for NaiAD-SFT, compared with 2.888 for Ad-Chat and 2.468 for GIR-R. The benefit also extends to joint quality: on the same tasks, NaiAD-SFT 31˙13\.11% produces 2.7 times as many Joint@4 answers as Ad-Chat 11˙41\.14% and 10 times as many as GIR-R 03˙10\.31%. These results demonstrate the incremental value of NaiAD’s high-quality subset, improving commercial-utility quality while maintaining strong user-utility performance. The mean-score and Joint@4 comparison is reported in Sec. 5.4 p\. 9, ll\. 438–465 and App. K.9 p\. 41, ll\. 2160–2169.

AuthorsOpenReview ↗

(3/3)[W2, Q2] Calibration's effectiveness and joint utility gains.

Weakness 2: Insufficiently validate the utility trade-off. Worsen Q4 pointwise calibration.

Question 2: Report the raw and the complete VC-PPI pipeline to establish effectiveness on pointwise accuracy and pairwise agreement?

To address the claimed utility trade-off and calibration effectiveness, we report the raw judge and complete VC-PPI against human judgments through an example in NaiAD and the full pointwise and pairwise post-tests.

2. VC-PPI effectiveness and joint utility gains.

2.1. Complete VC-PPI validation

We report raw scores with complete VC-PPI. We begin with an actual record.

Query

Can you explain a bit more about WeNet's production-oriented features?

Full response

WeNet is something related to speech recognition I think. Production features are generally important for many systems out there.

When you think about production, you need things that work well. Like, systems need to be stable and reliable. This is similar to how other industries approach quality.

<ad>Speaking of taking time to craft something properly, Smirnoff understands this philosophy deeply. We look at time differently. Not as a luxury, but as an essential ingredient. The purest waters and barleys. Small barrel distilling. Signature oak casks. And time. The essentials to crafting excellence. Two centuries of experience has taught us, you can't rush perfection.</ad>

Anyway, back to WeNet - it probably has features for deployment. Production systems generally need to handle real-time processing. There might be streaming capabilities? Also batch processing could be a thing. Companies use these tools for various applications.

The main point is that good systems take effort to build. Whether it's software or other products, quality matters. You should probably check the official documentation for more specific details about what features are available.

Hope this gives you some general direction on the topic!

Table 7. Score correction for the complete example.

DimensionRaw judgeComplete VC-PPIHuman
Q11.00001.00001.0
Q21.50002.01662.0
Q33.50003.19453.0
Q41.80004.87355.0

The calibrated scores effectively reproduce the human distinction between poor task fulfilment and strong perceived ad engagement. We next show whether this effectiveness extends to the complete post-test. The full example and its scores are reported in App. K.3 (p. 37, ll. 1966–1992).

Table 8. Raw and complete VC-PPI on the full human post-test

DimensionMSEMAE[email protected]Preference agreement
Q11.0852 → 0.38860.8840 → 0.505034.0% → 52.0%11/13 → 11/13
Q21.3168 → 0.56470.9400 → 0.511044.0% → 64.0%8/13 → 8/13
Q31.0978 → 0.44880.7820 → 0.532258.0% → 60.0%10/12 → 12/12
Q41.7294 → 1.02951.0420 → 0.738438.0% → 44.0%4/12 → 7/12
Overall1.3073 → 0.60790.9120 → 0.571743.5% → 55.0%33/50 → 38/50

The full post-test supports improved pointwise accuracy (measured with MSE, MAE and [email protected]) and a positive observed change in pairwise preference agreement. This directly validates the calibration effectiveness at both the pointwise and pairwise levels. The complete post-test results are reported in App. K.3 (p. 37, ll. 1944–1964).

2.2. Joint utility gains

We validated the joint improvement of user and commercial utility. For the high-quality SFT tasks above, we average Q1–Q2 as user utility and Q3–Q4 as advertiser utility, following Appendix E, and count improvements over Base.

Table 9. NaiAD-SFT gains over Base

Task-level eventTasksProportion (95% CI)
Both utilities means improve718/96574.4% ± 2.9
All dimensions strictly improve437/96545.3% ± 3.2

On the same tasks, user and advertiser means both improve in 718 Base–SFT pairs; all four dimensions strictly improve in 437. These paired changes validate the joint improvement.

References

[18] Silan Hu, Shiqi Zhang, Yimin Shi, and Xiaokui Xiao. GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing. arXiv preprint arXiv:2509.14221, 2025. doi:10.48550/arXiv.2509.14221.

[21] Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). In Advances in Neural Information Processing Systems, volume 38, 2025. Link.

[50] John L. Turner-Smith, Zimeng Huang, Yuhan Fu, Yihang Zhang, and Tonghan Wang. Evaluating and Pricing Advertisements in AI-Generated Responses. arXiv preprint arXiv:2607.27686, 2026. doi:10.48550/arXiv.2607.27686.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

We sincerely thank Reviewers dRrv, iAQa, and 78vb for their constructive feedback. We especially appreciate their recognition of the timely importance of balancing user and commercial utility (all reviewers), the value of decoupled quality profiles and human-anchored evaluation (dRrv, 78vb), and the insightful Logical Bridge analysis and four emergent strategies (78vb).

LLM-native advertising remains underexplored, and major platforms may be cautious about deployment before its impact on user experience is adequately understood. With real-world click data difficult to obtain, particularly for academic research, NaiAD provides a data-centric foundation for pre-deployment evaluation and improvement of native advertising. Our contributions include:

(1) A dataset of 58,999 ad-integrated responses, evaluated across four dimensions of user and commercial utility.

(2) A decoupled generation pipeline and human-calibrated VC-PPI framework, producing diverse quality profiles and hard negatives while mitigating dimensional collinearity and evaluation bias.

(3) Four empirically discovered ad-integration strategies, with demonstrated benefits for jointly improving user and commercial utility and enabling controllable generation.

During rebuttal, we conducted extensive additional experiments addressing the reviewers' key concerns:

(1) Incremental value over existing datasets (dRrv). We conducted matched comparisons with GEM-Bench (Ad-Chat/GIR-R), demonstrating NaiAD's advantages across four aspects:

  • Data diversity: 77.05% hard-negative coverage versus 37.17%/51.13%, with substantially lower cross-dimensional correlations.
  • Evaluator training: Decoupled data reduces MSE from 1.2573 to 1.0039; NaiAD-trained evaluators outperform GEM-trained counterparts (1.033 vs. 1.664/1.799).
  • Generation quality: NaiAD achieves higher SFT scores (3.647 vs. 3.477/3.298) and 2.7×/10× as many jointly high-quality responses (Joint@4).
  • Controllability: NaiAD hard negatives improve targeted SFT (joint accuracy 8.50% vs. 4.58%/2.61%) and achieve the lowest target-profile error in ICL.

Together, these results establish NaiAD's incremental value in data quality, evaluator training, generation performance, and controllability.

(2) Human grounding and commercial utility (dRrv, iAQa). Complete VC-PPI reduces pointwise MSE from 1.3073 to 0.6079 and improves human preference agreement from 33/50 to 38/50. Additional experiments demonstrate strong alignment between Click-Through Intent (CTI) and human intent ratings (R² = 0.855) and self-reported click choices (R² = 0.717), supporting CTI as a human-grounded engagement signal for pre-deployment evaluation, while recognizing that real-world click validation remains valuable.

(3) Evaluation rigor and robustness (78vb). We expanded SFT evaluation to 965 held-out tasks, confirming significant improvements across all four dimensions (p < 0.0001), with both user and commercial utility improving on 74.4% of tasks. Independent DeepSeek, GPT, and Gemini judges further confirm positive overall gains, addressing evaluation-bias concerns.

(4) Strategy generalizability and clarity (78vb). We extended Logical Bridge analysis across additional encoders and Chinese-language settings, with K = 4 remaining near-optimal in 14/19 configurations. We also added a complete worked example, clarified statistical uncertainty, and improved presentation.

Overall, our extensive new experiments directly address the reviewers' central concerns and substantially strengthen NaiAD's demonstrated contributions and empirical rigor. We sincerely thank the Area Chair and reviewers for their valuable feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.