- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
TGNN combines LLM-derived semantic links, CPV tags, heterogeneous GNN learning, distributed graph infrastructure, and a stream-batch deployment pipeline for industrial ad retrieval. The paper evaluates the integrated system with offline comparisons, ablations, and online A/B tests. The main question is which claimed benefits are established for the full system and which are established for its individual components.
Critical strengths
S1. Strong offline results, with an informative ablation. TGNN substantially outperforms HAN on both industrial datasets (AUC 83.24 vs. 64.69 on A, and 85.29 vs. 71.42 on B). In the reported w/o-SE ablation, AUC falls to 65.81/69.84—roughly HAN’s level, slightly higher on A and lower on B. This suggests that the removed semantic components account for much of the offline gain.
S2. Encouraging results from production deployment. The randomized A/B tests complement the offline results. With equally sized buckets and the same retrieval quota, the paper reports overall GMV gains of 3.43% in social feeds and 2.77% in short video, with larger gains for new ads (4.80% and 3.74%).
Critical weaknesses
W1. The cold-start claim needs a more direct test. The w/o-SE result makes a good case that semantic enrichment helps overall, but the aggregate results do not show how much newly launched ads benefit. The paper does not clearly state whether w/o-SE removes both I2I and I2CPV edges, and it does not isolate the LLM encoder's gain over a simpler content encoder. Results grouped by ad age or interaction count, together with a matched encoder comparison, would make the cold-start claim more convincing.
W2. The online gains are encouraging, but more detail are needed to assess them. The paper says the A/B test lasted “several weeks” and gives point estimates, but not the exact duration, sample size, randomization unit, or uncertainty around the GMV gains. The new-ad gains come from deploying the full system, therefore a component-level test would be needed to attribute them specifically to semantic enrichment or warm-start.
W3. The theoretical bound appears inconsistent with the model definition. Eq. 5 sets r_a^1 = s_a, but Proposition 2.1 assumes ||r_a^l|| <= rho^l ||s_a|| for rho < 1. At l = 1, this cannot hold for a nonzero s_a, leaving the stated bound unsupported under the current indexing. For Eq. 11, the differentiated variable and the treatment of cosine normalization and batch averaging are not specified clearly. The proposed link between that gradient weight and node degree also lacks supporting evidence.
W4. Repeated full-scale industrial training may be costly, but the paper does not state whether Tables 1, 5, and 6 show single runs or averages, what randomness was controlled, or whether variability estimates are available, especially for the public benchmarks. Evaluation candidate and negative-set construction and public-benchmark settings also remain unclear. On Dataset A, TGNN's recall R@1(0.1095) exceeds its precision P@1 (0.1082), which is unexpected if both metrics use conventional definitions on the same query population. The metric formulas, populations, and aggregation rules are needed to interpret these entries and reproduce the public results.
W5. The system's cost is unclear. The paper targets a latency below 20 ms per request, but its sub-10-ms TP-999 result covers only two-hop graph sampling, leaving total request latency unknown. It describes when embeddings are refreshed, yet does not quantify the compute and storage needed to encode new ads or maintain semantic IDs, codebooks, and trie-based I2I links. This makes TGNN's operational cost difficult to assess.
W6. Inaccurate comparisons. For example, Table 3 listed LiGNN, GiGL, and EmbedX as homogeneous, but their papers describe heterogeneous support. https://dl.acm.org/doi/pdf/10.1145/3637528.3671566, https://dl.acm.org/doi/epdf/10.1145/3711896.3737229, and https://www.vldb.org/pvldb/vol16/p3543-shi.pdf.
Questions for authors
Q1: What exactly does w/o-SE remove both I2I and I2CPV edges, and any node features? Are there existing results by ad age or interaction count that show how much semantic enrichment helps newly launched ads?
Q2. How are Precision@1 and Recall@1 defined and aggregated, and how are evaluation candidates selected? In particular, why does Recall@1 exceed Precision@1 on Dataset A?
Q3. How much do semantic links improve outcomes for new ads beyond using their existing content features alone? In the equal-quota A/B test, how did impressions and GMV change for established ads with similar content? Were the new-ad gains accompanied by reduced exposure or GMV for those ads?
Q4. What is TGNN’s incremental cost relative to the deployed system, including complete request latency and the compute and storage needed for LLM encoding, semantic link maintenance, and training?
2 comments
Response to Reviewer 9ot8 (part 2)
W5 / Q4 — Incremental cost and end-to-end latency.
Two-hop graph sampling stays under 10 ms TP-999 within our 20 ms serving budget. Incremental operational resources are:
- LLM encoding: Streaming (~ ads/day), capped at 192 GPU-hours/day on a dedicated 8-GPU H20 node, with ~30 GPU-hours/day actually consumed (see reply to Reviewer GKub).
- Semantic IDs and trie: 10 bytes per ad (~1 GB for ads) plus 128 KB of codebooks, requiring under 1 CPU-hour/day.
- Graph storage: The semantic edges account for 1.2% of all edges (~15 GB).
W6 — Table 3 and readability.
The reviewer is right: LiGNN, GiGL, and EmbedX all support heterogeneous graphs. We apologize for this mischaracterization and will correct Table 3, restricting comparisons to dimensions directly verifiable from each paper. We will also consolidate mathematical notation across sections and move the missing experimental details into the main text to improve readability.
Response to Reviewer 9ot8 (part 1)
We thank the reviewer for the thorough and precise comments.
W1 / Q1 / Q3 — Cold-start attribution.
w/o-SEremoves both semantic edge families (I2I and I2CPV, leaving CPV nodes disconnected) while retaining all node features. Because sa is never used as a node feature (only to construct edges), thew/o-SEdrop (−17.43 / −15.45 AUC on Datasets A / B) is strictly structural, on top of existing content features.Semantic edges add 1.2×109 edges covering 92% of ads (~13 edges per covered ad across ∣A∣>108); note that our logs do not break this average down by ad age, so we do not have a separate per-new-ad edge count. These edges reduce the sparse-node ratio from 34.0% to 23.7% (I2I) and 11.8% (+CPV). We will add offline results grouped by ad age and interaction count (full TGNN vs.
w/o-SE) in the revised version. We have not run a matched comparison against a smaller text encoder, so we cannot isolate the LLM's specific contribution from edge construction itself; we will state this explicitly.W2 / Q3 — A/B test details and displacement of established ads.
W3 — Corrected Proposition 2.1.
The reviewer is right. Since ra1=sa, the assumption must be written as ∥ral∥2≤ρl−1∥sa∥2. The resulting proof is both simpler and tighter: if ca1:ℓ=cb1:ℓ, Eq. 5 gives raℓ+1−rbℓ+1=sa−sb, which implies ∥sa−sb∥2≤2ρℓ∥s∥2 and cos(sa,sb)=1−2∥s∥22∥sa−sb∥22≥1−2ρ2ℓ. Hence, ℓmin=⌈logρκ/2⌉ guarantees cos(sa,sb)≥1−κ, remaining independent of ∣A∣. We will also explicitly clarify that sa in Eq. 5 denotes the 8-dim RQ-VAE latent (Fig. 1).
W3 — Gradient in Eq. 11.
Let u^i=ui/∥ui∥2 and pij=softmaxj(u^i⊤v^j/τ). Since ui appears only in row i of Eq. 10, the exact gradients are: ∂u^i∂L=−Nτ1((1−pii)v^i−∑j=ipijv^j),∂ui∂L=∥ui∥2I−u^iu^i⊤∂u^i∂L. We agree that the link between this gradient weight and node degree is an unsupported hypothesis, and we will remove that claim from the paper.
W4 / Q2 — Evaluation protocol and R@1 vs. P@1.