acceptodds
Withdrawn from ICLR 2027

Grounded Product Understanding in Livestream Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

E-commerce livestreams have emerged as an important channel for presenting products to online consumers, often featuring multiple products with relevant information distributed across different moments. This poses significant challenges for downstream product understanding applications, such as product-centric livestream clipping, where models need to identify the product and its relevant segments for information gathering. However, existing benchmarks for general product understanding typically evaluate product retrieval and temporal localization in isolation, leaving the critical correspondence between product identity and temporal evidence largely unassessed. To address this limitation, we introduce GPUB, a large-scale benchmark comprising 3,000 real-world e-commerce livestream instances with quality-controlled multi-moment temporal annotations and a catalog of over 31K fashion products. GPUB supports three evaluation tasks: given a livestream video and a candidate product set, the main task Grounded Product Understanding (GPrU) requires jointly identifying the product being presented and localizing its supporting moments; Product Retrieval and Product Moment Localization serve as two complementary subtasks. Evaluation of existing multimodal models shows that GPrU remains highly challenging, with the best-performing baseline achieving only 10.13 Pair [email protected]. To narrow the performance gap, we further develop UniPro, a unified product understanding model that derives product-aligned and temporally structured representations from shared multimodal encoding, improving Pair [email protected] to 24.58 while achieving 38.81 Joint R@[email protected] on GPrU.

Primary area:
datasets and benchmarks
Keywords:
Livestream video understanding, Grounded product understanding, Product retrieval, Temporal grounding, E-commerce

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9pTXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for joint video-to-product retrieval and temporal grounding. It supports product retrieval, temporal grounding, and their joint evaluation, with the aim of capturing the relationship between product identity and its temporal evidence in a video. The benchmark is challenging, with the evaluated open-source models generally achieving low performance. The paper also introduces UniPro, whose architecture and training approach substantially improve performance on these tasks.

Critical strengths
  1. The benchmark poses a challenging evaluation setting, as reflected in the generally low performance of the evaluated open-source models.
  2. UniPro substantially improves performance over the evaluated open-source models, demonstrating the effectiveness of the proposed approach on this benchmark.
Critical weaknesses
  1. The paper does not convincingly explain why these tasks need to be combined within a single benchmark. Training separate models for retrieval and grounding could still support the downstream applications discussed in the paper. It remains unclear what essential benefit the joint formulation provides over this alternative.
  2. As I understand the method, temporal information is not incorporated into the video embeddings themselves. Instead, the encoder is frozen, and an additional decoder performs temporal grounding. This does not appear to constitute an end-to-end solution, and it is unclear how the model learns the relationship between retrieval and grounding. This raises the question of whether optimizing the two tasks separately might be more effective.
  3. The evaluated open-source models were released more than a year ago. Including strong close-source models and more recent open-source models would provide a more informative assessment of the benchmark's difficulty and the proposed method's relative performance.
Questions for authors
  1. Could you clarify why retrieval and temporal grounding need to be combined, and what benefits this joint formulation offers over training and optimizing separate models for the two tasks?
  2. Could you provide additional results for more recent open-source models and strong close-source models?
Reviewer_Kpq1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GPUB evaluates whether models can identify products in livestream videos and locate supporting moments. The paper introduces a dataset and UniPro, with comparisons against general-purpose and fine-tuned models and component ablations.

Critical strengths

Joint evaluation of product identity and supporting moments addresses a useful application problem.

Critical weaknesses
  • Annotation quality lacks sufficient evidence, which is central to a benchmark paper. Section 3.2 describes human review and auditing but gives no audit size, error rate, or omission rate. Since candidate moments originate from ASR, visual evidence without clear spoken cues also needs a dedicated check. Missing labels can penalize valid predictions. Independent audit results and temporal-boundary agreement are needed.
  • The method's advantage is not sufficiently established. Four fine-tuned comparisons are included, but a direct product-identification-then-localization baseline is missing. In Table 4, UniPro improves over the strongest fine-tuned model by only 0.56 percentage points on the stricter Pair [email protected]. Comparisons with similar training data and compute are needed to identify the contribution of the joint design.
  • The main results cannot yet be reconciled with the stated evaluation. Figure 6 gives MiniCPM-V localization scores of 8.90 and 14.03 for the two query groups, while the main table reports 8.43 overall. A mean of per-query scores cannot be below both groups. If another aggregation is used, it should be specified. Joint Recall also needs effective sample counts, any run averaging, and reproducible results. These issues leave reservations about the main comparisons.
  • Some settings that affect task difficulty remain unclear. What input retrieves the 29 hard candidates? Are localization diagnostics computed with the correct product supplied in advance? These conditions affect the interpretation of the scores and require explicit evaluation rules.
  • Generally, I think the quality (writing and drawing) of this manuscript is way below the threshold of this conference.
Questions for authors

Pls refer to the comments above.

Reviewer_Q9ueOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GPUB, a large-scale benchmark containing over 31K fashion products and 3K real-world e-commerce livestream instances for the task of Grounded Product Understanding (GPrU). Instead of treating product retrieval and product moment localization as two separate tasks, GPrU jointly combines these two into a single task. The paper also proposes UniPro, a unified model focusing on GPrU. Through extensive experiments, the paper demonstrates that UniPro outperforms baselines on all three GPUB tasks.

Critical strengths
  • The paper introduces a carefully curated large-scale benchmark, GPUB, that contains 3,000 real-world livestream instances and 31,831 fashion products for the grounded product understanding task.
  • The paper has a good structure and is clearly written. The figures are informative and illustrative.
Critical weaknesses
  • Each benchmark instance is centered on a single target product. This setting limits the evaluation of the model’s ability when there are multiple target products appearing in a single video clip.
  • Although the paper retrieves 29 highest-ranked non-target products as hard negatives, it lacks sufficient assessment of the difficulty and target similarity of negative examples. For example, out of all benchmark instances, how many of them genuinely have all 30 targets difficult to distinguish? Or some instances have harder negative groups while others have easier ones. The paper lacks sufficient experiments and analysis of how the model performs under different difficulty levels.
Questions for authors
  • It is stated in Section 3.1 Task Definition that a benchmark instance consists of a livestream video V ‘centered on’ a single target product. Does it mean there is only one product in each video? Could there be other products inside the video but not annotated?
  • Following the previous question, in the real-world scenario, an e-commerce livestream often showcases multiple, even similar, products in a single stream. Since the paper is focusing on the single-product case, could the authors clarify the practicality of the dataset as well as the model ability under the multi-product setting?
  • Could the authors provide qualitative examples of how UniPro performs on GPrU, as well as some failure cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.