- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AdsCVR, a benchmark for cross-video comparison of e-commerce advertisements, and AdSeek, which actively invokes visual and audio tools to gather evidence. The method uses a training pipeline consisting of RL, trajectory-rectification SFT, and a second RL stage.
Critical strengths
- I think cross-video comparison of e-commerce advertisements is practically valuable, and AdsCVR provides a useful evaluation resource for this setting.
- The idea of actively combining visual and audio evidence is reasonable, and the tool ablations provide support for this design.
Critical weaknesses
- The gain from trajectory-rectification SFT in Table 3 is quite small, while most of the improvement appears after the second RL stage. I am therefore not fully convinced that the reported gain can be attributed to trajectory rectification without comparable continued-RL or ordinary-success-trajectory SFT baselines.
- Since Gemini generates the questions, answers, and textual clues, and GPT mainly checks the answers against those clues, I worry that errors in the initial video descriptions may carry through the verification process. The paper does not report the scale or error rate of independent human checks against the original videos, so the reliability of the dataset remains uncertain to me.
- The CrossVid results do not seem fully consistent with the claim of robust cross-domain generalization: average accuracy increases from 29.47% to 31.55%, but product identification drops from 62.5% to 45.4% and open-ended QA from 31.2% to 20.4%. I would appreciate an analysis explaining these task-specific declines.
Questions for authors
- I would like to know whether the authors have results from continued RL with a comparable training budget or SFT using ordinary successful trajectories, as these comparisons would help me assess the gains attributable to trajectory rectification itself.
- For the reliability of the dataset answers, I would like to better understand the scale and sampling procedure of the representative-sample checks, whether the original videos were inspected directly, and how many errors or disagreements were found.
- The marked declines on product identification and open-ended QA on CrossVid are somewhat puzzling to me. Do the authors have an error analysis indicating whether these issues mainly arise from evidence acquisition or from subsequent reasoning and answering?
3 comments
Response to Reviewer n5qN(2/2)
3. Clarifying the CrossVid Evaluation Results (weakness3 & question 3)
On CrossVid, AdSeek improves over the baseline by 2.08 percentage points on average, with substantial gains on tasks including BU (+15.4 percentage points), NC (+11.3 percentage points), and CC (+7 percentage points).
Some tasks, such as PI (plot inference) and MSR (multiview spatial reasoning), involve specialized scenarios requiring narrative or spatial reasoning. These differ substantially from e-commerce advertising videos, which are fast-paced, commercially focused, and lack complex plots; this domain difference accounts for the performance declines. Meanwhile, improvements on general tasks such as BU (behavioral understanding), CC (culinary comparison), and NC (narrative comprehension) support the transferability of AdSeek's capabilities. Overall, AdSeek achieves a substantial lead on e-commerce cross-video comparison tasks while also delivering overall performance gains on more general tasks in CrossVid, demonstrating strong generalization capability.
Response to Reviewer n5qN
Dear Reviewer,
As the discussion deadline approaches, we would appreciate it if you could let us know whether our rebuttal has addressed your concerns. If any questions remain, we would be happy to clarify. If our response has addressed your concerns, we would be grateful if you would consider revisiting your initial score in light of the clarifications.
Thank you for your time and thoughtful feedback. We hope you have a pleasant week.
Best regards, The Authors
Response to Reviewer n5qN(1/2)
We thank the reviewer for recognizing the value of cross-video analysis for e-commerce advertising and the effectiveness of our agentic framework. Below, we address your constructive comments in detail.
1. Evaluating the Effectiveness of Trajectory Rectification (weakness1 & question1)
AdSeek uses a three-stage training pipeline. We collect the RL trajectories from Stage 1, filter and rectify them to construct the SFT data for Stage 2, and thereby correct errors in the model's reasoning. Stage 3 then resumes RL exploration. This pipeline yields successive accuracy improvements of 71.27% → 71.53% → 74.30%.
To isolate the contribution of trajectory rectification in Stage 2, we provide the following ablations:
The training settings for each applicable stage are kept consistent with those of AdSeek.
With WeakCheck, Stage 2 performs similarly to AdSeek after Stage 1, while Stage 3 incurs a slight decline of 0.5 percentage points. Because the biases in these trajectories remain unrectified, this result supports the effectiveness of the trajectory filtering and rectification used in AdSeek's Stage 2.
Direct teacher distillation improves accuracy by 0.8 percentage points in Stage 2, but its subsequent Stage 3 gain (+1.7 percentage points) is smaller than AdSeek's (+2.77 percentage points). AdSeek rectifies reasoning errors in trajectories drawn from the student's own distribution, enabling the second RL stage to better realize the model's potential.
Continued RL alone performs similarly to AdSeek after Stage 1, indicating that trajectory rectification provides an advantage for subsequent reinforcement learning in the original three-stage pipeline.
Together, these results support the effectiveness of trajectory rectification within the three-stage training pipeline.
2. Benchmark Reliability (weakness2 & question2)
We took several measures during the construction of AdsCVR to reduce generation errors and potential bias:
We further clarify the human audit conducted during dataset construction. Using a fixed random seed, we drew a random sample of 100 questions without replacement, covering all six evaluation dimensions. Two annotators first answered the sampled questions independently without access to the reference answers. They then recorded (1) reference-answer correctness, (2) answer unambiguity, (3) sufficiency of video evidence, and (4) multi-video dependency , retaining evidence notes for each question. The audit results are as follows:
This type of human spot check has precedent in video-understanding research. VideoSAVi randomly audited 200 QA pairs from its generalization test set and reported video-based answerability, category-assignment correctness, and answer uniqueness of 97.5%, 94.0%, and 98.0%, respectively( VideoSAVi, Appendix A.5.4). The results of manual verification demonstrate that our benchmark is reliable, with the vast majority of questions requiring evidence from multiple videos to be answered.