acceptodds
Desk rejected at ICLR 2027

Evidence-Guided Constraint Reliability for Partially Relevant Video Retrieval

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Partially relevant video retrieval learns local query-video correspondence from video-level supervision. Unpaired relations induce uncertain training constraints because pairing labels leave local compatibility unresolved. Two challenges complicate constraint calibration: early optimization relies on immature relation evidence, and later responses conflate relation difficulty with positive-like local compatibility. These observations suggest that constraint calibration should evolve with relation maturity, using retriever-independent semantic evidence early and difficulty-conditioned task evidence as relations become informative. We propose Gradual Reliability-Aware Constraint Evolution (GRACE) to instantiate this principle. GRACE coordinates semantic prior evidence and evolving task evidence within a unified contrastive objective. Semantic Evidence Allocation Recalibration (SEAR) reuses each candidate video's paired queries to recalibrate candidate allocation through set-level semantic support. Positive-reference Attribution for Intervention Refinement (PAIR) compares paired and unpaired relations at similar difficulty to obtain relation-specific evidence. It then uses evidence above the conditional reference mean to refine constraint strength continuously. Together, the two mechanisms adjust candidate allocation and effective training pressure as task relations develop. Experiments on Charades-STA, ActivityNet Captions, and TVR demonstrate consistent improvements across retrieval settings and support the complementary roles of semantic prior and task-derived evidence.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Partially Relevant Video Retrieval, Text-Video Retrieval, Cross-Modal Retrieval, Weakly Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rzbmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies partially relevant video retrieval (PRVR), where training data provide query–video pairings but do not identify the relevant content inside each video. The authors propose GRACE, which uses evidence during training to calibrate the constraints on unpaired candidates. SEAR treats the queries paired with each candidate video as semantic prior evidence and uses them to recalibrate candidate allocation. PAIR compares paired and unpaired relations at similar difficulty and uses positive-reference evidence to adjust their constraint weights. Experiments on Charades-STA, ActivityNet Captions, and TVR show higher Rsum on all three benchmarks.

Critical strengths
  1. The method has a clear motivation and division of roles. The paper argues that local matching scores are unreliable early in training, so using them to adjust unpaired candidates can reinforce the model’s initial preferences. Later, a high score may reflect candidate difficulty rather than local relevance. GRACE uses SEAR to reallocate candidate weights from semantic priors early in training, then uses PAIR to adjust constraint strength from task-derived evidence after controlling for difficulty. Each module improves the baseline when used alone.
  2. The mechanism analysis is consistent with the proposed roles. The overlap between SEAR’s semantic evidence and the retriever’s candidate ranking grows during training, which supports using semantic priors early and reducing their influence later. For PAIR, paired relations still show stronger local specificity after difficulty is controlled, suggesting that the task evidence contains information beyond candidate difficulty.
Critical weaknesses
  1. PAIR’s core statistic lacks direct validation against local relevance. Appendix C.3 says that the responsibility estimated by PAIR represents distributional compatibility, not the probability that a particular relation is truly relevant. The paper shows paired and unpaired differences after controlling for difficulty, but does not use local relevance labels or human judgments to test whether PAIR assigns relevance correctly to individual candidates.
  2. PAIR’s key adaptive rules are not fully specified. The paper does not give the exact computation of the attenuation strength lambda_a or the rule for deciding when the conditional statistics are too weak to use and identity refinement should be triggered. These choices affect the constraint adjustment and make the method harder to reproduce and the gains harder to attribute.
Questions for authors
  1. Could the authors clarify how the attenuation strength lambda_a is computed in PAIR, and how they decide when the conditional statistics are insufficient to trigger identity refinement?
  2. Could the authors manually check a subset of candidate relations from the existing experiments to assess how well PAIR’s responsibility agrees with actual local relevance?
Reviewer_VK3LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GRACE, a framework for improving partially relevant video retrieval by adjusting how unpaired videos contribute to contrastive learning. GRACE separates negative weight allocation from total negative pressure. SEAR uses semantic evidence from paired training queries to redistribute negative weights, while PAIR uses difficulty-conditioned evidence to reduce penalties for potentially relevant videos. Experiments on Charades-STA, ActivityNet Captions, and TVR show Rsum improvements of 1.3, 1.5, and 2.3 over GMMFormerV2, respectively, without changing the inference model.

Critical strengths
  1. GRACE separates negative weight allocation from total negative pressure. SEAR and PAIR address different sources of uncertainty, and their combination provides a reasonable way to adjust training constraints.
  2. Experiments use matched random seeds and the same retrieval backbone. Ablations show that both components contribute, with their combination outperforming either component alone by 0.5~0.9. The method also preserves the original inference architecture and scoring function.
Critical weaknesses
  1. Although GRACE consistently imrpoves Rsum, its gains over the strongest listed alternatives are only 0.7~1.2. Furthermore, experiments use only GMMFormerV2, making it difficult to assess whether the method generalizes to other retrieval architectures.
  2. PAIR relies on estimated reference distributions, while SEAR depends on the available paired training queries. Their reliability under sparse or unbalanced data is not sufficiently evaluated. Sensitivity to key hyperparameters is also limited, and some details of PAIR's adaptive attenuation and fallback rules remain unclear.
Questions for authors
  1. How are PAIR's reference sets constructed? How is the attenuation parameter λa\lambda_a determined? and what criteria trigger the fallback?
  2. How sensitive are SEAR and PAIR to reference set sizes, the number of paired queries per video, and key hyperparameters such as the bin count and statistics refresh interval?
  3. Have the authors evaluated GRACE with other retrieval backbones to establish its general applicability?
Reviewer_mpPuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRACE (Gradual Reliability-Aware Constraint Evolution), a training framework for partially relevant video retrieval (PRVR) that addresses unreliable negative constraints arising from coarse video-level supervision. GRACE introduces two complementary mechanisms: Semantic Evidence Allocation Recalibration (SEAR), which leverages semantic priors from video-associated queries to redistribute negative sample weights, and Positive-reference Attribution for Intervention Refinement (PAIR), which uses difficulty-conditioned task evidence to identify potentially compatible unpaired relations and attenuate their negative constraints. Both mechanisms are integrated into a unified contrastive learning objective without modifying the inference architecture. Experiments on three PRVR benchmarks demonstrate consistent improvements over the GMMFormerV2 baseline

Critical strengths

1.The paper identifies two meaningful challenges in learning from uncertain negative relations under coarse video-level supervision,named Evidence-Intervention Mismatch and Difficulty-Reliability Entanglement. These challenges provide a clear motivation for distinguishing different sources of evidence during training. 2. GRACE provides a coherent formulation that separates negative gradient allocation from overall constraint strength. The integration of semantic prior evidence and difficulty-conditioned task evidence is supported by theoretical analysis and component ablations, while preserving the original inference architecture

Critical weaknesses
  1. Insufficient comparisons with simpler negative reweighting strategies. PAIR introduces a relatively complex statistical framework involving difficulty-conditioned distributions, mixture proportion estimation, and centered reference attribution. However, the current ablations do not include simpler alternatives, such as direct reweighting based on local specificity or a straightforward combination of difficulty and specificity.
  2. The implementation of gradual reliability-aware evolution primarily relies on a predefined semantic decay schedule and periodically updated task statistics. The experiments lack comparisons across different semantic decay durations, PAIR activation stages, or alternative transition strategies.
  3. GRACE involves several important hyperparameters, particularly the number of specificity bins, hardness kernel bandwidth, and reference-statistics refresh interval in PAIR. However, their influence on retrieval performance and statistical estimation stability is not investigated
Questions for authors

Does the estimated constraint reliability reflect actual local relevance? Figure 3 only shows distributional differences between paired and unpaired relations. Could the authors provide direct evidence that candidates downweighted by PAIR are more likely to contain relevant local content?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.