Evidence-Guided Constraint Reliability for Partially Relevant Video Retrieval

Partially relevant video retrieval learns local query-video correspondence from video-level supervision. Unpaired relations induce uncertain training constraints because pairing labels leave local compatibility unresolved. Two challenges complicate constraint calibration: early optimization relies on immature relation evidence, and later responses conflate relation difficulty with positive-like local compatibility. These observations suggest that constraint calibration should evolve with relation maturity, using retriever-independent semantic evidence early and difficulty-conditioned task evidence as relations become informative. We propose Gradual Reliability-Aware Constraint Evolution (GRACE) to instantiate this principle. GRACE coordinates semantic prior evidence and evolving task evidence within a unified contrastive objective. Semantic Evidence Allocation Recalibration (SEAR) reuses each candidate video's paired queries to recalibrate candidate allocation through set-level semantic support. Positive-reference Attribution for Intervention Refinement (PAIR) compares paired and unpaired relations at similar difficulty to obtain relation-specific evidence. It then uses evidence above the conditional reference mean to refine constraint strength continuously. Together, the two mechanisms adjust candidate allocation and effective training pressure as task relations develop. Experiments on Charades-STA, ActivityNet Captions, and TVR demonstrate consistent improvements across retrieval settings and support the complementary roles of semantic prior and task-derived evidence.