SwapCAST: Cross-Video Transition Supervision for Consistent Video Retrieval

Consistent Video Retrieval requires selecting a clip that follows a textual instruction while remaining consistent with the preceding visual history. Although residual state prediction provides a viable approach to this task, supervising the target embedding does not explicitly encourage transitions to generalize across visual contexts. We propose SwapCAST, a framework for learning transferable transition codes through cross-video transition swapping. SwapCAST represents the visual history as persistent context and encodes observed changes between clips into transition codes. An instruction-conditioned predictor estimates these codes from the available context, while a shared residual recomposer maps both observed and predicted codes to target embeddings. Our training strategy combines a compatible transition from a donor video with the receiver context and supervises the resulting representation using the receiver target. Donor-target discrimination and wrong-transition constraints encourage the model to preserve receiver-specific information while remaining sensitive to transition content. At inference, retrieval requires only the visual history and instruction, without any donor information. Using frozen CLIP features and calibrated score fusion, SwapCAST achieves retrieval accuracies of 58.53% on YouCook2 and 49.30% on CrossTask under our constructed CVR evaluation protocols. Across three training seeds, adding swapping supervision to an otherwise identical factorized base improves prediction-only retrieval accuracy by 2.39 and 3.86 percentage points, respectively. Controlled donor interventions further demonstrate the importance of transition compatibility and direction, supporting cross-video transition reuse as an effective supervisory signal for consistent retrieval.