acceptodds
Withdrawn from ICLR 2027

SwapCAST: Cross-Video Transition Supervision for Consistent Video Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Consistent Video Retrieval requires selecting a clip that follows a textual instruction while remaining consistent with the preceding visual history. Although residual state prediction provides a viable approach to this task, supervising the target embedding does not explicitly encourage transitions to generalize across visual contexts. We propose SwapCAST, a framework for learning transferable transition codes through cross-video transition swapping. SwapCAST represents the visual history as persistent context and encodes observed changes between clips into transition codes. An instruction-conditioned predictor estimates these codes from the available context, while a shared residual recomposer maps both observed and predicted codes to target embeddings. Our training strategy combines a compatible transition from a donor video with the receiver context and supervises the resulting representation using the receiver target. Donor-target discrimination and wrong-transition constraints encourage the model to preserve receiver-specific information while remaining sensitive to transition content. At inference, retrieval requires only the visual history and instruction, without any donor information. Using frozen CLIP features and calibrated score fusion, SwapCAST achieves retrieval accuracies of 58.53% on YouCook2 and 49.30% on CrossTask under our constructed CVR evaluation protocols. Across three training seeds, adding swapping supervision to an otherwise identical factorized base improves prediction-only retrieval accuracy by 2.39 and 3.86 percentage points, respectively. Controlled donor interventions further demonstrate the importance of transition compatibility and direction, supporting cross-video transition reuse as an effective supervisory signal for consistent retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Consistent Video Retrieval, Video-Text Retrieval, Visual State Transition Modeling, Temporal Consistency, Procedural Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TB88OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a method for consistent video retrieval. The authors try to seperate persistent visual context from procedural transitions. They transfer transition codes across different videos and use the receiver video continuation as supervision. The method also uses donor negatives, wrong-transition constraints, and relational losses. The authors test on YouCook2 and CrossTask with frozen CLIP features. The results show some clear improvements over their factorized baseline. However, the paper lacks fair comparison with CAST and does not fully prove that cross-video transition transfer causes the gains.

Critical strengths
  1. The authors address an interesting problem in video retrieval, learning transferable transitions across different visual contexts makes sense for procedural videos. The motivation is clear and the method has some novelty.

2.The authors provide quite detailed ablation studies. The swap objective improves accuracy by 2.39 and 3.86 percentage points on YouCook2 and CrossTask over the factorized base. The results across three seeds make the improvements more convincing.

3.The authors test compatible, random, wrong, and reversed transitions. The method also needs no donor videos during inference, which is practical.

Critical weaknesses

1 The authors do not rerun CAST under their reconstructed evaluation protocol. They report published CAST results from different candidate pools, so these numbers cannot support a direct comparison. The paper needs a fair experiment to show whether SwapCAST actually outperforms CAST.

  1. The authors do not compare their factorized architecture with a non-factorized residual prediction model. Their main ablations only compare different training objectives within the same architecture. It is unclear whether seperating persistent context and transition codes is actually necessary.

  2. The authors combine transition recomposition and donor-endpoint discrimination in their main swap ablation. Although they test removing donor discrimination, they do not add equivalent hard-negative supervision to the base model without swapping.

  3. The authors only evaluate on YouCook2 and CrossTask. Each retrieval query contains only ten candidates, which is quite limited for testing retrieval performance. The paper lacks larger candidate pools, unseen task evaluations, and cross-dataset generalization experiments.

5 The observed transition encoder takes both source and target visual embeddings, which allows the transition code to retain donor-specific appearance and context information. The relational losses and donor negatives do not explicitly remove this information, so the claimed separation between persistent context and transition is not so guaranteed.

Questions for authors

Address the weaknesses.

Reviewer_bog6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies consistent video retrieval and give a model SwapCAST. Given a video history and a text instruction, a model must find a clip that follows the instruction and fits the earlier video. SwapCAST separate the video context from the change between steps. During training, it takes a change from another video with similar steps and combines it with the current video's context. The target is still the correct next clip from the current video. At test time, the model only need the video history and the instruction; it does not need another video. The author test the method on YouCook2 and CrossTask. In their controlled tests, adding the cross-video training objective improves retrieval accuracy over their base model.

Critical strengths

1, Strong results with variation across runs. SwapCAST achieves good retrieval accuracy on both YouCook2 and CrossTask. The paper reports results includes the standard deviation. This make the results more informative than reporting a single run. The paper tests several training losses and their combinations. The results help show how the swap, wrong-transition, and relation losses affect performance.

2, An interesting swap design. The model uses a transition from another video while keeping the current video's context and target. This is an interesting way to encourage the model to learn reusable changes across videos. It improves prediction-only retrieval in the authors' controlled experiments.

Critical weaknesses

1, Missing a fair comparison with existing methods. In Table 1, SwapCAST and the baselines, including CAST, use different candidate pools and evaluation protocols. This makes it hard to tell whether the higher accuracy comes from the method or an easier evaluation setting. The authors clearly state this difference, but a direct comparison with CAST under the same setting is still needed to show the advantage of SwapCAST.

2, Figure 2 needs improvement. Several elements overlap, making the figure hard to follow. Only L_pred is labeled, while the other losses are not shown. Some variables, such as v_{d,pred}, do not seem to be defined in the main text. The three symbols under “Recomposed embeddings” also appear inconsistent with the equations: the last one should likely be \tilde{x}_i\ rather than another \hat{x}_i.

Questions for authors

What happens if the observed-transition encoder takes only the feature difference [x_i-a_i] instead of [a_i; x_i; a_i-x_i]?

Reviewer_E5sVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SwapCAST, which trains models to capture transferable procedural transitions by extracting a transition representation from a compatible video and recombining it with the visual context of a receiver video. The recomposed representation is supervised using the receiver video’s continuation, while inference requires only the receiver’s history and a text instruction—no donor video is needed.

Critical strengths

Reusing cross-video transitions as training supervision. The method does not merely ask the model to predict the target paired with the current sample. It also uses compatible state changes from other videos to test whether the model can predict the receiver video’s continuation while preserving its context.

Critical weaknesses

The main concerns are with the experimental evaluation:

  1. The experiments are not sufficiently comprehensive, and the individual contribution of each module is unclear.
  2. Evaluation on only two somewhat similar datasets provides limited evidence of generalizability.
  3. SwapCAST is not compared with CAST under matched evaluation conditions.
Questions for authors

The experimental analysis is insufficient, and some of the conclusions seem overstated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.