acceptodds
Under review as a conference paper at ICLR 2027

MORSE: Learning Component-Wise Residual Transformations for Composed Video Retrieval

Abstract

Composed Video Retrieval (CoVR) retrieves a target video given a reference video and a text instruction that describes the change. Existing methods learn composed representations through target alignment, but global target alignment alone is insufficient to distinguish the intended target from visually similar candidates that fail to satisfy the requested modification. How to coordinate changes across the components of the reference representation to capture the instruction remains underexplored. We propose MORSE, which formulates target-feature prediction as context-conditioned corrections over an ordered residual decomposition. This decomposition organizes a reference feature into additive components with cumulative dependencies. Residual Transformation predicts differentiated corrections conditioned on the instruction and each component’s residual context, then aggregates them with the reference feature to form a target-feature prediction. Target-supervised Refinement constrains the cumulative composition of the transformed components using the target decomposition under shared codebooks, connecting overall target discrimination with structural supervision without intermediate change annotations. Compared with recent methods on each benchmark, MORSE achieves relative improvements of 17.81% in R@1 on FineCVR-1M and 10.25% in mAP@50 on TF-CoVR. Further analyses support the effectiveness of context-conditioned corrections and target-side structural supervision in distinguishing instruction-specific changes. Code will be available upon acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Composed Video Retrieval, Residual Quantization, Video-Text Retrieval, Multimodal Retrieval, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SiMHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MORSE decomposes a reference video representation into ordered residual components using residual quantization, then learns instruction-conditioned corrections for each component to predict the target video feature. It further uses target-side prefix alignment to supervise how these component-wise corrections are progressively composed, improving fine-grained composed video retrieval.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The experiments show the effectiveness of the proposed method MORSE.
  3. The method is a natural way to achieve component supervision.
Critical weaknesses
  1. The paper lacks a sufficiently clear explanation of the learned components at different RQ levels. While the overall motivation is understandable—namely, learning modification residuals in a coarse-to-fine manner based on the hierarchical structure induced by residual quantization—it remains unclear what information is actually captured by the components at different levels. Since these components play a central role in the proposed transformation framework, a qualitative visualization or analysis showing their characteristics across levels would substantially improve interpretability and make the method easier to understand.

  2. The paper does not provide a comparison of training cost with existing methods. The proposed pipeline appears more complex than standard end-to-end training, particularly because the codebooks need to be pretrained before the modification learning stage. It would therefore be helpful to report the additional computational and training overhead introduced by codebook preparation, for example in terms of training time, GPU hours, or overall computational cost, and compare it with representative baselines.

Questions for authors

See the weaknesses.

Reviewer_2bxdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MORSE, a framework for Composed Video Retrieval (CoVR) that learns component-wise feature transformations through residual quantization. The method decomposes reference video representations into ordered residual components and predicts instruction-conditioned corrections using codeword, prefix, and residual information. A target-supervised refinement mechanism further constrains the transformed components through cumulative prefix alignment and contrastive learning. Experiments on WebVid-CoVR, FineCVR-1M, and TF-CoVR demonstrate improvements over existing methods, with ablation studies examining the contributions of different transformation inputs and training objectives.

Critical strengths
  1. MORSE achieves competitive results on three CoVR benchmarks, with particularly noticeable improvements on FineCVR-1M. These results demonstrate the practical effectiveness of the proposed framework.

  2. The paper integrates residual decomposition, instruction-conditioned transformations, and target-supervised refinement into a unified framework. The ablation studies provide useful evidence regarding the contributions of different inputs and training objectives.

Critical weaknesses
  1. The proposed framework largely combines established techniques, including residual quantization, feature correction, and contrastive learning. More importantly, it remains unclear why reconstruction-oriented residual decomposition is particularly suitable for modeling semantic modifications in CoVR. The component-wise corrections are predicted in parallel from the original reference decomposition and aggregated through summation, without explicit interactions among the predicted corrections. Therefore, the claimed coordination among components is not sufficiently substantiated. Comparisons with alternative feature decomposition strategies or capacity-matched transformation models are missing.

  2. In Table 5, removing the residual context actually improves FineCVR-1M R@1 from 31.56 to 32.71, questioning the consistent benefit of the proposed prefix–residual conditioning. Moreover, Table 4 shows that adding prefix alignment to the two discrimination objectives provides relatively modest improvements (0.64 R@1 on FineCVR-1M and 0.44 mAP@50 on TF-CoVR). While the overall retrieval results are promising, these findings leave the unique contributions of the proposed structural supervision insufficiently established, especially without comparisons against simpler auxiliary supervision strategies.

Questions for authors
  1. What makes residual quantization particularly suitable for learning semantic transformations in CoVR? Could the authors compare it with alternative feature decomposition strategies or capacity-matched multi-branch transformation models to better isolate the contribution of the ordered residual structure?

  2. How does MORSE explicitly coordinate corrections across different components, given that they are predicted independently from the original reference decomposition? Is there empirical evidence that the learned corrections capture meaningful complementary transformations rather than simply providing an alternative parameterization of a global feature update?

  3. Could the authors further clarify the contribution of the refinement mechanism? In particular, why does removing residual context improve performance on FineCVR-1M, and how does the proposed prefix alignment compare with simpler intermediate feature alignment or auxiliary supervision objectives?

Reviewer_hrqLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the composed video retrieval task and proposes MORSE: it decomposes reference video features via residual quantization into ordered components with cumulative dependencies, learns instruction-conditioned level-wise residual corrections on top of this decomposition, and provides target-side structured supervision for the transformed results through prefix alignment over the shared codebook . MORSE outperforms existing methods on three benchmarks — WebVid-CoVR, FineCVR-1M, and TF-CoVR — and ablation studies validate the effectiveness of the residual-context conditioning and the target-side structured supervision.

Critical strengths
  1. It formalizes the overlooked question of "how to coordinate the modifications of individual components of a reference feature" as context-conditioned correction over an ordered residual decomposition, bringing the structure of RQ into CoVR composition learning.

  2. It comprehensively outperforms recent state-of-the-art methods, including ReTrack, FDCA, and TF-CoVR-Base, on three public benchmarks, and the ablation studies clearly support the contributions of the two core designs.

Critical weaknesses

W1. Insufficient discussion of computational and engineering overhead: The method introduces L levels of RQ codebooks, level-wise predictors, and an additional Refinement branch, but training/inference time, memory footprint, and efficiency comparisons against baselines are missing. The sensitivity of performance to RQ codebook quality is also not adequately discussed, which undermines the reproducibility and practical value of the method.

W2. Unstable gains from the residual input in the ablation: Table 5 shows that removing the residual input actually improves performance on FineCVR-1M. The authors merely attribute this to a "dataset-dependent" effect without any explanation, which weakens the persuasiveness of the core claim that the prefix–residual context is necessary.

W3. Reproducibility: The main text repeatedly refers to Appendix A and Appendix B, but these appendices cannot be found after the main text of the PDF, so key implementation details cannot be verified, this affects the reviewer's assessment of the method's reliability. A discussion of limitations is also missing.

Questions for authors
  1. How are the number of RQ levels L and the codebook size K chosen? How sensitive is the method to them? Additional ablations on these hyperparameters are recommended.

  2. What is the overhead of MORSE relative to the strongest baseline in terms of training time, inference latency, and parameter count?

  3. In Table 5, the "w/o residual" variant outperforms the full model on FineCVR-1M. Can the authors explain why? Does this suggest that the residual input design needs to be adjusted on a per-dataset basis?

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your reviews. Please take some time to carefully examine their rebuttals and engage in the discussion by sharing your thoughts, remaining concerns, or follow-up questions.

As the discussion deadline is approaching, I would appreciate your timely participation to allow sufficient time for further exchanges with the authors.

Thank you for your time and valuable contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.