- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FaSTra, a multi-trajectory learning framework for reasoning video object segmentation (RVOS). The method aims to address the limitations of single-keyframe propagation, where trajectories initialized from a single frame may become unreliable under occlusion, deformation, and target reappearance. FaSTra employs a temporally diverse frame (TDF) selector to select multiple keyframes, generates candidate trajectories using a lightweight propagator, and uses a complementary trajectory (CT) router to select reliable predictions at each frame. To improve propagation and routing, the authors construct a best-of-trajectories (BoT) teacher to distill complementary knowledge across trajectories and introduce tri-level propagation distillation and utility-aware route distillation. In addition, the authors use FI-GRPO to optimize routing decisions through frame-level rewards. Experiments on multiple video segmentation benchmarks demonstrate improvements in segmentation accuracy and inference efficiency over the compared baselines.
Critical strengths
This paper presents a practical approach to addressing the limitations of single-keyframe propagation by exploiting temporal complementarity among trajectories initialized from different keyframes. The combination of the TDF selector and CT router provides a reasonable mechanism for adapting to changing target conditions throughout a video. The ablation results in Table 3 further support the effectiveness of this design.
FaSTra achieves competitive results on multiple video segmentation benchmarks. Compared with Sa2VA, the proposed method improves J&F by 12.9 and 6.5 points on MeViS and Ref-DAVIS17, respectively. The paper also analyzes inference efficiency in terms of FPS, computational cost, and memory usage. These results demonstrate the practical value of the framework in balancing segmentation accuracy and inference efficiency.
Critical weaknesses
Although FaSTra achieves improved segmentation performance through multi-stage training, the current ablation studies do not sufficiently distinguish the independent contributions of FI-GRPO and BoT knowledge distillation or verify the necessity of the proposed training strategies.
First, Table 4 shows that introducing FI-GRPO in Stage II improves J&F from 78.8 to 79.9, while adding KD in Stage III further increases J&F to 82.0. However, the paper already constructs a GT-guided BoT teacher and uses quality calibration, hard assignment, and ranking losses to train the CT router through utility-aware route distillation. The paper does not provide a controlled comparison using only these supervised losses without FI-GRPO. Therefore, it remains unclear how much additional improvement FI-GRPO provides over supervised route distillation.
Second, the BoT teacher uses GT masks to evaluate candidate trajectories and selects the highest-quality prediction at each frame as the distillation target. Although Table 5 demonstrates the overall effectiveness of tri-level propagation distillation, the paper does not compare BoT-predicted mask distillation with direct GT mask supervision. Since GT masks are already available during training, the additional benefit of using BoT predictions as distillation targets remains unclear. Moreover, tri-level propagation distillation includes prediction-level, state-level, and temporal-level supervision, but Table 5 only reports their combined effect. Thus, it is difficult to distinguish the contributions of mask prediction, internal propagation states, and temporal relation supervision.
Questions for authors
- What is the independent contribution of FI-GRPO compared with supervised route distillation?
Utility-aware route distillation already trains the CT router using GT-guided quality calibration, hard assignment, and ranking losses. Could the authors provide a controlled experiment that trains the CT router using only the route distillation loss (L_{rd}), and compare it with the setting that additionally employs FI-GRPO? The model architecture, TDF selector, candidate trajectories, and training budget should be kept consistent.
In addition, Appendix Eq. (9) and Table 9 include (L_{rd}) in the Stage II training objective, while Table 4 indicates that Stage II does not include KD. Please clarify the actual loss functions used in Stage II and the independent contribution of FI-GRPO to the performance improvement.
- What are the advantages of BoT-predicted mask distillation over direct GT mask supervision?
The BoT teacher uses GT masks to select the highest-quality candidate trajectory at each frame and uses its predicted masks for propagation distillation. Since GT masks are available during training, could the authors compare BoT-predicted masks with GT masks as prediction-level supervision targets under the same propagator architecture and training budget? Keeping the other distillation conditions unchanged would help clarify the additional value of BoT-predicted mask distillation.
- What are the individual contributions of the three levels in tri-level propagation distillation?
Table 5 shows a clear performance improvement after introducing the complete tri-level propagation distillation strategy. However, prediction-level, state-level, and temporal-level distillation are added together. Could the authors provide separate ablation results, such as Prediction-level only, Prediction + State-level, Prediction + Temporal-level, and the complete Tri-Level configuration? These comparisons would help clarify the individual contributions of the three levels and whether state-level and temporal-level distillation provide complementary information.
4 comments
Response to W2 & Q2: Advantages of BoT-Predicted Mask Distillation over Direct GT Supervision
W2 & Q2: Advantages of BoT-Predicted Mask Distillation over Direct GT Supervision:
We will address this question from two perspectives: the difference between the BoT teacher and GT masks in our framework, and additional controlled experiments.
1. Clarification of the role of GT masks in the current framework: Our Tri-level Propagation Distillation and Utility-aware Route Distillation are designed around the BoT teacher. Prediction-level Distillation takes the teacher's predicted logits and supervises the student with soft probability distributions. State-level Distillation requires the teacher's Propagation Memory, Object Pointer, and Object-Presence Score. Temporal-level Distillation requires cross-frame Feature Relations, and Route Distillation requires frame-wise trajectory quality comparisons. GT masks provide only hard segmentation labels and contain none of this information, so they cannot be directly plugged into the existing distillation framework as the teacher.
At each frame, the BoT teacher uses ground truth to select the highest-quality candidate prediction, while retaining the logits and propagation states associated with that prediction. In this way, it is close to ground truth in accuracy and also provides the intermediate teacher information that GT masks lack. This is why we use BoT as the distillation target.
2. Comparison between GT supervision and BoT teacher distillation: To answer your question, under the same propagator architecture and training budget, we use GT masks as the mask supervision target, add the temporal mask-difference supervision that GT can provide, and compare these settings with the complete BoT teacher distillation. Details are provided in Table 9 in the revised main text on page 10.
| Teacher | Mask Supervision | State-level Distillation | Temporal-level Distillation | J | F | J&F |
|---|---|---|---|---|---|---|
| GT Mask | ✓ | 75.2 | 82.5 | 78.8 | ||
| GT Mask | ✓ | ✓† | 75.5 | 82.7 | 79.1 | |
| BoT Teacher | ✓ | ✓ | ✓ | 78.5 | 85.6 | 82.0 |
† denotes that GT masks provide temporal mask-difference supervision but lack soft predictions, propagation-state features, and temporal feature relations.
The results show that the BoT teacher improves J&F from 79.1 to 82.0, clearly outperforming GT mask supervision. Unlike using GT masks as the teacher, the BoT teacher provides soft predictions, propagation states, and frame-wise trajectory quality comparisons, explicitly guiding when and which trajectory to select. Without such information, GT-based supervision cannot fully support route distillation or feature-level knowledge transfer.
Response to W3 & Q3: Contributions of Propagation States and Temporal Relations
W3 & Q3: Contributions of Propagation States and Temporal Relations:
We will address this question from two perspectives: what each level of supervision constrains, and a complete combinatorial ablation.
1. Roles of the three levels of supervision: Prediction-level aligns the frame-wise segmentation predictions of the teacher and the student. State-level transfers the Propagation Memory, Object Pointer, and Object-Presence Score, helping the student maintain effective target representations. Temporal-level constrains temporal changes during propagation through adjacent-frame Mask Changes and cross-frame Feature Relations. The three levels act on the output, the internal states, and the cross-frame relations, respectively.
2. Hierarchical combinatorial ablation: Keeping the model initialization, training data, FI-GRPO, and Route Distillation settings consistent, we change only the composition of Propagation Distillation and cover single levels, pairwise combinations, and the full combination, including the Prediction-level Only, Prediction + State-level, Prediction + Temporal-level, and complete Tri-Level configurations you suggested. Details are provided in Table 10 in the revised main text on page 10.
| Prediction-level Distillation | State-level Distillation | Temporal-level Distillation | J | F | J&F |
|---|---|---|---|---|---|
| ✓ | 76.3 | 83.5 | 79.9 | ||
| ✓ | 76.8 | 83.8 | 80.3 | ||
| ✓ | 76.5 | 83.7 | 80.1 | ||
| ✓ | ✓ | 76.9 | 84.0 | 80.5 | |
| ✓ | ✓ | 77.4 | 84.3 | 80.9 | |
| ✓ | ✓ | 78.0 | 85.1 | 81.6 | |
| ✓ | ✓ | ✓ | 78.5 | 85.6 | 82.0 |
Starting from 79.9 with Prediction-level Only, adding State-level reaches 80.5, adding Temporal-level reaches 80.9, and using all three levels reaches 82.0, showing that both the teacher's internal propagation states and its cross-frame relations provide effective guidance for the student. Removing Prediction-level, State-level, or Temporal-level from the complete configuration lowers J&F by 0.4, 1.1, and 1.5 points, respectively. Removing any component degrades performance, validating the complementary contributions of prediction-level, state-level, and temporal-level knowledge transfer. In particular, the combination of State-level and Temporal-level reaches 81.6, which also answers your question of whether these two levels provide complementary information.
Thank you again for your careful review and constructive suggestions. We have added these clarifications and supplementary experiments to Section 4.3 (Ablation Study) of the main text and corrected the description of the Stage II losses in the supplementary material. In addition, we add ablation studies on the number of propagation trajectories , keyframe selector strategies, the number of MLLM input frames, and the switching penalty in Appendix E.6–E.9 of the revised appendix material, further validating the effectiveness of each component in our multi-trajectory design. We have also improved the visualization of several tables for better clarity and readability and prepared a new supplementary video to provide a more intuitive demonstration of our method and its segmentation results. We hope these additions address your concerns and hope you can reconsider your assessment of our work in light of the supplementary results.
Follow-up on Our Rebuttal and Additional Experimental Evidence
Thank you again for your constructive feedback and careful evaluation of our work.
As the discussion deadline is approaching, we would like to kindly follow up on our response. We have carefully addressed all three of your concerns through additional controlled experiments on FI-GRPO, BoT teacher distillation, and tri-level propagation distillation. We have also substantially revised both the main paper and supplementary material, including new ablation studies, clearer explanations, and a supplementary video.
We have devoted considerable effort to these revisions, and your feedback and support would be truly invaluable to us. We sincerely hope this new evidence has addressed your concerns. If any questions remain, we would be grateful for the opportunity to clarify them before the deadline. If you find our responses convincing, we would deeply appreciate your consideration of revising your rating. Thank you sincerely for your time, understanding, and consideration.
Response to W1 & Q1: Independent Contributions of FI-GRPO and BoT Knowledge Distillation
Thank you for your careful review of the multi-stage training strategy and the BoT distillation design of FaSTra, and for your specific questions on FI-GRPO, GT supervision, and hierarchical distillation. We respond to each of them below.
W1 & Q1: Independent Contributions of FI-GRPO and BoT Knowledge Distillation:
Your question is similar to Reviewer j1ov's W2 & Q2 and Reviewer FFef's W1 & Q1. Here we focus on distinguishing the gains brought by FI-GRPO and BoT Knowledge Distillation, and on clarifying the loss functions actually used in Stage II. We will address it from two perspectives: the setting of each training stage and additional controlled experiments.
1. Clarification of the loss functions in Stage II: We thank you for pointing out the inconsistency between the supplementary material and the main text. Appendix Eq. (9) and Table 9 list Lrd in the Stage II training objective, which is an error. In actual training, Stage II uses the mask BCE loss Lbce together with the FI-GRPO objectives for routing and keyframe selection and the entropy and KL regularization terms, and does not include Lrd. Lrd is used in Stage III together with Propagation Distillation, which is consistent with the original Table 4 (now Table 8 in the revised main text on page 9), where KD is not checked for Stage II. We have corrected the corresponding equation and coefficient table. Details are provided in Table 12 in the revised supplementary material on page 20.
2. Controlled experiments using only BoT Knowledge Distillation: We keep the model architecture, TDF Selector, and candidate trajectories consistent. The training budgets of all variants are comparable. At each stage, we compare continued SFT, BoT Knowledge Distillation only (SFT + KD), and the variant with FI-GRPO. Details are provided in Table 8 in the revised main text on page 9.
Compared with continued SFT at the same stage, FI-GRPO brings a 0.9-point improvement in Stage II (79.0 → 79.9), and BoT Knowledge Distillation brings a 1.3-point improvement in Stage III (80.2 → 81.5), so each has its own independent contribution. Notably, directly applying KD in Stage II yields only 78.9, which is no better than continued SFT. This indicates that the router must first learn to exploit the temporal complementarity of candidate trajectories through FI-GRPO, allowing it to effectively distill knowledge from the BoT teacher. In Stage III, the complete scheme reaches 82.0, outperforming supervised route distillation alone at 81.5, which verifies the necessity of our training strategy.