acceptodds
Withdrawn from ICLR 2027

FaSTra: A Faster Yet Stronger Multi-Trajectory Learning Framework for Reasoning Video Object Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning Video Object Segmentation (RVOS) aims to identify target objects from complex language descriptions and generate temporally consistent pixel-level masks across video frames. Existing methods either rely on fixed keyframes, making propagation vulnerable to temporal variations, or iteratively search for keyframes at the cost of increased inference overhead. We observe that trajectories initialized from different temporal frames exhibit complementary reliability across different temporal regions. Motivated by this observation, we propose \ourmethod, a faster yet stronger multi-trajectory learning framework that reformulates keyframe selection as frame-wise trajectory selection. \ourmethod employs a temporally diverse frame selector to sample a small set of keyframes and generate multiple candidate trajectories with a lightweight propagator, while a complementary trajectory router dynamically selects the most reliable trajectory at each frame. To enhance multi-trajectory learning, we construct a best-of-trajectories teacher and introduce tri-level propagation distillation as well as utility-aware route distillation to strengthen propagation and routing, respectively. We further introduce GRPO-based frame-impact optimization to align frame-level routing decisions with segmentation improvements. \ourmethod-Qwen3-VL-4B outperforms Sa2VA-Qwen3-VL-4B by 12.9 and 6.5 & points on MeViS and Ref-DAVIS17, respectively, while improving frames per second (FPS) by 26.9-33.2% across videos of 60, 120, and 150 frames.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
knowledge distillation, reinforcement learning, reasoning video object segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hHzGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FaSTra, a multi-trajectory learning framework for reasoning video object segmentation (RVOS). The method aims to address the limitations of single-keyframe propagation, where trajectories initialized from a single frame may become unreliable under occlusion, deformation, and target reappearance. FaSTra employs a temporally diverse frame (TDF) selector to select multiple keyframes, generates candidate trajectories using a lightweight propagator, and uses a complementary trajectory (CT) router to select reliable predictions at each frame. To improve propagation and routing, the authors construct a best-of-trajectories (BoT) teacher to distill complementary knowledge across trajectories and introduce tri-level propagation distillation and utility-aware route distillation. In addition, the authors use FI-GRPO to optimize routing decisions through frame-level rewards. Experiments on multiple video segmentation benchmarks demonstrate improvements in segmentation accuracy and inference efficiency over the compared baselines.

Critical strengths

This paper presents a practical approach to addressing the limitations of single-keyframe propagation by exploiting temporal complementarity among trajectories initialized from different keyframes. The combination of the TDF selector and CT router provides a reasonable mechanism for adapting to changing target conditions throughout a video. The ablation results in Table 3 further support the effectiveness of this design.

FaSTra achieves competitive results on multiple video segmentation benchmarks. Compared with Sa2VA, the proposed method improves J&F by 12.9 and 6.5 points on MeViS and Ref-DAVIS17, respectively. The paper also analyzes inference efficiency in terms of FPS, computational cost, and memory usage. These results demonstrate the practical value of the framework in balancing segmentation accuracy and inference efficiency.

Critical weaknesses

Although FaSTra achieves improved segmentation performance through multi-stage training, the current ablation studies do not sufficiently distinguish the independent contributions of FI-GRPO and BoT knowledge distillation or verify the necessity of the proposed training strategies.

First, Table 4 shows that introducing FI-GRPO in Stage II improves J&F from 78.8 to 79.9, while adding KD in Stage III further increases J&F to 82.0. However, the paper already constructs a GT-guided BoT teacher and uses quality calibration, hard assignment, and ranking losses to train the CT router through utility-aware route distillation. The paper does not provide a controlled comparison using only these supervised losses without FI-GRPO. Therefore, it remains unclear how much additional improvement FI-GRPO provides over supervised route distillation.

Second, the BoT teacher uses GT masks to evaluate candidate trajectories and selects the highest-quality prediction at each frame as the distillation target. Although Table 5 demonstrates the overall effectiveness of tri-level propagation distillation, the paper does not compare BoT-predicted mask distillation with direct GT mask supervision. Since GT masks are already available during training, the additional benefit of using BoT predictions as distillation targets remains unclear. Moreover, tri-level propagation distillation includes prediction-level, state-level, and temporal-level supervision, but Table 5 only reports their combined effect. Thus, it is difficult to distinguish the contributions of mask prediction, internal propagation states, and temporal relation supervision.

Questions for authors
  1. What is the independent contribution of FI-GRPO compared with supervised route distillation?

Utility-aware route distillation already trains the CT router using GT-guided quality calibration, hard assignment, and ranking losses. Could the authors provide a controlled experiment that trains the CT router using only the route distillation loss (L_{rd}), and compare it with the setting that additionally employs FI-GRPO? The model architecture, TDF selector, candidate trajectories, and training budget should be kept consistent.

In addition, Appendix Eq. (9) and Table 9 include (L_{rd}) in the Stage II training objective, while Table 4 indicates that Stage II does not include KD. Please clarify the actual loss functions used in Stage II and the independent contribution of FI-GRPO to the performance improvement.

  1. What are the advantages of BoT-predicted mask distillation over direct GT mask supervision?

The BoT teacher uses GT masks to select the highest-quality candidate trajectory at each frame and uses its predicted masks for propagation distillation. Since GT masks are available during training, could the authors compare BoT-predicted masks with GT masks as prediction-level supervision targets under the same propagator architecture and training budget? Keeping the other distillation conditions unchanged would help clarify the additional value of BoT-predicted mask distillation.

  1. What are the individual contributions of the three levels in tri-level propagation distillation?

Table 5 shows a clear performance improvement after introducing the complete tri-level propagation distillation strategy. However, prediction-level, state-level, and temporal-level distillation are added together. Could the authors provide separate ablation results, such as Prediction-level only, Prediction + State-level, Prediction + Temporal-level, and the complete Tri-Level configuration? These comparisons would help clarify the individual contributions of the three levels and whether state-level and temporal-level distillation provide complementary information.

4 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Independent Contributions of FI-GRPO and BoT Knowledge Distillation

Thank you for your careful review of the multi-stage training strategy and the BoT distillation design of FaSTra, and for your specific questions on FI-GRPO, GT supervision, and hierarchical distillation. We respond to each of them below.

W1 & Q1: Independent Contributions of FI-GRPO and BoT Knowledge Distillation:

Your question is similar to Reviewer j1ov's W2 & Q2 and Reviewer FFef's W1 & Q1. Here we focus on distinguishing the gains brought by FI-GRPO and BoT Knowledge Distillation, and on clarifying the loss functions actually used in Stage II. We will address it from two perspectives: the setting of each training stage and additional controlled experiments.

1. Clarification of the loss functions in Stage II: We thank you for pointing out the inconsistency between the supplementary material and the main text. Appendix Eq. (9) and Table 9 list Lrd\mathcal{L}_{\mathrm{rd}} in the Stage II training objective, which is an error. In actual training, Stage II uses the mask BCE loss Lbce\mathcal{L}_{\mathrm{bce}} together with the FI-GRPO objectives for routing and keyframe selection and the entropy and KL regularization terms, and does not include Lrd\mathcal{L}_{\mathrm{rd}}. Lrd\mathcal{L}_{\mathrm{rd}} is used in Stage III together with Propagation Distillation, which is consistent with the original Table 4 (now Table 8 in the revised main text on page 9), where KD is not checked for Stage II. We have corrected the corresponding equation and coefficient table. Details are provided in Table 12 in the revised supplementary material on page 20.

2. Controlled experiments using only BoT Knowledge Distillation: We keep the model architecture, TDF Selector, and candidate trajectories consistent. The training budgets of all variants are comparable. At each stage, we compare continued SFT, BoT Knowledge Distillation only (SFT + KD), and the variant with FI-GRPO. Details are provided in Table 8 in the revised main text on page 9.

StageVariantSFTFI-GRPOKDJFJ&F
BaselineBase71.179.975.5
Stage ISFT✓75.282.578.8
Stage IISFT only✓75.382.779.0
Stage IISFT + FI-GRPO✓✓76.483.479.9
Stage IISFT + KD✓✓75.582.278.9
Stage IIISFT only✓76.683.780.2
Stage IIISFT + KD✓✓77.985.181.5
Stage IIISFT + FI-GRPO + KD✓✓✓78.585.682.0

Compared with continued SFT at the same stage, FI-GRPO brings a 0.9-point improvement in Stage II (79.0 → 79.9), and BoT Knowledge Distillation brings a 1.3-point improvement in Stage III (80.2 → 81.5), so each has its own independent contribution. Notably, directly applying KD in Stage II yields only 78.9, which is no better than continued SFT. This indicates that the router must first learn to exploit the temporal complementarity of candidate trajectories through FI-GRPO, allowing it to effectively distill knowledge from the BoT teacher. In Stage III, the complete scheme reaches 82.0, outperforming supervised route distillation alone at 81.5, which verifies the necessity of our training strategy.

AuthorsOpenReview ↗

Response to W2 & Q2: Advantages of BoT-Predicted Mask Distillation over Direct GT Supervision

W2 & Q2: Advantages of BoT-Predicted Mask Distillation over Direct GT Supervision:

We will address this question from two perspectives: the difference between the BoT teacher and GT masks in our framework, and additional controlled experiments.

1. Clarification of the role of GT masks in the current framework: Our Tri-level Propagation Distillation and Utility-aware Route Distillation are designed around the BoT teacher. Prediction-level Distillation takes the teacher's predicted logits and supervises the student with soft probability distributions. State-level Distillation requires the teacher's Propagation Memory, Object Pointer, and Object-Presence Score. Temporal-level Distillation requires cross-frame Feature Relations, and Route Distillation requires frame-wise trajectory quality comparisons. GT masks provide only hard segmentation labels and contain none of this information, so they cannot be directly plugged into the existing distillation framework as the teacher.

At each frame, the BoT teacher uses ground truth to select the highest-quality candidate prediction, while retaining the logits and propagation states associated with that prediction. In this way, it is close to ground truth in accuracy and also provides the intermediate teacher information that GT masks lack. This is why we use BoT as the distillation target.

2. Comparison between GT supervision and BoT teacher distillation: To answer your question, under the same propagator architecture and training budget, we use GT masks as the mask supervision target, add the temporal mask-difference supervision that GT can provide, and compare these settings with the complete BoT teacher distillation. Details are provided in Table 9 in the revised main text on page 10.

TeacherMask SupervisionState-level DistillationTemporal-level DistillationJFJ&F
GT Mask✓75.282.578.8
GT Mask✓✓†75.582.779.1
BoT Teacher✓✓✓78.585.682.0

† denotes that GT masks provide temporal mask-difference supervision but lack soft predictions, propagation-state features, and temporal feature relations.

The results show that the BoT teacher improves J&F from 79.1 to 82.0, clearly outperforming GT mask supervision. Unlike using GT masks as the teacher, the BoT teacher provides soft predictions, propagation states, and frame-wise trajectory quality comparisons, explicitly guiding when and which trajectory to select. Without such information, GT-based supervision cannot fully support route distillation or feature-level knowledge transfer.

AuthorsOpenReview ↗

Response to W3 & Q3: Contributions of Propagation States and Temporal Relations

W3 & Q3: Contributions of Propagation States and Temporal Relations:

We will address this question from two perspectives: what each level of supervision constrains, and a complete combinatorial ablation.

1. Roles of the three levels of supervision: Prediction-level aligns the frame-wise segmentation predictions of the teacher and the student. State-level transfers the Propagation Memory, Object Pointer, and Object-Presence Score, helping the student maintain effective target representations. Temporal-level constrains temporal changes during propagation through adjacent-frame Mask Changes and cross-frame Feature Relations. The three levels act on the output, the internal states, and the cross-frame relations, respectively.

2. Hierarchical combinatorial ablation: Keeping the model initialization, training data, FI-GRPO, and Route Distillation settings consistent, we change only the composition of Propagation Distillation and cover single levels, pairwise combinations, and the full combination, including the Prediction-level Only, Prediction + State-level, Prediction + Temporal-level, and complete Tri-Level configurations you suggested. Details are provided in Table 10 in the revised main text on page 10.

Prediction-level DistillationState-level DistillationTemporal-level DistillationJFJ&F
✓76.383.579.9
✓76.883.880.3
✓76.583.780.1
✓✓76.984.080.5
✓✓77.484.380.9
✓✓78.085.181.6
✓✓✓78.585.682.0

Starting from 79.9 with Prediction-level Only, adding State-level reaches 80.5, adding Temporal-level reaches 80.9, and using all three levels reaches 82.0, showing that both the teacher's internal propagation states and its cross-frame relations provide effective guidance for the student. Removing Prediction-level, State-level, or Temporal-level from the complete configuration lowers J&F by 0.4, 1.1, and 1.5 points, respectively. Removing any component degrades performance, validating the complementary contributions of prediction-level, state-level, and temporal-level knowledge transfer. In particular, the combination of State-level and Temporal-level reaches 81.6, which also answers your question of whether these two levels provide complementary information.

Thank you again for your careful review and constructive suggestions. We have added these clarifications and supplementary experiments to Section 4.3 (Ablation Study) of the main text and corrected the description of the Stage II losses in the supplementary material. In addition, we add ablation studies on the number of propagation trajectories KK, keyframe selector strategies, the number of MLLM input frames, and the switching penalty in Appendix E.6–E.9 of the revised appendix material, further validating the effectiveness of each component in our multi-trajectory design. We have also improved the visualization of several tables for better clarity and readability and prepared a new supplementary video to provide a more intuitive demonstration of our method and its segmentation results. We hope these additions address your concerns and hope you can reconsider your assessment of our work in light of the supplementary results.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experimental Evidence

Thank you again for your constructive feedback and careful evaluation of our work.

As the discussion deadline is approaching, we would like to kindly follow up on our response. We have carefully addressed all three of your concerns through additional controlled experiments on FI-GRPO, BoT teacher distillation, and tri-level propagation distillation. We have also substantially revised both the main paper and supplementary material, including new ablation studies, clearer explanations, and a supplementary video.

We have devoted considerable effort to these revisions, and your feedback and support would be truly invaluable to us. We sincerely hope this new evidence has addressed your concerns. If any questions remain, we would be grateful for the opportunity to clarify them before the deadline. If you find our responses convincing, we would deeply appreciate your consideration of revising your rating. Thank you sincerely for your time, understanding, and consideration.

Reviewer_FFefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FaSTra, a multi-trajectory learning framework for reasoning video object segmentation that integrates SAM2-based segmentation, temporally diverse keyframe selection, and complementary trajectory routing. I find the idea of generating multiple candidate trajectories and dynamically selecting among them particularly interesting. The combination of knowledge distillation and FI-GRPO is also a promising aspect of the framework. Overall, the paper is clearly written and well organized, and the experimental results demonstrate strong performance across several benchmarks. However, although the training-stage ablation shows improvements after introducing FI-GRPO, it does not establish whether these gains justify the additional complexity compared with simpler supervised routing approaches under matched training conditions.

Critical strengths
  1. I find the idea of selecting among multiple candidate trajectories particularly interesting. By dynamically choosing different trajectories at different frames, the framework can take advantage of their complementary strengths and improve segmentation reliability.
  2. The integration of knowledge distillation and FI-GRPO is another promising aspect of this work. These methods aim to improve propagation quality and routing decisions, respectively, providing a more comprehensive training strategy.
  3. The paper reports competitive results across multiple video object segmentation benchmarks. The improvements in both segmentation accuracy and inference efficiency provide encouraging evidence of the effectiveness of the proposed framework.
Critical weaknesses
  1. Table 4 shows performance improvements after introducing FI-GRPO. However, it remains unclear whether these gains are due to FI-GRPO itself or differences in training conditions.
  2. Since FI-GRPO is one of the main contributions, it should provide a comparison with a simpler supervised routing baseline under matched training budgets and supervision settings, including route-distillation losses. This would help clarify the additional benefit of FI-GRPO.
Questions for authors

Table 4 shows an improvement after introducing FI-GRPO. Could you clarify whether the comparison controls for the training budget and route-distillation losses?

3 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Contribution of FI-GRPO Under Matched Training Conditions

Thank you for recognizing our multi-trajectory selection, dynamic routing, and training design, and for your suggestions on the training conditions and the supervised routing baseline. We address these two issues below.

W1 & Q1: Contribution of FI-GRPO Under Matched Training Conditions:

Your question is similar to Reviewer j1ov's W2 & Q2 and Reviewer hHzG's W1 & Q1, which also concern the comparison between FI-GRPO and supervised training. Here we focus on whether the gain of FI-GRPO comes from differences in the training budget or the route-distillation losses. We will address it from two perspectives: the original training setting and an additional comparison in Stage II.

1. Clarification of the original training setting: We adopt a three-stage training strategy: SFT → FI-GRPO → FI-GRPO + KD. The original Table 4 (now Table 8 in the revised main text on page 9) presents the cumulative gains across training stages, rather than an independent ablation with strictly matched training budgets. Stage II performs 1,500 updates, in which FI-GRPO optimizes decisions according to the actual segmentation gains produced by routing actions. Stage III further performs 2,000 Propagation Distillation updates and 2,000 Route Distillation updates.

2. Comparison under a comparable training budget: We keep the training data, TDF Selector, and candidate trajectories identical, and add a continued-SFT variant at each stage to examine the effect. Details are provided in Table 8 in the revised main text on page 9.

StageVariantSFTFI-GRPOKDJFJ&F
BaselineBase71.179.975.5
Stage ISFT✓75.282.578.8
Stage IISFT only✓75.382.779.0
Stage IISFT + FI-GRPO✓✓76.483.479.9
Stage IISFT + KD✓✓75.582.278.9
Stage IIISFT only✓76.683.780.2
Stage IIISFT + KD✓✓77.985.181.5
Stage IIISFT + FI-GRPO + KD✓✓✓78.585.682.0

As shown in Table 8 in the revised main text on page 9, SFT largely saturates after Stage I: continued SFT in Stage II improves J&F only from 78.8 to 79.0, indicating that increasing the training budget alone brings little improvement. Under a comparable budget, SFT + FI-GRPO reaches 79.9, 0.9 points higher than continued SFT. Therefore, the improvement in Stage II mainly comes from FI-GRPO optimizing frame-wise routing decisions, rather than from differences in the training budget or the route-distillation losses.

AuthorsOpenReview ↗

Response to W2: Comparison with a Simpler Supervised Routing Baseline

W2: Comparison with a Simpler Supervised Routing Baseline:

The training budget and the original training setting have been clarified in W1 & Q1. Here we focus on the comparison with a supervised routing baseline under the same supervision settings, including the Route Distillation losses. Details are provided in Table 8 in the revised main text on page 9, and the table is not repeated here. We will address it from two perspectives: the setting of the supervised routing baseline and the comparison results.

1. Setting of the supervised routing baseline: SFT + KD without FI-GRPO in Table 8 in the revised main text on page 9 corresponds to the simpler supervised routing baseline you suggested. It trains the router through Route Distillation using the trajectory quality, selection, and ranking information provided by the BoT teacher, together with Propagation Distillation, and relies only on supervised signals. Its Route Distillation and Propagation Distillation losses are identical to those of our complete scheme. The only difference is whether FI-GRPO is used, and the training budgets are comparable.

2. Comparison between our training scheme and the supervised baseline: In Stage II, directly applying KD achieves only 78.9 J&F, lower than 79.9 for FI-GRPO. This suggests that limited trajectory complementarity restricts effective knowledge transfer from the BoT teacher when the router is trained only with supervised signals. In Stage III, the supervised baseline reaches 81.5, while our complete scheme, which combines FI-GRPO with KD, achieves 82.0. Under the same supervision settings, our training scheme consistently outperforms the simpler supervised routing baseline, showing that optimizing routing decisions according to their actual segmentation gains is more effective than just relying on supervised route distillation alone.

Thank you again for your careful review and constructive suggestions. We have added these clarifications and controlled experiments to Section 4.3 (Ablation Study) of the main text to better explain the settings of each training stage and the role of FI-GRPO. In addition, we have included ablation studies on the number of propagation trajectories KK, keyframe selector strategies, the number of MLLM input frames, and the switching penalty in Appendix E.6–E.9 of the revised appendix, further validating the key design choices of our framework. We have also improved the visualization of several tables for better clarity and readability and prepared a new supplementary video to provide a more intuitive demonstration of our method and its segmentation results. We hope these revisions and additional materials adequately address your concerns, and we sincerely hope you will reconsider your assessment of our work in light of the new evidence.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Evidence for FI-GRPO

Thank you again for your constructive feedback and recognition of our contributions.

As the discussion deadline is approaching, we would like to kindly follow up on our response. We have addressed your main concerns through additional controlled experiments under comparable training budgets and supervision settings, demonstrating the benefits of FI-GRPO over simpler supervised alternatives. We have also substantially revised both the main paper, appendix, and a supplementary video, adding further ablation studies as well as clearer explanations.

We sincerely hope these efforts have addressed your concerns and strengthened your confidence in our work. If you have any remaining questions, we would be happy to clarify them. If you find our additional evidence convincing, we would be truly grateful if you could consider revising your rating toward acceptance. Your support would mean a really really great deal to us. Thank you very much for your time, understanding, and consideration.

Reviewer_j1ovOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FaSTra for reasoning video object segmentation. The method generates several mask tracks from different keyframes and learns which track to use at each frame. It trains a lightweight propagator and a track selector using the best predictions among candidate tracks, identified with ground-truth masks. GRPO further optimizes keyframe and track selection. The paper reports accuracy gains across five benchmarks and faster inference than the corresponding Sa2VA baseline.

Critical strengths
  1. An interesting approach to track selection. I like the shift from choosing a keyframe to choosing a sequence of predictions from different tracks. A track may work well in one part of a video and fail elsewhere, so using its reliable portions makes sense. The use of GRPO is also appealing, as it trains the selection policy based on how much its choices affect segmentation quality.

  2. Strong accuracy and speed improvements. The results are impressive. With the same Qwen3-VL-4B backbone, FaSTra improves over Sa2VA by 12.9 J&F on MeViS and 6.5 on Ref-DAVIS17, while also running faster. Achieving both gains despite generating multiple tracks is particularly encouraging. Together with the results across five benchmarks, this makes a strong case for the practical value of the method.

Critical weaknesses
  1. Insufficient isolation of the routing contribution. I find the overall results strong, but I am not fully convinced that the experiments isolate the benefit of frame-wise routing. Table 3 compares the router with nearest-keyframe assignment, but does not compare against selecting one of the same candidate tracks for the entire video under matched training conditions. The best single trajectory is used as a reference during training, but its evaluation performance is not reported alongside the routed output. This makes it difficult to judge how much is gained by switching between tracks rather than producing and selecting better tracks.

  2. Unclear benefit of GRPO beyond supervised training. I appreciate the motivation for GRPO, but the available ground-truth masks already provide direct supervision for candidate quality and track selection. Table 4 shows an improvement from adding GRPO before distillation, but does not include an SFT+KD variant without GRPO. Without this comparison, I find it difficult to tell whether GRPO provides an additional benefit in the final model over a simpler supervised training approach.

Questions for authors
  1. Could you compare frame-wise routing with selecting a single track for the entire video, using the same candidate tracks and trained propagator? This would help clarify how much is gained from switching between tracks.

  2. Could you report an SFT+KD variant without GRPO under a comparable training budget? Table 4 shows that GRPO helps before distillation, but does not isolate its additional benefit when combined with KD.

3 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Insufficient Isolation of the Routing Contribution

Thank you for recognizing our overall experimental results and the motivation for FI-GRPO, and for your specific suggestions on the contribution of trajectory switching and the training comparisons. Your comments help us explain the role of each component more clearly. We address these two issues below.

W1 & Q1: Insufficient Isolation of the Routing Contribution:

We will address this question from two perspectives: the role of Best-Single Trajectory as a reference, and controlled experiments comparing trajectory-selection strategies.

1. Best-Single Trajectory vs. Predicted Single Trajectory: Best-Single Trajectory cannot directly serve as a baseline for comparison with our model because it uses ground-truth information. The appropriate direct comparison is Predicted Single, which does not use ground-truth information. In practice, Best-Single Trajectory and Predicted Single Trajectory are often different, and the model generally cannot determine whether Predicted Single Trajectory is Best-Single Trajectory.

2. Additional controlled experiments on trajectory selection: We fix the MLLM, TDF Selector, Propagator, model weights, and candidate trajectories, changing only the final trajectory-selection strategy to isolate the independent contribution of CT Router. Details are provided in Table 4 in the revised main text on page 8.

Trajectory SelectionJFJ&F
Baseline Model71.179.975.5
Predicted Single74.581.578.0
Nearest-Keyframe73.781.077.4
Best-Single (GT)79.183.981.5
CT Router (Ours)78.585.682.0
Best-of-Trajectories (BoT)80.585.783.1

The gains come from two steps. First, the TDF Selector selects reliable keyframes and produces better candidate trajectories, so Predicted Single improves J&F over the Baseline Model from 75.5 to 78.0. Second, CT Router switches between trajectories according to frame-wise reliability, improving over Predicted Single by a further 4.0 points and over Nearest-Keyframe, which assigns frames by temporal distance only, by 4.6 points. CT Router also outperforms Best-Single (GT), which is selected with ground truth, showing that frame-wise switching achieves gains beyond what any single trajectory can reach. Thus, the overall gain comes from the combined effects of our two designs: selecting better trajectories with the selector and switching trajectories with the router.

Moreover, BoT surpasses Best-Single (GT) by 1.6 points, confirming our motivation that trajectories exhibit temporal complementarity that cannot be fully exploited by a single trajectory. Note that BoT is obtained by comparing the teacher network's predictions with ground truth and cannot be generated directly by the model at inference time.

AuthorsOpenReview ↗

Response to W2 & Q2: Additional Benefit of FI-GRPO After KD

W2 & Q2: Additional Benefit of FI-GRPO After KD:

Your question is similar to Reviewer FFef's W1 & Q1 and Reviewer hHzG's W1 & Q1, which also concern the comparison between FI-GRPO and supervised training. We will address it from two perspectives: the roles of FI-GRPO and BoT Knowledge Distillation, and the training comparison without FI-GRPO.

1. Roles of FI-GRPO and BoT Knowledge Distillation: In Stage II, FI-GRPO optimizes decisions according to the actual segmentation gains produced by routing actions. Stage III then improves propagation and routing through KD. The original Table 4 (now Table 8 in the revised main text on page 9) presents the cumulative gains across training stages, rather than an independent ablation with strictly matched training budgets.

2. Additional training comparison without FI-GRPO: We fix the model initialization, training data, TDF Selector, candidate trajectories, and other settings. At each stage, we compare continued SFT, SFT + KD without FI-GRPO, and the variant with FI-GRPO. Details are provided in Table 8 in the revised main text on page 9.

StageVariantSFTFI-GRPOKDJFJ&F
BaselineBase71.179.975.5
Stage ISFT✓75.282.578.8
Stage IISFT only✓75.382.779.0
Stage IISFT + FI-GRPO✓✓76.483.479.9
Stage IISFT + KD✓✓75.582.278.9
Stage IIISFT only✓76.683.780.2
Stage IIISFT + KD✓✓77.985.181.5
Stage IIISFT + FI-GRPO + KD✓✓✓78.585.682.0

These results show that continued SFT after Stage I yields only marginal gains: in Stage II, it improves J&F only from 78.8 to 79.0. Directly applying KD without FI-GRPO in Stage II achieves only 78.9. This is because the router trained only with SFT in Stage I has not yet learned to exploit the temporal complementarity of candidate trajectories, which limits effective knowledge transfer from the BoT teacher to the student. Consequently, directly applying KD performs worse than FI-GRPO. In contrast, FI-GRPO improves J&F from 79.0 to 79.9 by optimizing frame-wise routing decisions, demonstrating its effectiveness in exploiting trajectory complementarity. In Stage III, SFT + KD reaches 81.5, while the complete scheme combining FI-GRPO and KD achieves 82.0, showing that our training strategy outperforms the supervised alternative without FI-GRPO.

Thank you again for your careful review and constructive suggestions. We have added these clarifications and controlled experiments to Section 4.3 of the main text to explain the contribution of trajectory switching and the role of FI-GRPO in the complete training procedure more clearly. In addition, we add ablation studies on the number of propagation trajectories KK, keyframe selector strategies, the number of MLLM input frames, and the switching penalty in Appendix E.6–E.9 of the revised supplementary material, further validating the trajectory setting and the effectiveness of keyframe selection and frame-wise routing. We have also improved the visualization of several tables for better clarity and readability and prepared a new supplementary video to provide a more intuitive demonstration of our method and its segmentation results.We hope these additions address your concerns and hope you can reconsider your assessment of our work in light of the supplementary results.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experimental Evidence

As the discussion deadline is approaching, we would like to kindly follow up on our response. We have addressed your concerns through additional controlled experiments and substantially revised both the main paper and supplementary material, including new ablation studies, improved explanations, and a supplementary video.

We believe these additions further strengthen the contributions of FaSTra. If you have any remaining concerns, we would be happy to clarify them. If you find our responses convincing, we would sincerely appreciate your consideration of a higher rating in light of these improvements. Thank you again for your positive assessment and constructive feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Request for Reviewer Follow-up Before the Discussion Deadline

Dear Program Chairs, Senior Area Chair, and Area Chair,

We sincerely appreciate your time and efforts in organizing the review process. We submitted our detailed rebuttal four days ago but have not yet received any follow-up responses from the three reviewers. As the discussion deadline is approaching, we are concerned that there may be limited opportunity for further discussion.

The three reviewers raised substantially overlapping concerns, particularly regarding the independent contributions of FI-GRPO and knowledge distillation. We have carefully addressed all their questions through additional controlled experiments, substantially revised both the main paper and supplementary material, and conducted further ablation studies beyond those explicitly requested. We have also prepared a new supplementary video to better demonstrate our method.

We have devoted considerable effort to these revisions and sincerely hope that the additional evidence can be considered during the discussion period. If possible, we would be very grateful if you could kindly remind the reviewers to revisit our responses and share any remaining concerns or feedback before the deadline. We would be more than happy to provide further clarification.

Your understanding and assistance would be deeply appreciated. Thank you very much for your time and consideration.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal as the Discussion Deadline Approaches

Thank you again for your time and thoughtful feedback on our work.

We are following up as it has been five days since we submitted our response, and the discussion deadline is now approaching. We have addressed all three of your concerns through additional controlled experiments and substantial revisions to the manuscript and supplementary material.

We understand that you may have a busy schedule. Nevertheless, we would greatly appreciate even a brief reply indicating whether you have had a chance to review our responses and whether any concerns remain. Your feedback at this stage is particularly important to us, as we hope to address any outstanding questions while there is still time for discussion.

If you have not yet been able to review the revisions, a brief update would also be very helpful. Thank you sincerely for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.