- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Align2Refine, a post-hoc framework for improving pretrained video representations using unlabeled videos. A shared token-wise MLP predicts spatially corresponding features from a frozen reference encoder, and consistency-guided reweighting emphasizes tokens that are easier to align. A two-component Gaussian mixture calibrates a threshold for identifying regular and spurious tokens. Building on UniRefiner, the refinement stage trains student LoRA parameters to preserve regular teacher features and redirect spurious features into boundary registers, with the original encoder supplying the feature targets. Using only RGB frames from the YouTube-VOS training split for refinement, the paper evaluates five video encoders on semantic segmentation, monocular depth, video object segmentation, and point tracking under frozen-backbone evaluation. Reported gains include 4.4-21.5 ADE20K mIoU and 7.7-13.2 DAVIS-M J&F. Component ablations, alternative DINOv3/VAE references, and UPerNet evaluation provide additional evidence for the method.
Critical strengths
Broad and substantial empirical improvements (Tables 2-4 and 13). All five video encoders improve across the reported spatial and temporal tasks. V-JEPA 2 gains 11.0 ADE20K mIoU and 13.2 DAVIS-M J&F. Transfer to DAVIS and MOSE extends the evidence beyond the refinement dataset. UPerNet also improves by 2.3 mIoU
Critical weaknesses
W1. Diagnostic interpretation needs clearer qualification (Secs. 3-4.2; Table 1). The partition is useful empirically, but low consistency can also reflect ordinary sample difficulty or reference mismatch. Score bimodality is partly induced by the reweighting procedure. Table 1 supports an association with segmentation quality, although training-token counts, subset composition, and the shared evaluation population should be clarified. These issues limit the specificity of the spurious-token interpretation; they do not invalidate the mask's demonstrated practical usefulness. W2. Comparison with the closest refinement method is not fully controlled (Sec. 4.3; Appendix C, Table 10). The refinement framework is inherited from UniRefiner. Table 10 uses its published image-model results without fully establishing matched data and adaptation budgets, and no matched video comparison changes only the diagnostic criterion. This limits conclusions about superiority over the prior criterion. Nevertheless, the random-mask and reweighting ablations provide meaningful evidence for the proposed mechanism, so I regard this as a moderate comparative-evidence gap rather than a sufficient reason for rejection.
Questions for authors
Q1. In Table 1, are all three probes evaluated on the same complete validation set? Please specify training-token counts and subset composition. This would help distinguish diagnostic quality from differences in training-sample difficulty. Q2. Which refinement data, training budget, and evaluation settings are matched between the published UniRefiner results and your image-model results in Table 10? Is the original criterion transferable to the video refinement pipeline, and what evidence is available under otherwise matched settings? Q3. How stable are the learned thresholds and mask proportions across videos and crop-space recomputation? Please report available variability for the key Table 5 comparisons and the probe/refinement computational cost. These would strengthen confidence in the reported recipe.
5 comments
Q3. Stable thresholds, crop-space masks, and lower computational cost
(1) Threshold and mask stability.
We measured threshold and mask stability on V-JEPA 2 ViT-g through repeated video sampling. For crop-space evaluation, we sampled 3 spatial crops per video and recomputed the masks using the same crop coordinates across frames.
| Statistic | Mean ± Std |
|---|---|
| GMM threshold | 0.0614 ± 0.0037 |
| Spurious-token ratio, full video (%) | 40.28 ± 1.36 |
| Spurious-token ratio, spatial crops (%) | 41.49 ± 1.74 |
The threshold remains stable across sampled videos, while the spurious-token ratio stays similar between full videos (40.28%) and spatial crops (41.49%).
(2) Computational cost.
We also measured computational cost on the YouTube-VOS training split using 8 NVIDIA RTX 5090 GPUs under matched experimental settings.
| Backbone | Method | Probe time(min) | Refinement time(min) | Total time(min) |
|---|---|---|---|---|
| V-JEPA 2 | UniRefiner | — | 77.4 | 77.4 |
| Align2Refine | 12.9 | 46.7 | 59.6 | |
| InternVideo-Next | UniRefiner | — | 35.3 | 35.3 |
| Align2Refine | 6.6 | 23.0 | 29.6 |
Including probe training, Align2Refine reduces total runtime by 23.0% on V-JEPA 2 and 16.1% on InternVideo-Next.
Hoping for a Discussion on Our Rebuttal for Paper 1130
Dear Reviewer,
I hope this message finds you well. We have carefully responded to the questions and concerns you previously raised, and we would greatly appreciate any further feedback you might be able to provide. Your insights are highly valuable to us, and we remain fully available to clarify or refine any remaining points.
As the discussion deadline approaches, we would be grateful for your response at your earliest convenience, as it would greatly help us move the discussion forward.
Thank you sincerely for your time and effort in reviewing our paper.
Best regards,
The Authors
Thank you for the detailed suggestions. We have clarified Table 1 and added further experiments on diagnostic accuracy, method comparison, and stability.
(1) Table 1 evaluation protocol.
In Table 1, we train a single linear segmentation probe on all tokens from the entire ADE20K training set, without subset-specific sampling. The same probe is then evaluated on the complete validation set, with results reported separately for all, regular, and spurious token locations. Regular and spurious tokens account for 57.1% and 42.9% of the validation tokens, respectively.
(2) Token-level error validation.
To directly validate whether low-consistency tokens are more likely to produce spatial prediction errors, we evaluated regular and spurious tokens against ground-truth labels on Cityscapes and DAVIS-S using InternVideo-Next ViT-L.
Regular tokens achieve substantially higher prediction accuracy than spurious tokens on both benchmarks, showing that low-consistency tokens are much more likely to produce errors at their corresponding spatial locations. Importantly, the DAVIS-S results demonstrate that this relationship also holds in dynamic video scenes involving motion, occlusion, and appearance changes.
(3) Correlation between alignment consistency and dense prediction performance.
We also examined the relationship between raw alignment consistency and ADE20K segmentation accuracy (Appendix C, Figure 8). We grouped tokens by their cosine consistency scores and measured the prediction accuracy within each interval. The plot shows a clear, near-monotonic increase in accuracy as consistency rises, demonstrating that alignment consistency effectively reflects the quality of local token representations.
The original Table 10 reports published UniRefiner results for image encoders. We have now added a direct video-encoder comparison using matched refinement data, training epochs, LoRA rank, optimizer, input resolution, and downstream evaluation protocols.
Align2Refine performs better across all five metrics on both video encoders, showing the benefit of our alignment-based token identification under matched refinement settings.