- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses the task of RVOS, i.e. segmentation of text-referred objects in a video. The recent dominant paradigm relies on using a MLLM, trained to produce a special token [SEG] that is in turn fed to a video segmentation model, typically SAM2. The paper identifies as a shortcoming of recent approaches the fact that the SEG token fails to integrate temporal cues, and in practice results in failure to disambiguate between multiple distractors. The proposed modifications include a Temporal Memory Reading module to guide segmentation across frames by reasoning on the MLLM tokens; in contrast to standard approaches in which after the first frame SAM2 alone is tasked with propagating the mask across frames.
Critical strengths
- Motivation. The preliminary analysis is interesting and delves into the failure modes of recent MLLM-based methods, providing evidence of how temporal cues weaken when projecting the LLM hidden states into the lower dim. space of SAM2 prompts
- Generality. The proposed method is agnostic to the specific framework and can be plugged in into methods relying on MLLM+SAM (e.g. SA2VA, VRS-HQ). The experiments provide these comparisons, coupling the proposed method with the respective baselines in the same setup.
- Performance. The proposed method provides consistent, non-negligible gains against all of the respective baselines
Critical weaknesses
- Contributions. While the preliminary analysis identifies a failure mode, the proposed contributions are not well linked to the highlighted problem, and they consist of convoluted architectural blocks. The explanation is also not clear; e.g. after the Cross attention in the TMR, there is a block in Fig.3 that says ‘Per-position retrieval’. The text (L253) is not more explicative. What does retrieval mean here? Is it just an interpretation of the cross attention? Or is there an actual retrieval step? Why would this follow the cross attention, and precede the per-channel gate? Overall, the contributions are not intuitive, and simpler alternatives are not discussed nor ablated. For example, the TMR introduces an additional cross attention before the Memory Attention, which is also a Cross Att. with past features. Could the 2 have been merged? This also implies changing the SAM2 inference at the anchor frame. While the ablation thoroughly evaluates the impact of each piece, it fails to provide actual insights given that the method has many architectural pieces. Removing each performs worse, but what would be a baseline for each of the proposed pieces? What about simply merging the MLLM tokens to the already existing memory bank SAM2, so it would naturally be integrated in the Memory Attention mechanism?
- Soundness. The experimental section contains unclear setups. All of the ablations report some subsets of MeViS-val (Appearance, Temporal, Same-cat) that are not standard, and seem to be introduced by the paper. This is not mentioned at all in the paper. While details are in the supplementary, this makes the paper confusing for the reader, and it leaves the question of whether we really need this novel splits, without any discussion or results for any prior baseline. The whole case study on failure cases, while potentially interesting, feels very brittle and based on arbitrary metrics and criteria. e.g. prediction-to-prediction mask threshold below 0.9 (?). Then, the reported metric is a percentage over a varying support, which therefore is not comparable metric across the considered scenarios. Tab.6 reports these numbers under scenarios not well characterized: ‘New collapses’, ‘Original-collapse’, ‘Original collapses separated’. What are we comparing in this table? It looks like a list of scenarios, with an unclear link between them, and without explanation of what each scenario represents. The whole set of experiments lacks scientific rigor.
- Clarity and paper design. The writing quality is suboptimal. Many sentences are unnecessarily convoluted, with questionable choice of vocabulary that falls outside standard language adopted in the literature (e.g. pretrained MLLM -> ‘host’, prompt projections -> ‘interface’ (?); mask encoding -> ‘write-back’) The worst one is ‘MLLM ablation -> ‘host augmentation’ (L306). These complaints might seem insignificant, but to me they are indication of an excessive use of generative tools without careful supervision, which ultimately impacts readability of the paper. Many sentence structures are typical of these tools, e.g. at the end of every paragraph ‘This is X, not Y’. Note that I am not against the use of Ai tools. However, without careful review it results in the paper having weird phrasing of common concepts, hence subpar clarity and reduced trust in the reader. Lastly, regarding paper design, all tables are together in page 8, which results in a messy layout and difficulty in following the experimental section. Below I put some more examples of convoluted sentences / weird phrasing. L045 (“decodability drops precipitously”); L47 (“preserves significantly higher decodability than the baseline interface”). What is the baseline interface?.: L138 (“Because preserving temporal ordering serves as the fundamental structural backbone for dynamic temporal cues, we operationalize this evaluation via frame-order decodability”), L157 (“This probe measures frame-order decodability, not whether the host correctly reasons about the referred target”) L264 (“At the anchor frame, no real predicted-mask memory has yet been written”)
Questions for authors
The paper has an interesting motivation, setting forth to study the failure cases of recent MLLM based methods for RVOS. It identifies potentially interesting areas to improve, and the proposed method achieves in practice good performance on benchmarks. However, the main limitations are the lack of clarity of the ablations, and of the preliminary experiments which underpin the motivation of the paper. These seem to lack a scientific protocol, and rely on seemingly arbitrary data splits without motivation.Overall, the paper writing lacks in clarity, failing to convincingly explain and motivate the specific technical contribution. Hence my rating is a BR; If the authors can provide convincing arguments for the above items, I would be willing to raise my rating. However, I would be inclined to believe that while the paper has potential, it would benefit from an extensive rewrite in order to be beneficial and informative for the community
2 comments
Response to Reviewer u5RM(2/2)
4. Are the diagnostic subsets needed, and how do prior models score on them?
The subsets are partitions of the standard MeViS valᵘ split, not new benchmarks, and all main comparisons use the official splits. They are defined in Appendix C.8, which the revision moves into Section 5.1:
- Temporal / Appearance. An expression is Temporal if its target can be identified only across several frames (motion, action order, state change), and Appearance if one frame is enough. Labels come from a rule-based pass and an LLM pass, with human review of about 150 borderline cases: 372 Temporal, 421 Appearance.
- Same-cat. Expression pairs from the same video that refer to different objects of the same category (327 pairs).
- Collapse. The 168 of those pairs for which the baseline predicts nearly the same mask for both expressions.
They are needed because the failure we study concerns targets that cannot be told apart in a single frame, and overall J&F mixes both kinds of expressions. Prior models show the same gap:
| MeViS valᵘ, J&F | Overall | Appearance | Temporal |
|---|---|---|---|
| VRS-HQ-7B | 56.6 | 68.1 | 43.6 |
| VRS-HQ-7B, matched fine-tuning | 56.8 | 68.1 | 44.0 |
| Sa2VA-8B | 60.4 | 70.5 | 49.0 |
| Sa2VA-8B + TMR | 64.0 | 72.1 | 54.8 |
| VRS-HQ-7B + TMR | 62.1 | 70.8 | 52.2 |
On Sa2VA-8B, TMR's gain also concentrates on Temporal expressions (+5.8 vs +1.6 on Appearance). Our response to Reviewer 4ggm (Q4) repeats the subset analysis on MeViS val, which was not used for any design choice.
5. Is the 0.9 collapse threshold arbitrary, and what does Table 6 compare?
Two predicted masks whose IoU averages at least 0.9 over the eight sampled frames cover nearly the same pixels, so we call the pair collapsed. The result does not depend on this threshold:
| Mask IoU threshold | Baseline collapsed pairs | TMR collapsed pairs |
|---|---|---|
| 0.80 | 198/327 | 113/327 |
| 0.85 | 184/327 | 96/327 |
| 0.90 | 168/327 | 75/327 |
| 0.95 | 122/327 | 48/327 |
At every threshold TMR removes 43–61% of the baseline collapses.
Table 6 follows the same 327 pairs from the baseline to TMR. The revision replaces its list with a 2 × 2 table with fixed denominators:
| TMR collapsed | TMR separated | Total | |
|---|---|---|---|
| Baseline collapsed | 65 | 103 (87 with higher target J&F) | 168 |
| Baseline separated | 10 | 149 | 159 |
| Total | 75 | 252 | 327 |
"New collapses" is the 10 in the lower-left cell. "Separated and improved" means the two masks no longer collapse and the average J&F of the two expressions, each against its own target, increases.
6. Writing and layout
We agree with your examples and have rewritten the paper:
- "host" is now "base model", "interface" is "token-to-prompt projection", "write-back" is "memory encoding", and "host augmentation" is "adding TMR to existing models".
- Paragraph-closing contrast sentences are removed, and long sentences are split. For example, L045 now reads "frame-order probe accuracy is lower in the final 256-d SAM2 prompt". L047 reads "a random projection to 256 dimensions keeps higher probe accuracy than the trained prompt". L264 reads "at the anchor frame, SAM2 has not yet stored memory from any predicted mask".
- Each table now appears next to the text that cites it, and Tables 3–8 no longer share one page.
Response to Reviewer u5RM(1/2)
1. What is "per-position retrieval", and why does it sit between the cross-attention and the gate?
There is no separate retrieval step. "Per-position retrieval" is the cross-attention itself: each spatial position of the current SAM2 feature map is a query, and the N grounded tokens are keys and values. The output is one 256-d vector per position, so each location reads its own mixture of the N tokens. The revision calls it "per-position cross-attention readout".
The gate scales this readout before it is added to the SAM2 feature (Eq. 8). It is a per-channel weight initialized at 0.1, so at the start of training SAM2 receives almost its original input. Gating the readout, rather than the sum, keeps SAM2's own feature at full strength. The pretrained model therefore starts unchanged and learns per channel how much grounded evidence to admit.
2. Could this cross-attention be merged into SAM2 memory attention, for example by adding the MLLM tokens to the memory bank?
We trained this alternative. The N grounded tokens are appended to the SAM2 memory as object-pointer tokens, so memory attention reads them directly and the extra cross-attention and gate are removed.
The merged form reaches 60.5, 1.6 below TMR. A likely reason is that in memory attention the N tokens share one softmax with the spatial memory of up to seven frames, each with 64 × 64 entries, so they receive little weight. A separate readout lets every position attend only to the N tokens.
The anchor frame does change. Standard SAM2 has no predicted-mask memory at the anchor frame, and TMR fills that memory from the visual bank. Table 3 isolates this step: 60.8 without it, 62.1 with it.
3. What is the simpler baseline for each component?
Each component replaces a simpler option, all on VRS-HQ-7B and MeViS valᵘ:
Whether these components address the diagnosed failure, rather than improving segmentation in general, is tested by the query-source controls in our response to Reviewer 4ggm (Q3) and by the Temporal and collapse results in item 4 below.