acceptodds
Under review as a conference paper at ICLR 2027

Boosting MLLM-SAM2 Referring Video Object Segmentation via Temporal Memory Reading

Abstract

Referring Video Object Segmentation (RVOS) aims to segment target objects in a video according to natural language expressions. Recent methods adopt an MLLM-SAM2 paradigm, where the MLLM token is projected into a sparse prompt for a SAM2 mask decoder. However, they suffer from object identification ambiguity when distinguishing visually similar instances via temporal cues. Through probing analysis, we trace this failure to the prompt interface: frame-order decodability degrades significantly during token-to-prompt conversion, yielding collapsed prompt representations that confuse same-category instances. We propose Temporal Memory Reading (TMR), which grounds expression-conditioned temporal queries in multi-frame SAM2 features and injects spatially adaptive query biases into memory attention throughout propagation. Experiments on multiple benchmarks show that TMR improves MLLM-SAM2 baselines on temporally ambiguous cases with similar distractors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video object segmentation, referring segmentation, video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u5RMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the task of RVOS, i.e. segmentation of text-referred objects in a video. The recent dominant paradigm relies on using a MLLM, trained to produce a special token [SEG] that is in turn fed to a video segmentation model, typically SAM2. The paper identifies as a shortcoming of recent approaches the fact that the SEG token fails to integrate temporal cues, and in practice results in failure to disambiguate between multiple distractors. The proposed modifications include a Temporal Memory Reading module to guide segmentation across frames by reasoning on the MLLM tokens; in contrast to standard approaches in which after the first frame SAM2 alone is tasked with propagating the mask across frames.

Critical strengths
  • Motivation. The preliminary analysis is interesting and delves into the failure modes of recent MLLM-based methods, providing evidence of how temporal cues weaken when projecting the LLM hidden states into the lower dim. space of SAM2 prompts
  • Generality. The proposed method is agnostic to the specific framework and can be plugged in into methods relying on MLLM+SAM (e.g. SA2VA, VRS-HQ). The experiments provide these comparisons, coupling the proposed method with the respective baselines in the same setup.
  • Performance. The proposed method provides consistent, non-negligible gains against all of the respective baselines
Critical weaknesses
  • Contributions. While the preliminary analysis identifies a failure mode, the proposed contributions are not well linked to the highlighted problem, and they consist of convoluted architectural blocks. The explanation is also not clear; e.g. after the Cross attention in the TMR, there is a block in Fig.3 that says ‘Per-position retrieval’. The text (L253) is not more explicative. What does retrieval mean here? Is it just an interpretation of the cross attention? Or is there an actual retrieval step? Why would this follow the cross attention, and precede the per-channel gate? Overall, the contributions are not intuitive, and simpler alternatives are not discussed nor ablated. For example, the TMR introduces an additional cross attention before the Memory Attention, which is also a Cross Att. with past features. Could the 2 have been merged? This also implies changing the SAM2 inference at the anchor frame. While the ablation thoroughly evaluates the impact of each piece, it fails to provide actual insights given that the method has many architectural pieces. Removing each performs worse, but what would be a baseline for each of the proposed pieces? What about simply merging the MLLM tokens to the already existing memory bank SAM2, so it would naturally be integrated in the Memory Attention mechanism?
  • Soundness. The experimental section contains unclear setups. All of the ablations report some subsets of MeViS-val (Appearance, Temporal, Same-cat) that are not standard, and seem to be introduced by the paper. This is not mentioned at all in the paper. While details are in the supplementary, this makes the paper confusing for the reader, and it leaves the question of whether we really need this novel splits, without any discussion or results for any prior baseline. The whole case study on failure cases, while potentially interesting, feels very brittle and based on arbitrary metrics and criteria. e.g. prediction-to-prediction mask threshold below 0.9 (?). Then, the reported metric is a percentage over a varying support, which therefore is not comparable metric across the considered scenarios. Tab.6 reports these numbers under scenarios not well characterized: ‘New collapses’, ‘Original-collapse’, ‘Original collapses separated’. What are we comparing in this table? It looks like a list of scenarios, with an unclear link between them, and without explanation of what each scenario represents. The whole set of experiments lacks scientific rigor.
  • Clarity and paper design. The writing quality is suboptimal. Many sentences are unnecessarily convoluted, with questionable choice of vocabulary that falls outside standard language adopted in the literature (e.g. pretrained MLLM -> ‘host’, prompt projections -> ‘interface’ (?); mask encoding -> ‘write-back’) The worst one is ‘MLLM ablation -> ‘host augmentation’ (L306). These complaints might seem insignificant, but to me they are indication of an excessive use of generative tools without careful supervision, which ultimately impacts readability of the paper. Many sentence structures are typical of these tools, e.g. at the end of every paragraph ‘This is X, not Y’. Note that I am not against the use of Ai tools. However, without careful review it results in the paper having weird phrasing of common concepts, hence subpar clarity and reduced trust in the reader. Lastly, regarding paper design, all tables are together in page 8, which results in a messy layout and difficulty in following the experimental section. Below I put some more examples of convoluted sentences / weird phrasing. L045 (“decodability drops precipitously”); L47 (“preserves significantly higher decodability than the baseline interface”). What is the baseline interface?.: L138 (“Because preserving temporal ordering serves as the fundamental structural backbone for dynamic temporal cues, we operationalize this evaluation via frame-order decodability”), L157 (“This probe measures frame-order decodability, not whether the host correctly reasons about the referred target”) L264 (“At the anchor frame, no real predicted-mask memory has yet been written”)
Questions for authors

The paper has an interesting motivation, setting forth to study the failure cases of recent MLLM based methods for RVOS. It identifies potentially interesting areas to improve, and the proposed method achieves in practice good performance on benchmarks. However, the main limitations are the lack of clarity of the ablations, and of the preliminary experiments which underpin the motivation of the paper. These seem to lack a scientific protocol, and rely on seemingly arbitrary data splits without motivation.Overall, the paper writing lacks in clarity, failing to convincingly explain and motivate the specific technical contribution. Hence my rating is a BR; If the authors can provide convincing arguments for the above items, I would be willing to raise my rating. However, I would be inclined to believe that while the paper has potential, it would benefit from an extensive rewrite in order to be beneficial and informative for the community

2 comments
AuthorsOpenReview ↗

Response to Reviewer u5RM(1/2)

1. What is "per-position retrieval", and why does it sit between the cross-attention and the gate?

There is no separate retrieval step. "Per-position retrieval" is the cross-attention itself: each spatial position of the current SAM2 feature map is a query, and the N grounded tokens are keys and values. The output is one 256-d vector per position, so each location reads its own mixture of the N tokens. The revision calls it "per-position cross-attention readout".

The gate scales this readout before it is added to the SAM2 feature (Eq. 8). It is a per-channel weight initialized at 0.1, so at the start of training SAM2 receives almost its original input. Gating the readout, rather than the sum, keeps SAM2's own feature at full strength. The pretrained model therefore starts unchanged and learns per channel how much grounded evidence to admit.

2. Could this cross-attention be merged into SAM2 memory attention, for example by adding the MLLM tokens to the memory bank?

We trained this alternative. The N grounded tokens are appended to the SAM2 memory as object-pointer tokens, so memory attention reads them directly and the extra cross-attention and gate are removed.

VRS-HQ-7B on MeViS valᵘJ&F
Grounded tokens appended to the SAM2 memory bank60.5
TMR (separate per-position readout)62.1

The merged form reaches 60.5, 1.6 below TMR. A likely reason is that in memory attention the N tokens share one softmax with the spatial memory of up to seven frames, each with 64 × 64 entries, so they receive little weight. A separate readout lets every position attend only to the N tokens.

The anchor frame does change. Standard SAM2 has no predicted-mask memory at the anchor frame, and TMR fills that memory from the visual bank. Table 3 isolates this step: 60.8 without it, 62.1 with it.

3. What is the simpler baseline for each component?

Each component replaces a simpler option, all on VRS-HQ-7B and MeViS valᵘ:

ComponentSimpler optionSimpler option J&FWith the component
Grounding against the bankTokens from the MLLM state only (Table 3)47.857.6
Per-position readoutOne global query per frame (Table 7)58.962.1
Separate readout before memory attentionTokens added to the memory bank (new)60.562.1
Bank-based anchor memoryPrompt-based anchor memory (Table 3)60.862.1
3D RoPE in the bankNone / frame only / row and column only (Table 7, new)57.7 / 59.8 / 60.262.1
Grounded tokens at every frameBase prompt repeated at every frame (Table 4)55.662.1
Grounded tokensEight learned anchor prompts (Table 4)57.162.1

Whether these components address the diagnosed failure, rather than improving segmentation in general, is tested by the query-source controls in our response to Reviewer 4ggm (Q3) and by the Temporal and collapse results in item 4 below.

AuthorsOpenReview ↗

Response to Reviewer u5RM(2/2)

4. Are the diagnostic subsets needed, and how do prior models score on them?

The subsets are partitions of the standard MeViS valᵘ split, not new benchmarks, and all main comparisons use the official splits. They are defined in Appendix C.8, which the revision moves into Section 5.1:

  • Temporal / Appearance. An expression is Temporal if its target can be identified only across several frames (motion, action order, state change), and Appearance if one frame is enough. Labels come from a rule-based pass and an LLM pass, with human review of about 150 borderline cases: 372 Temporal, 421 Appearance.
  • Same-cat. Expression pairs from the same video that refer to different objects of the same category (327 pairs).
  • Collapse. The 168 of those pairs for which the baseline predicts nearly the same mask for both expressions.

They are needed because the failure we study concerns targets that cannot be told apart in a single frame, and overall J&F mixes both kinds of expressions. Prior models show the same gap:

MeViS valᵘ, J&FOverallAppearanceTemporal
VRS-HQ-7B56.668.143.6
VRS-HQ-7B, matched fine-tuning56.868.144.0
Sa2VA-8B60.470.549.0
Sa2VA-8B + TMR64.072.154.8
VRS-HQ-7B + TMR62.170.852.2

On Sa2VA-8B, TMR's gain also concentrates on Temporal expressions (+5.8 vs +1.6 on Appearance). Our response to Reviewer 4ggm (Q4) repeats the subset analysis on MeViS val, which was not used for any design choice.

5. Is the 0.9 collapse threshold arbitrary, and what does Table 6 compare?

Two predicted masks whose IoU averages at least 0.9 over the eight sampled frames cover nearly the same pixels, so we call the pair collapsed. The result does not depend on this threshold:

Mask IoU thresholdBaseline collapsed pairsTMR collapsed pairs
0.80198/327113/327
0.85184/32796/327
0.90168/32775/327
0.95122/32748/327

At every threshold TMR removes 43–61% of the baseline collapses.

Table 6 follows the same 327 pairs from the baseline to TMR. The revision replaces its list with a 2 × 2 table with fixed denominators:

TMR collapsedTMR separatedTotal
Baseline collapsed65103 (87 with higher target J&F)168
Baseline separated10149159
Total75252327

"New collapses" is the 10 in the lower-left cell. "Separated and improved" means the two masks no longer collapse and the average J&F of the two expressions, each against its own target, increases.

6. Writing and layout

We agree with your examples and have rewritten the paper:

  • "host" is now "base model", "interface" is "token-to-prompt projection", "write-back" is "memory encoding", and "host augmentation" is "adding TMR to existing models".
  • Paragraph-closing contrast sentences are removed, and long sentences are split. For example, L045 now reads "frame-order probe accuracy is lower in the final 256-d SAM2 prompt". L047 reads "a random projection to 256 dimensions keeps higher probe accuracy than the trained prompt". L264 reads "at the anchor frame, SAM2 has not yet stored memory from any predicted mask".
  • Each table now appears next to the text that cites it, and Tables 3–8 no longer share one page.
Reviewer_W4rMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Memory Reading (TMR), a framework designed to improve referring video object segmentation (RVOS) by addressing object identification ambiguity in existing MLLM-SAM2 architectures. TMR introduces a Mask-Free Visual Bank, Temporal Token Grounding, and Temporal Memory Reading. These components ground expression-conditioned queries in multi-frame SAM2 features and inject the resulting information into SAM2 memory attention during video propagation.

Critical strengths
  • The paper identifies an important limitation of existing MLLM-SAM2 architectures, i.e., the tendency to produce nearly identical segmentation masks for visually similar objects despite different referring expressions. The probing analysis provides interesting evidence that frame-order information becomes less linearly decodable during the conversion from MLLM representations to SAM2 prompts. This analysis offers a useful perspective on potential limitations of the existing prompt-based interface.

  • The proposed method improves segmentation performance across multiple benchmarks and model configurations

Critical weaknesses
  • My primary concern is the limited methodological novelty of the proposed framework. TMR largely builds upon existing architectural components, including learnable query tokens, cross-attention-based visual grounding, positional encoding, and feature injection into memory attention. Although these components are integrated in a technically reasonable manner, the resulting framework appears to be an incremental extension of the existing MLLM-SAM2 architecture.

  • The paper attributes the object identification ambiguity primarily to the loss of temporal information during prompt construction. However, the probing experiments measure frame-order decodability rather than the ability to understand object motion or temporal relationships. The fact that frame permutations can be distinguished from intermediate representations does not necessarily establish that those representations encode the temporal information required for accurate object identification. Furthermore, the proposed TMR module introduces an additional pathway for multi-frame visual features to influence SAM2 memory attention. Therefore, the observed performance improvements may arise from stronger visual conditioning and feature retrieval rather than improved temporal reasoning.

Questions for authors
  • What is the fundamental methodological distinction between TMR and existing approaches that inject additional visual features or language-conditioned representations into SAM2 memory attention? Beyond the specific implementation, what new principle does TMR introduce that cannot be achieved through conventional cross-attention-based conditioning?

  • Can the authors provide a controlled experiment that preserves the same multi-frame visual features and spatial positional information while removing or randomizing temporal ordering?

1 comment
AuthorsOpenReview ↗

Response to Reviewer W4rM

Q1. What principle distinguishes TMR from conventional cross-attention conditioning?

The components are standard, and we do not claim a new attention operator. The contribution is where the referring condition enters SAM2 propagation. SAM2 encodes its memory from its own predicted masks, so a wrong selection at the anchor frame is written into memory and read back at every later frame. TMR therefore conditions the query side of memory attention at every frame, with tokens grounded in a bank that is built without any prompt or mask. Each simpler placement scores lower (VRS-HQ-7B, MeViS valᵘ):

Where the condition entersSourceJ&F
Anchor-frame prompt only (matched fine-tuning)Table 456.8
Anchor-frame prompt, expanded to 8 learned tokensTable 457.1
Same prompt repeated at every frameTable 455.6
Grounded tokens appended to SAM2 memory as object-pointer tokensnew60.5
Grounded tokens, one vector for all positions and frames, no bank anchor memoryTable 357.6
Grounded tokens, one global query per frameTable 7(iii)58.9
Grounded tokens, per-position readout at every frame (TMR)Table 362.1

The pointer variant is the conventional way to add conditioning tokens to SAM2 memory attention, and it uses the same grounded tokens. It reaches 60.5, 1.6 below TMR. Conditioning the query side therefore adds a gain beyond what the same tokens give when they are read as memory.

Prior methods use the other placements. VRS-HQ and Sa2VA condition only the anchor-frame prompt. SAMWISE adapts image features with text inside the encoder. SAM2 object pointers are produced by the mask decoder together with each predicted mask, so they inherit its errors. The revised related work states these differences.

Q2. Does the gain come from temporal order or from stronger visual conditioning?

We kept the same bank features and the spatial positional encoding and removed frame order in two ways:

VRS-HQ-7B on MeViS valᵘOverall J&FTemporal J&FAppearance J&F
TMR (RoPE over frame, row, column)62.152.270.8
Frame coordinates shuffled at test time (same checkpoint, mean ± std over 3 permutations)59.7 ± 0.347.570.5
Trained without the frame coordinate (RoPE over row, column)60.248.570.5
Trained with RoPE over frame only (Table 7)59.8––
Trained without positional encoding (Table 7)57.7––

Removing frame order costs 2.4 J&F at test time and 1.9 when the model is trained without it. Almost all of the loss is on Temporal expressions (−4.7 and −3.7), while Appearance drops by only 0.3 in both cases. The trained model uses the order of the sampled frames, and it uses it where the target must be identified over time. Frame and spatial coordinates are complementary: row and column alone give 60.2, frame alone 59.8, and all three 62.1.

On the probe, we agree. It shows only that frame order is less decodable after prompt construction, not that the MLLM state encodes which object moved. Section 3 now states this scope. Our response to Reviewer 4ggm (Q3) tests directly whether TMR's gain comes from the MLLM state or from the visual bank, by changing the TMR query source. Expression-free queries reach 57.9, the final 256-d prompt 59.0, and the 4096-d MLLM state 62.1. The bank and per-frame conditioning alone account for 1.1 of the 5.3-point gain over matched fine-tuning.

Reviewer_4ggmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses MLLM-SAM2 RVOS models predicting nearly the same mask for expressions that refer to different same-category objects. Probing VRS-HQ-7B, the authors find that same-category prompts become more similar and frame-order cues become harder to decode after the MLLM token is converted into the 256-d SAM2 prompt. They propose Temporal Memory Reading (TMR), which feeds visually grounded MLLM queries into SAM2 memory attention at every frame. With the MLLM frozen, TMR improves four hosts on four benchmarks, outperforms matched fine-tuning and prompt-based controls on VRS-HQ-7B, and reduces collapse on 327 diagnostic pairs from 168 to 75.

Critical strengths
  1. The gains on the diagnosed failure are large and well controlled. On VRS-HQ-7B, matched fine-tuning without TMR changes scores by at most 0.3 points, while TMR gains 5.3 on MeViS valid_u (95% CI [4.95, 5.71] over three paired seeds). Eight trained anchor prompts with more parameters (57.1) and a repeated prompt (55.6) do not match TMR (62.1). The gains are concentrated on temporal expressions (+8.6 vs. +2.7 for appearance), and collapse on the 327 same-category pairs drops from 168 to 75, with 87 pairs both separating and improving. This is consequential because it shows that the improvement is not explained by extra training or extra prompt capacity.
  2. TMR is a light plug-in that helps several hosts. With the host MLLM frozen, it adds 15.49M parameters and about 0.41 ms per frame (1.6% lower SAM2-path throughput), and it improves all four hosts on all four benchmarks (e.g., 14B: 61.2 on MeViS val, 66.9 on ReVOS). This is consequential because the method can be added to existing MLLM-SAM2 systems at little cost.
Critical weaknesses
  1. The diagnosis does not establish what TMR fixes. The temporal probe decodes which permutation of the eight input frames was given, which is not the same as the cue needed to tell a moving object from a still one, and even a random 256-d projection loses much of this accuracy (62.8 to 33.5 for K = 8). The prompt-to-collapse link is a moderate correlation (r = 0.41). More importantly, TMR does not only reuse the pre-projection MLLM token: it adds new visual evidence (SAM2 features of eight frames) and conditions every propagation frame. The prompt-side controls (eight prompts, repeated prompt) have neither, so they cannot separate recovering lost host information from adding visual grounding and per-frame conditioning, as the authors also note in Appendix F. This is consequential because the paper's framing, that the failure is traced to the prompt interface and TMR addresses it, is its main contribution beyond a memory-attention adapter.
  2. Model selection and most evidence share one split, and the cross-host gains are only partly controlled. N, d_t, the pseudo mask, the diversity loss, and the positional encoding are all chosen on MeViS valid_u (Tables 7–8), and the component ablations, subset gains, and collapse analysis are reported on the same split. Matched fine-tuning exists only for VRS-HQ-7B, so for the three Sa2VA hosts the gains also include fine-tuning SAM2 modules on a mixture weighted toward MeViS and ReVOS; the largest gain (Sa2VA-2B, +9.1 on Ref-YouTube-VOS) is not controlled. Some held-out results are also hard to read: Ours-7B gains +0.2 J but +5.1 F on Ref-YouTube-VOS, and Ours-8B loses 1.2 F on Ref-DAVIS17 (+0.2 J&F overall). This is consequential because the claim of consistent improvement across hosts rests mainly on these uncontrolled comparisons.
Questions for authors
  1. How is the ungrounded temporal token in Table 3 injected, and why does it fall far below the baseline (47.8 vs. 56.6)? Similarly, how is the grounding-only variant (57.6) delivered to SAM2 without memory reading?
  2. What explains the J/F asymmetry on Ref-YouTube-VOS for Ours-7B (+0.2 J, +5.1 F) and the F drop for Ours-8B on Ref-DAVIS17?
  3. Can you train TMR with the 256-d post-projection prompt as the query source instead of the pre-projection MLLM token? If the gains stay similar, they come from visual grounding and per-frame conditioning rather than from recovering information lost at the interface.
  4. Can you report matched fine-tuning for at least one Sa2VA host (ideally Sa2VA-2B), and results on a split not used for design selection for the subset and collapse analyses (e.g., MeViS val)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4ggm (1/2)

Q1. How do the ungrounded and grounding-only variants reach SAM2?

The rows of Table 3 are cumulative, and the anchor-frame prompt is kept in every row. Without memory reading, the N tokens are mean-pooled into one 256-d vector and added through the gate g to the memory-attention query at every position of every frame. This injection is identical everywhere, so it carries expression information but no information about location.

Ungrounded. The tokens are computed from the MLLM state alone, Z = φ_m(φ_q(h_seg) + E_t), with no attention to the visual bank. The score drops to 47.8, below both the original baseline (56.6) and matched fine-tuning (56.8). The vector has never been aligned with SAM2 features, and adding it to every query shifts all memory-attention matches at once. It disrupts SAM2's pretrained matching without indicating where the target is.

Grounding only. The tokens first attend to the eight sampled frames, so the injected vector is built from SAM2 features. This removes the drop and reaches 57.6, 0.8 above matched fine-tuning. Relative to the original baseline, the gain is on Temporal expressions (43.6 → 45.6), with no change on Appearance (68.1 → 68.2).

The remaining gain requires the injection to vary across positions. Replacing the uniform vector with the per-position readout gives 60.8 (Table 3). In Table 7(iii), a readout that varies across frames but not across positions gives 58.9, against 62.1 for the full model.

Q2. What explains the J/F asymmetry?

If TMR moves a mask from a wrong object to the right one, J and F should rise together. A gain only in F means the same object is segmented with sharper boundaries. Table 1 shows that the asymmetry depends on the benchmark:

J / F gain over the base modelMeViS valRef-YouTube-VOS
Sa2VA-2B+3.7 / +4.2+9.0 / +9.0
VRS-HQ-7B+5.6 / +5.8+0.2 / +5.1
Sa2VA-8B+3.3 / +3.8+0.9 / +3.5
Sa2VA-14B+6.2 / +6.4+3.5 / +6.1

On MeViS, where same-category distractors make target identification the main difficulty, J and F rise by similar amounts for all four models. Ref-YouTube-VOS has far fewer expressions with temporal cues (42.4% vs 61.6% on MeViS val, Appendix D.6), so fewer of the base models' errors come from picking the wrong object. There, the J gain is smaller than the F gain for the three stronger models (+0.2, +0.9, +3.5 J), and much of the improvement is at the boundary. The exception is Sa2VA-2B, the weakest base model on this benchmark (62.9 J&F), whose J and F rise equally.

The boundary gain does not come from fine-tuning alone. Matched fine-tuning leaves Ref-YouTube-VOS unchanged (70.4 → 70.2 J&F), and TMR adds 2.9 J&F over it.

On Ref-DAVIS17, Sa2VA-8B with TMR changes J from 72.1 to 73.7 and F from 81.1 to 79.9, for +0.2 J&F. We have a single training run for this configuration and cannot distinguish +0.2 from zero. The revision reports it as no measurable change and no longer counts it as an improvement.

AuthorsOpenReview ↗

Response to Reviewer 4ggm(2/2)

Q3. How much does the pre-projection query source matter?

We compare three query sources within TMR: learned queries shared across expressions, the final 256-d post-TDA prompt, and the 4096-d pre-projection state. All three use the same eight sampled frames, grounding architecture, anchor memory, and per-frame readout, and all keep the original expression-conditioned anchor prompt. The shared-query variant therefore measures what the visual bank and per-frame conditioning contribute without expression information in the TMR branch.

VRS-HQ-7B on MeViS valᵘOverall J&F ↑Temporal J&F ↑Appearance J&F ↑Collapse pairs ↓
Original baseline56.643.668.1168/327
Matched fine-tuning56.844.068.1163/327
TMR, queries shared across expressions57.945.169.2147/327
TMR, final 256-d prompt59.047.169.5135/327
TMR, pre-projection state62.152.270.875/327

With the final 256-d prompt as query, TMR reaches 59.0, 3.1 below the pre-projection state. The gap is 5.1 on Temporal expressions (47.1 vs 52.2) and 1.3 on Appearance (69.5 vs 70.8), and the final prompt leaves 60 more collapsed pairs (135 vs 75). The pre-projection state therefore carries information that TMR uses and the final prompt lacks, mostly where the target must be identified from temporal evidence.

The 5.3-point gain over matched fine-tuning splits into three parts. Shared queries, which add the visual bank and per-frame conditioning without expression information in the TMR branch, give 1.1 (56.8 → 57.9). Using the final prompt as query adds 1.1 (→ 59.0). Using the pre-projection state instead adds the remaining 3.1 (→ 62.1), and on Temporal expressions this last step is the largest (+5.1 of +8.2).

On the probes, we accept your three points and have narrowed Section 3:

  • Predicting which of K frame permutations was given is not the cue that separates a moving object from a still one. Section 3 now claims only that frame order is less decodable after prompt construction.
  • At K = 8, accuracy is 62.8% for the pre-projection state, 33.5% after a random 256-d projection, and 22.7% for the final prompt. Dimensionality reduction alone accounts for 48%, 46%, and 73% of the drop at K = 2 / 4 / 8. Only the remaining gap below the random projection (8.6 / 18.0 / 10.8 points at K = 2 / 4 / 8) is attributed to the trained prompt.
  • Prompt similarity and prediction-to-prediction mask IoU have a moderate association (Pearson r = 0.41, Spearman ρ = 0.46, n = 327). The revision calls it an association and no longer describes it as strong.

Q4. Do the gains persist under matched fine-tuning and outside the design split?

Matched fine-tuning for Sa2VA-2B. Same training data, schedule, and trainable SAM2 modules as TMR, without the TMR modules.

Sa2VA-2B, J&FMeViS valRef-YouTube-VOSRef-DAVIS17ReVOS
Reproduced baseline44.762.972.252.7
Matched fine-tuning45.565.072.853.5
TMR48.672.075.756.6
TMR gain over matched fine-tuning+3.1+7.0+2.9+3.1

TMR improves on matched fine-tuning on all four benchmarks. Of the largest gain you point out, 9.1 points on Ref-YouTube-VOS, 7.0 remain over matched fine-tuning. The revision states gains across base models against the matched control where one exists, and labels the remaining comparisons as gains over released checkpoints.

Outside the design split. Model settings, expression-labeling rules, and the 0.9 collapse threshold were fixed on MeViS valᵘ and not changed. Temporal and Appearance labels on MeViS val use the same rule-based and LLM labelling passes as on MeViS valᵘ. Collapse pairs on ReVOS valid follow the rules of Appendix A.2.

SplitTemporal gainAppearance gainCollapse pairs (baseline → TMR)
MeViS valᵘ (design split)+8.6+2.7168 → 75 of 327
MeViS val+6.0+2.0–
ReVOS valid––90 → 54

Both findings hold outside the design split. On MeViS val the Temporal gain is three times the Appearance gain (+6.0 vs +2.0), and on ReVOS valid TMR removes 36 of the 90 baseline collapses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.