- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MindSight first scans a video at low resolution, then selects a temporal interval and chooses STOP, LOW, or HIGH. Paired continuations at matched timestamps provide cost-aware action targets. Replay-verified SFT is followed by TB-GMPO, which balances contributions across turns and bounds log-ratio updates. Experiments on two backbones and four benchmarks report accuracy gains and reduced visual-token use.
Critical strengths
- Well-motivated action supervision. Matching timestamps for LOW and HIGH distinguishes visual fidelity from temporal localization. Figure 3 reports comparable or better accuracy than always-HIGH with substantially fewer acquired visual tokens.
- Informative ablations and cost accounting. Three-seed component ablations and separate accounting for acquired tokens, replayed prefill, and serving latency make the evaluation informative. The cached and full-replay comparisons are particularly useful for understanding deployment costs.
- Coherent training pipeline. The method combines replay-verified trajectories, action targets that account for downstream observation costs while excluding ambiguous comparisons, and turn-balanced policy optimization (Sec. 3.2). These choices address trajectory validity, supervision quality, and uneven turn lengths.
Critical weaknesses
- Cost-aware action preferences may shift during RL. Labels are generated with a frozen SFT snapshot, whereas TB-GMPO changes the continuation policy and uses a different reward (Eqs. 3 and 5). The paper acknowledges that targets can become stale, but does not quantify how their cost-adjusted action preferences change. Such changes need not be harmful, but their effect on the final accuracy-cost trade-off is unclear.
- The speedup is conditional on cache reuse. Table 3 reports 3.46 seconds to the first final-answer token with prefix caching and 7.92 seconds with full replay, compared with 4.72 seconds for Uniform Sparse. The paper clearly demonstrates this dependence. The remaining question is which cache settings and reuse rates are needed to obtain the speedup in practice.
Questions for authors
- On fixed development states, how often does the best action under the cost-adjusted value criterion change from the SFT snapshot to the final RL policy? How do these changes affect the accuracy-cost trade-off?
- What cache settings and prefix-reuse rates produce Table 3's results, and under what deployment conditions would the speed advantage diminish? Please clarify applicable caching optimizations for each method, without assuming equal cross-turn reuse for single-pass baselines.
- What fraction of paired-supervision states pass filtering, and what is their STOP/LOW/HIGH label distribution? How do actual action frequencies evolve during RL relative to the benchmark proportions already reported in Figure 3?
4 comments
Response to Reviewer 91FZ (2/3)
2. Cache settings, reuse rates, and deployment conditions
Table 3 in the paper was measured on four A100-80GB GPUs, TP=4, batch size=1. MindSight’s prefix-cache setting reuses KV states for the identical multimodal history across turns, whereas full replay recomputes that history.
As Table R6 shown, under this setup, TTFT is 3.46 s with prefix reuse, 7.92 s with full replay, and 4.72 s for Uniform Sparse. Thus, cached MindSight has 26.7% lower TTFT than Uniform Sparse, whereas full replay has 67.8% higher TTFT. MindSight’s latency advantage over Uniform Sparse is conditional on prefix reuse; it reverses in the reported full-replay configuration.
TTFT includes scan, tool, and reasoning calls before the first final-answer token. The measured complete-answer latency is 4.91 s for cached MindSight versus 6.25 s for Uniform Sparse, a 21.4% reduction. These timings belong to the original serving configuration and are not measurements of the updated 488-frame evaluation.
Table R6. Reported serving results and reuse statistics.
| Method | TTFT (s) | TPOT (ms/token) | Complete answer (s) | Peak total GPU memory (GB/GPU) | Intra-episode KV hit rate (%) | Mean executed prefill tokens per episode |
|---|---|---|---|---|---|---|
| MindSight, prefix cache | 3.46 | 28.4 | 4.91 | 32.1 | 64.8 | 18,432 |
| MindSight, full replay | 7.92 | 30.8 | 9.84 | 41.2 | 0 (reuse disabled) | 39,936 |
| Uniform Sparse | 4.72 | 29.1 | 6.25 | 28.5 | N/A | 24,576 |
| Uniform Dense | 16.90 | 29.7 | 18.42 | 64.2 | N/A | 61,440 |
Intra-episode KV hit rate is the ratio of eligible history tokens actually reused in later turns to all eligible history tokens in those turns, excluding newly appended observations and text. Executed prefill tokens are summed within each episode and then averaged across episodes. Peak total GPU memory denotes the peak total memory occupied on each GPU, including allocated KV memory, model weights, and other runtime allocations.
The reuse statistics show why the latency gain is plausible under the reported serving setup. The cached configuration reuses a substantial fraction of eligible history states and executes much less prefill work than full replay. At the same time, the comparison does not assign MindSight’s intra-episode reuse opportunity to the single-pass baselines. For Uniform Sparse and Uniform Dense, intra-episode KV hit rate is marked as N/A, because those methods do not have the same multi-turn browsing structure.
Effective reuse requires an unchanged serialized multimodal prefix and KV retention or availability across turns. The advantage can diminish under cache-capacity pressure, concurrent eviction, prefix changes, or routing to a worker without the required KV states. Therefore, the revised manuscript limits the speed claim to the measured serving conditions and defines reuse statistics separately from visual-encoding reuse and cross-question reuse.
Table R7. Method-specific optimization scope and configuration record
| Method | Intra-episode reuse | Additional applicable optimization | Engine/version | Initial cache state | Routing/eviction |
|---|---|---|---|---|---|
| MindSight | Identical history KV across browsing turns | Common text-prefix and repeated visual-encoding cache where supported | vLLM 0.6.0 | Cold at request start | Sticky routing, LRU eviction |
| Uniform Sparse | N/A: single pass | Common text-prefix and repeated-video/cross-question reuse where supported | vLLM 0.6.0 | Cold at request start | Sticky routing, LRU eviction |
| Uniform Dense | N/A: single pass | Common text-prefix and repeated-video/cross-question reuse where supported | vLLM 0.6.0 | Cold at request start | Sticky routing, LRU eviction |
Under the reported serving configuration, MindSight achieves lower latency than Uniform Sparse with cross-turn prefix reuse, but higher latency without it. The speed claim is therefore conditional on effective KV retention and reuse.
Response to Reviewer 91FZ (3/3)
3. Training-label acceptance and actual action frequencies during RL
Of 6,840 attempted states , 6,651 completed all action evaluations and 4,218 received clear labels. Among the completed states, 2,433 were ambiguous and 189 were incomplete. The acceptance rate is therefore 61.7% of attempted states , or 63.4% of completed states.
Table R8a. Accepted training labels
| Action | Labels | Rate (%) |
|---|---|---|
| STOP | 1,772 | 42.0 |
| LOW | 1,401 | 33.2 |
| HIGH | 1,045 | 24.8 |
The labeling configuration uses 32 continuations per action , λC=0.10 , τV=0.02 , and 1,000 bootstrap resamples , with nominal 95% simultaneous confidence after Bonferroni correction. Ambiguous states contribute no action-label loss. Figure 3 in the paper describes final-policy inference actions under the original evaluation budget. It uses different populations and denominators from Table R8a, so the two tables should not be directly interpreted as a drift estimate.
Table R9. Checkpoint action audit on the common held-out RL validation set
| Checkpoint | Validation accuracy (%) | Parseable decisions | STOP/LOW/HIGH (%) | LOW/HIGH calls per episode | Acquired visual tokens | Executed prefill tokens | False STOP (%) |
|---|---|---|---|---|---|---|---|
| Label-generating SFT snapshot | 55.8 | 1,187 | 41.8 / 35.4 / 22.8 | 1.84 | 17,920 | 31,488 | 11.2 |
| Post-label SFT / RL initialization | 60.0 | 1,191 | 39.9 / 36.0 / 24.1 | 1.97 | 18,784 | 30,112 | 9.4 |
| Intermediate RL | 66.1 | 1,193 | 37.6 / 36.8 / 25.6 | 2.11 | 19,968 | 28,736 | 7.1 |
| Final RL, 10k updates | 72.5 | 1,194 | 36.9 / 37.1 / 26.0 | 2.19 | 20,352 | 27,904 | 5.8 |
Table R9 tracks actual action frequencies across checkpoints on the same held-out validation questions under a fixed evaluation budget . During RL, the action shares change consistently from initialization through the intermediate checkpoint to the final checkpoint:
- STOP decreases: 39.9% → 37.6% → 36.9% ( −3.0 percentage points overall).
- LOW increases: 36.0% → 36.8% → 37.1% ( +1.1 percentage points ).
- HIGH increases: 24.1% → 25.6% → 26.0% ( +1.9 percentage points ).
Together, LOW/HIGH account for 60.1% → 62.4% → 63.1% of parseable decisions , while their combined calls per episode rise from 1.97 to 2.19 (+0.22) . Thus, RL shifts the policy moderately toward additional observations rather than earlier stopping. This shift accompanies an increase in validation accuracy from 60.0% to 72.5% (+12.5 percentage points) and a decrease in False STOP from 9.4% to 5.8% (−3.6 percentage points).
We hope our responses have addressed your concerns and clarified the relevant aspects of our work. Please feel free to raise any additional questions; we would be glad to provide further clarification.
We would be grateful if you would consider reassessing our work in light of these clarifications.
Thank you for the clarifications. The new analyses address my main concerns, I would like to raise my score from 2 to 3.
Response to Reviewer 91FZ (1/3)
We sincerely thank the reviewer for the careful and constructive assessment. The comments helped us clarify three important issues: cost-adjusted action preference changes after RL, cache requirements and deployment conditions, and the distinction between training-label statistics and inference-time action frequencies. In the revision, we have added the confirmed supervision and budget results and clarified the limitations of the current evidence. For clarity, we respond point by point below.
1. Cost-adjusted action preferences after RL
The reviewer is correct that paired labels are defined relative to the frozen SFT continuation policy that generated them. TB-GMPO then changes this continuation policy by optimizing answer correctness, temporal grounding, and format rewards. Its return does not directly include the acquired-token penalty used in the paired value criterion. Therefore, reference-policy KL alone does not guarantee unchanged action rankings.
Under an identical TB-GMPO training setup, adding paired labels improves accuracy from 38.4/54.2% to 39.5/56.0%, a gain of 1.1/1.8 points on LVBench/VideoMME-Long, respectively. This directly shows that paired supervision remains beneficial in the tested RL pipeline. To answer the reviewer’s question about the frequency and consequences of individual stale labels directly, we re-estimate action values on the same fixed development states, using the same cost-adjusted criterion and confidence rule under both continuation policies. Visible evidence, candidate actions, timestamps, resolutions, and remaining budgets are held fixed.
This uses the same cost-adjusted criterion under each policy, rather than comparing argmax actions under different reward definitions. From the label-generating SFT snapshot to RL initialization, the preferred action changes in 8.9% of 1,012 jointly confident states. From RL initialization to the final RL policy, it changes in 11.7% of 1,084 jointly confident states (Table R5a).
Table R5a. Fixed-state preference audit
These results exhibit the pattern expected from an RL stage that is both effective and policy-improving. The cumulative flip rate from the label-generating SFT snapshot to the final RL policy is moderate rather than catastrophic. Moreover, ambiguous→confident transitions occur more frequently than confident→ambiguous transitions , suggesting that the final policy does not simply drift away from the original labels. This suggests that the final policy does not simply drift away from the original labels; instead, it makes a substantial subset of previously uncertain states more decisively evaluable.
Table R5b. Consequences under the final continuation policy
These results support two points. First, stale labels can matter: the changed-action subset shows a larger mean old-label value regret ( 0.0314 ) and a larger correctness difference ( +4.6 pp ). Second, the aggregate effect remains limited because changed confident actions account for only a subset of all evaluable states. This is consistent with our main claim: paired supervision remains useful in the tested RL pipeline, while individual stale labels should be audited rather than assumed harmless.
In the updated evaluation reported in our response to Reviewer xjPX, MindSight achieves 48.5% accuracy, outperforming the local official-style Qwen baseline by 3.8 percentage points (44.7%). Compared with fixed-resolution Qwen, MindSight achieves higher accuracy (48.5% vs. 43.8%) with a 23.4% lower maximum visual-token budget (50,176 vs. 65,536), half the context limit (64K vs. 128K), and fewer permitted frame observations (488 vs. 512). These results demonstrate a favorable accuracy–cost trade-off: MindSight delivers better accuracy under tighter visual-token and context budgets.