- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes "residual test-time agentic computation": supplementary computation targets only requirements unresolved by the base model, and after a localized failure, only the affected states and their dependencies are recomputed. The authors instantiate this formulation as ReTAC—a training-free multiagent framework consisting of structured native-state generation, residual routing and planning, targeted spatial perception, stage-aligned verification, and dependency-based selective recovery. The paper validates the approach across 3 MLLMs and 2 video spatial reasoning benchmarks, and verifies the effectiveness of its core designs through routing-intervention and component-replacement ablation experiments.
However, the core contribution is closer to a system integration of existing components plus a residual organization principle than to a methodological breakthrough. Several substantive limitations remain: (1) the direct-return branch fully trusts unverified self-reported labels; (2) the efficiency evaluation only covers invocation counts, lacking time/token dimensions; (3) the gains on strong base models are limited and remain unanalyzed.
Critical strengths
-
The motivation is grounded in concrete evidence, and the central claim gets a direct test. The paper opens with a sharp motivating counterexample—providing a ground-truth cognitive map to Qwen2.5-VL hurts its performance—which frames the real question as "what is still missing, given what the base model already resolves?" The routing intervention then tests this head-on: forcing extra computation on instances that should return directly, and suppressing refinement on instances that need it, both degrade performance across all three models. This is a clean way to show that where computation is spent matters more than how much of it is spent.
-
The ablations are well-designed, and the results hold consistently across models. Replacing the residual planner with a generic one, and the stage-aligned verification/selective recovery with generic reflection, both clearly hurt performance, so the two core designs are doing real work rather than being decorative. On top of that, ReTAC tops all six model–benchmark combinations, with role instructions fixed across models and a cost-accounting scheme that charges failed and retried calls separately. Together this reduces the worry that the gains are a single-model accident or an artifact of selective counting.
Critical weaknesses
-
The direct-return branch trusts unverified self-reported labels. In Eq. 2, when is empty the system returns as-is, and the paper explicitly says DIRECT is a routing decision, not a verification outcome. If the native model misses a requirement because of its own blind spot, the pipeline terminates early and hands back an answer that was never checked. The authors acknowledge this risk in D.2, but offer no mitigation—no spot-checking of the direct-return branch, no confidence calibration, no analysis of how often self-reported SUPPORTED/UNRESOLVED actually matches ground truth. Since the whole residual premise rests on these self-reports being reliable, this feels like a fairly central gap.
-
The experiments don't really establish that residual allocation beats existing adaptive methods. The comparison is limited to Direct, CoT, S&T, and Think3D. None of the adaptive agent methods discussed in the related work—AdaTooler-V, AdaReasoner, and others—are included, so it's hard to support the claim that a residual signal is better than the context/feedback signals those methods already use.
-
Efficiency is measured only through mean model calls. There's no end-to-end wall-clock time, token consumption, or tool latency, and the gains on the strongest model (GPT-5.5) are modest without any analysis of why.
Questions for authors
-
When , the system directly returns results without verification. How can we guarantee the reliability of the native self-reported labels (SUPPORTED/UNRESOLVED)? Has the consistency between self-reported labels and the true missing requirements (mislabel rate) been measured? Can a fallback mechanism be added to handle early termination caused by missed requirements?
-
Tables 1 and 2 show that the gain shrinks as the base model gets stronger. This pattern deserves analysis rather than just reporting: does the benefit track the native model's raw capability, or its self-report quality? The authors should at least state where they expect the method to stop paying off, since on the strongest model the improvements are already modest.
-
The comparison is limited to Direct, CoT, S&T, and Think3D. The adaptive agent methods mentioned in the related work, such as AdaTooler-V and AdaReasoner, are not included. The claim that residual allocation is preferable to generic adaptive allocation can only be established by comparing against the closest comparable methods under the same protocol.
-
Efficiency is measured only in terms of mean model calls. End-to-end wall-clock time, token consumption, and tool latency relative to Direct/CoT/S&T/Think3D would be more informative. The authors should also report accuracy-cost curves for recovery budgets T = 0, 1, 2, 3, 4; this would both justify the default choice of T = 2 and show the concrete shape of the claim that larger budgets do not yield monotonic gains.
3 comments
Response to Reviewer 3 (2/3)
W2&Q3:Compare ReTAC with closely related adaptive methods, such as AdaTooler-V and AdaReasoner, under the same protocol to establish the advantage of residual allocation.
A: Thank you for suggesting comparisons with additional adaptive agent methods. We agree that these comparisons would strengthen the empirical evidence for ReTAC. Although AdaTooler-V and AdaReasoner are discussed in our related work, both involve additional training.
To compare inference-time methods while keeping the backbone models unchanged, we therefore add AlloSpatial (Ruan et al., 2026), a concurrent work first released in June 2026, using its training-free setting. This adaptive tool-use framework has publicly available code and is already cited in our Introduction and the “Video Spatial Reasoning with MLLMs” paragraph of Section 2. It combines query-conditioned spatial representations with adaptive tool-use decisions, making it a relevant comparison for video spatial reasoning.
As shown in Table 3, ReTAC consistently outperforms AlloSpatial across all three backbones on both benchmarks. These results extend our comparison to a relevant adaptive agent method.
Table 3. Comparison with AlloSpatial across three backbones on VSI-Bench-Debiased and STI-Bench.
| Method | Qwen3-VL-8B VSI ↑ | Qwen3-VL-8B STI ↑ | Gemini 3.1 Pro VSI ↑ | Gemini 3.1 Pro STI ↑ | GPT-5.5 VSI ↑ | GPT-5.5 STI ↑ |
|---|---|---|---|---|---|---|
| Direct | 47.73 | 39.87 | 48.77 | 49.47 | 61.01 | 49.61 |
| AlloSpatial | 51.75 | 42.14 | 51.47 | 50.36 | 61.97 | 50.81 |
| ReTAC | 53.49 | 44.50 | 53.80 | 51.33 | 62.38 | 52.14 |
In the revised manuscript, we have included AlloSpatial as an additional baseline in Tables 1 and 2 and added the corresponding discussion in the Experiment section.
W3: Efficiency evaluation lacks latency, token usage, and tool-cost measurements.
R3: Thank you for pointing out the limitations of our efficiency evaluation. We therefore supplement this metric with input and output token consumption, end-to-end latency, and tool execution time. Table 4 compares ReTAC with Direct, CoT, See&Trek, Think3D, and AlloSpatial using Qwen3-VL-8B on VSI-Bench-Debiased. Token counts are accumulated across all model calls for each question and then averaged over the full evaluation set. End-to-end latency covers the complete process from input preprocessing to the final answer, including model calls, tool execution, and recovery. All cost metrics are reported as averages per question.
Table 4. Performance and computational cost on VSI-Bench-Debiased with Qwen3-VL-8B.
| Method | Score ↑ | Mean calls ↓ | Input tokens ↓ | Output tokens ↓ | E2E latency (s/question) ↓ | Tool time (s/question) ↓ |
|---|---|---|---|---|---|---|
| Direct | 47.73 | 1.0 | 9767.5 | 2.6 | 1.14 | 0 |
| CoT | 48.59 | 1.0 | 9791.5 | 1155.7 | 2.62 | 0 |
| See&Trek | 48.27 | 1.0 | 29568.7 | 3.1 | 6.44 | 5.39 |
| Think3D | 48.99 | 2.7 | 23035.5 | 992.9 | 15.92 | 7.12 |
| AlloSpatial | 51.75 | 3.8 | 23380.9 | 897.3 | 22.03 | 9.39 |
| ReTAC | 53.49 | 4.2 | 22576.4 | 925.1 | 3.34 | 1.07 |
Compared with See&Trek, Think3D, and AlloSpatial, ReTAC achieves a higher score with lower end-to-end latency and fewer input tokens, despite making more model calls. This illustrates why call counts alone are insufficient to characterize computational cost.
Response to Reviewer 3 (3/3)
Q2: Clarify whether diminishing gains on stronger models reflect native capability or self-report quality, and when ReTAC’s benefits no longer justify its cost.
A2: Thank you for highlighting this distinction. ReTAC’s gains depend on both the requirements left unresolved by native inference and how reliably these requirements are identified.
Our manual label experiment shows high agreement with human annotations, which suggests that poorer self-report quality is unlikely to be the main explanation for GPT-5.5’s smaller gains. Meanwhile, only 30.7% of GPT-5.5 questions enter refinement. On this fixed refinement subset, suppressing refinement reduces the score by 2.80 points. These findings are consistent with a smaller need for supplementary computation, while showing that refinement remains useful on the selected questions.
We expect the benefit to diminish when native inference already resolves most requirements. This is an encouraging sign that stronger foundation models already possess substantial video spatial understanding capabilities. ReTAC’s selective routing aims to reduce unnecessary computation in these cases.
ReTAC is particularly useful for smaller models with limited native spatial understanding, improving their spatial reasoning through targeted inference-time computation without requiring a larger backbone. Its training-free design also requires no access to model weights or gradients, making it applicable to both open-weight and proprietary models.
Q4: Report accuracy–cost curves for recovery budgets to justify and assess non-monotonic gains.
A4: Thank you for suggesting a broader analysis of recovery budgets. Figure 5(b) already reports scores and mean model calls for all three backbones at . Increasing the budget from to reduces the scores of GPT-5.5 and Gemini 3.1 Pro, while providing only a marginal gain for Qwen3-VL-8B. These results show that larger recovery budgets do not necessarily improve performance.
Following your suggestion, we extend the evaluation to and . The results are summarized in Table 5.
Table 5. Performance and mean model calls of ReTAC under different recovery budgets on VSI-Bench-Debiased.
| Recovery budget T | Qwen3-VL-8B Score ↑ | Mean calls ↓ | Gemini 3.1 Pro Score ↑ | Mean calls ↓ | GPT-5.5 Score ↑ | Mean calls ↓ |
|---|---|---|---|---|---|---|
| 0 | 51.48 | 2.92 | 50.28 | 2.78 | 60.37 | 2.81 |
| 1 | 53.13 | 3.82 | 52.77 | 3.80 | 61.25 | 3.46 |
| 2 | 53.49 | 4.21 | 53.80 | 4.13 | 62.38 | 3.81 |
| 3 | 53.57 | 4.40 | 53.12 | 4.28 | 60.86 | 3.97 |
| 4 | 53.17 | 4.45 | 52.85 | 4.30 | 60.04 | 3.99 |
As shown in Table 5, increasing the recovery budget from to improves scores across all three backbones. Beyond , mean model calls continue to increase, while performance gains diminish or reverse. At , Qwen3-VL-8B shows only a marginal improvement, while scores decline for both Gemini 3.1 Pro and GPT-5.5. Increasing the budget to further reduces scores for all three models. These results demonstrate that larger recovery budgets do not consistently improve performance and support as a practical balance between performance and computational cost.
Response to Reviewer 3 (1/3)
Thank you for recognizing the motivation of residual computation. Your main concerns focus on label reliability, comparisons with more closely related methods, broader efficiency metrics, and a wider range of recovery budgets. In response, we have reported our prior label reliability assessment, added a recent comparable method with publicly available code, expanded the efficiency evaluation, and evaluated additional recovery budgets. These results provide further evidence for the effectiveness of ReTAC. We have made the following additions and improvements based on your feedback.
W1.1&Q1.1: How reliable are the native SUPPORTED/UNRESOLVED labels, and has their mislabel rate been measured?
A: Thank you for raising this thoughtful concern. During the initial development of ReTAC, we assessed their reliability through manual annotation. For each of the three backbones, we randomly sampled 200 examples from the full evaluation set of 2,362 examples and compared the model’s labels with human annotations. Table 1 reports the agreement rates.
Table 1. Agreement between native self-reported labels and human annotations
The self-reported labels show high agreement with human annotations. These results supported our decision to use self-reported labels for residual routing. However, high agreement does not guarantee correct labels for every example, which is why we explicitly acknowledge this limitation in Appendix D.2.
W1.2&Q1.2: Can a fallback mechanism detect missed requirements and prevent premature direct return?
A: We propose a lightweight safeguard for the direct-return branch. Before returning an answer, a deterministic check requires every SUPPORTED requirement to have a nonempty evidence record and a valid source reference to the sampled input frames. If either is missing or invalid, the corresponding requirement is marked UNRESOLVED and passed to the existing residual refinement pipeline. This check applies to every direct-return candidate and requires no additional model calls.
We evaluate this mechanism on the same manually annotated examples used in our label reliability analysis. Table 2 compares agreement with human annotations before and after the check. We also report the false-SUPPORTED rate, defined as the proportion of human-labeled UNRESOLVED examples incorrectly labeled SUPPORTED.
Table 2. Native label reliability before and after the fallback check
As shown in Table 2, agreement increases and false-SUPPORTED rate decreases. These results show that the fallback can further reduce errors that cause the model to return an answer while some requirements remain unresolved.