acceptodds
Withdrawn from ICLR 2027

ReTAC: Residual Test-Time Agentic Computation for Video Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time agentic computation can improve multimodal large language models (MLLMs) through planning, tool use, iterative perception, and reflection, but indiscriminate computation may duplicate resolved work, introduce noisy evidence, and perturb correct predictions. We propose that such computation should be residual: allocate additional computation only to requirements unresolved by the base model and, after a localized failure, recompute only the affected states. We instantiate this formulation in ReTAC, a training-free multiagent framework for video spatial reasoning. ReTAC organizes supplementary computation around unresolved requirements. It plans ordered steps to acquire missing spatial evidence and combines the observations with compatible native evidence for reasoning. In addition, it uses stage-aligned verification to localize failures to planning, perception, or reasoning. Dependency-based recovery recomputes only affected states while reusing unaffected ones. Extensive experiments show that ReTAC achieves the highest overall score among the compared methods across three mainstream MLLMs and two widely adopted benchmarks. Further analyses show that gains from additional computation depend not only on how much is used, but also on where it is allocated and which valid intermediate states are preserved.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Spatial Reasoning, Multimodal Large Language Models, Tool-Augmented Reasoning, Capability-Aware Planning, Selective Error Correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cn2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes "residual test-time agentic computation": supplementary computation targets only requirements unresolved by the base model, and after a localized failure, only the affected states and their dependencies are recomputed. The authors instantiate this formulation as ReTAC—a training-free multiagent framework consisting of structured native-state generation, residual routing and planning, targeted spatial perception, stage-aligned verification, and dependency-based selective recovery. The paper validates the approach across 3 MLLMs and 2 video spatial reasoning benchmarks, and verifies the effectiveness of its core designs through routing-intervention and component-replacement ablation experiments.

However, the core contribution is closer to a system integration of existing components plus a residual organization principle than to a methodological breakthrough. Several substantive limitations remain: (1) the direct-return branch fully trusts unverified self-reported labels; (2) the efficiency evaluation only covers invocation counts, lacking time/token dimensions; (3) the gains on strong base models are limited and remain unanalyzed.

Critical strengths
  1. The motivation is grounded in concrete evidence, and the central claim gets a direct test. The paper opens with a sharp motivating counterexample—providing a ground-truth cognitive map to Qwen2.5-VL hurts its performance—which frames the real question as "what is still missing, given what the base model already resolves?" The routing intervention then tests this head-on: forcing extra computation on instances that should return directly, and suppressing refinement on instances that need it, both degrade performance across all three models. This is a clean way to show that where computation is spent matters more than how much of it is spent.

  2. The ablations are well-designed, and the results hold consistently across models. Replacing the residual planner with a generic one, and the stage-aligned verification/selective recovery with generic reflection, both clearly hurt performance, so the two core designs are doing real work rather than being decorative. On top of that, ReTAC tops all six model–benchmark combinations, with role instructions fixed across models and a cost-accounting scheme that charges failed and retried calls separately. Together this reduces the worry that the gains are a single-model accident or an artifact of selective counting.

Critical weaknesses
  1. The direct-return branch trusts unverified self-reported labels. In Eq. 2, when UnatU^{\text{nat}} is empty the system returns ynaty^{\text{nat}} as-is, and the paper explicitly says DIRECT is a routing decision, not a verification outcome. If the native model misses a requirement because of its own blind spot, the pipeline terminates early and hands back an answer that was never checked. The authors acknowledge this risk in D.2, but offer no mitigation—no spot-checking of the direct-return branch, no confidence calibration, no analysis of how often self-reported SUPPORTED/UNRESOLVED actually matches ground truth. Since the whole residual premise rests on these self-reports being reliable, this feels like a fairly central gap.

  2. The experiments don't really establish that residual allocation beats existing adaptive methods. The comparison is limited to Direct, CoT, S&T, and Think3D. None of the adaptive agent methods discussed in the related work—AdaTooler-V, AdaReasoner, and others—are included, so it's hard to support the claim that a residual signal is better than the context/feedback signals those methods already use.

  3. Efficiency is measured only through mean model calls. There's no end-to-end wall-clock time, token consumption, or tool latency, and the gains on the strongest model (GPT-5.5) are modest without any analysis of why.

Questions for authors
  1. When Unat=∅U^\text{nat}=\emptyset, the system directly returns results without verification. How can we guarantee the reliability of the native self-reported labels (SUPPORTED/UNRESOLVED)? Has the consistency between self-reported labels and the true missing requirements (mislabel rate) been measured? Can a fallback mechanism be added to handle early termination caused by missed requirements?

  2. Tables 1 and 2 show that the gain shrinks as the base model gets stronger. This pattern deserves analysis rather than just reporting: does the benefit track the native model's raw capability, or its self-report quality? The authors should at least state where they expect the method to stop paying off, since on the strongest model the improvements are already modest.

  3. The comparison is limited to Direct, CoT, S&T, and Think3D. The adaptive agent methods mentioned in the related work, such as AdaTooler-V and AdaReasoner, are not included. The claim that residual allocation is preferable to generic adaptive allocation can only be established by comparing against the closest comparable methods under the same protocol.

  4. Efficiency is measured only in terms of mean model calls. End-to-end wall-clock time, token consumption, and tool latency relative to Direct/CoT/S&T/Think3D would be more informative. The authors should also report accuracy-cost curves for recovery budgets T = 0, 1, 2, 3, 4; this would both justify the default choice of T = 2 and show the concrete shape of the claim that larger budgets do not yield monotonic gains.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/3)

Thank you for recognizing the motivation of residual computation. Your main concerns focus on label reliability, comparisons with more closely related methods, broader efficiency metrics, and a wider range of recovery budgets. In response, we have reported our prior label reliability assessment, added a recent comparable method with publicly available code, expanded the efficiency evaluation, and evaluated additional recovery budgets. These results provide further evidence for the effectiveness of ReTAC. We have made the following additions and improvements based on your feedback.

W1.1&Q1.1: How reliable are the native SUPPORTED/UNRESOLVED labels, and has their mislabel rate been measured?

A: Thank you for raising this thoughtful concern. During the initial development of ReTAC, we assessed their reliability through manual annotation. For each of the three backbones, we randomly sampled 200 examples from the full evaluation set of 2,362 examples and compared the model’s labels with human annotations. Table 1 reports the agreement rates.

Table 1. Agreement between native self-reported labels and human annotations

BackboneAudited examplesHuman-labeled SUPPORTEDAgreement on SUPPORTEDHuman-labeled UNRESOLVEDAgreement on UNRESOLVEDOverall agreement
Qwen3-VL-8B200103100/103 (97.09%)9795/97 (97.94%)195/200 (97.50%)
Gemini 3.1 Pro200126125/126 (99.21%)7472/74 (97.30%)197/200 (98.50%)
GPT-5.5200136134/136 (98.53%)6463/64 (98.44%)197/200 (98.50%)

The self-reported labels show high agreement with human annotations. These results supported our decision to use self-reported labels for residual routing. However, high agreement does not guarantee correct labels for every example, which is why we explicitly acknowledge this limitation in Appendix D.2.

W1.2&Q1.2: Can a fallback mechanism detect missed requirements and prevent premature direct return?

A: We propose a lightweight safeguard for the direct-return branch. Before returning an answer, a deterministic check requires every SUPPORTED requirement to have a nonempty evidence record and a valid source reference to the sampled input frames. If either is missing or invalid, the corresponding requirement is marked UNRESOLVED and passed to the existing residual refinement pipeline. This check applies to every direct-return candidate and requires no additional model calls.

We evaluate this mechanism on the same manually annotated examples used in our label reliability analysis. Table 2 compares agreement with human annotations before and after the check. We also report the false-SUPPORTED rate, defined as the proportion of human-labeled UNRESOLVED examples incorrectly labeled SUPPORTED.

Table 2. Native label reliability before and after the fallback check

BackboneAgreement before ↑Agreement after ↑False-SUPPORTED rate before ↓False-SUPPORTED rate after ↓
Qwen3-VL-8B195/200 (97.50%)197/200 (98.50%)2/97 (2.06%)0/97 (0.00%)
Gemini 3.1 Pro197/200 (98.50%)198/200 (99.00%)2/74 (2.70%)1/74 (1.35%)
GPT-5.5197/200 (98.50%)198/200 (99.00%)1/64 (1.56%)0/64 (0.00%)

As shown in Table 2, agreement increases and false-SUPPORTED rate decreases. These results show that the fallback can further reduce errors that cause the model to return an answer while some requirements remain unresolved.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/3)

W2&Q3:Compare ReTAC with closely related adaptive methods, such as AdaTooler-V and AdaReasoner, under the same protocol to establish the advantage of residual allocation.

A: Thank you for suggesting comparisons with additional adaptive agent methods. We agree that these comparisons would strengthen the empirical evidence for ReTAC. Although AdaTooler-V and AdaReasoner are discussed in our related work, both involve additional training.

To compare inference-time methods while keeping the backbone models unchanged, we therefore add AlloSpatial (Ruan et al., 2026), a concurrent work first released in June 2026, using its training-free setting. This adaptive tool-use framework has publicly available code and is already cited in our Introduction and the “Video Spatial Reasoning with MLLMs” paragraph of Section 2. It combines query-conditioned spatial representations with adaptive tool-use decisions, making it a relevant comparison for video spatial reasoning.

As shown in Table 3, ReTAC consistently outperforms AlloSpatial across all three backbones on both benchmarks. These results extend our comparison to a relevant adaptive agent method.

Table 3. Comparison with AlloSpatial across three backbones on VSI-Bench-Debiased and STI-Bench.

MethodQwen3-VL-8B VSI ↑Qwen3-VL-8B STI ↑Gemini 3.1 Pro VSI ↑Gemini 3.1 Pro STI ↑GPT-5.5 VSI ↑GPT-5.5 STI ↑
Direct47.7339.8748.7749.4761.0149.61
AlloSpatial51.7542.1451.4750.3661.9750.81
ReTAC53.4944.5053.8051.3362.3852.14

In the revised manuscript, we have included AlloSpatial as an additional baseline in Tables 1 and 2 and added the corresponding discussion in the Experiment section.

W3: Efficiency evaluation lacks latency, token usage, and tool-cost measurements.

R3: Thank you for pointing out the limitations of our efficiency evaluation. We therefore supplement this metric with input and output token consumption, end-to-end latency, and tool execution time. Table 4 compares ReTAC with Direct, CoT, See&Trek, Think3D, and AlloSpatial using Qwen3-VL-8B on VSI-Bench-Debiased. Token counts are accumulated across all model calls for each question and then averaged over the full evaluation set. End-to-end latency covers the complete process from input preprocessing to the final answer, including model calls, tool execution, and recovery. All cost metrics are reported as averages per question.

Table 4. Performance and computational cost on VSI-Bench-Debiased with Qwen3-VL-8B.

MethodScore ↑Mean calls ↓Input tokens ↓Output tokens ↓E2E latency (s/question) ↓Tool time (s/question) ↓
Direct47.731.09767.52.61.140
CoT48.591.09791.51155.72.620
See&Trek48.271.029568.73.16.445.39
Think3D48.992.723035.5992.915.927.12
AlloSpatial51.753.823380.9897.322.039.39
ReTAC53.494.222576.4925.13.341.07

Compared with See&Trek, Think3D, and AlloSpatial, ReTAC achieves a higher score with lower end-to-end latency and fewer input tokens, despite making more model calls. This illustrates why call counts alone are insufficient to characterize computational cost.

AuthorsOpenReview ↗

Response to Reviewer 3 (3/3)

Q2: Clarify whether diminishing gains on stronger models reflect native capability or self-report quality, and when ReTAC’s benefits no longer justify its cost.

A2: Thank you for highlighting this distinction. ReTAC’s gains depend on both the requirements left unresolved by native inference and how reliably these requirements are identified.

Our manual label experiment shows high agreement with human annotations, which suggests that poorer self-report quality is unlikely to be the main explanation for GPT-5.5’s smaller gains. Meanwhile, only 30.7% of GPT-5.5 questions enter refinement. On this fixed refinement subset, suppressing refinement reduces the score by 2.80 points. These findings are consistent with a smaller need for supplementary computation, while showing that refinement remains useful on the selected questions.

We expect the benefit to diminish when native inference already resolves most requirements. This is an encouraging sign that stronger foundation models already possess substantial video spatial understanding capabilities. ReTAC’s selective routing aims to reduce unnecessary computation in these cases.

ReTAC is particularly useful for smaller models with limited native spatial understanding, improving their spatial reasoning through targeted inference-time computation without requiring a larger backbone. Its training-free design also requires no access to model weights or gradients, making it applicable to both open-weight and proprietary models.

Q4: Report accuracy–cost curves for recovery budgets T=0,1,2,3,4T = 0,1,2,3,4 to justify T=2T = 2 and assess non-monotonic gains.

A4: Thank you for suggesting a broader analysis of recovery budgets. Figure 5(b) already reports scores and mean model calls for all three backbones at T=1,2,3T=1,2,3. Increasing the budget from T=2T=2 to T=3T=3 reduces the scores of GPT-5.5 and Gemini 3.1 Pro, while providing only a marginal gain for Qwen3-VL-8B. These results show that larger recovery budgets do not necessarily improve performance.

Following your suggestion, we extend the evaluation to T=0T=0 and T=4T=4. The results are summarized in Table 5.

Table 5. Performance and mean model calls of ReTAC under different recovery budgets on VSI-Bench-Debiased.

Recovery budget TQwen3-VL-8B Score ↑Mean calls ↓Gemini 3.1 Pro Score ↑Mean calls ↓GPT-5.5 Score ↑Mean calls ↓
051.482.9250.282.7860.372.81
153.133.8252.773.8061.253.46
253.494.2153.804.1362.383.81
353.574.4053.124.2860.863.97
453.174.4552.854.3060.043.99

As shown in Table 5, increasing the recovery budget from T=0T=0 to T=2T=2 improves scores across all three backbones. Beyond T=2T=2, mean model calls continue to increase, while performance gains diminish or reverse. At T=3T=3, Qwen3-VL-8B shows only a marginal improvement, while scores decline for both Gemini 3.1 Pro and GPT-5.5. Increasing the budget to T=4T=4 further reduces scores for all three models. These results demonstrate that larger recovery budgets do not consistently improve performance and support T=2T=2 as a practical balance between performance and computational cost.

Reviewer_Z68VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on using MLLMs as agents without any fine-tuning, with experiments on Qwen3-VL-8B, Gemini 3.1 Pro, and GPT-5.5. When additional computation is needed, it divides video QA into four roles: a Planner that specifies the steps for collecting evidence to answer the question; a Perceiver that follows these steps to collect evidence and record its sources; a Reasoner that infers the answer from the existing evidence; and a Verifier that checks the plan, evidence, and reasoning and can request replanning, reperception, or rereasoning. The experiments are comprehensive and compare the method with direct answering and different inference and agentic approaches.

Critical strengths

The motivation is clear: computation should focus on the parts of the problem that still need to be solved. More computation does not necessarily improve reasoning, and rerunning the entire pipeline may disturb already correct intermediate results or incur unnecessary computational costs.

The method requires no training and can be used with off-the-shelf models.

The pipeline integrates external tools with the prompt and agent design.

Critical weaknesses

The results are not very convincing, as the differences between the proposed method and the baselines seem small on several tasks. Table 3 also does not provide sufficiently clear evidence of where the improvements come from.

It would be helpful to describe and quantify the role of the Verifier more clearly. In practice, when and how often does it trigger replanning, reperception, or rereasoning? Since the same model is used throughout, why does it not consider these issues during the initial reasoning pass?

The types of QA tasks seem limited, and there appear to be no causal reasoning tasks in the evaluation.

The limitations and potential extensions are mentioned in the appendix but not in the conclusion. It would be helpful to discuss them more clearly in the main text.

The inference cost could be reported more clearly. Although the paper reports model-call counts, it would be helpful to provide the time for each stage, GPU setup, memory usage, and FLOPs.

Questions for authors

Please see the Weaknesses section.

Even when ChatGPT or other models are used directly, could they already perform internal reasoning or agentic steps? It would be worth clarifying how the proposed method differs from such direct use or better positioning the direct use.

Does using this agentic method make computation more efficient? I am curious how its computational load is measured and how this cost compares with the performance gains.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/5)

We appreciate your thoughtful review and your recognition of our motivation and the adaptability of our method. Reviewers 1 and 3 also recognized our motivation and highlight our design and controlled analyses. Your main concerns focus on the clarity of the ablation studies, the role of the Verifier, runtime and GPU usage, and computational cost evaluation. In response, we have expanded the computational cost evaluation, reorganized and supplemented the ablation studies, provided detailed runtime and GPU information, and clarified the Verifier’s role. We have also addressed each point you identified as requiring further clarification. Our responses are as follows.

W1: Small gains on several tasks and unclear evidence in Table 3 about which components drive the improvements.

R1: The gains vary across task types. As discussed in Section 4.2, the clearest gains on VSI-Bench-Debiased occur on tasks requiring spatial evidence across frames or viewpoints. These tasks align with the object localization, temporal evidence gathering, and observer-relative spatial reasoning supported by residual refinement. Gains on distance and room size estimation are smaller, reflecting limitations in the evidence available through the tools. This pattern exactly reflects ReTAC’s core design principle: adapting supplementary computation to the unresolved requirements of each model–instance pair, rather than base model strength alone.

Regarding the sources of improvement in Table 3 of original version, general reflection changes both verification and recovery scope. To clarify their individual contributions, we separate this comparison into ablations of stage-aligned verification and selective recovery. We reorganize the results from Table 3 and Fig. 5(a) below and report the mean number of model calls per question.

Table 1. Component ablations on VSI-Bench-Debiased.

SettingResidual planningStage-aligned verificationSelective recoveryQwen Score ↑Qwen Calls ↓GPT Score ↑GPT Calls ↓
Full ReTACYesYesYes53.494.262.383.8
General planningGeneral planningYesYes51.374.861.274.3
Full restartYesYesFull restart52.875.161.494.5
General reflection with full restartYesGeneral reflectionFull restart51.584.660.853.9

Rows 1 and 2: Residual planning. Replacing routing and planning based on the native state with General Planning, which does not access native evidence or support labels, reduces scores and increases the model calls. This demonstrates the important role of explicitly using existing evidence and unresolved requirements to organize supplementary computation.

Rows 1 and 3: Selective recovery. With stage-aligned verification fixed, selective recovery improves scores while reducing mean model calls. These results show that reusing unaffected states can reduce repeated computation while also improving accuracy.

Rows 3 and 4: Stage-aligned verification. With full restart fixed, stage-aligned verification improves scores over general reflection, which assesses the entire inference trajectory at once. Although mean model calls increase by 0.5 and 0.6, respectively. Gain/Call, which is calculated as (Score−ScoreDirect)/(Calls−1)(\mathrm{Score}-\mathrm{Score}_{\mathrm{Direct}})/(\mathrm{Calls}-1) improves from 1.069 to 1.254 on Qwen and from -0.055 to 0.137 on GPT. These results support the value of stage-specific error localization and targeted feedback, producing higher scores and greater gains per additional call over baseline.

Thank you for your suggestions on the ablation study. We have revised the ablation table and the accompanying discussion in the latest version of the paper to clearly distinguish the contributions of residual planning, stage-aligned verification, and selective recovery.

W2.1: It would be helpful to describe and quantify the role of the Verifier more clearly.

R2.1: As described in the first paragraph of Section 3.3, the Verifier checks the current intermediate states and localizes issues to specific stages. It runs only on questions routed to residual refinement. After the initial planning, perception, and reasoning steps, it checks these stages in order and repeats verification after each recovery.

The checks at each stage are summarized below.

  • Planning: Whether the plan covers unresolved requirements and the selected operations can acquire the required evidence.
  • Perception: Whether observations are supported by source frames or tool outputs and valid evidence covers the requirements.
  • Reasoning: Whether reasoning is supported by evidence, the inference steps are logically sound, and the final answer agrees with the conclusion.
AuthorsOpenReview ↗

Response to Reviewer 2 (2/5)

W2.2: Clarify when and how often the Verifier triggers replanning, reperception, or rereasoning

R2.2: Trigger timing for recovery. As described in the Recovery Scope paragraph of Section 3.4 and Appendix A.3:

  • REPLAN is triggered when requirements are missing or the evidence acquisition plan needs revision.
  • REPERCEIVE is triggered when the plan remains valid, but a perception operation must be repeated to obtain valid evidence.
  • REREASON is triggered when planning and perception checks pass, but reasoning misuses the evidence or the answer conflicts with the conclusion.

Regarding recovery frequency, Table 2 summarizes the data from Fig. 4. Approximately 30% of questions entering residual refinement trigger at least one recovery across all three models.

Table 2. Frequency of recovery among questions entering residual refinement

ModelQuestions entering refinementQuestions with at least one recoveryPercentage of refinement questions
Qwen3-VL-8B1,08733430.7%
Gemini 3.1 Pro91929632.2%
GPT-5.572521529.7%

Table 3 provides a breakdown by recovery action. Each percentage is the number of executions of that action divided by the total number of executed recovery actions.

Table 3. Distribution of executed recovery actions

ModelREPLAN (%)REPERCEIVE (%)REREASON(%)
Qwen3-VL-8B20.4235.5344.05
Gemini 3.1 Pro17.5738.4144.02
GPT-5.514.8951.3733.74

W2.3: Why does the same model miss issues during initial reasoning that it later identifies during verification?

R2.3: Initial generation and verification differ in their objectives, inputs, and criteria. Generation constructs plans, observations, and reasoning, while verification checks specific states that have already been produced. For example, reasoning verification does not access the original video. It checks whether the reasoning follows from the evidence and whether the answer agrees with the conclusion. Thus, although both stages use the same base model, verification performs a different task under a focused context and explicit criteria. This can help identify issues missed during generation.

W3: Limited QA task diversity and no evaluation of causal reasoning.

R3: Our study focuses on video spatial reasoning, an established research direction that examines how models understand geometry, spatial relations, viewpoints, and motion from visual observations.

Our evaluation scope is consistent with recent work in this area. For example, AlloSpatial [1] evaluates on VSI-Bench and MindCube, Think3D [2] uses BLINK Multi-view, MindCube-1K, and VSI-Bench-Tiny, and See&Trek [3] evaluates on VSI-Bench and STI-Bench. These benchmarks cover diverse spatial abilities, including object counting, size and distance estimation, spatial relations, 3D localization and reasoning across viewpoints. They also assess temporal and motion understanding through tasks such as appearance order, route planning, displacement and path length, speed and acceleration, orientation, trajectory description, and pose estimation.

Within this research scope, our evaluation uses VSI-Bench-Debiased and STI-Bench, each containing eight task types.

We agree that causal reasoning is another important dimension of video understanding. It further concerns why events occur, the effects of interventions, and counterfactual outcomes. These questions fall outside the scope of our current study, so we do not include causal reasoning benchmarks. We appreciate this suggestion and regard extending residual computation to causal video reasoning as a valuable direction for future work.

References

[1] Shouwei Ruan, Bin Wang, Zhenyu Wu, Qihui Zhu, Yuxiang Zhang, Jingzhi Li, Yubin Wang, and Xingxing Wei. Allospatial: Agentic harness framework for spatial reasoning in foundation models. arXiv preprint arXiv:2606.08952, 2026.

[2] Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Zhenfei Yin, et al. Think3d: Thinking with space for spatial reasoning. arXiv preprint arXiv:2601.13029, 2026.

[3] Pengteng Li, Pinhao Song, Wuyang Li, Huizai Yao, Weiyu Guo, Yijie Xu, Dugang Liu, and Hui Xiong. See&trek: Training-free spatial prompting for multimodal large language model. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.

AuthorsOpenReview ↗

Response to Reviewer 2 (3/5)

W4: The limitations and potential extensions are mentioned in the appendix but not in the conclusion. It would be helpful to discuss them more clearly in the main text.

R4: Thank you for this suggestion. In the original submission, we placed the discussion of limitations and potential extensions in the appendix due to space constraints. We agree that these points should also be clearly stated in the main text. We have therefore expanded the Conclusion (Section 5) to discuss this part. The added paragraph is provided below:

ReTAC has some limitations that motivate future work. Our evaluation focuses on geometric and spatiotemporal reasoning, leaving its applicability to causal reasoning and broader multimodal tasks untested. The gains achievable with ReTAC are also constrained by the capabilities of its external tools. Future work could extend residual computation to tasks requiring causal or counterfactual inference and incorporate more capable spatial tools.

W5: The inference cost could be reported more clearly. Although the paper reports model-call counts, it would be helpful to provide the time for each stage, GPU setup, memory usage, and FLOPs.

R5: Thank you for suggesting a more detailed breakdown of inference cost. We use two NVIDIA L20Y GPUs, each with 80 GB of memory. Table 4 compares the computational cost and latency of Direct inference and ReTAC. ReTAC’s mean model calls are taken from Fig. 5.

We report MLLM inference FLOPs only for Qwen3-VL-8B, including visual encoding, prefill, and decoding across all model calls. External tool computation is excluded because its FLOPs are difficult to estimate reliably. GPT-5.5 and Gemini 3.1 Pro are accessed through APIs, so their inference FLOPs are unavailable and marked as N/A.

For Qwen3-VL-8B, we measure end-to-end wall-clock latency from the start of input preprocessing to the final answer, including all model calls, tool executions, and recovery steps. For the proprietary API models, latency is measured on the client side. Peak local GPU memory covers locally executed models and tools. Direct inference through APIs involves no local GPU execution, so its local GPU memory usage is marked as N/A.

Table 4. Computational cost and end-to-end latency of Direct inference and ReTAC.

BackboneMethodMean model callsMLLM FLOPs/question (TFLOPs)End-to-end latency (s)Peak local GPU memory (GiB)
Qwen3-VL-8BDirect1.01601.1426
Qwen3-VL-8BReTAC4.23203.3470
Gemini 3.1 ProDirect1.0N/A7.37N/A
Gemini 3.1 ProReTAC4.1N/A12.1544
GPT-5.5Direct1.0N/A37.76N/A
GPT-5.5ReTAC3.8N/A41.6044

Table 5 breaks down ReTAC’s runtime into native inference, planning, perception, reasoning, verification, and other overhead. All values are mean elapsed times in seconds per question over the full evaluation set, with skipped stages contributing zero. Perception includes external tool execution, and verification includes checks at all three stages. Operations repeated during recovery are attributed to their corresponding stages. Other overhead covers preprocessing and orchestration time not assigned to a specific stage. Total denotes measured end-to-end latency.

Table 5. ReTAC runtime breakdown by stage (seconds per question)

BackboneNative inferencePlanningPerceptionReasoningVerificationOther overheadTotal
Qwen3-VL-8B1.040.271.160.280.490.103.34
Gemini 3.1 Pro5.460.813.210.901.370.4012.15
GPT-5.521.042.448.223.005.920.9841.60

Q1: Even when ChatGPT or other models are used directly, could they already perform internal reasoning or agentic steps?

A1: The models may perform internal reasoning during direct inference. But they receive only the video and question, without access to external tools. “Direct” does not imply an absence of internal reasoning. It denotes a single model call without external tool execution.

Reviewer_M21sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary. ReTAC is a training-free multi-agent framework for video spatial reasoning where one MLLM acts as Planner, Perceiver, Reasoner, and Verifier. Its central claim is that test-time computation should be residual: add compute only for the model's unresolved requirements and recompute only invalidated states, so gains depend on where compute is allocated, not how much.

Contributions.

  1. Residual test-time agentic computation: add compute only for the model's unresolved requirements, keeping compatible native evidence.

  2. Stage-aligned verification and selective recovery: localize the failed stage and recompute only invalidated states and their dependencies, reusing the rest.

  3. Allocation over amount: controlled analyses such as routing interventions show where compute goes matters more than how much.

Validation. Evaluated on three models (Qwen3-VL-8B, Gemini 3.1 Pro, GPT-5.5) and two benchmarks (VSI-Bench-Debiased, STI-Bench), ReTAC attains the highest overall score in all six settings, beating Direct, CoT, See&Trek, and Think3D. Component ablations, routing interventions, answer-transition analysis, and a cost study further support the residual-allocation principle.

Critical strengths
  1. Residual test-time agentic computation: add compute only for the model's unresolved requirements, keeping compatible native evidence.

  2. Stage-aligned verification and selective recovery: localize the failed stage and recompute only invalidated states and their dependencies, reusing the rest.

  3. Allocation over amount: controlled analyses such as routing interventions show where compute goes matters more than how much.

Critical weaknesses
  1. The method is incremental and largely engineering. At its core it combines adaptive on-demand tool use with stage-aligned reflection and localized correction. Both already exist in prior work such as AdaTooler-V and ViSRA. The main difference is routing via the model's self-reported unresolved requirements plus dependency tracking to limit recomputation. This reads as a careful combination and refinement rather than a new paradigm. Dependency tracking also contributes little to accuracy, only about −0.08 to +0.89 over full restart, and mainly saves calls, which weakens its standing as a core contribution.

  2. The baseline set is thin and lacks a compute-matched comparison. Only See&Trek and Think3D are method-level baselines, while Direct and CoT are weak. More critically, ReTAC averages about 4 model calls per question and roughly 8 to 10 on refined questions, yet is never compared against self-consistency or majority voting under the same call budget. Given that "allocation matters more than amount" is the central claim, the absence of this control leaves the roughly 1-point gains unattributable to the selective mechanism and the evidence chain incomplete.

Questions for authors
  1. Beyond existing adaptive tool-use methods such as AdaTooler-V and multi-role reflection frameworks such as ViSRA, what is ReTAC's core conceptual increment? Can you ablate stage-aligned verification and dependency-aware selective recovery separately, to show which one drives the accuracy gain rather than their combined replacement by full restart? Dependency tracking yields only about −0.08 to +0.89 over full restart and mainly saves calls.

  2. Baselines. The only method-level baselines are See&Trek and Think3D. Were these run with their original implementations or re-implemented on your operation interface, and under what frame count and tool budget? More critically, ReTAC averages about 4 calls per question, so why is there no comparison against self-consistency or majority voting under the same call budget? Given that "allocation matters more than amount" is the central claim, the absence of this control leaves the roughly 1-point gains unattributable.

  3. Negligible gains on strong models. On GPT-5.5, ReTAC improves over the best baseline by only +0.98 on VSI, and routing interventions show that suppressing refinement on GPT's refinement subset costs just 2.80 points, far below Qwen's 11.06. This suggests strong models rarely need supplementary compute and that the residual mechanism mainly patches weaker models. For frontier models like GPT-5.5, how much meaningful and statistically significant improvement does ReTAC actually deliver over a single direct inference, and is this heavier multi-agent pipeline still worthwhile on such models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/3)

Thank you very much for your constructive feedback and for recognizing our residual computation principle, selective recovery design, and controlled analyses. Your main concerns focus on the experiments, particularly budget control, the completeness of the ablation studies, the comparison settings, and the meaning for strong model. In response, we have added comparisons under matched computational budgets, reorganized the ablation results to clarify individual component contributions, and clarified the comparison settings.The following are our responses to your suggestions and questions.

W1: Better clarify the novelty of ReTAC.

R1: Prior methods focus on calling tools when needed. We raised a question that has received less attention: for a given problem, how should additional computation depend on the evidence and intermediate states the model has already formed? We explicitly define how existing states constrain the scope of additional computation, which is more specific than just selecting tools based on context. This involves two aspects:

  • Initial supplementary computation: When existing evidence is sufficient to support a requirement, acquiring it again wastes computation and may introduce conflicting evidence.
  • Computation after failure: An incorrect answer does not require restarting the entire solution process. A localized error affects only certain states and their dependents.

We study the allocation of test-time computation and the reuse of intermediate states, aiming to improve final accuracy and reduce unnecessary repeated computation.

W2&Q2.2: In addition to See&Trek and Think3D, comparisons should include self-consistency and majority voting under the same model call budget.

R2: Thank you for highlighting the limitations in baseline coverage and compute budget control. To strengthen our conclusions, we have added the following experiments.

First, we compare ReTAC with two sampling baselines under the same total model call budget. Majority voting generates answers directly, while self-consistency generates stepwise reasoning followed by a final answer for each sample, as described in the original self-consistency paper. We match the budgets as follows:

For a test set of NN questions, let CiC_i denote the number of model calls made by ReTAC for question ii. Its total call budget is B=∑i=1NCiB=\sum_{i=1}^{N}C_i. This count includes the initial response and all model calls during planning, perception, reasoning, verification, and recovery. External tool executions that do not call the base model are excluded.

We assign each baseline the same total budget BB. Let k=⌊B/N⌋k=\left\lfloor B/N\right\rfloor and r=B−kNr=B-kN. Before evaluating the baselines, we use a fixed random seed to select rr questions for k+1k+1 samples each. The remaining questions receive kk samples each. Both baselines use the same allocation.

For both baselines, we aggregate answers by majority vote for multiple-choice questions and by the median after unit normalization for numerical questions. The results are reported in the table below, showing that ReTAC outperforms majority voting and self-consistency under the same model call budget.

Table 1. Comparison under the same model call budget on VSI-Bench-Debiased.

ModelMethodCalls/qVSI Score ↑
Qwen3-VL-8BMajority voting4.247.97
Qwen3-VL-8BSelf-consistency4.248.02
Qwen3-VL-8BReTAC4.253.49
Gemini 3.1 ProMajority voting4.149.89
Gemini 3.1 ProSelf-consistency4.150.73
Gemini 3.1 ProReTAC4.153.80
GPT-5.5Majority voting3.861.12
GPT-5.5Self-consistency3.861.46
GPT-5.5ReTAC3.862.38
AuthorsOpenReview ↗

Response to Reviewer 1 (2/3)

Q1: Clarify ReTAC's core conceptual increment and separate ablations of stage-aligned verification and dependency-aware selective recovery to clarify their individual contributions to accuracy.

A1: Regarding our methodological contribution, prior methods enable tool use when needed or introduce reflection frameworks. As discussed in our response to W1, we address a question that has received less attention: for a given problem, how should additional computation depend on the evidence and intermediate states the model has already formed?

Our contribution is not adaptive tool use, multiagent collaboration, or reflection itself. We explicitly represent which requirements remain unresolved and which states are invalidated by a failure, and use these representations to guide supplementary computation and recovery.

Regarding the ablations, our previous analysis treated residual recovery as a whole. Although Fig. 5a and Table 3 contain relevant results, we did not present them together to distinguish the contributions of stage-aligned verification and selective recovery. We now organize these results in the table below:

  • Row 1: Full ReTAC.
  • Row 2: Replace selective recovery with a full restart while retaining stage-aligned verification. The model receives the localized errors and repeats the solution process from the beginning to avoid these errors. This corresponds to Fig. 5a.
  • Row 3: Further replace stage-aligned verification with general reflection. The model repeats the solution process without explicit error localization. This corresponds to “gen reflection” in Table 3.

In our framework, selective recovery relies on stage-aligned verification to identify the failed states. Removing this verification while retaining selective recovery is therefore not a valid configuration of the current design.

Table 2. Verification and recovery ablations: scores, mean model calls per question, and score gains per additional call over Direct(calculated as (Score−ScoreDirect)/(Calls−1)(\mathrm{Score}-\mathrm{Score}_{\mathrm{Direct}})/(\mathrm{Calls}-1)).

VerificationRecoveryQwen3-VL-8B Score ↑Calls ↓Gain/Call ↑Gemini 3.1 Pro Score ↑Calls ↓Gain/Call ↑GPT-5.5 Score ↑Calls ↓Gain/Call ↑
Stage-aligned verificationSelective recovery53.494.21.80053.804.11.62362.383.80.489
Stage-aligned verificationFull restart52.875.11.24253.884.71.38161.494.50.138
General reflectionFull restart51.584.61.06952.364.21.12260.853.9-0.055

Comparing Rows 2 and 3, stage-aligned verification improves scores while increasing mean model calls. Gain/Call also increases for all three models. With the recovery strategy fixed, these results support the value of stage-specific error localization and targeted feedback over general reflection.

Comparing Rows 1 and 2, selective recovery reduces mean model calls and improves Gain/Call across all three models. These results show that selective recovery reduces repeated computation while maintaining or improving performance.

Overall, stage-aligned verification provides consistent score gains in this comparison, while selective recovery improves computational efficiency and can also improve accuracy.

Q2.1: Clarify baseline implementations, frame counts, and tool budgets.

A2.1: We evaluated See&Trek and Think3D using their official codebases, preserving their original method components. Both baselines used 32 video frames per video. See&Trek retains its semantic keyframe selection and camera-trajectory prompting, while Think3D retains its 3D reconstruction and iterative viewpoint exploration. Our adaptations were limited to integrating the same base models and evaluation datasets used for ReTAC.

We have added the above description to Appendix B.1 (“Shared Evaluation Protocol”) in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1 (3/3)

Q3: For frontier models like GPT-5.5, how much meaningful and statistically significant improvement does ReTAC actually deliver over a single direct inference?

A3: Thank you for raising this concern. We agree that the overall gains on GPT-5.5 are modest. However, the aggregate scores conceal larger improvements on specific tasks, and ReTAC invokes the multiagent pipeline selectively. We provide two observations below.

First, the benefits vary across task types. ReTAC improves 12 of the 16 task scores across the two benchmarks. The largest gains occur in relative direction on VSI-Bench-Debiased (+4.64 points), and displacement and path length (+6.98), dimensional measurement (+5.19), and 3D video grounding (+3.15) on STI-Bench. These results show that supplementary computation remains useful for several spatial reasoning tasks even with a strong backbone.

The smaller gains and slight declines in other categories limit the overall improvement. However, the larger gains in relative direction, dimensional measurement, and displacement and path length show that ReTAC remains useful for these tasks even with a strong backbone such as GPT-5.5.

Second, ReTAC applies the multiagent pipeline selectively. For GPT-5.5, only 725 of 2,362 questions (30.7%) enter this pipeline. The remaining questions return directly after native inference. Weaker models use supplementary computation more often and obtain larger gains, while stronger models return directly more often and obtain smaller gains from a higher starting score. This routing behavior reflects our design goal: adapt computation to the capabilities of different open and closed models, improve final accuracy, and reduce unnecessary repeated computation.

These results support the value of selective computation for frontier models: ReTAC improves tasks that still benefit from additional evidence while avoiding the full multiagent pipeline for most questions.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your thoughtful feedback and constructive suggestions, which have helped us strengthen our work. During the rebuttal period, we believe we have addressed your concerns through additional experiments and clarifications, including those regarding the completeness of our ablations, computational trade-offs, comparisons with closely related methods, and broader efficiency metrics.

We have uploaded a revised manuscript with the following updates in response to your comments:

  • Additional method comparison (Tables 1 and 2): We added AlloSpatial, a closely related concurrent work first released in June 2026, to the main experimental comparisons.
  • Baseline descriptions (Section 4.1): We added introductions to the compared methods to clarify their approaches and differences.
  • Clearer ablation studies: We reorganized the ablation table to make the effect of each proposed component easier to assess.
  • Computational cost and budget (Section 4.5): We added a section with two complementary evaluation metrics to better characterize the trade-off between performance gains and computational cost.
  • Future work (Section 5): We moved the discussion of future work from the appendix to the main text to make research directions more explicit.
  • Baseline configurations (Appendix B.1): We added detailed execution configurations for the compared methods.
  • Cost measurement details (Appendix B.3): We added the subsection “Measurement of Computational Cost,” explaining the metrics and procedures used to evaluate computational cost.

As the discussion period draws to a close, we would greatly appreciate your feedback on our responses and revisions and welcome the opportunity to discuss and clarify any remaining concerns. If you believe that our responses and revisions address your concerns, we would be grateful if you could reconsider your assessments and scores accordingly.

Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.