acceptodds
Withdrawn from ICLR 2027

IROHA: Long-Horizon Visual Reasoning with an Agentic Harness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Visual reasoning asks a generative model to solve a task by generating its execution. Direct generation produces a candidate solution, but leaves open whether it satisfies the task and how to proceed if it fails. In long-horizon tasks, deciding what to do next also requires checking whether earlier actions have established the conditions needed by later ones. We introduce vIsual Reasoning Oriented Harness for Agents (IROHA) that makes this control explicit around frozen video and world-action models. Planning defines task objectives, while executable tools produce candidate visual solutions and first-frame guidance for structured tasks, or visual grounding for natural scenes. Verification checks whether generated outcomes realize these requirements; its feedback guides candidate selection or revision of the task analysis and generation setup. With frozen state-of-the-art video and world-action backbones, IROHA improves the VBVR score from 0.65 to 0.78 and LIBERO-Plus simulator success rate from 72.8% to 74.8%. It also improves the RBench scores across three backbones, including gains on long-horizon instructions. Intriguingly, our RBench analysis shows substantial gains in task accomplishment alongside nearly unchanged rendering consistency. We further demonstrate how IROHA assembles multistep video solutions to the challenges like Hanoi Tower in real-world style scenes, using state checks, retries, and backtracking. These findings show that explicit reasoning and feedback unlock the potential of existing generative capabilities in complex tasks, without additional training or parameter updates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic AI, harness engineering, video reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xKhrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IROHA, a training-free agentic harness that integrates planning, executable tools, video generation, and verification to improve visual reasoning with frozen backbones. Experiments across VBVR-Bench, RBench, and LIBERO-Plus demonstrate consistent performance improvements.

Critical strengths

The paper addresses an interesting problem and demonstrates promising improvements across multiple benchmarks and generative backbones. The experiments are relatively comprehensive, including ablation studies and comparisons with existing methods.

Critical weaknesses
  1. The proposed IROHA framework largely follows established agentic workflows, including planning, tool execution, verification, and feedback-driven refinement, which are already standard capabilities of existing agent frameworks and harnesses. The paper mainly applies these existing components to video generation without introducing a substantially new algorithm or mechanism. In particular, it remains unclear what technical challenges specific to visual reasoning are addressed by IROHA that cannot be handled by existing agent frameworks.

  2. Despite emphasizing long-horizon reasoning as its central contribution, the main experiments do not convincingly evaluate this capability. For instance, the RBench evaluation primarily relies on prompt rewriting and best-of-three candidate selection, rather than sequential reasoning with state tracking and error recovery. Although the paper includes multistep demonstrations and a rule-based planner benchmark, these do not provide sufficient quantitative evidence of long-horizon visual reasoning in generated videos.

Questions for authors

I think this paper should provide a more explicit comparison with existing agent frameworks and harnesses, ideally through a controlled baseline using a general-purpose agent framework with the same tools and backbones, to demonstrate what IROHA uniquely contributes beyond existing agentic workflows.

Reviewer_UmZoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IROHA, a training-free “agentic harness” that wraps frozen video generators and world-action models with a planner, a tool registry, and a verifier. The motivation is that a visually plausible clip can still fail the task, for example an object placed inside a bin instead of to its left. A planner (gpt-5.4) analyses the first frame and instruction and rewrites the prompt. Hand-written executable tools (A* search, contour ranking, analytic physics, grid edits, etc.) compute a symbolic solution and supply a guided starting frame. Grounding and tracking modules recover entity states from the generated frames, and an MLLM-plus-heuristic verifier accepts candidates, rejects them, or triggers revisions and backtracking. The control loop differs by benchmark: segment-level revision on VBVR-Bench, best-of-three selection on RBench, and per-control-block trajectory selection on LIBERO-Plus. Reported gains are 0.65 -> 0.781 on VBVR-Bench (matching VLM-as-Teacher without parameter updates, with near ID/OOD parity), RBench improvements on three backbones, and +2.02 pp on LIBERO-Plus. A tool-free RBench diagnostic shows task accomplishment rising while rendering consistency barely changes. The authors also introduce R2bench, a rule-checked planner benchmark in synthetic “real-world style” scenes, and show multistep Tower-of-Hanoi videos with state checks and backtracking. A held-out V-ReasonBench evaluation with test-time tool synthesis reports 0.351 -> 0.484.

Critical strengths
  1. The paper separates task correctness from visual plausibility and argues convincingly, with good qualitative examples, that frozen generators fail on logical constraints that external computation can resolve.
  2. It covers three public benchmarks with different scoring regimes (rule-based, VLM-judged, simulator success), three backbones on RBench, a held-out benchmark obtained after the registry was fixed, and a new planner benchmark.
  3. The paper include a re-run paired baseline at identical settings, best-of-K with the same verifier, component ablations, a tool-disabled probe, still-image-edit and held-frame controls, task-type clustered bootstrap, and cross-family judge rescoring.
  4. The VBVR score matches a test-time LoRA method, and the OOD score is on par with the ID score.
  5. Task accomplishment improves while rendering consistency stays flat, and some consistency criteria even drop. This supports the paper’s argument that task completion should be evaluated as a separate objective.
  6. The paper includes an analysis of the 27 task types below 0.70, a synthesized-tool failure mode, selector–judge agreement caveats, and cost caveats.
  7. The multistep Hanoi/assembly traces show legality checks and backtracking, with rollout accounting.
Critical weaknesses
  1. The tools have been engineered on the evaluation suite. Appendix B.5 acknowledges that VBVR-Bench is “the suite used to develop the registry”. The ten tools are task-family solvers whose output becomes the generator’s starting frame (Alg. 1, line 9). The verifier proxy also hard-codes VBVR conventions, such as counting newly red pixels with fixed RGB thresholds. The “OOD” split is OOD only for the backbone’s fine-tuning corpus, not for tool design, so ID/OOD parity is not evidence of generalization. The comparison in Table 1 with training-based methods that never saw benchmark-specific solvers is therefore not like-for-like.
  2. Without tools, the harness adds about what plain sampling adds. Removing tool guidance gives 0.675, versus 0.650 for the backbone. Best-of-3 with the same verifier gives 0.676. This suggests that planning, verification, and revision alone contribute little beyond extra samples on VBVR, and that most of the 0.131 gain comes from the hand-coded solvers. Perception tasks alone contribute 57% of that gain.
  3. The structural check (App. L) can replace the committed video with pixel-composited “restored” variants of I0, or with the frozen rollout. The ablation effect is small (−0.005), but it raises the question of what exactly is scored as a “generated” video, and it should be disclosed in the main text.
  4. The paper presents modest and confounded gains. Indeed, a) on RBench, three candidates are compared against one, with MLLM selection judged by an MLLM scorer. Tools add only +0.006 to +0.023 over the tool-free harness. The margin over NEWTON (+0.02 to +0.05) is against a version stripped of its trained planner. b) on LIBERO-Plus, the gain is +2.0 pp with a wide CI ([+0.43, +4.45]), at three imagined rollouts plus an MLLM call per control block, and with a drop on texture perturbations.
  5. The “long-horizon” claim is under-supported quantitatively. VBVR episodes are mostly short (8/500 reach the 5-segment cap). R2bench measures text-level planner legality, not video generation. The Hanoi success figure (90/100) is certified by the same gpt-5.4 that acts as planner and verifier, with no independent symbolic or human check of the final videos. Fig. 7 compares methods under different budgets.
  6. Limited conceptual novelty and a benchmark-specific system. The approach combines known ingredients: visual programming (VISPROG/ViperGPT), reflexion-style revision, best-of-N selection, and NEWTON-style plan-and-verify. Each benchmark gets a different loop and many hand-set thresholds (0.7/0.3 blend, v≥6, q≥0.5, 0.95 correlation stop). The result reads more as a strong engineered pipeline than as a general method.
  7. Reproducibility and statistical scope. Each configuration is a single end-to-end run with one seed and depends on closed APIs (gpt-5.4/5.2). The CIs cover case variation only, and the paper notes that repeated API calls may yield different outputs. V-ReasonBench is evaluated with one run instead of the released pass@5 protocol, and its synthesized programs are cached in an order-dependent way. End-to-end compute is unmeasured: one RBench trace takes about 15 minutes of generation for three calls.
Questions for authors

Q1) Discuss in details the critical weaknesses discussed above.

Q2) What exactly does the “tool-provided starting frame” contain? Does it already depict the final answer (e.g., the red ring or the completed path), so that the generator mainly animates toward it? Please show score breakdowns by route (tool / fallback / prompt revision) and the fraction of samples on each.

Q3) What access did tool developers have to VBVR-Bench instances, and was there a held-out development/test split within VBVR? How were the verifier thresholds and proxy weights chosen, and how sensitive is the 0.781 to them?

Q4) In what fraction of VBVR samples is the final scored output a restored composite or the frozen rollout rather than the harness’s committed generation?

Q5) On V-ReasonBench, how many of the 13 families were handled by pre-existing registry tools versus test-time synthesized programs, and what are the results under the official pass@5 protocol?

Q6) For LIBERO-Plus, what do a random choice among the three candidates and an MLLM-only selector achieve? Is the texture regression due to the detector or to the MLLM signal?

Q7) Which model is the official RBench judge, and how does it relate to the gpt-family selector? Can the full configuration (with tools), not only the tool-free one, be rescored by a different-family judge?

Q8) How much does the VBVR gain depend on the planner? Does the harness still help with an open-weight planner, given that R2bench shows large planner-dependent variation (e.g., Haiku +4 pp)?

Q9) What is the variance across seeds and across repeated end-to-end runs? Can you report at least three seeds for the main VBVR and RBench rows?

Q10) For the Hanoi demonstrations, can the 90/100 goal-reaching rate be confirmed by an independent symbolic state extractor or by human raters? How is object identity and appearance kept consistent across 15 or more concatenated segments?

Q11) What are the end-to-end GPU-hours and API cost per sample for IROHA, compared with the frozen baseline and with VLM-as-Teacher’s ~16 LoRA steps?

Q12) The best-of-K bootstrap intervals in Table 15 are strongly asymmetric (e.g., +0.021 [+0.009, +0.062]). How were they computed?

Q13) Could the 18 sub-0.70 task types that lack a tool be addressed by test-time synthesis on VBVR itself? This would directly test whether the approach scales beyond a hand-built registry.

Reviewer_8bQQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper reformulate long-horizon execution as a task-state management problem and propose LongHorizon-Harness, which maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment.

Critical strengths

1.The harness updates no model parameters, so it is broadly applicable and cheap to adopt. 2.VBVR-Bench improves from 0.650 to 0.781. RBench improves by 0.08–0.15. 3.Task accomplishment versus rendering consistency is meaningful.

Critical weaknesses

1.VBVR-Bench are largely attributable to hand-engineered solvers. Removing tool guidance lower the score from 0.781 to 0.675 (Tab13), which accounts for roughly 81% of the +0.131. 2.LIBERO-Plus gain is small and fragile, and largely explained by resampling rather than verification.

Questions for authors

1.In Appendix K, the success rates on the 1,737 common cases are 74.67% (frozen), 76.34% (random selection), and 77.37% (harness). These imply a harness-vs-random difference of +1.03 pp, yet the reported 95% interval for this contrast is [+1.38, +3.31], which not contain the point estimate. Are the two intervals mislabeled, or computed for a different contrast (e.g., harness vs. frozen, +2.70 pp)? I want to know the point estimate, interval, and computation procedure for all three pairwise contrasts.

2.Why do roughly 736 instances within the larger clusters lack random-selection results?

3.Why does random replacement of the default yield a significant improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.