acceptodds
Under review as a conference paper at ICLR 2027

Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents

Abstract

Omni-modal agents must seek evidence across video, audio, web pages, and computation to answer questions. Their main bottleneck is planning: noisy multimodal observations accumulate in conversation history and disrupt later decisions, while multimodal models have limited capacity for multi-step planning. Controlled backend replacements support this diagnosis: replacing the planner causes a much larger performance loss than replacing the perception backend. We present Omni-Decision, an omni-modal agent built on evidence-ledger planning: it replaces the growing dialogue history with an explicit evidence ledger that records what evidence is still missing, what has been confirmed, and where records conflict. A critic reads each noisy observation and passes only the usable content to the ledger, discarding the rest, so the planner works from a compact context throughout the task. Each run records the state, action, and verdict at every step, and supervised fine-tuning and decision-level reinforcement learning on these trajectories further improve the planner. Omni-Decision achieves state-of-the-art accuracy of 81.4% on OmniGAIA at approximately 43% of Gemini-3.1-Pro's cost per question, and 65.0% on WorldSense long-video understanding, level with the strongest end-to-end model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal agents, agentic planning, long-video question answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XYBPOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Omni-Decision, an omni-modal agent that uses a typed evidence ledger to track unresolved needs, verified facts, and conflicts. A planner, critic, and reducer share this state to guide evidence acquisition and finalization. The system is evaluated on OmniGAIA and WorldSense, and its trajectories support supervised fine-tuning and decision-level reinforcement learning.

Critical strengths
  1. The evidence ledger provides a clear way to track missing information and conflicting evidence across heterogeneous tool outputs, whihc provides a useful design for tasks requiring several dependent evidence-gathering steps.
  2. The experiments cover both open-world evidence seeking and question answering. The competitive results and measured computational costs demonstrate the system’s practical performance.
  3. The recorded states, actions, and verdicts provide reusable supervision without step-level manual annotation. Training weaker planners on these trajectories is a useful extension of the inference framework.
Critical weaknesses
  1. The evidence-readiness mechanism is not consistently described across the method and prompts. Allowing answers despite insufficient evidence metadata and permitting forced finalization leave the scope of the claimed evidence-closure control unclear.
  2. The authors can also discuss if the planner makes the wrong decision at the beginning, and the way to mitigate the impact.
  3. The insights of the LEDGER can be detailed and made clearer. For example, if it helps drop unrelated information, and if it help models make new and correct plans.
Questions for authors

See weaknesses.

Reviewer_qUCeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Omni-Decision is an omni-modal agent that organizes evidence-seeking tasks through a structured ledger of unresolved needs, facts, supporting evidence, and conflicts. A critic filters tool observations, while a deterministic reducer updates the ledger to guide subsequent actions and answer readiness. The paper also uses recorded execution trajectories for supervised fine-tuning and decision-level reinforcement learning of weaker planners. Experiments report 81.39% accuracy on OmniGAIA and 65.0% on WorldSense, with lower inference costs than direct Gemini-3.1-Pro answering, alongside backend replacements, state ablations, and failure analysis.

Critical strengths
  1. The evidence ledger integrates requirement tracking, evidence provenance, conflict handling, and stopping decisions into an explicit state, making the agent’s execution easier to inspect.
  2. The system achieves competitive accuracy at substantially lower reported inference cost, while comparisons against raw-history and summary-memory controls support the practical value of its state-management design.
Critical weaknesses
  1. Table 2 does not establish the broad claim that planning is the dominant bottleneck of omni-modal agents, because the measured losses depend on which models replace the original backends. Replacing GPT-5.2 with GLM-5.2 reduces accuracy by only 6.39 points, whereas every perception replacement reduces it by 15.83–22.50 points; the much larger planning losses arise with weaker Qwen alternatives. Moreover, the highlighted comparison uses different Qwen3.5-Plus and Qwen3.5-Omni-Plus variants, starting from different reference backends. These results demonstrate sensitivity to particular backend choices, but do not adequately separate an intrinsic planning limitation from differences in model capability and compatibility with the framework.
  2. The main text does not identify the sources of the training tasks or explain how their questions, media, and trajectories are separated from the 360 OmniGAIA evaluation questions, leaving the generalization significance of Table 4 unclear. In addition, the reinforcement-learning stage scores proposed decisions without executing them, using closure checks and an LLM judge, but does not validate whether these scores predict successful evidence acquisition. This matters because apparent progress toward closing the ledger need not produce correct evidence. Together, these omissions prevent a clear assessment of whether post-training improves transferable planning ability or primarily fits the training tasks and reward heuristic.
Questions for authors
  1. Figure 3 includes 22 incorrect cases with fully closed ledgers. What caused these failures, and how can the system reopen a closed need, correct an accepted fact, and update dependent computations when that fact is subsequently found to be wrong?
  2. How does the planner’s context remain approximately constant as evidence and conflicts accumulate? Please clarify what information is omitted from the bounded ledger representation and whether the planner can recover omitted evidence when it becomes relevant to a later decision.
2 comments

Dear Area Chair and Authors,

I would appreciate it if the Area Chair could first confirm whether the submission complies with the ICLR 2027 formatting requirements. I have raised this question in previous confidential comments and would welcome clarification.

Thank you to the authors for their response. My remaining concern is whether a tool call that receives a higher reward is more likely to return correct evidence. In Table 4, adding RL to SFT yields a net gain of three correct answers across the 238 held-out Medium and Hard questions. The planner could achieve this gain by learning when to keep searching, even if its ability to choose an effective tool call stays the same. This distinction matters because alignment with the teacher’s reference decision accounts for 60% of the reward.

We would be happy to help the Area Chair clarify the formatting question. If there is a specific concern, we would appreciate a pointer to the relevant requirement and the passage or page in our submission so that we can address it directly.

Table 4 shows improved planner performance on held-out tasks. With the perception model, available tools, and execution budget held fixed, adding RL to the SFT planner increased the number of correct answers from 98 to 101 across the 238 held-out Medium and Hard questions. You suggest that the planner may have learned when to keep searching. Recognizing from the current evidence that a key question remains unresolved and deciding whether to gather more evidence or answer is itself a planning decision. As §3.4 explains, improving this decision is an explicit training objective. Even if the entire gain arose through the mechanism you propose, it would still represent an improvement in the planning capability we set out to train.

The 60% figure in Appendix F.3 is the weight assigned to alignment with the teacher’s reference decision. The rubric also gives high scores to reasonable alternative actions that address the current evidence need. Prior controlled experiments support this training approach. With the same framework, backbone, and tools, Orchestra-o1 reports 28.6% task accuracy for SFT and 30.0% for DA-GRPO (Table 2). Your question about whether higher-scoring calls more often return correct evidence concerns the reward at the level of individual tool calls. Table 4 evaluates the learned planning policy as a whole under actual tool execution and shows that it answered more held-out questions correctly within the same budget.

Reviewer_mnqvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Omni-Decision, an agent architecture designed to solve the planning and context-pollution bottlenecks in omni-modal, long-horizon tasks. Through controlled experiments, the authors demonstrate that current multimodal agents fail primarily because noisy multimodal observations (e.g., long transcripts, cluttered web pages) accumulate in the dialogue history, overwhelming the planner. To solve this, Omni-Decision replaces the traditional chronological dialogue history with an Evidence Ledger, a structured, typed state that tracks open needs, confirmed facts, and conflicting information. The system separates cognitive load: a Critic evaluates single, raw observations and extracts relevant facts, a deterministic Reducer updates the ledger, and the Planner only reads this clean, compact ledger to decide the next action. Because the ledger naturally structures the reasoning process, the authors also introduce a post-training recipe that uses ledger states to generate dense, step-level rewards to train open-source models (like Qwen) without human annotation. The system achieves State-of-the-Art (81.4%) on OmniGAIA, outperforming Gemini-3.1-Pro (79.4%) at just 43% of the cost, and demonstrates strong performance on WorldSense.

Critical strengths
  • Because the Planner only reads a condensed state rather than the entire cumulative history of web pages and video transcripts, the token usage per step remains flat (Figure E.1). This leads to a better-performing agent that costs less than half of a direct Gemini-3.1-Pro prompt.
  • The system's failures are highly transparent. Researchers can look at the terminal ledger and see exactly what remained unfilled or which conflict was unresolved, making debugging and failure attribution much easier than standard black-box agent traces.
Critical weaknesses
  • Currently, the Critic only evaluates the textual output returned by perception tools. It does not look at the raw media itself. Therefore, if a video-captioning tool hallucinates or misses a visual detail in its text output, it is hard for the Critic to cross-reference the actual video, passing the error downstream.
  • The paper does not cite comprehensive papers in related work, e.g., one hierarchical multi-agent multimodal systems, such as DEPART [1] also observes the noise of the vision input and build mulit-agent systems as well. Integrating this would contextualize the work better.
  • It is unclear from the experiments how reliably the system recovers if the initial checklist generated at Step 1 is fundamentally flawed.

[1] Hsu et al (2025). DEPART: A hierarchical multi-agent system for multi-turn interaction. NeurIPS 2025 Workshop on Multimodal Algorithmic Reasoning.

Questions for authors
  • How often does the system successfully recognize that it needs to add a new dependency to the ledger mid-task? If the initial parse of the query misses a crucial step, how robust is the system at recovering?
  • Have you experimented with providing the Critic direct access to the multimodal inputs (e.g., passing the image/video clip directly to the Critic along with the tool's text output)? How much would this improve the Media Perception failures, and how would it impact the cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.