acceptodds
Desk rejected at ICLR 2027

OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal large language models (MLLMs) and multi-agent systems have advanced visual generation. However, three limitations remain. (1) Existing methods often distill task-specific experience with limited generalizability. (2) Reflection is often deferred until task completion. (3) Knowledge is often acquired only in response to downstream task demands. To address these limitations, we introduce OmniHarness, a framework for generalizable visual generation via symbolic policy learning. OmniHarness abstracts verified executions into symbolic policies for visual generation task families, capturing shared procedures and applicability conditions while removing instance-specific inputs. The harness instantiates, adapts, and composes these policies for new tasks. Intermediate verification guides refinement and failure recovery during execution. Through self-directed inquiry, OmniHarness autonomously generates and executes practice tasks near its capability limits before downstream objectives are specified. Execution feedback continually refines the policies while model parameters remain fixed. Experiments across six benchmarks, three MLLM backbones, and three visual agent frameworks demonstrate strong performance and continual capability expansion. On ComfyBench's Creative tasks, OmniHarness achieves a 95.0% resolve rate, exceeding the strongest baseline by 27.5 percentage points. Frozen policy snapshots improve existing visual agent systems through plug-and-play reuse.

Primary area:
generative models
Keywords:
Generalizable Visual Generation, Symbolic Policy Learning, Agent Harnesses, Multi-Agent Systems, Self-Directed Inquiry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f4uWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OmniHarness, an agent framework that improves visual generation by accumulating experience in an external library. Before downstream tasks arrive, the system autonomously generates and selects practice tasks, abstracts verified execution workflows into reusable symbolic policies, and stores failure experience and corrective strategies. The authors evaluate the framework through multiple visual generation benchmarks, component ablations, and experience-library transfer experiments.

Critical strengths
  1. The experiments cover a broad range of benchmarks and include extensive ablation studies, providing good support for the system's effectiveness.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Some claims about the contributions are overstated. The paper primarily integrates existing mechanisms into a system and validates it through extensive experiments, making the contribution incremental overall. Symbolic workflow abstraction, experience accumulation, and feedback-driven recovery have already been explored in prior work.
  2. The paper lacks a complete cross-task example showing how experience is distilled and subsequently applied to another task.
  3. The performance gains across task categories on ComfyBench are uneven. Complex tasks mainly involve composing Vanilla workflows and would theoretically be expected to be easier to learn. However, why does experience accumulation substantially improve Creative tasks without demonstrating an advantage on Complex tasks under the same underlying model?
  4. Although practice for GenEval, GenEval2, and WISE does not access their evaluation tasks, the paper does not sufficiently clarify whether the experience library continues to be updated during evaluation or whether it is reset between benchmarks.
Questions for authors

See Weaknesses

Reviewer_R7xYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To my best understanding about this paper, OmniHarness is a ComfyUI-based visual agent that stores successful workflows as reusable templates. Before evaluation, it generates practice tasks, executes them with intermediate verification and recovery, and builds workflow and failure libraries. For a new task, it retrieves, modifies, or combines stored workflows. The paper evaluates the system on six benchmarks and reports 92.5% Total Resolve and 95.0% Creative Resolve on ComfyBench.

Critical strengths
  • The paper integrates practice-task generation, workflow reuse, intermediate verification, and local recovery into one working system. The failure library is also useful in practice.

  • The evaluation is broad. The ablations show that self-directed inquiry, planning, and intermediate verification improve ComfyBench performance.

Critical weaknesses
  • The paper is difficult to follow. The core idea appears to be structured ComfyUI workflow memory, but it is described using many modules and broad terms such as "symbolic policy learning." Important implementation details are scattered across the appendix. Besides, Section 3.1 introduces too many symbols without enough explanation or examples. The notation hides a relatively simple task-selection idea and makes the method difficult to follow.

  • The main ComfyBench result allows the policy library to keep learning during evaluation. This means that later test tasks may benefit from earlier test tasks. The paper includes an ablation without online updates, but it does not test different task orders.

  • OmniHarness also uses 50 practice rounds, intermediate checks, and multiple retries. Since the baselines may use less computation, it is unclear whether the gain comes from better policies or simply more computation.

Questions for authors
  • What is the exact difference between a symbolic policy and a stored parameterized ComfyUI workflow? What part enables transfer beyond ordinary workflow retrieval?
1 comment
AuthorsOpenReview ↗

We Would Be Grateful to Hear from You Before the Discussion Closes

Dear Reviewer R7xY,

As the discussion deadline approaches, we sincerely hope to hear from you while there is still time to address any remaining concerns. Your feedback would mean a great deal to us, and we would be grateful to know whether our response has answered your questions. If anything remains unclear, we are ready to provide further clarification or additional information promptly.

We deeply appreciate the time and care you have devoted to reviewing our work and helping us improve it.

Reviewer_YCRwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

OmniHarness is a visual-generation agentic framework which optimizes reusable symbolic workflow policies. It combines self-directed inquiry, feedback-guided execution, and persistent workflow/failure libraries. The paper evaluates six benchmarks covering workflow construction, compositional generation, knowledge-informed synthesis, and image editing. On ComfyBench, Codex GPT-4o + OmniHarness achieves 92.5% Total Resolve and 95.0% Creative Resolve. It also achieves 0.997 on GenEval (which is the highest score in the leaderboard), 89.78 on GenEval2, and 0.86 on WISE.

Critical strengths
  1. Practical integration of workflow reuse and local recovery The method describes policy preconditions, dependencies, reliability statistics, failure records, a restricted Code-as-Policy representation, and intermediate-output verification. These details make the framework more actionable than a purely conceptual proposal. The ComfyBench gains, especially on Creative tasks, suggest that pre-task practice and structured reuse can improve complex workflow execution.

  2. Strong performance across diverse visual-generation benchmarks. OmniHarness achieves 95.0% Creative Resolve on ComfyBench, exceeding SymbOmni by 27.5 percentage points (Table 1). It also achieves overall scores of 0.997 on GenEval, 89.78 on GenEval2, and 0.86 on WISE (Tables 2–4). These results demonstrate its effectiveness across workflow construction, compositional generation, and knowledge-informed synthesis.

Critical weaknesses
  1. The benefit of symbolic abstraction is not sufficiently isolated. Removing inquiry also removes experience and computation. Matched workflow-retrieval, experience-memory, and practice-budget baselines are needed to distinguish symbolic learning from richer memory and additional inference. Comparative end-to-end costs should also be reported.

  2. The reliability of verifier-guided policy learning remains insufficiently evaluated. In my opinion, OmniHarness needs verification/reward outcomes to retain workflows, update reliability estimates, and guide future reuse. Therefore, incorrect judgments may affect subsequent policy selection. Verifier accuracy, robustness to noisy feedback, and independent policy evaluation should be assessed.

  3. The ablation results lack a clear aggregation protocol. Tables 7–8 report totals that cannot be reconstructed from the displayed modality-specific results. Explicit subgroup counts, aggregation rules, and uncertainty estimates are needed to assess the component contributions.

Questions for authors
  1. What task groups and denominators are included in the Creative/Complex totals in Tables 7-8?

  2. Can you report the cost of self-directed inquiry, end-to-end latency, and cost–quality trade-offs? These measurements would help assess the framework’s practical utility.

1 comment
AuthorsOpenReview ↗

We Would Be Grateful to Hear from You Before the Discussion Closes

Dear Reviewer YCRw,

With the discussion period coming to an end, we very much hope to hear your thoughts on our response before the deadline. We would appreciate knowing if any questions remain or if there is anything we could explain more clearly. We are ready to respond promptly and would welcome the chance to discuss any outstanding concerns with you.

Thank you for your careful review and helpful suggestions. We sincerely appreciate your time and the valuable input you have provided to improve our manuscript.

Comments on the paper

1 comment

We sincerely thank the Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs for their time and effort throughout the review process. Upon receiving the reviews, we carefully considered each concern, conducted additional experiments, and improved the manuscript accordingly. We posted our responses as soon as these updates were complete. We greatly value this opportunity and have done our best to address the feedback with care.

We also appreciate the reviewers’ thoughtful evaluations and their recognition of the following strengths of our work:

  1. The experiments cover a broad range of benchmarks, providing strong evidence of the system’s effectiveness. (Reviewers YCRw, R7xY, and f4uW)
  2. Extensive ablation studies demonstrate the contributions of the system’s core components. (Reviewers YCRw, R7xY, and f4uW)
  3. Symbolic policy learning demonstrates practical value in visual generation. (Reviewers YCRw and R7xY)
  4. The paper is clearly written and easy to follow. (Reviewer f4uW)

A major concern shared by the reviewers is how symbolic policy learning differs from retrieving parameterized workflows and what enables its advantages. Another common concern is computational efficiency, including the tradeoff between cost and quality and comparisons under matched computational budgets. Reviewers also requested stronger evidence of system robustness, particularly verifier reliability and sensitivity to task order during online policy updates.

To address these concerns, we conducted additional experiments and analyses, including:

  1. We focused the contribution statements on Self-Directed Inquiry and Symbolic Policy Learning, reducing emphasis on established execution mechanisms. The method now follows Self-Directed Inquiry, Symbolic Abstraction, and Symbolic Retrieval, with essential details moved from the appendix to the main text. We reduced unnecessary notation while retaining key definitions and formulas, and added a complete cross-task reuse example in Figure 4, making task selection and the overall learning process easier to follow. (Reviewers R7xY and f4uW)
  2. We added symbolic policy learning ablations in Tables 7 and 8. The results show that abstraction across task families, applicability conditions, adaptive instantiation, and composition enable more effective reuse than storing parameterized workflows alone. Symbolic retrieval over metadata and reliability prioritization further improve retrieval quality while reducing downstream reasoning costs. (Reviewers YCRw and R7xY)
  3. We added an analysis of inquiry costs and downstream reasoning efficiency in Figure 9 and Table 8. The results support 50 inquiry rounds as a practical balance between cost and quality. With the same initial library and retry budget, OmniHarness achieves higher Resolve with fewer tokens and shorter reasoning latency than the compared agents on Vanilla and Creative tasks. (Reviewers YCRw and R7xY)
  4. We evaluated verifier reliability under four levels of policy corruption across five randomized task orders. Figure 8 shows high verifier accuracy against independent human judgments and stable official Resolve rates with limited variation across task orders. These results support the robustness of both the verifier and the overall system to noisy stored knowledge and changes in task order during online policy updates. (Reviewers YCRw and R7xY)
  5. We added Section 4.1, Experimental Setup and Evaluation Protocol, and Appendix A.2, Additional Details and Clarifications, to specify implementation settings, comparison conditions, and policy library updates within benchmarks and resets between benchmarks. Table 14 provides the complete modality breakdown, subgroup counts, and resolved counts, clarifying the denominators and aggregation of the Creative and Complex totals. (Reviewers YCRw, R7xY, and f4uW)

All these results are consistently encouraging, providing strong empirical evidence of the effectiveness and generalizability of OmniHarness while demonstrating its efficiency and robustness.

Our rebuttal received positive recognition from Reviewer f4uW (confidence: 4), who explicitly confirmed that our revisions addressed the concerns raised in the review. Although the other two reviewers have not yet participated in the discussion, we have carefully considered their feedback and made every effort to address it through detailed responses and additional experiments. We are confident that these efforts have addressed the issues they raised.

We sincerely thank the reviewers for their thoughtful comments, which have helped us strengthen this work. We have learned a great deal from this process, and these lessons will continue to guide our future research. We hope our work can provide useful insights and contribute to the broader development of agent research. As the discussion draws to a close, we would greatly appreciate any further responses from the reviewers. We deeply value this opportunity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.