acceptodds
Under review as a conference paper at ICLR 2027

CodeOmni: A Code-Driven Harness for Extensible Omni-Modal Reasoning

Abstract

The code-as-action/tool paradigm has been widely explored in prior work. However, existing approaches either underutilize the expressiveness of code or remain tailored to a single scenario. To address these limitations, we propose CodeOmni, a code-driven harness that uses executable programs to orchestrate interactions between models and harness components, providing a unified execution substrate for image, video, audio, and audio-visual tasks. In CodeOmni, frozen models generate Python programs that jointly support tool orchestration, control flow, exact computation, and task-local state management. A variable-mediated workspace further stores intermediate artifacts outside the main conversational context and exposes them only when needed, reducing unnecessary context accumulation. New tasks can be incorporated through lightweight runtime profiles and task-grounded skills, which guide tool selection and program structure. Extensive experiments involving eight models and seven multimodal benchmarks show that unconstrained code generation alone is insufficient and can lead to suboptimal performance. In contrast, equipping the harness with task-grounded skills consistently improves performance over direct model inference in most evaluation settings. As a preliminary exploration of code-driven harnesses, CodeOmni demonstrates the potential of executable code as a shared and extensible substrate for multimodal model interaction, and provides insights toward the development of more capable and general harnesses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
harness,code-based,omni

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_geDxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CodeOmni, a multimodal agent framework that lets frozen models solve tasks by writing and executing Python code. Instead of directly answering a question, the model can use code to call tools and sub-agents, perform deterministic operations, and store intermediate results outside the main context. The framework also uses runtime profiles and task-specific skills to constrain how the model solves different tasks. The authors evaluate CodeOmni on image, video, audio, and audio-visual benchmarks. A key finding is that simply giving the model code execution and tools is often not enough and can even hurt performance, while adding task-specific skills leads to much more consistent improvements, especially on structured visual tasks.

Critical strengths
  1. The paper provides a broad empirical study of code-driven multimodal orchestration and identifies a useful negative result. Across image, video, audio, and audio-visual tasks, the experiments consistently show that simply giving a model access to code execution and tools does not reliably improve performance and can substantially hurt it. The comparison between DIRECT, HARNESS, and HARNESS+SKILL therefore provides useful evidence that enlarging the action space alone is insufficient, and that stronger task structure is necessary for reliable agentic execution. This empirical finding is practically relevant for the design of multimodal tool-using systems, even if the proposed solution relies heavily on task-specific skills.

  2. CodeOmni offers a coherent and reasonably general engineering abstraction for integrating code execution, tools, sub-agents, and external state across multiple modalities without model training. The separation between a shared executable core, runtime profiles, registered tools, and task skills gives the framework a modular structure that can accommodate image, video, audio, and audio-visual tasks within the same planner–executor interface. Although the individual components are not especially novel, the unified design is practically useful and provides a concrete systems framework for studying how frozen multimodal models interact with executable tools and persistent task state.

Critical weaknesses
  1. The main gains are difficult to attribute to CodeOmni itself because they depend heavily on handcrafted benchmark-specific skills. In Appendix H, several “skills” already implement the core solution procedure rather than merely guiding the agent: VSP-Grid uses a fixed BFS-based solver after extracting the grid, BLINK-Jigsaw uses a deterministic seam-matching algorithm, and Word Search uses a dedicated image-processing pipeline based on grid estimation, glyph masks, and Hamming-distance matching. This matters because the unskilled HARNESS is often no better than direct inference and can degrade substantially, while the large improvements appear only after these task-specific procedures are added. The current evaluation therefore shows that CodeOmni can execute carefully engineered solvers, but does not cleanly demonstrate that the proposed harness itself provides a general reasoning or extensibility advantage. This concern is amplified by the absence of comparisons with closely related frameworks such as MUSE, ViperGPT/VisProg, CodeAct, or OmniAgent, or with simpler pipelines running the same handcrafted procedures outside CodeOmni.

  2. The verifier-guided repair setting uses reference-answer-dependent feedback, which introduces test-time label leakage. Appendix H.8 states that the verifier performs exact or normalized matching against the reference answer when available, and failed attempts are then retried with knowledge that the previous answer did not pass the acceptance checks. Although the paper argues that the correct option is never explicitly revealed, this distinction is not sufficient: on a binary task such as BLINK-Jigsaw, rejecting one of the two choices uniquely determines the other as correct, while repeated rejection on multi-choice tasks still reveals partial label information. As a result, the strong HARNESS+REPAIR results cannot be interpreted as ordinary inference improvements produced by better reasoning or verification; they are partly oracle-assisted. Since the paper repeatedly uses repair performance as evidence of improved reliability, this substantially weakens that part of the empirical validation.

  3. The paper contains a concerning citation error despite claiming that AI-assisted literature search was manually verified. The Qwen3-Omni Technical Report citation lists multiple incorrect author names, not just a single typo. Given the paper’s explicit statement that all references were manually checked, this raises concerns about the reliability of the citation verification process.

Questions for authors

Clarification Requests

  1. The main gains seem to come from the task-specific skills rather than from the harness itself. In particular, the VSP, Jigsaw, and Word Search skills already contain fairly complete task solvers. Can the authors clarify what benefit CodeOmni provides beyond simply running these handcrafted procedures? A comparison against a simpler pipeline using the same solvers would be very helpful here.

  2. I am concerned about the repair setting. The verifier is allowed to check the prediction against the reference answer and then tells the next attempt that the previous answer failed. For a binary task such as BLINK-Jigsaw, this effectively reveals the correct answer. Can the authors provide results with a verifier that does not use the ground-truth answer at all?

  3. What is the relationship between CodeOmni and MUSE? Several profiles are named muse_*, and the repair mechanism is also described as following MUSE. Were the profiles or task procedures adapted from MUSE? If so, why is MUSE not included as a baseline on the overlapping tasks?

  4. For Video-MME and MMAU, some reported gains over direct inference are very small. Are these differences statistically significant? It would also be useful to report token or inference cost, since the current "Calls" metric does not capture how expensive different calls are.

11 comments
AuthorsOpenReview ↗

W1/Q1: Separating the Contributions of Task-Specific Solvers and the Harness

Thank you for your thorough and insightful review, and for recognizing the practical relevance of our negative findings and the usefulness of our unified, modular design. We appreciate your questions about contribution attribution, reference-independent verification, as well as your careful attention to citation accuracy.

To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.

For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).

For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).

For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.

When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.

Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult.

To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, Word Search.

Backbone modelMethodVSP Valid / Optimal (%)Jigsaw solved (150)Word Search solved (100)
GPT-6 LunaDirect inference100.0 / 97.511846
GPT-6 LunaCodeOmni, no skills80.0 / 75.08718
GPT-6.1 SolDirect inference100.0 / 100.013765
GPT-6.1 SolCodeOmni, no skills100.0 / 100.014779
GPT-6 AstraDirect inference100.0 / 100.014669
GPT-6 AstraCodeOmni, no skills100.0 / 100.015086
DeepSeek 4.1 FlashDirect inference70.0 / 70.011018
DeepSeek 4.1 FlashCodeOmni, no skills70.0 / 70.013128
GLM 5.3 FlashDirect inference57.5 / 57.512827
GLM 5.3 FlashCodeOmni, no skills40.0 / 40.010028

The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance. DeepSeek 4.1 Flash also improves on Jigsaw and Word Search while maintaining its VSP results.

The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.

Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.

AuthorsOpenReview ↗

W1/Q1: Separating the Contributions of Task-Specific Solvers and the Harness

Thank you for your thorough and insightful review, and for recognizing the practical relevance of our negative findings and the usefulness of our unified, modular design. We appreciate your questions about contribution attribution, reference-independent verification, as well as your careful attention to citation accuracy.

To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.

For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).

For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).

For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.

When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.

Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult.

To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, Word Search.

Backbone modelMethodVSP Valid / Optimal (%)Jigsaw solved (150)Word Search solved (100)
GPT-6 LunaDirect inference100.0 / 97.511846
GPT-6 LunaCodeOmni, no skills80.0 / 75.08718
GPT-6.1 SolDirect inference100.0 / 100.013765
GPT-6.1 SolCodeOmni, no skills100.0 / 100.014779
GPT-6 AstraDirect inference100.0 / 100.014669
GPT-6 AstraCodeOmni, no skills100.0 / 100.015086
DeepSeek 4.1 FlashDirect inference70.0 / 70.011018
DeepSeek 4.1 FlashCodeOmni, no skills70.0 / 70.013128
GLM 5.3 FlashDirect inference57.5 / 57.512827
GLM 5.3 FlashCodeOmni, no skills40.0 / 40.010028

The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance. DeepSeek 4.1 Flash also improves on Jigsaw and Word Search while maintaining its VSP results.

The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.

Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.

AuthorsOpenReview ↗

W2/Q2: Answer Leakage in Verifier-Guided Repair

Thank you for raising this concern. Our repair experiments were designed following the verifier-guided retry procedure used in MUSE. In this procedure, an initial response is evaluated by a task-specific verifier. If verification fails, the model receives the original task, its previous responses, and verifier feedback, and is asked to produce a revised response. This continues until verification succeeds or the repair budget is exhausted. Following this design, our implementation allowed up to three additional attempts for VSP, Jigsaw, Word Search, CoMT, and AVUT, and one additional attempt for Video-MME and MMAU.

We audited the available implementations to clarify the information used by each verifier:

BenchmarkVerification procedureAccess to reference information
VSP-GridExecutes the predicted action sequence in the simulator to check whether it avoids traps and reaches the goal.Uses benchmark-provided grid annotations, including the start, goal, and obstacles, rather than directly matching a reference action sequence.
BLINK-JigsawChecks the selected candidate against the pixel-seam scoring procedure.The generic verifier reads the reference answer before its result is overwritten by the benchmark-specific seam check. The final gate was label-invariant in our audit, but the process was not fully isolated from reference access.
Word SearchCompares the predicted count or row–column coordinates with the reference value.Yes, the reference answer is used.
CoMTCompares the parsed option with the reference answer.Yes, the reference answer is used.
Video-MMEChecks the predicted option against the reference, alongside the applicable format and evidence requirements.Yes, the reference answer is used.
MMAUChecks the predicted option against the reference, alongside the applicable format and evidence requirements.Yes, the reference answer is used.
AVUTThe audited verification path compares the parsed prediction with the reference answer.Yes, the reference answer is used.

Although the failure messages do not explicitly reveal the correct answer, we agree that reference-based rejection still leaks answer information. For multiple-choice questions, providing the previous choice together with a correctness-based rejection allows the model to eliminate that choice. On a binary task, this identifies the correct alternative. Furthermore, a deterministic solver can provide similarly discriminating feedback even without reading a reference label. Consequently, improvements under this procedure cannot be attributed solely to the model’s ability to independently identify and correct its errors.

We acknowledge that following an existing repair procedure does not resolve this evaluation concern. We will therefore remove the HARNESS+REPAIR experiments and their associated results, discussion, and performance claims from the revised manuscript and supplementary material. These results will no longer be used to support claims about improved reasoning or reliability.

Reviewer_PgXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CodeOmni, a training-free, code-driven harness for multimodal reasoning. A frozen foundation model generates executable Python programs that can invoke registered tools and modality-specific sub-agents, perform deterministic computation, and maintain task-local state through Python variables, a VariableStore, and an ArtifactStore. The framework further introduces runtime profiles that restrict available actions and task-grounded skills that provide task-specific strategies or executable procedures.

The system is evaluated on image, video, audio, and audio-visual benchmarks with eight models. An interesting empirical observation is that the generic HARNESS configuration frequently fails to improve over direct inference and can substantially degrade performance. Adding task-grounded skills leads to much stronger results, especially on structured image tasks such as VSP-Grid, BLINK-Jigsaw, and Word Search. The supplementary material additionally studies verifier-guided repair.

Overall, the paper provides a fairly comprehensive implementation and empirical study of code as a common runtime substrate for multimodal model interaction. My main concern is that the strongest improvements are tightly coupled to benchmark-specific handcrafted skills, making it difficult to determine how much of the gain should be attributed to the proposed general harness.

Critical strengths
  1. Broad and unusually systematic empirical evaluation. The paper evaluates multiple model families across image, video, audio, and audio-visual tasks, and separates DIRECT, HARNESS, and HARNESS+SKILL conditions. In particular, showing that HARNESS alone can substantially underperform direct inference is valuable: it demonstrates that executable code and additional tools do not automatically improve reasoning and exposes a real failure mode of code-agent systems.

  2. A practical and clearly documented execution architecture. The combination of executable Python, persistent task-local state, external artifact storage, registered modality tools, and sub-agents provides a coherent implementation for multimodal orchestration. The supplementary material exposes prompts, runtime policies, and task skills in considerable detail, which makes the system easier to understand and reproduce.

Critical weaknesses
  1. The experiments do not isolate the contribution of the shared harness and workspace from task-specific solvers. Several skills directly execute fixed BFS, seam-comparison, or pixel-grid procedures and can terminate without model-generated planning. A simple pipeline using identical skills, tools, and budgets is missing. Media comparisons also change calls or decoding settings; for example, the MMAU skill enables thinking and changes temperature (Appendix D.3). The results therefore support the combined configurations more strongly than the general orchestration mechanism.
  2. The BLINK-Jigsaw execution description is not reconciled with the results. Appendix D.2 says exact skills execute without a planner call, and Listing 6 shows seam scoring followed by a final answer. Nevertheless, Table 2 reports approximately one model call per example and scores ranging from 79/150 to 150/150 across backbones. Ties or execution errors can invoke fallback model code, so this is a clarification issue rather than proof that the results are wrong. Actual routing and fallback frequencies are essential for interpreting and reproducing this comparison.
Questions for authors
  1. For Table 2’s Jigsaw runs, was the exact-skill marker active? Please report per-model counts for deterministic success, ties, execution errors, and model fallback, reconciling them with the reported Calls.
  2. How does CodeOmni compare with a simple pipeline executing the same frozen skills and tools under matched model-call and decoding budgets? Which measured gains specifically require the persistent workspace or iterative planner?
8 comments
AuthorsOpenReview ↗

W1/Q2: Separating the Contributions of Task-Specific Solvers and the Harness

Thank you for your thoughtful and detailed review, and for recognizing the breadth of our empirical evaluation and the practicality of our execution architecture. We particularly appreciate your careful examination of the Jigsaw execution details and your suggestions for separating the contributions of the task-specific procedures, persistent workspace, and iterative planner through controlled comparisons. Below, we respond to your questions point by point.

To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.

For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).

For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).

For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.

When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.

Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult.

To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, Word Search.

Backbone modelMethodVSP Valid / Optimal (%)Jigsaw solved (150)Word Search solved (100)
GPT-6 LunaDirect inference100.0 / 97.511846
GPT-6 LunaCodeOmni, no skills80.0 / 75.08718
GPT-6.1 SolDirect inference100.0 / 100.013765
GPT-6.1 SolCodeOmni, no skills100.0 / 100.014779
GPT-6 AstraDirect inference100.0 / 100.014669
GPT-6 AstraCodeOmni, no skills100.0 / 100.015086
DeepSeek 4.1 FlashDirect inference70.0 / 70.011018
DeepSeek 4.1 FlashCodeOmni, no skills70.0 / 70.013128
GLM 5.3 FlashDirect inference57.5 / 57.512827
GLM 5.3 FlashCodeOmni, no skills40.0 / 40.010028

The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance.

The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.

Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.

AuthorsOpenReview ↗

W2/Q1: Clarifying How Jigsaw Was Executed

Thank you for pointing out this inconsistency. The planner-free exact-skill mode was not enabled in any of the eight historical Jigsaw runs. We verified that the required activation marker was absent from their frozen skills.

The actual execution sequence was:

Model generates Python → runtime executes that code → the seam solver runs if the generated code calls it.

The skill instructed the model to call the solver, but did not automatically execute it. This distinction explains both the model calls and the backbone-dependent results.

BackboneCorrect /150Items with solver-named codeFailed execution stepsCalls
Qwen3.5-27B1501500150
Qwen3.5-35B1501490150
Qwen3.6-27B1501500150
Qwen3.6-35B1501500150
Qwen3-Omni-Thinking1441350150
GPT-5.4-mini7900150
GPT-5.6-luna13812581238
Gemini-3.1-Pro1501501151

For example, GPT-5.6-luna’s 125 solver-named items were all correct; its other 25 items contributed 13 correct answers, producing 138/150. GPT-5.4-mini’s saved programs contained no named solver calls and scored 79/150.

Calls includes the planning needed to generate code. It counts planner steps plus sub-agent dispatches. Dispatches were zero here, so GPT-5.6-luna’s 238 Calls comprise 150 initial planning steps and 88 additional steps. Execution errors and additional steps are separate counts.

Saved seam scores showed no ties. However, helper entry, successful return, and fallback were not separately instrumented. We therefore report solver-named code as a routing indicator; verified deterministic-success and fallback counts are unavailable.

We will correct Appendix D.2, Listing 6, and the Table 2 explanation to distinguish the supported planner-free mode from the planner-generated execution actually evaluated, and include these audit results and limitations.

Dear Reviewer,

Thank you again for your thoughtful review. We would like to kindly follow up on our rebuttal and the additional experiments we conducted in response to the reviewers’ concerns.

The new results reveal a capability-dependent transition in the effect of CodeOmni. For models with weaker coding, instruction-following, and reasoning abilities, the additional planning, code execution, and tool coordination required by the harness can introduce errors, and CodeOmni may perform worse than direct inference. As these model capabilities improve, the effect changes from negative to positive: with our strongest backbones, CodeOmni improves the performance ceiling even without handcrafted task-specific solvers. We believe this transition from suppression to improvement is an important finding for understanding when general-purpose model harnesses are effective.

We also compared CodeOmni with MUSE and the DeepSeek harness. On the strong backbones, CodeOmni consistently outperforms MUSE on Jigsaw and Word Search. With GPT-6 Astra, it achieves the best overall results among all compared harnesses, also outperforming the DeepSeek harness. With GPT-6.1 Sol, CodeOmni is better on Jigsaw, while the DeepSeek harness performs better on Word Search. We therefore do not claim universal superiority, but the results indicate that CodeOmni can exploit the capabilities of strong models more effectively than the alternative harnesses in several settings.

As the discussion period approaches its end, we would greatly appreciate knowing whether these additional results address your main concerns regarding the contribution of the harness, the role of handcrafted solvers, and the fairness of the comparisons. If any concern remains insufficiently addressed, please let us know, and we will respond as specifically as possible.

Thank you again for your time and consideration.

Reviewer_CK4yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CodeOmni is a training free harness where a frozen model writes Python that drives 32 tools and 6 modality sub agents across image, video, audio and audio visual tasks, with a variable store keeping large artifacts out of context, runtime profiles fixing tool access and step budgets, and task skills supplying either prompt guidance or a frozen executable block. Four conditions are compared, namely direct inference, the harness alone, the harness plus a skill, and the harness plus verifier gated repair. The reported finding is that the unguided harness often falls below direct inference while skill guided execution improves on it.

Critical strengths
  1. The negative result deserves to be published. Handing a model code execution and a tool registry often makes it worse, sometimes catastrophically, with MMAU falling from 75.70 to 26.19 and speech from 73.27 to 9.31, and the effect holds across all eight models and all seven settings. Most harness papers report only the cases where the harness helped.

  2. Reproducibility is good. The appendix gives the full system prompt, the sub agent prompts, the repair template, per profile budgets, the tool inventory, and the exact first step program for each executable skill, which is enough to rebuild the system.

Critical weaknesses
  1. The skills that produce the gains are hand written solvers that route around the model. VSP calls the authors' own BFS, Jigsaw calls a seam discontinuity scorer that picks the lower loss patch, and Word Search assumes a 25 or 35 pixel grid pitch and does template matching on glyph masks. The tables give this away, since four different Qwen models all score exactly 91 out of 100 on Word Search and five models all score exactly 150 out of 150 on Jigsaw, which only makes sense if a deterministic solver is answering. CoMT is the one image benchmark whose skill is only a prompt, and there the skill adds 0.4 points over the base harness, which is the paper quietly refuting its own generalization.

  2. Repair looks gold gated and there is no external baseline. Section 3.6 says the verifier compares the answer with a reference when available and that failure triggers a fresh attempt, so up to three retries fire only when the system is already known to be wrong, which is best of four with an oracle selector and explains Jigsaw hitting exactly 1200 out of 1200 across all eight models. Separately, the profiles are literally named after MUSE and the metrics follow it, yet MUSE never appears as a baseline, nor does CodeAct or ViperGPT, so the only comparisons are against direct inference and against the authors' own ablation.

Questions for authors
  1. Please run the hand coded solvers from a single perception call with no planner, no profile and no budget. If that matches HARNESS+SKILL, the gains belong to the solvers.

  2. Per benchmark, does the verifier that triggers repair read the reference answer? If so the repair numbers are best of K under an oracle and need relabeling, with a label free variant alongside.

  3. Why do four distinct models land on exactly 91 out of 100 on Word Search? What is varying the backbone measuring in that column?

  4. Please add one external harness baseline, such as MUSE (Lu et al., 2026).

6 comments
AuthorsOpenReview ↗

W1/Q1: Separating the Contributions of Task-Specific Solvers and the Harness

To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.

For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).

For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).

For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.

When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.

Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult. We therefore introduced these skills as scaffolding to support reliable task completion.

To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, and Word Search. The no-skill configuration retains general-purpose code execution and image-analysis tools, while excluding the supplied task-specific solver instructions and implementations. The results are presented below.

Backbone modelMethodVSP Valid / Optimal (%)Jigsaw solved (150)Word Search solved (100)
GPT-6 LunaDirect inference100.0 / 97.511846
GPT-6 LunaCodeOmni, no skills80.0 / 75.08718
GPT-6.1 SolDirect inference100.0 / 100.013765
GPT-6.1 SolCodeOmni, no skills100.0 / 100.014779
GPT-6 AstraDirect inference100.0 / 100.014669
GPT-6 AstraCodeOmni, no skills100.0 / 100.015086
DeepSeek 4.1 FlashDirect inference70.0 / 70.011018
DeepSeek 4.1 FlashCodeOmni, no skills70.0 / 70.013128
GLM 5.3 FlashDirect inference57.5 / 57.512827
GLM 5.3 FlashCodeOmni, no skills40.0 / 40.010028

The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance. DeepSeek 4.1 Flash also improves on Jigsaw and Word Search while maintaining its VSP results.

The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.

Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.

AuthorsOpenReview ↗

W2/Q2: Answer Leakage in Verifier-Guided Repair

Thank you for raising this concern. Our repair experiments were designed following the verifier-guided retry procedure used in MUSE. In this procedure, an initial response is evaluated by a task-specific verifier. If verification fails, the model receives the original task, its previous responses, and verifier feedback, and is asked to produce a revised response. This continues until verification succeeds or the repair budget is exhausted. Following this design, our implementation allowed up to three additional attempts for VSP, Jigsaw, Word Search, CoMT, and AVUT, and one additional attempt for Video-MME and MMAU.

We audited the available implementations to clarify the information used by each verifier:

BenchmarkVerification procedureAccess to reference information
VSP-GridExecutes the predicted action sequence in the simulator to check whether it avoids traps and reaches the goal.Uses benchmark-provided grid annotations, including the start, goal, and obstacles, rather than directly matching a reference action sequence.
BLINK-JigsawChecks the selected candidate against the pixel-seam scoring procedure.The generic verifier reads the reference answer before its result is overwritten by the benchmark-specific seam check. The final gate was label-invariant in our audit, but the process was not fully isolated from reference access.
Word SearchCompares the predicted count or row–column coordinates with the reference value.Yes, the reference answer is used.
CoMTCompares the parsed option with the reference answer.Yes, the reference answer is used.
Video-MMEChecks the predicted option against the reference, alongside the applicable format and evidence requirements.Yes, the reference answer is used.
MMAUChecks the predicted option against the reference, alongside the applicable format and evidence requirements.Yes, the reference answer is used.
AVUTThe audited verification path compares the parsed prediction with the reference answer.Yes, the reference answer is used.

Although the failure messages do not explicitly reveal the correct answer, we agree that reference-based rejection still leaks answer information. For multiple-choice questions, providing the previous choice together with a correctness-based rejection allows the model to eliminate that choice. On a binary task, this identifies the correct alternative. Furthermore, a deterministic solver can provide similarly discriminating feedback even without reading a reference label. Consequently, improvements under this procedure cannot be attributed solely to the model’s ability to independently identify and correct its errors.

We acknowledge that following an existing repair procedure does not resolve this evaluation concern. We will therefore remove the HARNESS+REPAIR experiments and their associated results, discussion, and performance claims from the revised manuscript and supplementary material. These results will no longer be used to support claims about improved reasoning or reliability.

AuthorsOpenReview ↗

Q3: Explaining Identical Word Search Scores Across Backbones

Thank you for raising this question. We inspected the per-instance predictions and execution traces of the four Qwen models. All four correctly answered the same 83 instances using the shared pixel-based solver without visual sub-agent calls. The remaining 17 instances required visual-model fallback, on which each model correctly answered eight questions. Their identical aggregate accuracy therefore decomposes into 83 solver-supported successes plus eight fallback successes, yielding 91/100. No repair was enabled in these runs.

However, identical aggregate accuracy does not imply identical per-instance predictions or error sets. The incorrect instances for each model are listed below. All IDs refer to the original dataset instance IDs, and all four models were evaluated on the same 100 instances.

ModelCorrect / TotalIncorrect instance IDs
Qwen3.5-27B91/100519, 736, 762, 852, 865, 995, 1011, 1093, 1196
Qwen3.5-35B91/100178, 234, 736, 762, 852, 865, 995, 1093, 1196
Qwen3.6-27B91/100519, 736, 762, 852, 865, 995, 1011, 1093, 1196
Qwen3.6-35B91/100178, 762, 852, 860, 865, 995, 1011, 1093, 1196

Six instances—762, 852, 865, 995, 1093, and 1196—were answered incorrectly by all four models. Across the full evaluation set, there are eight instances on which the four predictions are not all identical. Their predictions are shown below; bold entries indicate correct answers.

Instance IDReference answerQwen3.5-27BQwen3.5-35BQwen3.6-27BQwen3.6-35B
17844343
23444344
519[9, 9][10, 8][9, 9][9, 10][9, 9]
736[2, 19][2, 16][2, 18][2, 17][2, 19]
762[16, 4][18, 4][20, 3][19, 4][20, 4]
860[8, 3][8, 3][8, 3][8, 3][9, 3]
1011[16, 8][18, 7][16, 8][17, 8][17, 8]
1093[11, 9][1, 1][11, 10][1, 1][11, 10]

For example, both 27B models answer instance 178 correctly, whereas both 35B models answer it incorrectly; the opposite pattern occurs on instance 519. Some shared errors also contain different predictions, as illustrated by instance 762. We note that Qwen3.5-27B and Qwen3.6-27B have the same nine error IDs, but their predictions differ on four of those instances: 519, 736, 762, and 1011.

Regarding what backbone variation measures, this column evaluates the models within a shared solver-backed pipeline. The models generate the code that invokes the prescribed procedure and provide visual perception when the pixel solver cannot handle the input. In these four runs, all 100 instances completed in one planner step with no recorded execution errors, and each run used 17 visual sub-agent calls. Consequently, the observed prediction differences are concentrated in the visual-fallback cases, while the shared deterministic solver accounts for most of the aggregate score.

We therefore agree that the identical 91/100 scores should not be interpreted as evidence of identical underlying reasoning ability, nor as an independent comparison of general reasoning across backbones. We will clarify this decomposition and report the per-instance differences to make the roles of the shared solver and model-dependent visual fallback explicit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Reviewers and Area Chair

Dear Reviewers and Area Chair,

Thank you for your careful reviews, constructive criticism, and follow-up questions. Your feedback helped us clarify the contribution of CodeOmni and add more controlled evidence.

Our rebuttal makes three points. First, we separate task-specific skills from the general-purpose harness. We acknowledge that handcrafted skills provide the largest gains for weaker models, whose instruction-following and coding abilities may be insufficient for unconstrained code-driven execution. The regressions observed for such models are an important limitation and lesson from our study.

Second, the added no-skill experiments show that CodeOmni can improve strong models without benchmark-specific solvers. GPT-6.1 Sol improves from 137 to 147 solved Jigsaw instances and from 65 to 79 Word Search instances; GPT-6 Astra improves from 146 to 150 and from 69 to 86, respectively, while preserving its VSP ceiling. With the same Astra backbone, CodeOmni obtains 236/250 solved instances, compared with 215/250 for direct inference, 222/250 for MUSE, and 224/250 for the DeepSeek harness. These results show that CodeOmni can raise the performance ceiling of capable models.

Third, the matched ablations explain why it helps. Removing iterative planning, persistent workspace, or structured verification causes distinct failures: premature commitment, loss of reusable evidence, and shallow or insufficiently checked code interaction. Code execution, planning, and persistent state therefore work jointly: visual hypotheses become measurable procedures, intermediate artifacts remain available, and uncertain conclusions can be revisited and verified. This is the main architectural advantage over one-shot reasoning or a bare ReAct code-tool loop.

Overall, the revised evidence shows that task-specific skills can provide strong task-dependent gains, while CodeOmni itself enables strong models to use code as an iterative, stateful, and verifiable reasoning process. This improves their ability to solve complex visual problems involving global comparison, counting, localization, and compositional consistency, and gives CodeOmni an advantage over direct inference and alternative general-purpose harnesses.

Thank you again for your time, thoughtful feedback, and careful consideration of our work.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.