- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CodeOmni, a multimodal agent framework that lets frozen models solve tasks by writing and executing Python code. Instead of directly answering a question, the model can use code to call tools and sub-agents, perform deterministic operations, and store intermediate results outside the main context. The framework also uses runtime profiles and task-specific skills to constrain how the model solves different tasks. The authors evaluate CodeOmni on image, video, audio, and audio-visual benchmarks. A key finding is that simply giving the model code execution and tools is often not enough and can even hurt performance, while adding task-specific skills leads to much more consistent improvements, especially on structured visual tasks.
Critical strengths
-
The paper provides a broad empirical study of code-driven multimodal orchestration and identifies a useful negative result. Across image, video, audio, and audio-visual tasks, the experiments consistently show that simply giving a model access to code execution and tools does not reliably improve performance and can substantially hurt it. The comparison between
DIRECT,HARNESS, andHARNESS+SKILLtherefore provides useful evidence that enlarging the action space alone is insufficient, and that stronger task structure is necessary for reliable agentic execution. This empirical finding is practically relevant for the design of multimodal tool-using systems, even if the proposed solution relies heavily on task-specific skills. -
CodeOmni offers a coherent and reasonably general engineering abstraction for integrating code execution, tools, sub-agents, and external state across multiple modalities without model training. The separation between a shared executable core, runtime profiles, registered tools, and task skills gives the framework a modular structure that can accommodate image, video, audio, and audio-visual tasks within the same planner–executor interface. Although the individual components are not especially novel, the unified design is practically useful and provides a concrete systems framework for studying how frozen multimodal models interact with executable tools and persistent task state.
Critical weaknesses
-
The main gains are difficult to attribute to CodeOmni itself because they depend heavily on handcrafted benchmark-specific skills. In Appendix H, several “skills” already implement the core solution procedure rather than merely guiding the agent: VSP-Grid uses a fixed BFS-based solver after extracting the grid, BLINK-Jigsaw uses a deterministic seam-matching algorithm, and Word Search uses a dedicated image-processing pipeline based on grid estimation, glyph masks, and Hamming-distance matching. This matters because the unskilled
HARNESSis often no better than direct inference and can degrade substantially, while the large improvements appear only after these task-specific procedures are added. The current evaluation therefore shows that CodeOmni can execute carefully engineered solvers, but does not cleanly demonstrate that the proposed harness itself provides a general reasoning or extensibility advantage. This concern is amplified by the absence of comparisons with closely related frameworks such as MUSE, ViperGPT/VisProg, CodeAct, or OmniAgent, or with simpler pipelines running the same handcrafted procedures outside CodeOmni. -
The verifier-guided repair setting uses reference-answer-dependent feedback, which introduces test-time label leakage. Appendix H.8 states that the verifier performs exact or normalized matching against the reference answer when available, and failed attempts are then retried with knowledge that the previous answer did not pass the acceptance checks. Although the paper argues that the correct option is never explicitly revealed, this distinction is not sufficient: on a binary task such as BLINK-Jigsaw, rejecting one of the two choices uniquely determines the other as correct, while repeated rejection on multi-choice tasks still reveals partial label information. As a result, the strong
HARNESS+REPAIRresults cannot be interpreted as ordinary inference improvements produced by better reasoning or verification; they are partly oracle-assisted. Since the paper repeatedly uses repair performance as evidence of improved reliability, this substantially weakens that part of the empirical validation. -
The paper contains a concerning citation error despite claiming that AI-assisted literature search was manually verified. The Qwen3-Omni Technical Report citation lists multiple incorrect author names, not just a single typo. Given the paper’s explicit statement that all references were manually checked, this raises concerns about the reliability of the citation verification process.
Questions for authors
Clarification Requests
-
The main gains seem to come from the task-specific skills rather than from the harness itself. In particular, the VSP, Jigsaw, and Word Search skills already contain fairly complete task solvers. Can the authors clarify what benefit CodeOmni provides beyond simply running these handcrafted procedures? A comparison against a simpler pipeline using the same solvers would be very helpful here.
-
I am concerned about the repair setting. The verifier is allowed to check the prediction against the reference answer and then tells the next attempt that the previous answer failed. For a binary task such as BLINK-Jigsaw, this effectively reveals the correct answer. Can the authors provide results with a verifier that does not use the ground-truth answer at all?
-
What is the relationship between CodeOmni and MUSE? Several profiles are named
muse_*, and the repair mechanism is also described as following MUSE. Were the profiles or task procedures adapted from MUSE? If so, why is MUSE not included as a baseline on the overlapping tasks? -
For Video-MME and MMAU, some reported gains over direct inference are very small. Are these differences statistically significant? It would also be useful to report token or inference cost, since the current "Calls" metric does not capture how expensive different calls are.
11 comments
W1/Q1: Separating the Contributions of Task-Specific Solvers and the Harness
Thank you for your thorough and insightful review, and for recognizing the practical relevance of our negative findings and the usefulness of our unified, modular design. We appreciate your questions about contribution attribution, reference-independent verification, as well as your careful attention to citation accuracy.
To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.
For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).
For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).
For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.
When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.
Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult.
To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, Word Search.
| Backbone model | Method | VSP Valid / Optimal (%) | Jigsaw solved (150) | Word Search solved (100) |
|---|---|---|---|---|
| GPT-6 Luna | Direct inference | 100.0 / 97.5 | 118 | 46 |
| GPT-6 Luna | CodeOmni, no skills | 80.0 / 75.0 | 87 | 18 |
| GPT-6.1 Sol | Direct inference | 100.0 / 100.0 | 137 | 65 |
| GPT-6.1 Sol | CodeOmni, no skills | 100.0 / 100.0 | 147 | 79 |
| GPT-6 Astra | Direct inference | 100.0 / 100.0 | 146 | 69 |
| GPT-6 Astra | CodeOmni, no skills | 100.0 / 100.0 | 150 | 86 |
| DeepSeek 4.1 Flash | Direct inference | 70.0 / 70.0 | 110 | 18 |
| DeepSeek 4.1 Flash | CodeOmni, no skills | 70.0 / 70.0 | 131 | 28 |
| GLM 5.3 Flash | Direct inference | 57.5 / 57.5 | 128 | 27 |
| GLM 5.3 Flash | CodeOmni, no skills | 40.0 / 40.0 | 100 | 28 |
The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance. DeepSeek 4.1 Flash also improves on Jigsaw and Word Search while maintaining its VSP results.
The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.
Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.
W2/Q2: Answer Leakage in Verifier-Guided Repair
Thank you for raising this concern. Our repair experiments were designed following the verifier-guided retry procedure used in MUSE. In this procedure, an initial response is evaluated by a task-specific verifier. If verification fails, the model receives the original task, its previous responses, and verifier feedback, and is asked to produce a revised response. This continues until verification succeeds or the repair budget is exhausted. Following this design, our implementation allowed up to three additional attempts for VSP, Jigsaw, Word Search, CoMT, and AVUT, and one additional attempt for Video-MME and MMAU.
We audited the available implementations to clarify the information used by each verifier:
| Benchmark | Verification procedure | Access to reference information |
|---|---|---|
| VSP-Grid | Executes the predicted action sequence in the simulator to check whether it avoids traps and reaches the goal. | Uses benchmark-provided grid annotations, including the start, goal, and obstacles, rather than directly matching a reference action sequence. |
| BLINK-Jigsaw | Checks the selected candidate against the pixel-seam scoring procedure. | The generic verifier reads the reference answer before its result is overwritten by the benchmark-specific seam check. The final gate was label-invariant in our audit, but the process was not fully isolated from reference access. |
| Word Search | Compares the predicted count or row–column coordinates with the reference value. | Yes, the reference answer is used. |
| CoMT | Compares the parsed option with the reference answer. | Yes, the reference answer is used. |
| Video-MME | Checks the predicted option against the reference, alongside the applicable format and evidence requirements. | Yes, the reference answer is used. |
| MMAU | Checks the predicted option against the reference, alongside the applicable format and evidence requirements. | Yes, the reference answer is used. |
| AVUT | The audited verification path compares the parsed prediction with the reference answer. | Yes, the reference answer is used. |
Although the failure messages do not explicitly reveal the correct answer, we agree that reference-based rejection still leaks answer information. For multiple-choice questions, providing the previous choice together with a correctness-based rejection allows the model to eliminate that choice. On a binary task, this identifies the correct alternative. Furthermore, a deterministic solver can provide similarly discriminating feedback even without reading a reference label. Consequently, improvements under this procedure cannot be attributed solely to the model’s ability to independently identify and correct its errors.
We acknowledge that following an existing repair procedure does not resolve this evaluation concern. We will therefore remove the HARNESS+REPAIR experiments and their associated results, discussion, and performance claims from the revised manuscript and supplementary material. These results will no longer be used to support claims about improved reasoning or reliability.
W1/Q1: Separating the Contributions of Task-Specific Solvers and the Harness
Thank you for your thorough and insightful review, and for recognizing the practical relevance of our negative findings and the usefulness of our unified, modular design. We appreciate your questions about contribution attribution, reference-independent verification, as well as your careful attention to citation accuracy.
To assess the contribution of the handcrafted solvers, we conducted fixed-pipeline experiments that directly execute these procedures without iterative planning.
For VSP, we use a single multimodal model call to extract the grid dimensions and the coordinates of the start, goal, and obstacles from the image. These observations are then passed to a breadth-first search solver to compute a shortest path that avoids the obstacles. In these experiment, this perception-then-search pipeline produced correct solutions for all instances (100%).
For Jigsaw, we directly compare the pixel continuity of each candidate patch with the incomplete image along the upper and left boundaries of the missing region. The solver selects the candidate with the lower average absolute RGB difference across these boundaries. This procedure requires no model calls and correctly solves all 150 instances (100%).
For Word Search, we identify the supported grid geometry, segment the image into individual cells, and binarize the character images. We construct a majority-character template using the pixel-wise median and compare each cell against this template. For location questions, we select the cell with the largest difference; for counting questions, we separate the character groups at the largest gap in the sorted difference scores and count the outliers. This procedure correctly solves 83 of the 100 instances without any model calls.
When these solvers are executed as intended, results are largely consistent across models. Taken together, these controls indicate that the substantial gains on these structured visual tasks arise primarily from the supplied task-specific solvers. We explicitly acknowledge this contribution and agree that these gains alone do not establish an independent reasoning advantage of the harness.
Our motivation for introducing handcrafted solver skills was primarily practical: we observed limitations in instruction following and multi-step reasoning in the backbones used in our initial experiments, which made reliable use of the harness difficult.
To examine the contribution of CodeOmni without these supplied procedures, we compared direct inference against CodeOmni with handcrafted task-specific solver skills disabled across five backbone models on VSP, Jigsaw, Word Search.
The results show that CodeOmni can improve the performance of stronger backbones without the supplied handcrafted solvers. For GPT-6.1 Sol, the number of solved questions increases from 137 to 147 on Jigsaw and from 65 to 79 on Word Search. For GPT-6 Astra, it increases from 146 to 150 on Jigsaw and from 69 to 86 on Word Search. On VSP, both models already achieve 100% validity and 100% optimality with direct inference, and CodeOmni preserves this ceiling performance. DeepSeek 4.1 Flash also improves on Jigsaw and Word Search while maintaining its VSP results.
The benefit is nevertheless model- and task-dependent. GPT-6-Luna performs worse with the no-skill harness across all three tasks, while GLM-5.3-flash regresses on VSP and Jigsaw and improves only slightly on Word Search. These observations are consistent with our motivation for providing scaffolding to models that struggle to use the harness effectively, although this comparison alone does not establish the precise causes of their difficulties.
Taken together, these results support the conclusion that handcrafted solver skills are not a prerequisite for CodeOmni to improve accuracy with capable backbones on Jigsaw and Word Search. They do not imply that such skills provide no additional benefit. We will clarify this distinction in the manuscript and report the no-skill comparison alongside the solver-assisted results, separating the evidence for the general harness from the contribution of the supplied task-specific procedures.