- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a self-improving harness for multimodal SWE tasks, called DuoAgent. DuoAgent with Fable 5.1 sets a new state of the art score on SWE-Bench Multimodal, beating out Fable 5.1 with Claude Code. In addition, a novel method that can be used to generate SWE-Bench Multimodal-style tasks is shown to be effective when paired with DuoOPSD, a self-distillation algorithm that improves scores for Qwen3.5-4B as well as Gemma-4-E4B on that same multimodal benchmark.
Critical strengths
I found the paper generally easy to follow, with nice figures and well organized tables. The paper presents strong results (beating out Fable 5.1) in a competitive domain that is of high interest to the community. The paper is comprehensive in that it not only presents a harness but also a method for generating new tasks and finetuning on them. Experiments feature a diverse set of harnesses, including the original Multimodal agent, Codex, ClaudeCode, OpenHands, and pi.
Critical weaknesses
I think the paper's main weakness is that only one benchmark is used. I think that having a second benchmark, for example [1], and showing that results translate to other multimodal software engineering domains, would help strengthen the case of the paper. For multimodal task generation, I think that this is a very interesting part of the paper, that should've perhaps gotten more of the spotlight. For example, I think Table 9 should've been in the main paper along with some analysis on the types of tasks that were synthesized, if there were any interesting observations about them, etc. I would be interested to see how a simple baseline, of prompting a strong model to create new instances, would fare against the method in the paper (maybe something along the lines of [2]). My assumption is that a simple baseline wouldn't be so great, which would again strengthen the results.
Minor: I think that the abstract and intro are perhaps a bit too verbose, and could be made shorter and easier to read. For example, the introduction's 3rd paragraph goes over the contributions, which are then repeated at the end of the section. The paper presents two similar-but-separate methods: 1) a harness, and 2) a finetuning pipeline. This is not easy, as both of them contain a lot of material, but I think the flow between those can be improved.
[1] He, Zehai, et al. "Vision2web: A hierarchical benchmark for visual website development with agent verification." arXiv preprint arXiv:2603.26648 (2026).
[2] Yang, John, et al. "Swe-smith: Scaling data for software engineering agents." Advances in Neural Information Processing Systems 38 (2026).
Questions for authors
- Do you think the results could generalize to other SWE + multimodal benchmarks, apart from SWE-Bench Multimodal?
- How would a baseline of getting a large model to generate synthetic multimodal instances fare against the method presented in the paper?
- In Table 1, how would Astra/Fable fare when using high/extra high reasoning?
4 comments
Rebuttal 2/2
Q3: How would Astra/Fable perform with high or extra-high reasoning effort?
Thank you for this suggestion. We evaluated GPT-6 Astra at xhigh reasoning effort under both Codex and DuoAgent, and compared it with the medium setting used in the paper. To provide more details about the results, we also report the localization and hidden test results. File and function localization and test-failure rates are percentages. P2P failures indicate regressions in previously passing tests; F2P failures indicate that the target failing tests remain unresolved.
Table R11: Medium versus xhigh reasoning effort under Codex and DuoAgent.
| Harness | Reasoning effort | Median reasoning tokens | Resolved (517) | File loc. | Function loc. | P2P fail | F2P fail |
|---|---|---|---|---|---|---|---|
| Codex | Medium | 167 | 202 (39.1%) | 61.1 | 57.4 | 2.7 | 57.9 |
| Codex | Xhigh | 3,444 | 186 (36.0%) | 61.3 | 57.9 | 2.1 | 60.8 |
| DuoAgent | Medium | 120 | 211 (40.8%) | 62.3 | 59.3 | 2.5 | 58.3 |
| DuoAgent | Xhigh | 2,728 | 199 (38.5%) | 62.6 | 59.2 | 1.9 | 61.2 |
Xhigh performs worse under both harnesses despite comparable or better localization. By checking the agent traces, we observe that xhigh tends to make overly cautious edits: the model preserves existing behavior but can restrict the fix so narrowly that the reported failure remains.
For example, in grommet__grommet-6239, tab underlines become misplaced when tabs contain scrollable content inside a fixed-height section. Both DuoAgent runs correctly locate src/js/components/Tabs/Tabs.js and identify the header's flex behavior.
Medium: passed. It prevents the header from shrinking by setting flex={false} directly on the header container:
- <Box direction={overflow ? 'row' : 'column'} {...tabsHeaderStyles}>
+ <Box
+ flex={false}
+ direction={overflow ? 'row' : 'column'}
+ {...tabsHeaderStyles}
+ >
Xhigh: failed. It adds the same setting only within the custom-border condition:
if (theme.tabs.header && theme.tabs.header.border) {
// ...resolve border color...
+ tabsHeaderStyles.flex = false;
tabsHeaderStyles.border = {
// ...border settings...
};
}
Xhigh identifies the correct change but makes it depend on an unrelated theme setting. Without a custom header border, the layout fix is never applied. Medium fixes the layout regardless of border configuration. The recorded evaluation confirms that medium passed the Tabs test suite while xhigh failed it; both passed all 77 regression suites, and neither run had an infrastructure failure. This illustrates how a narrowly guarded edit can avoid regressions while leaving the target issue unresolved. Despite that, DuoAgent continues to outperform Codex at xhigh.
Minor comments: Shorten the abstract and introduction, and improve the flow between the harness and fine-tuning pipeline.
Thank you very much for these suggestions. We will shorten the abstract and introduction, remove the repeated contribution summary, and clarify how the harness's execution feedback supports synthetic task generation and subsequent training.
Thank you for your efforts; my concerns have been addressed. Including the new results would strengthen the paper’s contribution. I also think that including the improved text in the revised paper before the rebuttal period ends would be helpful.
Thank you very much for your constructive and detailed suggestions. We have revised the manuscript accordingly and incorporated the new results presented in our rebuttal. All changes are highlighted in blue. Please let us know if you have any further comments or suggestions!
Rebuttal 1/2
We thank the reviewer for the thoughtful and constructive feedback, and for appreciating the paper's clarity, strong results, and coverage of repair, task generation, and training. Following your suggestions, we conducted additional experiments on Vision2Web, direct synthetic task generation, and increased reasoning effort.
Thank you for suggesting Vision2Web. We evaluated Codex and DuoAgent with GPT-6 Astra across the three task levels of Vision2Web.
Table R10: Codex versus DuoAgent across three levels of Vision2Web.
DuoAgent improves performance across all three levels, supporting its usefulness beyond repository-level bug repair. The common requirement is to check whether generated code produces the intended interface and behavior. The interactive verifier gives the agent feedback on this requirement that code inspection alone cannot provide. We will include these results in the revision.
Thank you for suggesting this comparison. We compared our generation pipeline with directly prompting the same strong model to generate new instances from the same 98 seeds, and evaluated both using the same checks.
Table R3: Synthetic task generation versus direct prompting on the same 98 seeds.
The main difficulty is producing an observable defect, rather than generating applicable code that still renders. Direct prompting usually satisfies the latter requirements, but most generated changes do not manifest as bugs under replay. Such instances provide little useful supervision for repairing visual behavior, even though their patches and applications appear valid.
Our method grounds mutations in visual elements reached by the seed's interaction sequence and checks their effects through execution. This connects the generated code change to a behavior the solver can actually observe and repair. The comparison supports the importance of this grounding beyond prompting a strong model to propose plausible bugs.
Thank you also for encouraging more analysis of the generated tasks. We will bring the task-generation statistics from the appendix into the main paper and discuss the mutation types and the distinction between code validity and observable behavioral defects.