acceptodds
Under review as a conference paper at ICLR 2027

DuoAgent: Self-Improving on Software Engineering from Visual and Structural Feedback

Abstract

LLM agents are increasingly used for real-world software engineering (SWE), yet they still struggle to resolve challenging practical software issues involving visual behavior, such as broken layouts or incorrect UI states, due to weak visual grounding, scarce training data, and inadequate supervision. To address these challenges, we propose a self-improving agentic framework based on visual and structure feedback to address multimodal SWE problems, where we ground agents in interactive browser environments, exploited during deployment, data generation, and training. In particular, we first propose DuoAgent, a with specialized agents: orchestrator, reproducer, solver, and verifier. The reproducer builds a browser sandbox and a fixed action sequence that triggers the reported defect. The solver analyzes the repository and proposes candidate fixes. For each candidate, the verifier rebuilds the software, replays the same actions, compares the resulting trace with the buggy one, and returns a visual and a structural diagnosis. This allows the solver to reason about the effect of its changes on the reported behavior. Then we propose a algorithm that localizes visual elements and injects perturbations into clean sandboxes with various strategies for data augmentation. We scale up the training data from 100 seed instances to 513 synthetic instances, each verified by re-executing the seed's action sequence on the mutated and fixed programs and checking that the injected defect is visible, stable, and removed by the inverse mutation. In addition, we design a novel post-training algorithm based on on-policy self-distillation, DuoOPSD, which trains a student model on its own rollouts with teachers conditioned on diagnoses of visual and structural discrepancies as privileged information. On SWE-bench Multimodal, DuoAgent establishes a new state of the art at 52.0%, substantially surpassing the previous best result of 36.0%. Meanwhile, training with DuoOPSD on our synthetic tasks significantly improves student models and consistently outperforms existing post-training algorithms trained on the same rollouts with the same number of optimizer steps. These results establish a new paradigm for self-improving multimodal SWE agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Software Engineering, LLM Agents, Visual Grounding, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ubuGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a self-improving harness for multimodal SWE tasks, called DuoAgent. DuoAgent with Fable 5.1 sets a new state of the art score on SWE-Bench Multimodal, beating out Fable 5.1 with Claude Code. In addition, a novel method that can be used to generate SWE-Bench Multimodal-style tasks is shown to be effective when paired with DuoOPSD, a self-distillation algorithm that improves scores for Qwen3.5-4B as well as Gemma-4-E4B on that same multimodal benchmark.

Critical strengths

I found the paper generally easy to follow, with nice figures and well organized tables. The paper presents strong results (beating out Fable 5.1) in a competitive domain that is of high interest to the community. The paper is comprehensive in that it not only presents a harness but also a method for generating new tasks and finetuning on them. Experiments feature a diverse set of harnesses, including the original Multimodal agent, Codex, ClaudeCode, OpenHands, and pi.

Critical weaknesses

I think the paper's main weakness is that only one benchmark is used. I think that having a second benchmark, for example [1], and showing that results translate to other multimodal software engineering domains, would help strengthen the case of the paper. For multimodal task generation, I think that this is a very interesting part of the paper, that should've perhaps gotten more of the spotlight. For example, I think Table 9 should've been in the main paper along with some analysis on the types of tasks that were synthesized, if there were any interesting observations about them, etc. I would be interested to see how a simple baseline, of prompting a strong model to create new instances, would fare against the method in the paper (maybe something along the lines of [2]). My assumption is that a simple baseline wouldn't be so great, which would again strengthen the results.

Minor: I think that the abstract and intro are perhaps a bit too verbose, and could be made shorter and easier to read. For example, the introduction's 3rd paragraph goes over the contributions, which are then repeated at the end of the section. The paper presents two similar-but-separate methods: 1) a harness, and 2) a finetuning pipeline. This is not easy, as both of them contain a lot of material, but I think the flow between those can be improved.

[1] He, Zehai, et al. "Vision2web: A hierarchical benchmark for visual website development with agent verification." arXiv preprint arXiv:2603.26648 (2026).‏

[2] Yang, John, et al. "Swe-smith: Scaling data for software engineering agents." Advances in Neural Information Processing Systems 38 (2026).

Questions for authors
  1. Do you think the results could generalize to other SWE + multimodal benchmarks, apart from SWE-Bench Multimodal?
  2. How would a baseline of getting a large model to generate synthetic multimodal instances fare against the method presented in the paper?
  3. In Table 1, how would Astra/Fable fare when using high/extra high reasoning?
4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for the thoughtful and constructive feedback, and for appreciating the paper's clarity, strong results, and coverage of repair, task generation, and training. Following your suggestions, we conducted additional experiments on Vision2Web, direct synthetic task generation, and increased reasoning effort.

Q1: Do the results generalize to other multimodal SWE benchmarks beyond SWE-bench Multimodal?

Thank you for suggesting Vision2Web. We evaluated Codex and DuoAgent with GPT-6 Astra across the three task levels of Vision2Web.

Table R10: Codex versus DuoAgent across three levels of Vision2Web.

HarnessModelL1L2L3
CodexGPT-6 Astra84.367.165.4
DuoAgentGPT-6 Astra92.869.675.3

DuoAgent improves performance across all three levels, supporting its usefulness beyond repository-level bug repair. The common requirement is to check whether generated code produces the intended interface and behavior. The interactive verifier gives the agent feedback on this requirement that code inspection alone cannot provide. We will include these results in the revision.

Q2: How would directly prompting a strong model to generate synthetic multimodal instances compare with the proposed method?

Thank you for suggesting this comparison. We compared our generation pipeline with directly prompting the same strong model to generate new instances from the same 98 seeds, and evaluated both using the same checks.

Table R3: Synthetic task generation versus direct prompting on the same 98 seeds.

Generation methodPatch appliesPage rendersBug observable in replay
Ours552 (98.6%)551 (98.4%)513 (91.6%)
Direct prompting579 (99.5%)577 (99.1%)114 (19.6%)

The main difficulty is producing an observable defect, rather than generating applicable code that still renders. Direct prompting usually satisfies the latter requirements, but most generated changes do not manifest as bugs under replay. Such instances provide little useful supervision for repairing visual behavior, even though their patches and applications appear valid.

Our method grounds mutations in visual elements reached by the seed's interaction sequence and checks their effects through execution. This connects the generated code change to a behavior the solver can actually observe and repair. The comparison supports the importance of this grounding beyond prompting a strong model to propose plausible bugs.

Thank you also for encouraging more analysis of the generated tasks. We will bring the task-generation statistics from the appendix into the main paper and discuss the mutation types and the distinction between code validity and observable behavioral defects.

AuthorsOpenReview ↗

Rebuttal 2/2

Q3: How would Astra/Fable perform with high or extra-high reasoning effort?

Thank you for this suggestion. We evaluated GPT-6 Astra at xhigh reasoning effort under both Codex and DuoAgent, and compared it with the medium setting used in the paper. To provide more details about the results, we also report the localization and hidden test results. File and function localization and test-failure rates are percentages. P2P failures indicate regressions in previously passing tests; F2P failures indicate that the target failing tests remain unresolved.

Table R11: Medium versus xhigh reasoning effort under Codex and DuoAgent.

HarnessReasoning effortMedian reasoning tokensResolved (517)File loc.Function loc.P2P failF2P fail
CodexMedium167202 (39.1%)61.157.42.757.9
CodexXhigh3,444186 (36.0%)61.357.92.160.8
DuoAgentMedium120211 (40.8%)62.359.32.558.3
DuoAgentXhigh2,728199 (38.5%)62.659.21.961.2

Xhigh performs worse under both harnesses despite comparable or better localization. By checking the agent traces, we observe that xhigh tends to make overly cautious edits: the model preserves existing behavior but can restrict the fix so narrowly that the reported failure remains.

For example, in grommet__grommet-6239, tab underlines become misplaced when tabs contain scrollable content inside a fixed-height section. Both DuoAgent runs correctly locate src/js/components/Tabs/Tabs.js and identify the header's flex behavior.

Medium: passed. It prevents the header from shrinking by setting flex={false} directly on the header container:

- <Box direction={overflow ? 'row' : 'column'} {...tabsHeaderStyles}>
+ <Box
+   flex={false}
+   direction={overflow ? 'row' : 'column'}
+   {...tabsHeaderStyles}
+ >

Xhigh: failed. It adds the same setting only within the custom-border condition:

  if (theme.tabs.header && theme.tabs.header.border) {
    // ...resolve border color...
+   tabsHeaderStyles.flex = false;
    tabsHeaderStyles.border = {
      // ...border settings...
    };
  }

Xhigh identifies the correct change but makes it depend on an unrelated theme setting. Without a custom header border, the layout fix is never applied. Medium fixes the layout regardless of border configuration. The recorded evaluation confirms that medium passed the Tabs test suite while xhigh failed it; both passed all 77 regression suites, and neither run had an infrastructure failure. This illustrates how a narrowly guarded edit can avoid regressions while leaving the target issue unresolved. Despite that, DuoAgent continues to outperform Codex at xhigh.

Minor comments: Shorten the abstract and introduction, and improve the flow between the harness and fine-tuning pipeline.

Thank you very much for these suggestions. We will shorten the abstract and introduction, remove the repeated contribution summary, and clarify how the harness's execution feedback supports synthetic task generation and subsequent training.

Thank you for your efforts; my concerns have been addressed. Including the new results would strengthen the paper’s contribution. I also think that including the improved text in the revised paper before the rebuttal period ends would be helpful.

Thank you very much for your constructive and detailed suggestions. We have revised the manuscript accordingly and incorporated the new results presented in our rebuttal. All changes are highlighted in blue. Please let us know if you have any further comments or suggestions!

Reviewer_FpTQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses visual software issues (e.g., broken layouts, incorrect UI states) in SWE-bench Multimodal, which unit tests rarely catch and which often appear only after a sequence of interactions. It proposes three components:

  1. DuoAgent, a multi-agent harness. A reproducer builds a browser sandbox and an action sequence that reproduces the reported defect. A solver proposes patches. A verifier replays the actions on each candidate and returns visual and structural diagnoses by comparing traces step by step. The reproducer's and verifier's skills are updated on the development split.
  2. Verified synthetic task generation. Bugs are injected into fixed programs under ten mutation strategies and validated through replay and a behavioral check, yielding 513 tasks from 93 seeds.
  3. DuoOPSD, an on-policy self-distillation method. Two EMA teachers are conditioned on visual and structural expert diagnoses, and a JSD loss is applied only on verification turns.

DuoAgent improves all four evaluated models over their native harnesses, reaching 52.0% with Claude Fable 5.1 (vs. 46.8% under Claude Code). DuoOPSD improves Qwen3.5-4B from 20.3% to 24.2% and Gemma-4-E4B from 13.2% to 17.2%, outperforming SFT, GRPO, and OPSD under the same number of optimizer steps.

Critical strengths

1. Turning visual issue reports into executable specifications.

The paper converts a textual and visual bug report into a browser sandbox plus a fixed action sequence. This gives each candidate patch an execution-grounded check that unit tests cannot provide, including for dynamic issues that require interaction. The pipeline is fully automated, unlike GUIRepair, which needs per-instance human setup. It reproduces 477 of 480 test instances (Table 5), and the improvements hold on both static and dynamic issues across four models (Table 7).

2. A verifiable source of training data for visual bugs, with a useful insight on privileged information.

Existing synthetic SWE pipelines verify injected bugs with unit tests, which visual defects lack. Here, replay-based validation with a behavioral check produces verified tasks that also accept alternative repairs. Tables 4 and 13 further show that abstracted diagnoses are more useful privileged information for a small model than the raw gold trace or screenshots.

Critical weaknesses
  1. The sources of DuoAgent's gains are insufficiently isolated, and the reliability of its feedback remains unclear.

Section 4.4 ablates DuoOPSD but not DuoAgent itself. The contributions of replay-based verification, visual versus structural feedback, learned skills, and the termination policy are therefore unclear. Moreover, verifier accuracy against hidden-test outcomes is not reported, despite documented disagreements (Appendix C) and 31 test sandboxes where the reference repair leaves the execution trace unchanged (Table 5). Finally, Table 5 uses Claude Fable 5.1 for reproduction, but the model assignments and possible sharing of sandboxes across other configurations are unclear. These omissions make it difficult to attribute the reported gains to the proposed feedback mechanism rather than additional system capabilities.

  1. The self-improvement claim relies substantially on external expert supervision, while the advantage of DuoOPSD is not convincingly established.

Although DuoOPSD's teachers share the student's EMA weights, their privileged diagnoses are generated by a stronger model (GPT-6 Astra, Appendix H). Synthetic tasks also rely on strong-model generation and filtering, while skill updates use ground-truth feedback from the development split. The extent to which the framework achieves autonomous self-improvement, rather than benefiting from externally generated expert supervision, is therefore unclear. Moreover, DuoOPSD outperforms GRPO by only 1.8 and 0.8 percentage points on the two students, without reported variance across training runs, leaving the robustness of its advantage uncertain.

Questions for authors
  1. For each non-Fable row in Tables 1, 2, 6, and 7, which model serves as the reproducer, the verifier, and the skill-update step UU? Are the 477 approved test sandboxes built once with Claude Fable 5.1 and shared across all models? Does the reported cost include the reproducer, verifier, and skill-update calls? If a frontier model is involved, could you report a same-model configuration (reproducer, solver, and verifier all the same model), at least for Qwen3.5-4B and o3?

  2. On the test split, what is the agreement between the verifier's verdict sis^i and the hidden-test outcome r(Δi)r(\Delta_i) (precision and recall, over submitted candidates and over all verified candidates)? How do DuoAgent and the native harness compare on the 31 instances where τgold=τbug\tau_{\text{gold}} = \tau_{\text{bug}}?

  3. Could you provide a component ablation of DuoAgent on at least one model (for example, o3, which shows the largest gain, would be most informative)? Specifically: (i) the native harness with DuoAgent's solver instructions but no reproducer or verifier; (ii) DuoAgent with a visual-only or structural-only verifier; (iii) initial versus learned skills; (iv) DuoAgent without the forced-verification requirement.

  4. Which model writes the mutations, the synthetic issues, and the expert diagnoses? Does the diagnosis writer see the mutation diff or the inverse mutation? Have you tried diagnoses written by the student itself, or by a model of the same size? If the gains disappear in that setting, the claim that the solver improves "without a stronger teacher" should be revised.

9 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of executable reproduction, verified synthetic tasks, and diagnosis-based supervision. We address the concerns below with additional experiments and clarifications.

W1/Q1: Which models serve as the reproducer, verifier, and skill-update step? Are the sandboxes shared across configurations, and what does the reported cost include? How does a same-model configuration perform?

Thank you for pointing out the ambiguity. Claude Fable 5.1 constructs the reproduction sandboxes shared across different configurations, as reported in Table 5. The solver, verifier, and orchestrator use the model listed for each configuration. As described in Appendix C, the skill-update step uses the same model as the agent whose skills it updates. We will make these assignments explicit in the experimental setup.

The costs in Table 1 include all components except reproduction, which is completed once before evaluation and shared across solver configurations. We report this one-time cost separately below; it is not included in the per-run costs.

Table R6: One-time reproduction costs for development and test environments.

SplitMean cost/environmentMedianEstimated total for full split
Dev$5.33$3.37$533
Test$2.73$2.43$1,309

Development environments cost more because construction includes the additional C_fix gate, which compares τ_gold with τ_bug and requires further sandbox refinement when the reference repair does not produce an observable change. Test-environment construction does not use this gate because the gold repair is unavailable to the reproducer. We will clarify this cost accounting and include it in the main table in the revision.

Following your suggestion, we evaluated configurations in which the same model performs reproduction and repair. We selected the first 10 test instances from each repository and all instances from repositories with fewer than 10, totaling 105 instances. The reproducer includes sandbox construction, action generation, and the reproduction check.

Table R7: Shared frontier-model reproduction versus same-model reproduction.

ReproducerSolver, verifier, orchestratorSandboxes passing C_reproResolved (105)
Claude Fable 5.1Qwen3.5-4B10525 (23.8%)
Qwen3.5-4BQwen3.5-4B515 (14.3%)
Claude Fable 5.1o310534 (32.4%)
o3o31528 (26.7%)

Qwen3.5-4B and o3 struggle to construct sandboxes that reproduce the reported behavior. As a result, their opportunities to benefit from verification are also limited. When reproduction fails, the solver falls back to its native workflow. These results confirm that the stronger reproducer materially contributes to performance. We will clarify that the main experiments evaluate different solvers with shared reproduction infrastructure.

W1/Q2: How well do verifier verdicts agree with hidden-test outcomes? How do the harnesses compare on the 31 instances where τ_gold = τ_bug?

Thank you for suggesting this evaluation. We scored visual-only, structural-only, and combined verdicts against benchmark ground truth over the final submitted patches from Claude Fable 5.1, GPT-6 Astra, o3, and Qwen3.5-4B.

Table R5: Verifier agreement with hidden-test outcomes on final submitted patches.

Verifier evidenceF1PrecisionRecall
Visual only0.8000.7490.858
Structural only0.7130.7280.699
Visual + structural0.8230.7490.912

Combining both views reduces missed successful repairs without sacrificing precision relative to visual-only verification. Screenshots reveal visual changes, while browser states can expose token boundaries or enabled controls that look unchanged.

We also evaluated the 31 instances from Table 5 where applying the reference repair leaves the recorded trace unchanged.

Table R8: Repair performance on 31 instances with identical buggy and gold traces.

ModelNative harnessDuoAgent
Qwen3.5-4B2/31 (6.5%)3/31 (9.7%)
o36/31 (19.4%)6/31 (19.4%)
GPT-6 Astra, medium8/31 (25.8%)9/31 (29.0%)
Claude Fable 5.1, medium13/31 (41.9%)14/31 (45.2%)

As expected, DuoAgent offers little improvement when reproduction does not expose the effect of a correct repair. In the additional successful cases, the forced-verification loop returns a failure verdict and prompts further inspection, allowing the solver to continue reasoning about the code. This does not resolve the underlying observability limitation: useful verification depends on reproducing behavior that distinguishes a successful repair from the buggy program.

AuthorsOpenReview ↗

Rebuttal 2/2

W1/Q3: Could you provide component ablations separating solver instructions, evidence modalities, learned skills, and forced verification?

Thank you for suggesting these experiments. We ablated the harness using the untrained Qwen3.5-4B and Gemma-4-E4B on the same 105-instance subset. We consider: 1) No fixed action sequence: removes fixed interaction replay during verification; 2) Visual evidence only: verifies using screenshots alone; 3) Structural evidence only: verifies using browser states alone; 4) No skills: removes learned procedural skills from the harness; 5) No forced verification: removes the hard-coded verification requirement; 6) No verifier: removes verification and its diagnostic feedback entirely.

Table R1: Harness component ablations on 105 test instances.

VariantQwen3.5-4BGemma-4-E4B
Full harness25/105 (23.8%)13/105 (12.4%)
No fixed action sequence21/105 (20.0%)8/105 (7.6%)
Visual evidence only24/105 (22.9%)11/105 (10.5%)
Structural evidence only23/105 (21.9%)13/105 (12.4%)
No skills19/105 (18.1%)12/105 (11.4%)
No forced verification22/105 (21.0%)12/105 (11.4%)
No verifier21/105 (20.0%)12/105 (11.4%)

The full harness performs best on both models. Removing fixed replay is particularly harmful: for Gemma, it produces the largest degradation and performs worse than removing the verifier entirely. Without the interactions needed to reach the reported defect, verification observes only the static page and can provide misleading feedback. This highlights that verification must reproduce the relevant interaction to be useful; simply inspecting the running application is insufficient.

Procedural guidance and enforced verification also contribute. Neither evidence modality consistently suffices across both models, motivating their combination. We will include these results and clarify the scope of each ablation.

W2/Q4: Which model generates mutations, synthetic issues, and expert diagnoses? Does the diagnosis writer see the mutation or inverse mutation? How does external expert supervision affect the self-improvement claim?

Thank you for asking us to clarify the source of supervision. The mutations, synthetic issues, and expert diagnoses are all generated with GPT-6 Astra. The diagnoses use aligned buggy and correct executions, without access to the mutation diff or inverse mutation.

Our self-improvement claim refers to learning from the student's own rollouts through self-distillation, with externally supplied privileged information. This follows the original OPSD work [1], where a stronger LLM generates privileged reference solutions while a frozen copy of the initial student serves as the distillation teacher. Self-distillation does not require privileged information to be self-generated. Similarly, Astra supplies our diagnoses, while the student's EMA models provide token-level supervision on its own continuations.

We also tried student-generated diagnoses but found them much lower in quality than Astra's, particularly in identifying visual discrepancies. Extracting these discrepancies remains challenging for the student, even when it can use an explicit diagnosis to guide repair. Training diagnosis generation is therefore a complementary direction: at test time, the verifier must generate diagnoses, whereas DuoOPSD trains the solver to use them. We will clarify this limitation and revise “without a stronger teacher” to distinguish the distillation teachers from the external source of privileged information.

W2: The advantage over GRPO is modest, and its robustness across training runs is unclear.

Thank you for raising this concern. We repeated training three times for Qwen3.5-4B and report the mean resolution rate and standard deviation below.

Table R9: Post-training performance across three Qwen3.5-4B training runs.

MethodMean resolution rate (%)Std (pp)
SFT21.411.91
GRPO21.921.73
OPSD21.791.45
DuoOPSD24.311.67

DuoOPSD retains the highest average performance, with variability comparable to the baselines. This supports its advantage beyond the originally reported run, although the variation across runs warrants caution about the precise size of the improvement. We will include these results in the revision.

[1] Self-distilled reasoner: On-policy self-distillation for large language models. arXiv 2026.

Thanks for the detailed response and additional experiments. I appreciate the effort put into addressing our concerns. I have two remaining questions and would be grateful for further clarification.

  1. Attribution of gains. Table R7 suggests that the performance of non-Fable models benefits substantially from sandboxes constructed by Claude Fable 5.1. I would suggest explicitly reporting the reproducer model for each DuoAgent row in Table 1 and clarifying the comparison with GUIRepair using o3 in Section 4.2, since DuoAgent's o3 configuration also relies on Fable for reproduction.

  2. Missing matched baseline. Would it be possible to report the native-harness results (pi for Qwen3.5-4B, Codex for o3) on the same 105-instance subset used in Table R7? As I understand it, the full-test results in Table 1 are not directly comparable to this subset. These additional results would help clarify whether DuoAgent still improves over the native harness when reproduction is performed by the same model rather than Fable.

Please correct me if I have misunderstood any part of the experimental setup or the reported results.

Reviewer_5jyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-improvement of SWE-agent that addresses repository-level software issues whose symptoms appear in rendered interfaces or interaction sequences. It coordinates four agents: an orchestrator manages repairs, a reproducer constructs a browser environment and a fixed action sequence, a solver proposes code changes, and a verifier replays the sequence after each patch. Comparing candidate and buggy execution traces produces separate visual and structural diagnoses. Procedural skills are refined on development instances and frozen for evaluation. The same environments support synthetic task generation through code mutations, with checks for observable, stable defects and recoverable behavior. The pipeline produces 513 synthetic tasks from 100 development instances. For post-training, DuoOPSD uses two teachers sharing moving-average student parameters but conditioned separately on expert visual and structural diagnoses derived from correct executions. The student learns from its own trajectories, with distillation applied to responses following verification feedback. On SWE-bench Multimodal, DuoAgent improves repair rates across four backbones; Claude Fable 5.1 increases from 46.8% with its native harness to 52.0%. DuoOPSD improves Qwen3.5-4B from 20.3% to 24.2% and Gemma-4-E4B from 13.2% to 17.2%. Final patches are evaluated using benchmark hidden tests, with additional experiments examining localization, training methods, synthetic data selection, and teacher inputs.

Critical strengths
  • A successful application of self-distillation to multimodal SWE agents: The paper demonstrates that diagnosis-conditioned on-policy self-distillation can improve agents performing repository-level repair involving code inspection, editing, tool execution, and repeated browser verification. DuoOPSD improves Qwen3.5-4B from 20.3% to 24.2% and Gemma-4-E4B from 13.2% to 17.2%, outperforming SFT, GRPO, and reference-patch-conditioned OPSD under a shared training pool and optimizer-step budget. These results provide useful empirical evidence for self-distillation in a demanding interactive setting.
Critical weaknesses
  • The contributions of the individual harness components remain insufficiently isolated. Table 1 establishes the benefit of the complete system, but does not distinguish the effects of fixed interaction replay, specialized verifier feedback, or development-trained procedural skills; Table 4 instead examines training choices. A simpler browser-enabled baseline with comparable tools and resources, together with targeted component ablations, would clarify which distinctive design choices are responsible for the gains (§§3.2, 4.2–4.4).
  • Limited novelty in both harness design and OPSD algorithm design. Contributions are mainly on the empirical application.
Questions for authors
  • DuoOPSD’s teachers know the correct target behavior through expert diagnoses, while the student receives feedback comparing its candidate with the buggy execution. Given that teacher–student disagreement can reflect this information gap rather than an incorrect student decision, what justifies concentrating distillation on post-verification turns, and when would you expect this supervision to transfer poorly?
  • Could you analyze a representative sample of existing repair trajectories to show when visual and structural diagnoses provide complementary or conflicting information, and how these cases relate to the solver’s subsequent edits and final patch correctness?
5 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of diagnosis-conditioned self-distillation in interactive multimodal SWE. We address the concerns below with new component ablations, verifier evaluation, and case studies.

W1: The contributions of the individual harness components remain insufficiently isolated.

Thank you for suggesting these experiments. We conducted component ablations for Qwen3.5-4B and Gemma-4-E4B on a 105-instance subset of the test split, which consists of the first 10 instances from each repository and all instances from repositories with fewer than 10. We consider: 1) No fixed action sequence: removes fixed interaction replay during verification; 2) Visual evidence only: verifies using screenshots alone; 3) Structural evidence only: verifies using browser states alone; 4) No skills: removes learned procedural skills from the harness; 5) No forced verification: removes the hard-coded verification requirement; 6) No verifier: removes verification and its diagnostic feedback entirely.

Table R1: Harness component ablations on 105 test instances.

VariantQwen3.5-4BGemma-4-E4B
Full harness25/105 (23.8%)13/105 (12.4%)
No fixed action sequence21/105 (20.0%)8/105 (7.6%)
Visual evidence only24/105 (22.9%)11/105 (10.5%)
Structural evidence only23/105 (21.9%)13/105 (12.4%)
No skills19/105 (18.1%)12/105 (11.4%)
No forced verification22/105 (21.0%)12/105 (11.4%)
No verifier21/105 (20.0%)12/105 (11.4%)

The full harness performs best on both models. Removing fixed replay is particularly harmful: for Gemma, it performs even worse than removing the verifier entirely. Without the interactions needed to reach the reported defect, verification observes only the static page and can provide misleading feedback. This highlights that verification must reproduce the relevant interaction to be useful. The other ablations show that procedural guidance, learned skills, and enforced verification also contribute. Neither evidence modality consistently suffices across both models, motivating their combination.

Besides, we evaluated a browser-enabled Pi harness, adding the Playwright MCP server to the standard Pi harness. We report results for Qwen3.5-4B on the same 105-instance subset.

Table R2: Browser-enabled Pi versus DuoAgent on 105 test instances.

ConfigurationResolvedResolution rate
Base15/10514.29%
Base + Playwright MCP15/10514.29%
DuoAgent25/10523.81%

Providing browser tools alone does not improve performance in this setting. Inspection of the agent traces shows that it attempted environment reproduction on only three tasks. Of these, two involved page navigation and screenshots, and none was resolved. This suggests that browser access alone does not reliably lead the agent to reproduce the reported behavior or use execution feedback to guide repair, motivating an explicit reproduction and verification workflow.

W2: Limited novelty in both harness design and OPSD algorithm design. Contributions are mainly on the empirical application.

Thank you for raising this concern. A central challenge in visual SWE is obtaining execution feedback that captures the user-visible defect. Such defects may require interaction and escape existing tests. We turn issue reports into replayable browser executions and interpret their visual and structural changes to verify candidate repairs and validate synthetic defects that appear under replay and disappear after repair. Simply prompting a strong model to generate bugs is insufficient: on the same seeds, direct prompting produces applicable patches and renderable pages, but most defects are not observable under replay.

Table R3: Synthetic task generation versus direct prompting on the same 98 seeds.

Generation methodPatch appliesPage rendersBug observable in replay
Ours552 (98.6%)551 (98.4%)513 (91.6%)
Direct prompting579 (99.5%)577 (99.1%)114 (19.6%)

A second challenge is turning execution feedback into transferable supervision. Providing the correct trace does not help the untrained student (Table 13), while reference-patch-conditioned OPSD yields weaker transfer (Table 3). DuoOPSD instead conditions separate visual and structural teachers on diagnoses of execution differences and supervises post-verification decisions. The contribution is identifying what execution-derived information the student can use and where it should guide learning, beyond applying standard OPSD to another benchmark.

AuthorsOpenReview ↗

Rebuttal 2/2

Q1: What justifies concentrating distillation on post-verification turns, and when would this supervision transfer poorly?

Thank you for this insightful question. Modern SWE agents already have strong code navigation and editing capabilities, yet struggle with multimodal issues. A key bottleneck is verifying whether their edits actually fix the behavior users observe. This requires interacting with the application and interpreting both its appearance and browser state. We therefore concentrate supervision on post-verification turns, where the solver must use this evidence to decide whether its patch worked and what to change next. This targets the capability missing from conventional SWE workflows.

We agree that teacher–student disagreement can nevertheless reflect an information gap. Transfer can fail when an expert diagnosis identifies a complex event-ordering error that the student cannot translate into code changes. For example, a teacher may know the exact intended layout from the correct execution, while the issue description and buggy screenshots leave that layout ambiguous to the student. Thus, effective supervision requires both a replay that exposes the relevant behavior and a diagnosis the student can understand and apply.

Q2: Could you analyze repair trajectories to show when visual and structural diagnoses provide complementary or conflicting information, and how these cases relate to edits and final patch correctness?

Thank you for suggesting this analysis. We traced a repair trajectory with several Replay calls, following the verdict of each view at every turn, the edit the solver made in between, and how its submitted patch fared on the hidden tests. In openlayers__openlayers-14414 (Claude Fable 5.1), lowering the level of a GeoTIFF flood layer should shrink the white flooded area, but the rendered pixels stop changing.

Table R4: Visual and structural feedback across an OpenLayers repair trajectory.

EvidenceReplay 1Replay 2
VisualNot fixed. Screenshots are identical to the buggy run; the white area does not shrink when the level is lowered.Fixed. When the level is lowered, the white flooded area shrinks to the low-lying pixels, unlike the buggy run.
StructuralNot fixed. The recorded browser state matches the buggy run.Not fixed. No observable change in the recorded browser state for either level change.
CombinedNot fixed. Neither view shows the flooded area responding to the lower level.Fixed. The flood layer is drawn on a canvas, which the recorded browser state doesn't capture, so "no change" there means "not visible to this view". The screenshot shows the white area shrinking after the level drops, and neither view shows an error or regression.

After replay 2, the patch resolves the issue on the hidden tests. On the submitted patch they conflict, and the combined verdict resolves the conflict correctly by treating the structural "no change" as a blind spot rather than a failed repair. Apart from this example, there are also cases where there is no change in the screenshots, while the errors in console logs are fixed (which can only be captured in structural diagnosis).

We additionally evaluated verifier verdicts against benchmark ground truth over the final submitted patches from Claude Fable 5.1, GPT-6 Astra, o3, and Qwen3.5-4B.

Table R5: Verifier agreement with hidden-test outcomes on final submitted patches.

Verifier evidenceF1PrecisionRecall
Visual only0.8000.7490.858
Structural only0.7130.7280.699
Visual + structural0.8230.7490.912

Combining both views reduces missed successful repairs without sacrificing precision relative to visual-only verification. This supports the case-study finding that the views have complementary blind spots. We will add these analyses to the revision.

For the harness component ablation, it seems overall the difference is very small and within noise range?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for the constructive feedback. Below, we summarize the additional experiments and clarifications in our responses.

Additional experiments

  1. We added component ablations and a browser-enabled Pi baseline to isolate the contributions of interaction, verification, procedural skills, and browser access (Table R1 in the responses to Reviewers 5jyU and FpTQ; Table R2 in the response to Reviewer 5jyU).
  2. We added case studies showing how visual and structural feedback resolve conflicting observations, and evaluated verifier agreement with hidden-test outcomes on final submitted patches (Table R4 in the response to Reviewer 5jyU; Table R5 in the responses to Reviewers 5jyU and FpTQ).
  3. We evaluated configurations using the same model for reproduction and repair, identifying reproduction as a separate capability bottleneck (Table R7 in the response to Reviewer FpTQ).
  4. We compared DuoAgent with native harnesses on the 31 instances with identical buggy and gold traces, finding little improvement when reproduction does not expose the repair's effect (Table R8 in the response to Reviewer FpTQ).
  5. We repeated Qwen3.5-4B training three times and reported the mean and standard deviation for DuoOPSD, OPSD, GRPO, and SFT. DuoOPSD retains the highest mean performance with comparable variability (Table R9 in the response to Reviewer FpTQ).
  6. We tried student-generated diagnoses and observed lower quality than Astra's, particularly in identifying visual discrepancies (Reviewer FpTQ).
  7. We evaluated DuoAgent on Vision2Web and found improvements over Codex across all three task levels (Table R10 in the response to Reviewer ubuG).
  8. We compared synthetic task generation with direct prompting of the same strong model. Both produce valid code, but our method more reliably produces defects observable under browser interactions (Table R3 in the responses to Reviewers 5jyU and ubuG).
  9. We evaluated Astra at xhigh reasoning effort and analyzed its failures. Increased reasoning can produce overly cautious edits that avoid regressions but leave the target bug unresolved (Table R11 in the response to Reviewer ubuG).

Clarifications

Reviewer 5jyU

  1. We clarified the methodological contributions of interactive verification, verified synthetic task generation, and diagnosis-conditioned training, supported by comparisons with simpler alternatives.
  2. We explained why supervision focuses on post-verification decisions and when the student's capabilities limit transfer from expert diagnoses.

Reviewer FpTQ

  1. We clarified the model assignments, shared reproduction sandboxes, and skill-update models, and separately reported the one-time reproduction costs excluded from Table 1 of the paper (Table R6 in the response to Reviewer FpTQ).
  2. We clarified that self-distillation uses the student's own rollouts and EMA teachers while relying on externally supplied privileged information, consistent with prior OPSD work.
  3. We explained why expert diagnoses are used and identified training the verifier to generate better diagnoses as complementary future work. We will revise the phrase “without a stronger teacher” accordingly.

Reviewer ubuG

  1. We clarified the benefit of execution-based validation for synthetic tasks and will bring the task-generation statistics and analysis into the main paper.
  2. We will shorten the abstract and introduction, remove repeated contributions, and improve the connection between the harness, task generation, and training.

We hope this summary facilitates the reviewers' evaluation and discussion. We welcome further questions and would appreciate reconsideration of the scores if our responses have satisfactorily addressed the concerns. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.