- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents DuoDiscover, a multi-agent code optimization framework for settings where evaluation is expensive, aiming to improve search efficiency under limited time and evaluation budgets. Each worker independently selects a code region to optimize and constructs a matching low-cost proxy evaluator, called a yardstick. It explores candidates through fast cycles of local edits and proxy scoring, while full evaluations check correctness and confirm performance gains before they are accepted. After each full evaluation, the framework uses a noise-aware Kendall rank correlation test to reassess whether the yardstick ranks candidates consistently with the true objective, requiring workers to rebuild invalidated yardsticks or switch targets. An orchestrator allocates full-evaluation resources according to search progress and manages exploration branches by spawning and terminating workers, briefing them, transferring confirmed solutions, and combining edits. The authors evaluate DuoDiscover on two repository-level optimization benchmarks and four GPU/TPU kernel suites, alongside case studies including vLLM, SQLite, and data-prefetcher design. The paper reports a 15.02× geometric-mean speedup on 100 SWE-fficiency-Lite tasks, a mean speed-of-light (SOL) score of 79.74% on SOL-Exec, and a 1.92× geometric-mean improvement across four serving metrics on vLLM. Ablations and search analyses further examine the roles of local proxies, fidelity checks, and multi-agent coordination, while discussing proxy drift, initialization overhead, and limitations when reliable proxies cannot be constructed.
Critical strengths
I see the paper’s main strength as its compelling system design for a practical bottleneck: generating optimization candidates is relatively inexpensive, but evaluating them can be very costly. The authors connect the questions of where to optimize and how to assess progress at low cost. Each agent independently identifies an optimization target and builds a dedicated low-cost evaluator, or yardstick, turning a search constrained by full correctness and performance evaluations into a local optimization process with fast feedback and many more opportunities to try edits. I find this particularly appealing for large codebases and complex systems, where the value of exploring a direction depends both on its potential gains and on whether useful feedback can be obtained at an affordable cost.
I also appreciate that the framework takes seriously the risk of proxy evaluations misleading the search. It repeatedly checks whether yardsticks rank candidates consistently with the true objective, invalidates those that drift, and requires gains to be confirmed by full evaluation before they are accepted. This keeps the acceptance of improvements grounded in measurements of actual performance. Multi-agent coordination serves the same purpose: agents explore different regions, while the orchestrator allocates scarce full-evaluation opportunities according to progress. Briefings, transfers of confirmed solutions, and combinations of edits allow findings from one branch to inform others. These mechanisms support broader exploration, reduce duplicated effort, and enable reuse of confirmed improvements.
The experiments provide concrete support for these design choices. On the analysis samples of 20 repository tasks and 50 kernels, yardsticks evaluate candidates approximately 22× and 53× faster than full evaluations, respectively. The system explores more candidates and obtains more confirmed improvements with fewer full evaluations. The ablation results also show that introducing yardsticks without fidelity checks reduces performance, whereas adding those checks restores performance and brings it above the baseline. I take this as evidence that managing the reliability of the evaluators is a key part of the system’s effectiveness. The vLLM case study further illustrates its practical value: under the same 20-hour budget with four agents, DuoDiscover achieves a 1.92× geometric-mean improvement across four serving metrics using seven full evaluations, compared with 1.07× from twelve evaluations for Claude Code.
For me, the paper’s most convincing contribution is how it brings target discovery, low-cost evaluator construction, fidelity maintenance, and team coordination into a coherent optimization loop, and shows how this loop can lead to more effective search in real systems where evaluation is expensive.
Critical weaknesses
I recognize the effectiveness of the framework, the soundness of its design, and the strong results it achieves. My main concern is that, despite the connections illustrated in individual cases, the paper does not provide a sufficiently systematic analysis linking its coordination operations to concrete code optimizations, or sufficiently detailed run records to make the optimization trajectories and their relationship to the framework clear.
The paper describes the roles of the worker agents and the orchestrator, as well as operations such as spawn, terminate, brief, transfer, and combine. However, the material available to me for this review does not include the full prompts actually used, runnable code, or a complete implementation-level interaction protocol. The authors state that they plan to release the code and agent prompts. At the review stage, the actual prompts and the organization of interactions among agents would do more to convince me of the framework's effectiveness than descriptions alone. Even with the rules and examples provided, I still find it difficult to determine exactly how these operations are triggered in practice: how stalled or redundant exploration is identified, how the decision to terminate an agent is made, and exactly what information a brief contains and how it shapes subsequent behavior. I would like the authors to provide the corresponding anonymized implementation and configuration materials, so that these core designs can be independently reproduced and serve as concrete references for future coding-agent development.
The case narratives in the appendix help explain the system's behavior, but they are still insufficient to reconstruct a complete optimization run. I would like to see a complete record of at least one task, from the initial code to the final result, including each agent's targets, the construction and revision of its yardsticks, candidate patch diffs, yardstick scores and true-evaluation results, and the process by which edits are accepted, rejected, transferred, and combined. More importantly, I want the authors' analysis of these records. In particular, I would appreciate to the expatiation which changes to functions, algorithms, data structures, or system configurations account for the final gains, and compare those changes with the human and baseline patches on the same task. The component ablations and selected case studies already demonstrate the value of yardsticks and coordination. Nevertheless, the analysis still centers largely on final performance and aggregate search statistics. A systematic analysis of the search process across successful explorations, failed explorations, and those terminated early is still missing. I want to understand which attempts at constructing or revising yardsticks changed the direction of search, whether termination effectively reduced redundant or low-value exploration, whether briefs led agents to discover new targets or revise their existing plans, and which concrete code changes and gains confirmed by true evaluation ultimately resulted from these behaviors. I want to see a complete analysis tracing coordination decisions, changes in search trajectories, code modifications, and resulting gains, rather than only score drops after removing components. This would help distinguish designs with transferable mechanistic value from gains that depend on particular cases or implementation choices. Beyond helping verify the results, such an analysis would explain what more valuable optimization opportunities DuoDiscover actually discovers, and how we should design agents' abilities to locate optimization targets, modify code, and collaborate.
Reflecting on this through the lens of policy optimization, I see a potential problem in DuoDiscover's underlying design. Let me first make clear that I already consider its design ideas and agent coordination structure innovative, and that both the paper and the framework offer substantial value to the community. However, from a policy optimization or reinforcement learning perspective, the reliance on cheap yardstick screening and early termination may well lead to myopic search. True evaluation can confirm that accepted edits produce real gains, but it cannot establish that candidates or directions screened out by yardsticks have no value. In particular, some refactorings across modules may require intermediate states that offer no immediate gain or even temporarily degrade performance. Screening and termination based on immediate progress may end the exploration before those gains emerge. Such coordination could therefore cut off search paths that would need to cross a temporary performance valley to escape a local optimum. Improving heuristic evaluation and the way coordination guides the search strategy may help address this issue. Of course, this work is not about training an LLM agent or constructing an RL setting. My point is to articulate a potential limitation that, in my view, may be nearly unavoidable in DuoDiscover's current design, and to suggest a direction for its further development. I would welcome a discussion of whether this issue is evident in the system's current behavior, whether the authors have considered it in their design, and whether they would explicitly identify it as a direction for future work. This could provide useful guidance for the development of agents that coordinate exploration. Building on the joint analysis requested above, I would also encourage the authors to provide evidence and discussion from the perspective of search myopia: for example, by auditing a sample of rejected candidates, analyzing tasks that require non-monotonic improvement or refactoring across modules, and explaining how the system preserves promising exploration that has not yet produced immediate gains. These analyses could inform the design of more effective systems. I hope this offers the authors a useful new perspective. This concern will not affect my assessment of the paper.
The existing cross-backbone experiments provide encouraging evidence, but they do not yet establish whether DuoDiscover retains its advantage when the underlying model itself becomes stronger at identifying bottlenecks, constructing evaluators, and planning over long horizons. Both agents and their underlying models are evolving rapidly. In many other domains, increasingly capable models have shown remarkable performance, and the advantages or substantial gains once provided by specialized frameworks may be eroded by improvements in the underlying models. I do not think coding agents have already reached that point, but I do think we should remain curious and skeptical about how substantial DuoDiscover's advantage remains with stronger models. Put simply, the main experiments use Claude Opus 5, and I would like to know whether DuoDiscover remains more effective than the existing baseline agents when built on stronger flagship models such as GPT-6 Astra and Claude Opus 5.5. For this extension, I would like the authors to keep the other conditions as consistent as possible, compare the underlying coding agent with and without DuoDiscover, and analyze which changes in search behavior account for any changes in the gains. Such experiments would not only test generalization, but also help identify which coordination and evaluation mechanisms are worth retaining as explicit framework components, and which capabilities may increasingly be supplied by the underlying model. This is an extension I would be very interested to see. I would not lower my assessment of the paper if the authors choose not to provide it, and a qualitative analysis on a subset of tasks would be sufficient.
Overall, I recognize the value of the problem formulation and the empirical effectiveness demonstrated in this paper. Frankly, I see this as a new paradigm for balancing effective search with low-cost search, and as a well-designed agent framework. I also find the rigor of the experiments and analyses convincing. My main reservations concern implementation transparency and evidence of the optimization process at the code level: the current results show that the framework can produce improvements, but they still do not allow readers to fully understand how those improvements arise or translate that understanding into design principles for future agents. The further analyses I have suggested on local optima, screened-out search directions, and stronger underlying models are studies of mechanism and generalization that I would be very interested to see. I do not intend to place undue demands on the authors. I hope instead to have an intellectually stimulating discussion.
Questions for authors
Have the authors considered studying LLM coding agents through the lens of policy optimization?
In my view, DuoDiscover already provides a promising experimental platform for examining how coordination policies guide the search for better code. If the authors could provide a complete analysis tracing coordination decisions, changes in search trajectories, code modifications, and resulting gains, this would also offer a foundation for understanding the longer-term consequences of accepting, rejecting, or terminating particular lines of exploration. In particular, I would be interested in how these decisions balance immediate progress against the potential value of exploration whose benefits only emerge after several intermediate changes. Could this perspective help improve coordination policies, avoid the myopic search discussed in my weaknesses, and ultimately discover more valuable code optimizations? I see this as a promising direction for further analysis and development of the existing framework. I would find such a discussion particularly interesting, both for understanding DuoDiscover's current behavior and for informing the design of future coding agents.
5 comments
(continued from the previous comment)
W3 and Q: Myopic search and policy-optimization lens
W3. ... from a policy optimization or reinforcement learning perspective, the reliance on cheap yardstick screening and early termination may well lead to myopic search. ... some refactorings across modules may require intermediate states that offer no immediate gain or even temporarily degrade performance.
Q. Have the authors considered studying LLM coding agents through the lens of policy optimization?
Thank you for this insightful point. At a high level, a central challenge shared by policy optimization and search is balancing exploration and exploitation. This is also a key consideration in DuoDiscover's orchestrator design: agents independently explore different targets, while the orchestrator expands exploration to new targets and exploits promising discoveries by building on or combining confirmed changes (§2.3). While policy optimization learns a policy, DuoDiscover manages this trade-off through heuristic orchestration across agents.
Regarding myopic search, DuoDiscover explicitly validates yardstick fidelity against true evaluations and invalidates yardsticks that fail the fidelity test, mitigating the risk of discarding promising candidates due to inaccurate screening. However, we note that myopic search is a fundamental challenge in both search and policy optimization, not specific to yardsticks. Even with perfectly accurate true evaluations, a candidate may be discarded before further exploration reveals its potential. To mitigate premature termination, DuoDiscover marks an agent as stalled only when its best score remains unchanged for more than twice the fleet's median time between improvements, and lets a new structure mature even when it shows no immediate gain (App. A.2 and A.3).
To further check whether yardstick screening cut off useful candidates, we ran true evaluations on randomly chosen screened-out candidates from a sample of the tasks of §4 (App. F.2). Only 14 of 238 on kernels and 9 of 146 on repositories beat the best confirmed candidate at that point. The runs found 11 and 8 of these later, and the other 3 and 1 were variants of confirmed edits that did not beat the final result. These findings suggest that yardstick screening was not a major bottleneck in our experiments.
Nevertheless, search can still drop a direction that offers no immediate gain and may pay off only after other changes (see the example in App. F.4). We thank the reviewer for the policy-optimization insights, which offer a promising direction for future work, for example learning the long-term value of a target from run logs. We have added this limitation and this direction to §6.
W4: Stronger backbones
W4. I would like to know whether DuoDiscover remains more effective than the existing baseline agents when built on stronger flagship models such as GPT-6 Astra and Claude Opus 5.5.
Yes, it does. We have now run Claude Opus 5.5 and GPT-6 Astra, which both came out after our experiments, with and without DuoDiscover on the 20 repository tasks and 50 kernels of our ablations (new rows in Table 5).
| Backbone and method | Kernels (% SOL) | Repositories (×) |
|---|---|---|
| Claude Code/Opus 5 (Table 5) | 69.12 | 12.85 |
| Claude Code/Opus 5 + DuoDiscover (Table 5) | 80.80 (+11.7) | 14.93 (1.16×) |
| Claude Code/Opus 5.5 | 72.31 | 13.53 |
| Claude Code/Opus 5.5 + DuoDiscover | 82.49 (+10.2) | 15.29 (1.13×) |
| Codex/GPT-6 Astra | 68.34 | 12.91 |
| Codex/GPT-6 Astra + DuoDiscover | 78.96 (+10.6) | 14.86 (1.15×) |
DuoDiscover also reaches each base agent's final kernel score 2.7× sooner with Claude Opus 5.5 and 2.3× sooner with GPT-6 Astra, against 2.4× with Claude Opus 5 (§4). So DuoDiscover still helps frontier models. Even as models write better candidates, each true evaluation takes just as long, and that is where its yardsticks and coordination of parallel workers save time.
We thank the reviewer again for the constructive feedback and the new perspective. We hope these additions address the points raised and strengthen the paper, and we would be grateful if the reviewer would consider them in the final score. We are happy to answer any further questions.
Best,
The Authors
Thank you for the clear and helpful discussion, will share more detailed thoughts soon.
I sincerely appreciate the authors’ thoughtful responses and careful discussion, which have addressed most of my concerns and increased my confidence in the paper. I plan to provide more detailed comments soon. I apologize for the brief reply due to other commitments, but wanted to acknowledge that I find the authors’ responses convincing and thank them for their efforts.
We thank the reviewer for the careful review of the paper and the thoughtful perspective on the design. We appreciate the reviewer's recognition of target discovery, yardstick construction, fidelity maintenance and coordination as one loop for balancing effective search with low-cost search. We are also glad the reviewer found that the experiments provide concrete support for these design choices. We address the comments below.
W1: Prompts, code and agent interactions
Thank you for this suggestion. We have added the worker and orchestrator prompts, the configuration and how the agents interact to App. A.2 and A.3, and we plan to release the code upon acceptance.
We summarize the triggers here, and App. A.2 gives the full details. The orchestrator identifies stalled and redundant agents from a short progress summary of each agent (§2.3). An agent is stalled when its best score stays flat for more than twice the fleet's median time between improvements, and redundant when it shares another agent's target and approach. The orchestrator briefs a stalled agent toward a new target or approach, and terminates an agent that keeps stalling or is redundant, deferring a termination while a true evaluation is in flight.
A brief shares what the fleet has found, such as a confirmed gain or an uncovered target. For example, in our vLLM case study, the orchestrator terminated one agent and briefed the other three to move away from throughput, and they found new targets in p99 decode latency, GPU power and first-token latency (App. D.1). The orchestrator can seed agents with the confirmed state of the best agent (a transfer). After a termination, it spawns a new agent that starts fresh, starts from that state, or merges the changes of several agents, keeping the merge only if a true evaluation confirms it (a combine).
W2: A full run record and where the gains come from
Thank you for the suggestion. We have added the complete record of one task, the FreeToken run, to App. G. We note that part of the discovered optimization was also approved upstream.
App. G includes every target and yardstick, every candidate sent to a true evaluation with its scores and decision, and the orchestrator actions that changed the course of the run, from the initial code to the final result. Below we summarize it, following the reviewer's questions.
Of the changes that account for the final gains, the largest is an algorithm change, a CPU expert loop that visits each expert's weights once while they are in cache. A function change fixes the fp8 dispatch rule so that batch-8 calls go to a new split-K kernel instead of a general GEMM, cutting the fp8 layer's time per batch-8 step from 38.2 to 24.1 ms. The combined edits confirmed at hour 15 add two more kinds of change (App. D.2). The hot-expert cache policy is a data structure change, and the fetch fraction set per model and GPU is a system configuration change. On the same task, the Claude Code baseline instead rewrote the batch-8 GPU kernels and ended at 1.05× after 14 true evaluations, against our 1.44× after 6. We compare with a human expert patch on a SciPy task in App. F.4.
Yardsticks and coordination decisions both shaped this search. The yardstick that scored the fp8 layer against the bytes it must move exposed the misrouted dispatch, which no test caught. When two agents converged on the CPU expert loop, the orchestrator terminated one, and its replacement found the shared-expert schedule (App. D.2).
Across the runs of §4, within five candidates after a brief, agents revised their plans in 31% of cases and started new targets in 24%. Targets without a validated yardstick also redirected the search, sending the agent to another target in 76% of cases on kernels and 82% on repositories. Termination reduced redundant exploration, keeping the share of agents on the same or a related target at 5 to 15% on kernels against 40 to 60% for Claude Code, and at 18% on repositories against 55%. Targets that never confirmed a gain took only 19% of the true evaluations on kernels and 20% on repositories (App. F.2).
(continued in the next comment)