acceptodds
Under review as a conference paper at ICLR 2027

DuoDiscover: Code Optimization by Discovering What to Optimize and How to Evaluate

11 views

Abstract

Most discovery and optimization frameworks assume that candidate solutions can be evaluated cheaply over many iterations. In many real settings, however, a single true evaluation, such as a full benchmark or serving trace, can take hours, so a search can afford only a handful of true evaluations. When evaluation is the bottleneck, search must discover both promising targets to optimize and cheap ways to measure progress on them to find a solution. We present DuoDiscover, a search system that builds on this idea in three ways. First, each worker agent in a fleet of coding agents co-discovers a target and builds a cheap proxy evaluator for it, which we call a yardstick. The agent then hill-climbs the yardstick rather than the true evaluation. Second, because search can exploit any gap between a yardstick and the true objective, fidelity tests are introduced to continually check that each yardstick still ranks edits like the true objective and invalidate any that drift. Every gain is confirmed by the true evaluation before it is accepted. Third, the worker fleet is bottom-up, with no central decomposition of the problem. A lightweight orchestrator spawns and terminates agents, briefs running ones, and seeds new agents with others' confirmed work. We evaluate DuoDiscover on 2 repository-level benchmarks requiring multi-file edits and on 4 GPU and TPU kernel suites covering hundreds of kernels. On SWE-fficiency, it doubles the average speedup of human experts. On SOL-Exec, it ranks first on 121 of 235 kernels and in the top three on 212. In addition to those benchmarks, on the heavily optimized vLLM and SQLite engines, DuoDiscover achieves and gains over the base code, respectively. On a CPU microarchitecture design task with evaluations of up to 28 hours, DuoDiscover surpasses both AlphaEvolve and the best human design in 9 days, shorter search time, a big step toward autonomous chip design.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
code optimization, LLM coding agents, proxy evaluation, multi-agent systems, AI for systems, surrogate-guided search, kernel optimization, AI for chip design

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_FAdfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents DuoDiscover, a multi-agent code optimization framework for settings where evaluation is expensive, aiming to improve search efficiency under limited time and evaluation budgets. Each worker independently selects a code region to optimize and constructs a matching low-cost proxy evaluator, called a yardstick. It explores candidates through fast cycles of local edits and proxy scoring, while full evaluations check correctness and confirm performance gains before they are accepted. After each full evaluation, the framework uses a noise-aware Kendall rank correlation test to reassess whether the yardstick ranks candidates consistently with the true objective, requiring workers to rebuild invalidated yardsticks or switch targets. An orchestrator allocates full-evaluation resources according to search progress and manages exploration branches by spawning and terminating workers, briefing them, transferring confirmed solutions, and combining edits. The authors evaluate DuoDiscover on two repository-level optimization benchmarks and four GPU/TPU kernel suites, alongside case studies including vLLM, SQLite, and data-prefetcher design. The paper reports a 15.02× geometric-mean speedup on 100 SWE-fficiency-Lite tasks, a mean speed-of-light (SOL) score of 79.74% on SOL-Exec, and a 1.92× geometric-mean improvement across four serving metrics on vLLM. Ablations and search analyses further examine the roles of local proxies, fidelity checks, and multi-agent coordination, while discussing proxy drift, initialization overhead, and limitations when reliable proxies cannot be constructed.

Critical strengths

I see the paper’s main strength as its compelling system design for a practical bottleneck: generating optimization candidates is relatively inexpensive, but evaluating them can be very costly. The authors connect the questions of where to optimize and how to assess progress at low cost. Each agent independently identifies an optimization target and builds a dedicated low-cost evaluator, or yardstick, turning a search constrained by full correctness and performance evaluations into a local optimization process with fast feedback and many more opportunities to try edits. I find this particularly appealing for large codebases and complex systems, where the value of exploring a direction depends both on its potential gains and on whether useful feedback can be obtained at an affordable cost.

I also appreciate that the framework takes seriously the risk of proxy evaluations misleading the search. It repeatedly checks whether yardsticks rank candidates consistently with the true objective, invalidates those that drift, and requires gains to be confirmed by full evaluation before they are accepted. This keeps the acceptance of improvements grounded in measurements of actual performance. Multi-agent coordination serves the same purpose: agents explore different regions, while the orchestrator allocates scarce full-evaluation opportunities according to progress. Briefings, transfers of confirmed solutions, and combinations of edits allow findings from one branch to inform others. These mechanisms support broader exploration, reduce duplicated effort, and enable reuse of confirmed improvements.

The experiments provide concrete support for these design choices. On the analysis samples of 20 repository tasks and 50 kernels, yardsticks evaluate candidates approximately 22× and 53× faster than full evaluations, respectively. The system explores more candidates and obtains more confirmed improvements with fewer full evaluations. The ablation results also show that introducing yardsticks without fidelity checks reduces performance, whereas adding those checks restores performance and brings it above the baseline. I take this as evidence that managing the reliability of the evaluators is a key part of the system’s effectiveness. The vLLM case study further illustrates its practical value: under the same 20-hour budget with four agents, DuoDiscover achieves a 1.92× geometric-mean improvement across four serving metrics using seven full evaluations, compared with 1.07× from twelve evaluations for Claude Code.

For me, the paper’s most convincing contribution is how it brings target discovery, low-cost evaluator construction, fidelity maintenance, and team coordination into a coherent optimization loop, and shows how this loop can lead to more effective search in real systems where evaluation is expensive.

Critical weaknesses

I recognize the effectiveness of the framework, the soundness of its design, and the strong results it achieves. My main concern is that, despite the connections illustrated in individual cases, the paper does not provide a sufficiently systematic analysis linking its coordination operations to concrete code optimizations, or sufficiently detailed run records to make the optimization trajectories and their relationship to the framework clear.

The paper describes the roles of the worker agents and the orchestrator, as well as operations such as spawn, terminate, brief, transfer, and combine. However, the material available to me for this review does not include the full prompts actually used, runnable code, or a complete implementation-level interaction protocol. The authors state that they plan to release the code and agent prompts. At the review stage, the actual prompts and the organization of interactions among agents would do more to convince me of the framework's effectiveness than descriptions alone. Even with the rules and examples provided, I still find it difficult to determine exactly how these operations are triggered in practice: how stalled or redundant exploration is identified, how the decision to terminate an agent is made, and exactly what information a brief contains and how it shapes subsequent behavior. I would like the authors to provide the corresponding anonymized implementation and configuration materials, so that these core designs can be independently reproduced and serve as concrete references for future coding-agent development.

The case narratives in the appendix help explain the system's behavior, but they are still insufficient to reconstruct a complete optimization run. I would like to see a complete record of at least one task, from the initial code to the final result, including each agent's targets, the construction and revision of its yardsticks, candidate patch diffs, yardstick scores and true-evaluation results, and the process by which edits are accepted, rejected, transferred, and combined. More importantly, I want the authors' analysis of these records. In particular, I would appreciate to the expatiation which changes to functions, algorithms, data structures, or system configurations account for the final gains, and compare those changes with the human and baseline patches on the same task. The component ablations and selected case studies already demonstrate the value of yardsticks and coordination. Nevertheless, the analysis still centers largely on final performance and aggregate search statistics. A systematic analysis of the search process across successful explorations, failed explorations, and those terminated early is still missing. I want to understand which attempts at constructing or revising yardsticks changed the direction of search, whether termination effectively reduced redundant or low-value exploration, whether briefs led agents to discover new targets or revise their existing plans, and which concrete code changes and gains confirmed by true evaluation ultimately resulted from these behaviors. I want to see a complete analysis tracing coordination decisions, changes in search trajectories, code modifications, and resulting gains, rather than only score drops after removing components. This would help distinguish designs with transferable mechanistic value from gains that depend on particular cases or implementation choices. Beyond helping verify the results, such an analysis would explain what more valuable optimization opportunities DuoDiscover actually discovers, and how we should design agents' abilities to locate optimization targets, modify code, and collaborate.

Reflecting on this through the lens of policy optimization, I see a potential problem in DuoDiscover's underlying design. Let me first make clear that I already consider its design ideas and agent coordination structure innovative, and that both the paper and the framework offer substantial value to the community. However, from a policy optimization or reinforcement learning perspective, the reliance on cheap yardstick screening and early termination may well lead to myopic search. True evaluation can confirm that accepted edits produce real gains, but it cannot establish that candidates or directions screened out by yardsticks have no value. In particular, some refactorings across modules may require intermediate states that offer no immediate gain or even temporarily degrade performance. Screening and termination based on immediate progress may end the exploration before those gains emerge. Such coordination could therefore cut off search paths that would need to cross a temporary performance valley to escape a local optimum. Improving heuristic evaluation and the way coordination guides the search strategy may help address this issue. Of course, this work is not about training an LLM agent or constructing an RL setting. My point is to articulate a potential limitation that, in my view, may be nearly unavoidable in DuoDiscover's current design, and to suggest a direction for its further development. I would welcome a discussion of whether this issue is evident in the system's current behavior, whether the authors have considered it in their design, and whether they would explicitly identify it as a direction for future work. This could provide useful guidance for the development of agents that coordinate exploration. Building on the joint analysis requested above, I would also encourage the authors to provide evidence and discussion from the perspective of search myopia: for example, by auditing a sample of rejected candidates, analyzing tasks that require non-monotonic improvement or refactoring across modules, and explaining how the system preserves promising exploration that has not yet produced immediate gains. These analyses could inform the design of more effective systems. I hope this offers the authors a useful new perspective. This concern will not affect my assessment of the paper.

The existing cross-backbone experiments provide encouraging evidence, but they do not yet establish whether DuoDiscover retains its advantage when the underlying model itself becomes stronger at identifying bottlenecks, constructing evaluators, and planning over long horizons. Both agents and their underlying models are evolving rapidly. In many other domains, increasingly capable models have shown remarkable performance, and the advantages or substantial gains once provided by specialized frameworks may be eroded by improvements in the underlying models. I do not think coding agents have already reached that point, but I do think we should remain curious and skeptical about how substantial DuoDiscover's advantage remains with stronger models. Put simply, the main experiments use Claude Opus 5, and I would like to know whether DuoDiscover remains more effective than the existing baseline agents when built on stronger flagship models such as GPT-6 Astra and Claude Opus 5.5. For this extension, I would like the authors to keep the other conditions as consistent as possible, compare the underlying coding agent with and without DuoDiscover, and analyze which changes in search behavior account for any changes in the gains. Such experiments would not only test generalization, but also help identify which coordination and evaluation mechanisms are worth retaining as explicit framework components, and which capabilities may increasingly be supplied by the underlying model. This is an extension I would be very interested to see. I would not lower my assessment of the paper if the authors choose not to provide it, and a qualitative analysis on a subset of tasks would be sufficient.

Overall, I recognize the value of the problem formulation and the empirical effectiveness demonstrated in this paper. Frankly, I see this as a new paradigm for balancing effective search with low-cost search, and as a well-designed agent framework. I also find the rigor of the experiments and analyses convincing. My main reservations concern implementation transparency and evidence of the optimization process at the code level: the current results show that the framework can produce improvements, but they still do not allow readers to fully understand how those improvements arise or translate that understanding into design principles for future agents. The further analyses I have suggested on local optima, screened-out search directions, and stronger underlying models are studies of mechanism and generalization that I would be very interested to see. I do not intend to place undue demands on the authors. I hope instead to have an intellectually stimulating discussion.

Questions for authors

Have the authors considered studying LLM coding agents through the lens of policy optimization?

In my view, DuoDiscover already provides a promising experimental platform for examining how coordination policies guide the search for better code. If the authors could provide a complete analysis tracing coordination decisions, changes in search trajectories, code modifications, and resulting gains, this would also offer a foundation for understanding the longer-term consequences of accepting, rejecting, or terminating particular lines of exploration. In particular, I would be interested in how these decisions balance immediate progress against the potential value of exploration whose benefits only emerge after several intermediate changes. Could this perspective help improve coordination policies, avoid the myopic search discussed in my weaknesses, and ultimately discover more valuable code optimizations? I see this as a promising direction for further analysis and development of the existing framework. I would find such a discussion particularly interesting, both for understanding DuoDiscover's current behavior and for informing the design of future coding agents.

5 comments

We thank the reviewer for the careful review of the paper and the thoughtful perspective on the design. We appreciate the reviewer's recognition of target discovery, yardstick construction, fidelity maintenance and coordination as one loop for balancing effective search with low-cost search. We are also glad the reviewer found that the experiments provide concrete support for these design choices. We address the comments below.

W1: Prompts, code and agent interactions

W1. At the review stage, the actual prompts and the organization of interactions among agents would do more to convince me of the framework's effectiveness than descriptions alone. Even with the rules and examples provided, I still find it difficult to determine exactly how these operations are triggered in practice ...

Thank you for this suggestion. We have added the worker and orchestrator prompts, the configuration and how the agents interact to App. A.2 and A.3, and we plan to release the code upon acceptance.

We summarize the triggers here, and App. A.2 gives the full details. The orchestrator identifies stalled and redundant agents from a short progress summary of each agent (§2.3). An agent is stalled when its best score stays flat for more than twice the fleet's median time between improvements, and redundant when it shares another agent's target and approach. The orchestrator briefs a stalled agent toward a new target or approach, and terminates an agent that keeps stalling or is redundant, deferring a termination while a true evaluation is in flight.

A brief shares what the fleet has found, such as a confirmed gain or an uncovered target. For example, in our vLLM case study, the orchestrator terminated one agent and briefed the other three to move away from throughput, and they found new targets in p99 decode latency, GPU power and first-token latency (App. D.1). The orchestrator can seed agents with the confirmed state of the best agent (a transfer). After a termination, it spawns a new agent that starts fresh, starts from that state, or merges the changes of several agents, keeping the merge only if a true evaluation confirms it (a combine).

W2: A full run record and where the gains come from

W2. I would like to see a complete record of at least one task, from the initial code to the final result ... which changes to functions, algorithms, data structures, or system configurations account for the final gains, and compare those changes with the human and baseline patches on the same task. ... I want to see a complete analysis tracing coordination decisions, changes in search trajectories, code modifications, and resulting gains ...

Thank you for the suggestion. We have added the complete record of one task, the FreeToken run, to App. G. We note that part of the discovered optimization was also approved upstream.

App. G includes every target and yardstick, every candidate sent to a true evaluation with its scores and decision, and the orchestrator actions that changed the course of the run, from the initial code to the final result. Below we summarize it, following the reviewer's questions.

Of the changes that account for the final gains, the largest is an algorithm change, a CPU expert loop that visits each expert's weights once while they are in cache. A function change fixes the fp8 dispatch rule so that batch-8 calls go to a new split-K kernel instead of a general GEMM, cutting the fp8 layer's time per batch-8 step from 38.2 to 24.1 ms. The combined edits confirmed at hour 15 add two more kinds of change (App. D.2). The hot-expert cache policy is a data structure change, and the fetch fraction set per model and GPU is a system configuration change. On the same task, the Claude Code baseline instead rewrote the batch-8 GPU kernels and ended at 1.05× after 14 true evaluations, against our 1.44× after 6. We compare with a human expert patch on a SciPy task in App. F.4.

Yardsticks and coordination decisions both shaped this search. The yardstick that scored the fp8 layer against the bytes it must move exposed the misrouted dispatch, which no test caught. When two agents converged on the CPU expert loop, the orchestrator terminated one, and its replacement found the shared-expert schedule (App. D.2).

Across the runs of §4, within five candidates after a brief, agents revised their plans in 31% of cases and started new targets in 24%. Targets without a validated yardstick also redirected the search, sending the agent to another target in 76% of cases on kernels and 82% on repositories. Termination reduced redundant exploration, keeping the share of agents on the same or a related target at 5 to 15% on kernels against 40 to 60% for Claude Code, and at 18% on repositories against 55%. Targets that never confirmed a gain took only 19% of the true evaluations on kernels and 20% on repositories (App. F.2).

(continued in the next comment)

(continued from the previous comment)

W3 and Q: Myopic search and policy-optimization lens

W3. ... from a policy optimization or reinforcement learning perspective, the reliance on cheap yardstick screening and early termination may well lead to myopic search. ... some refactorings across modules may require intermediate states that offer no immediate gain or even temporarily degrade performance.

Q. Have the authors considered studying LLM coding agents through the lens of policy optimization?

Thank you for this insightful point. At a high level, a central challenge shared by policy optimization and search is balancing exploration and exploitation. This is also a key consideration in DuoDiscover's orchestrator design: agents independently explore different targets, while the orchestrator expands exploration to new targets and exploits promising discoveries by building on or combining confirmed changes (§2.3). While policy optimization learns a policy, DuoDiscover manages this trade-off through heuristic orchestration across agents.

Regarding myopic search, DuoDiscover explicitly validates yardstick fidelity against true evaluations and invalidates yardsticks that fail the fidelity test, mitigating the risk of discarding promising candidates due to inaccurate screening. However, we note that myopic search is a fundamental challenge in both search and policy optimization, not specific to yardsticks. Even with perfectly accurate true evaluations, a candidate may be discarded before further exploration reveals its potential. To mitigate premature termination, DuoDiscover marks an agent as stalled only when its best score remains unchanged for more than twice the fleet's median time between improvements, and lets a new structure mature even when it shows no immediate gain (App. A.2 and A.3).

To further check whether yardstick screening cut off useful candidates, we ran true evaluations on randomly chosen screened-out candidates from a sample of the tasks of §4 (App. F.2). Only 14 of 238 on kernels and 9 of 146 on repositories beat the best confirmed candidate at that point. The runs found 11 and 8 of these later, and the other 3 and 1 were variants of confirmed edits that did not beat the final result. These findings suggest that yardstick screening was not a major bottleneck in our experiments.

Nevertheless, search can still drop a direction that offers no immediate gain and may pay off only after other changes (see the example in App. F.4). We thank the reviewer for the policy-optimization insights, which offer a promising direction for future work, for example learning the long-term value of a target from run logs. We have added this limitation and this direction to §6.

W4: Stronger backbones

W4. I would like to know whether DuoDiscover remains more effective than the existing baseline agents when built on stronger flagship models such as GPT-6 Astra and Claude Opus 5.5.

Yes, it does. We have now run Claude Opus 5.5 and GPT-6 Astra, which both came out after our experiments, with and without DuoDiscover on the 20 repository tasks and 50 kernels of our ablations (new rows in Table 5).

Backbone and methodKernels (% SOL)Repositories (×)
Claude Code/Opus 5 (Table 5)69.1212.85
Claude Code/Opus 5 + DuoDiscover (Table 5)80.80 (+11.7)14.93 (1.16×)
Claude Code/Opus 5.572.3113.53
Claude Code/Opus 5.5 + DuoDiscover82.49 (+10.2)15.29 (1.13×)
Codex/GPT-6 Astra68.3412.91
Codex/GPT-6 Astra + DuoDiscover78.96 (+10.6)14.86 (1.15×)

DuoDiscover also reaches each base agent's final kernel score 2.7× sooner with Claude Opus 5.5 and 2.3× sooner with GPT-6 Astra, against 2.4× with Claude Opus 5 (§4). So DuoDiscover still helps frontier models. Even as models write better candidates, each true evaluation takes just as long, and that is where its yardsticks and coordination of parallel workers save time.

We thank the reviewer again for the constructive feedback and the new perspective. We hope these additions address the points raised and strengthen the paper, and we would be grateful if the reviewer would consider them in the final score. We are happy to answer any further questions.

Best,

The Authors

Reviewer_FAdfOpenReview ↗

Thank you for the clear and helpful discussion, will share more detailed thoughts soon.

I sincerely appreciate the authors’ thoughtful responses and careful discussion, which have addressed most of my concerns and increased my confidence in the paper. I plan to provide more detailed comments soon. I apologize for the brief reply due to other commitments, but wanted to acknowledge that I find the authors’ responses convincing and thank them for their efforts.

Reviewer_RzyrOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DuoDiscover, a multi-agent code optimization framework that jointly discovers promising optimization targets and inexpensive proxy evaluators called yardsticks. Agents use these yardsticks to efficiently explore candidate modifications, while periodic true evaluations and fidelity tests ensure that the proxies remain reliable. A lightweight orchestrator coordinates agents and combines confirmed improvements. Experiments across repository-level optimization, GPU/TPU kernels, and real-world systems demonstrate substantial improvements in search efficiency and optimization performance.

Critical strengths
  1. The paper identifies expensive evaluation as a key bottleneck in code optimization. Jointly discovering optimization targets and inexpensive evaluators is an interesting and practical contribution.
  2. The combination of target–yardstick co-discovery, fidelity testing, and multi-agent orchestration provides a systematic solution. The fidelity mechanism is particularly valuable for addressing unreliable proxy evaluations.
  3. The paper evaluates diverse workloads, including real-world systems such as vLLM and SQLite. The improvements over competitive baselines are convincing, and the ablation studies in Tables 3 and 4 provide useful evidence for the contributions of individual components.
Critical weaknesses
  1. Section 2.4 evaluates yardstick reliability using candidates that receive true evaluations. However, promising candidates incorrectly ranked poorly by a yardstick may never be evaluated. This limitation could be discussed more explicitly, although it does not undermine the overall contribution.
  2. Constructing yardsticks and validating their fidelity introduce additional overhead. As acknowledged in the limitations, the approach may be less beneficial when true evaluations are relatively inexpensive or the search budget is short. A clearer discussion of this trade-off would improve its practical applicability.
Questions for authors
  1. Does DuoDiscover occasionally perform true evaluations on candidates ranked poorly by the yardstick? If not, how does the framework mitigate the risk of missing promising candidates?
  2. Could the authors briefly clarify under what conditions the cost of constructing and validating yardsticks is justified by the reduction in true evaluations?
1 comment

We thank the reviewer for the encouraging review and the thoughtful questions. We appreciate the reviewer's recognition of expensive evaluation as a key bottleneck, and of joint target and yardstick discovery as a practical contribution. We are also glad the reviewer found the fidelity mechanism valuable and the improvements over competitive baselines convincing. We address the comments below.

W1 and Q1: Candidates the yardstick ranks poorly

W1. ... promising candidates incorrectly ranked poorly by a yardstick may never be evaluated. This limitation could be discussed more explicitly ...

Q1. Does DuoDiscover occasionally perform true evaluations on candidates ranked poorly by the yardstick? ...

Thank you for this thoughtful point. The reviewer is right that DuoDiscover does not run true evaluations on candidates that the yardstick ranks poorly, so it may miss a promising one. The fidelity test reduces this risk by checking each yardstick on every state that also has a true score, from the base code and rejected candidates to the best version so far. A yardstick that cannot tell good candidates from poor ones fails this test. If the agent rebuilds it, the candidates the old yardstick ranked poorly are re-scored, so good ones can be rescued.

To measure how often promising candidates are missed, we ran true evaluations on randomly chosen poorly ranked candidates from a sample of the §4 tasks. Only 14 of 238 on kernels and 9 of 146 on repositories beat the best confirmed candidate at that point, and the runs found 11 and 8 of these later. The remaining 3 and 1 were variants of edits the runs had already confirmed, and none of them beat the final result.

We agree this limitation deserves more explicit discussion, though it is also a general challenge of search. Even with perfect true evaluations, a candidate that scores poorly at an intermediate step may be dropped before it leads to a better solution, which is the usual exploration-exploitation trade-off.

A promising follow-up is to complement candidate ranking with diversity or information-gain scores, and to spend a few true evaluations on poorly ranked candidates that score high on them. We have added this discussion to §6, the sample check to App. F.2 and the re-scoring after a rebuild to App. A.1.

W2 and Q2: When the yardstick cost is justified

W2. ... the approach may be less beneficial when true evaluations are relatively inexpensive or the search budget is short ...

Q2. Could the authors briefly clarify under what conditions the cost of constructing and validating yardsticks is justified by the reduction in true evaluations?

We designed DuoDiscover to push the frontier on hard real-world systems, where true evaluations are expensive and runs are long, such as vLLM serving and CPU design (up to 28 hours per evaluation). However, we agree that a clearer discussion of this trade-off helps practical use. The cost is justified when a yardstick saves more true-evaluation time than it takes to build.

This condition holds across our 2 repository benchmarks, 4 kernel benchmarks and 5 case studies, where true evaluations take minutes to hours. We find that, on the §4 tasks, building a yardstick takes less time than one true evaluation, and it then runs 53× faster on kernels and 22× on repositories (table below). These savings outweigh the upfront cost even under short budgets. Including this build time, DuoDiscover leads Claude Code from the first hour and reaches Claude Code's 8-hour score in 2.8 hours on kernels and a median of 2.5 hours on repositories (§4).

KernelsRepositoriesvLLM
True evaluation (median)14 min15 minabout 85 min
Yardstick run (median)16 s40 sseconds to 13 min
Building a yardstick (median)8 min11 min7 to 29 min
Targets and yardsticks (% tokens)13%19%22%

DuoDiscover does not rely only on yardsticks. The orchestration of its agent fleet (§2.3) is another core component. When no cheaper useful yardstick is available, DuoDiscover falls back to true evaluations. We show this on circle packing, where a true evaluation takes less than a second. Even without yardsticks, DuoDiscover reaches AlphaEvolve's score of 2.635 for 26 circles 1.25× faster than Claude Code, showing the benefit of the orchestrator that coordinates its bottom-up fleet. Without the orchestrator, with yardsticks and the fidelity test only, it reaches the same score in 1.82 hours (vs. 1.8 for Claude Code), so building and dropping yardsticks costs almost nothing when true evaluations are fast. Overall, DuoDiscover gives large speedups when true evaluations are expensive and stays competitive when they are cheap.

We have added this discussion to §2.2 and §6, the new table numbers to §4 and App. F.3, and the circle-packing run to App. E.2.

We thank the reviewer again and are happy to run further analysis during the discussion period.

Best,

The Authors

Reviewer_XTpSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies code-optimization settings where true evaluations are costly enough to permit only a small number of search iterations. It jointly discovers an optimization target (a code region) and a target-specific, cheap proxy evaluator called a yardstick, then lets a worker agent rapidly hill-climb the yardstick instead of repeatedly running the full evaluation. This joint discovery matters because a proxy is useful only if it tracks meaningful improvements to its target, and a target is useful only if it can be measured cheaply and reliably. DuoDiscover continually runs fidelity tests to check whether the yardstick still ranks edits consistently with the true evaluation. Drifting yardsticks are invalidated, and no gain is accepted without confirmation by a true evaluation. The system uses a bottom-up fleet of agents rather than centrally decomposing the problem: agents independently explore targets and yardsticks, while a lightweight orchestrator allocates expensive evaluations, supports agents making progress, stops stalled searches, and seeds new agents with confirmed work. Experiments show improvements compare to baseline methods.

Critical strengths

Large efficiency gains under expensive evaluation. Yardsticks score candidate edits roughly 22× faster than true evaluations on repositories and 53× faster on kernels.

Every reported gain is confirmed by a true evaluation, and drifting yardsticks are invalidated rather than trusted.

Its margin widens as the number of agents increases, and it searches more broadly with cheap yardsticks while using fewer true evaluations.

Critical weaknesses

When no faithful yardstick can be constructed for a target, DuoDiscover falls back to standard search over the expensive true objective.

Constructing yardsticks and running initial fidelity tests adds cost, making it less advantageous under extremely short evaluation budgets.

As code evolves, rank drift invalidates 41% of repository yardsticks and 45% of kernel yardsticks, which wastes budget.

Questions for authors

see above

2 comments

We thank the reviewer for the careful review and the nice summary. We appreciate the reviewer's recognition of why jointly discovering a target and its yardstick matters. We are also glad the reviewer found the efficiency gains large. We address the comments below.

W1: Targets with no faithful yardstick

W1. When no faithful yardstick can be constructed for a target, DuoDiscover falls back to standard search ...

Thank you for this point. We acknowledge it as a limitation in §6. However, such cases are rare in practice. A full fallback occurred in only 3 of the 50 kernels and 2 of the 20 repository tasks in our ablations.

When one target has no validated yardstick, the agent stops optimizing it and usually moves to another target, because each of its candidates would otherwise need a slow true evaluation (see the table below).

KernelsRepositories
Targets per run (average, Table 2)159.2
Without a validated yardstick43.9
Of those, the agent moved to another target76%82%

A full fallback happens only when no target in a task has a validated yardstick. In these tasks, the search runs on true evaluations alone, like Claude Code's, but the orchestrator still coordinates the agents (App. F.5).

Falling back is safer than using a yardstick that fails the fidelity test, because such yardsticks do not track the true evaluation (median τ of 0.03 on kernels, App. F.1). In our ablation without the fidelity test, the repository speedup drops from 13.25× to 10.33×, below Claude Code's 12.85× (Table 3). We have added the fallback numbers to App. F.2 and made this limitation clearer in App. A.1 and §6.

W2: Setup cost under short budgets

W2. ... running initial fidelity tests adds cost, making it less advantageous under extremely short evaluation budgets.

We agree that building and testing yardsticks adds an upfront cost, and §6 already lists this as a limitation. However, this cost is small and amortized over long runs, especially when true evaluations are expensive. For example, our vLLM optimization runs 20 hours, while prior CPU design work (ArchAgent v2) searched for over 40 days.

Across our benchmarks, the overhead remains small relative to true evaluation costs. Median yardstick construction takes 8 minutes for kernels and 11 minutes for repositories, compared to 14 and 15 minutes per true evaluation, respectively (§4, Table 2). Finding targets and building yardsticks take about 11% of the agents' time on kernels and 14% on repositories (13% and 19% of the tokens, App. F.3), and the fidelity tests reuse true evaluations the search already runs (App. F.1). Once built, a yardstick scores a candidate 53× faster than a true evaluation on kernels and 22× on repositories (Table 2), so it recovers its build time as soon as it screens out its first candidate. DuoDiscover leads from the first hour and keeps the lead at 2 and 8 hours (table below).

1 h2 h8 h (Table 3)
Kernels (% SOL), DuoDiscover50.8661.3380.80
Kernels (% SOL), Claude Code41.0551.6069.12
Repositories (×), DuoDiscover6.2111.8414.93
Repositories (×), Claude Code5.388.2412.85

We have added these numbers to §4 and App. F.3.

W3: Budget lost to rank drift

W3. ... rank drift invalidates 41% of repository yardsticks and 45% of kernel yardsticks, which wastes budget.

Yes, drift wastes some budget, as we note in §4, but this waste is limited. Drift happens because the agent's later edits change the code a yardstick was built on, often after the yardstick has done its job. Of the yardsticks later invalidated, 63% on kernels and 48% on repositories had already led to a confirmed gain.

The fidelity test rechecks each yardstick after every true evaluation and invalidates it when it drifts, and these rechecks need no extra true evaluations. The agent then rebuilds it in a median of 8 minutes on kernels and 11 on repositories, or moves to another target.

Every search spends some true evaluations on candidates that bring no gain, Claude Code included. In our runs, drift wasted at most 11.9% of all true evaluations on kernels and 9.2% on repositories. Despite this small overhead, DuoDiscover is still much more efficient than Claude Code, since yardsticks screen out 81% of kernel candidates and 84% of repository candidates before any true evaluation. It needs 2.6 vs. 4.8 true evaluations per confirmed gain on kernels and 4.8 vs. 11.6 on repositories. App. F.1 and F.2 give the details. We also added these counts to App. F.2 and made the cost of drift clearer in §4.

We thank the reviewer again for the constructive feedback. We hope these additions address the points raised, and we would be grateful if the reviewer would consider them when finalizing the score. We are happy to run further analysis during the discussion period.

Best,

The Authors

Dear Reviewer XTpS,

Thank you for raising several insightful points, some of which other reviewers shared. They helped us strengthen the paper. In the revision, we now report how often a full fallback occurs, how the setup cost compares with a true evaluation, and how much drift costs (§4, §6 and App. F). We were glad to address the other reviewers’ questions on these related points, and the revision has since led to an improved rating.

We hope these revisions address your concerns, and we would be happy to provide any further clarification. If everything looks good to you, we would be very grateful if you would consider these updates when finalizing your rating.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers and Overview of the Revision

We thank the reviewers and the AC for the careful reviews and the constructive feedback. We are glad that the reviewers found joint target and yardstick discovery a practical contribution (Rzyr), noted the large efficiency gains under expensive evaluation (XTpS), and found the rigor of the experiments and analyses convincing (FAdf).

We address the comments below. The changes in the revised PDF are shown in blue.

Yardstick cost (XTpS, Rzyr). Building a yardstick takes less time than one true evaluation, and DuoDiscover leads Claude Code from the first hour (§4). A full fallback to true evaluations occurred in only 3 of the 50 kernels and 2 of the 20 repository tasks, and the cost of drift is small, at most 11.9% of true evaluations on kernels and 9.2% on repositories (App. F.2). On circle packing, where a true evaluation takes less than a second, DuoDiscover reaches AlphaEvolve's score 1.25× faster than Claude Code (App. E.2).

Candidates the yardstick ranks poorly (Rzyr, FAdf). We ran true evaluations on randomly chosen poorly ranked candidates. Only 14 of 238 on kernels and 9 of 146 on repositories beat the best confirmed candidate at that point, and the runs found 11 and 8 of these later (App. F.2).

Prompts and a full run record (FAdf). We have added the worker and orchestrator prompts, the configuration and how the agents interact to App. A.2 and A.3, and the complete record of one task, the FreeToken run, to App. G.

Stronger backbones (FAdf). We have now run Claude Opus 5.5 and GPT-6 Astra, which both came out after our experiments. DuoDiscover raises repository speedup by 1.13× and 1.15× and kernel SOL score by 10.2 and 10.6 points (Table 5).

We thank the reviewers again and are happy to run further analysis during the discussion period.

Best,

The Authors

AuthorsOpenReview ↗

Thank You to the Reviewers and the AC

Dear Reviewers and Area Chair,

We sincerely thank the reviewers and the AC for their time and constructive feedback. We are encouraged that all three reviewers recommend acceptance, with strong recognition of the paper's contributions and an increased rating following our rebuttal. The feedback has helped us further strengthen the paper and clarify its contributions.

We have carefully addressed the feedback from all three reviewers. While we did not have the opportunity to continue the discussion with Reviewer XTpS, the points they raised largely align with limitations already acknowledged in our original submission. We have further elaborated on these points in §4, §6, and App. F, and hope these clarifications satisfactorily answer the questions raised.

Beyond benchmark improvements, DuoDiscover advances the frontier of autonomous discovery by making previously impractical optimization challenges practical. From production LLM serving to CPU microarchitecture design, it substantially accelerates real-world optimization, shortening discovery cycles from over a month to under two weeks, pushing the boundaries of what AI agents can achieve and opening new opportunities for AI-driven discovery across the systems stack.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.