acceptodds
Under review as a conference paper at ICLR 2027

Can Agents Design Libraries for Other Agents?

Abstract

Agents increasingly build on code written by other agents, and they reimplement rather than reuse, growing the codebases later agents must work in. To measure how well agents design libraries for other agents, we introduce LibraryDesignBench, a two-phase benchmark in which an agent implements a full-featured library from a specification that defines required capabilities and potential use cases without prescribing the design. We evaluate the library through the correctness and simplicity of programs written by three user agents from different model families. Across 242 expert-validated programming problems spanning 15 library-design tasks in four languages, agent designers reproduce the abstractions of their human-authored counterparts on eleven of fifteen tasks, and downstream agents adopt both kinds of library but underuse them, reimplementing capabilities the library already provides. Our audit identifies rigid or hard-to-use interfaces as the leading obstacles in the sampled excess-code failures. We experiment with a targeted intervention: giving designers more prescriptive agent-first guidance and having them test their library with subagents improves downstream scores and yields simpler programs. LibraryDesignBench provides both a testbed for evaluating library-design practices for agent users and an initial design baseline that improves downstream reuse.

Primary area:
datasets and benchmarks
Keywords:
Agentic Coding Benchmark, Agent Agent Evaluations, Library Design, Long Horizon Evaluation, Coding, Code Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VwjCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LibraryDesignBench, a two-phase benchmark in which an agent implements a library from a specification that defines required capabilities and potential use cases. The authors evaluated the library through the correctness and simplicity of programs written by other agents. They found the designed libraries reproduce the abstractions of their human-authored counterparts and also analyzed some failure modes.

Critical strengths
  1. The research question is important and timely. Designing libraries is a critical way to improve the usability and efficiency for AI and AI-human interaction in the long run.
  2. The benchmark and evaluation is thorough and run at scale, involving problems from different tasks, coding languages, and many evaluation trails.
  3. Some analyses are useful, for example, how and why the user agents underuse the libraries.
Critical weaknesses

I think the main issue of this paper is the soundness of its evaluation. The main score is given by the correctness ×\times simplicity. Later experiments show that the correctness (pass rate) using libraries is similar to that not using libraries. Therefore, the performance gain mainly comes from the improved simplicity. Although it is an important aspect, it alone cannot fully reflect the quality and functionality of the generated libraries. The evaluation therefore lacks direct and internal evaluation of these libraries, for example, whether the code is correct and simple, whether the design of API is reasonable and clear, which are important for future their future reuse.

Questions for authors

No questions.

2 comments

Thank you for taking the time to write a thoughtful review of our work. We appreciate your highlighting the importance of measuring an agent’s ability to design a library as a crucial way to improve usability and efficiency. We are glad our analysis of how and why agents underuse libraries was useful. That analysis is, itself, the direct evaluation of the library, and we believe that clarifications and reframing of the manuscript will resolve the soundness concerns you raised.

Observing users on fixed tasks is the standard way to evaluate human preferences for API design, and we apply this approach to LDB to find agent design preferences. Software engineering evaluates API usability by observing developers solve fixed tasks, often under competing designs [1, 2, 3], because inspecting an API does not reveal where its users struggle [4, 5]. We follow this by viewing agents as the users, because agents will increasingly work with purely agent-designed code. Scoring the library directly requires injecting human biases through either prescribed testing interfaces or a rubric for what “good” design looks like. Instead, we use complexity and size metrics following existing analyses [6,7].

We do directly analyze the library directly through our failure taxonomy (Sec 3.2). We use our failure taxonomy to better understand the libraries without incorporating them into our scoring. Through these we can answer the following questions:

  • Is the library code correct? Library bugs cause 16% of test failures, ranging from 9 contract violations for GPT-6 Astra to 33 for Grok 4.6 (Table 9).
  • Is the API design reasonable? Rigid interfaces cause 35% of excess code, and verbose ones cause 29%.
  • Is the API clear? 43% of test failures occur where the library could solve the problem, but its names, documentation, or contract did not communicate how.

The same audit on production-library solutions found rigid interfaces cause 13% of excess code there vs. 35% for agent-written libraries. Library bugs cause 7% of test failures vs. 16% (new Table 10). Sec. 3.2 now frames the audit explicitly as a direct evaluation of the libraries.

Our library score penalizes incorrectness more than simplicity to punish library defects more. By design, every problem must be solvable without the library with the specification and supporting environment alone. (Sec 2.3). So LDB is measuring what the library adds rather than if the problem itself is possible. We updated section 3.1 to clarify this. Library defects can still reduce score through:

  • A bug that breaks a test is penalized quadratically and simpler code can not mitigate that penalty.
  • When the implementer agent encounters broken library code, they end up reimplementing it. We find that 41% of reimplemented code is caused by broken library functionality.
  • Design gaps surface in the exact same way as implementers need to implement that missing functionality.

We updated the limitations section to highlight that code no downstream problem uses is not evaluated.

[1] B. Ellis, J. Stylos, and B. Myers. The Factory Pattern in API Design: A Usability Evaluation. ICSE, 2007.
[2] M. Piccioni, C. A. Furia, and B. Meyer. An Empirical Study of API Usability. ESEM, 2013.
[3] I. Rauf, E. Troubitsyna, and I. Porres. A Systematic Mapping Study of API Usability Evaluation Methods. Computer Science Review, 33:49–68, 2019.
[4] M. P. Robillard. What Makes APIs Hard to Learn? Answers from Developers. IEEE Software, 26(6):27–34, 2009.
[5] B. A. Myers and J. Stylos. Improving API Usability. Communications of the ACM, 59(6):62–69, 2016.
[6] C. Xie, X. Gu, Y. Shi, and B. Shen. Rethinking Code Complexity Through the Lens of Large Language Models. arXiv:2602.07882, 2026.
[7] S. Patel, B. L. Hou, A. Purohit, K. Xu, J. Pan, H. He, and V. Chen. Is Agent Code Less Maintainable Than Human Code? arXiv:2606.21804, 2026.

Thank you for your response. I will maintain my score.

Reviewer_QMCTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LibraryDesignBench grades an agent-designed library only by how well other agents can use it. A designer agent builds a library from a capability spec that gives no interfaces and no tests. An example is clirs, a Rust CLI parser. Three fixed implementer agents then use that library to solve 242 problems across 15 tasks.

Each problem is scored as (test-pass fraction)² × simplicity. Simplicity is the size of an expert reference solution divided by the size of the implementer's solution, capped at 1. The reference is written with the production library (e.g., clap).

Critical strengths
  1. The problem itself is intriguing. The experimental design and dataset construction are solid.
Critical weaknesses
  1. The comparison with the production library favors production, so it's unclear what "agents design as well as production" means.
  • It avoids the real choice. clap isn't installed, so implementers have to use clirs. The benchmark never tests whether an agent would pick an agent-written library over a mature one.
  • Parametric knowledge. Designers and implementers both know clap from pretraining. Copying clap's builder API may be recall, not design. Implementers that guess familiar names also favor libraries that copy production.
  • Why using less capable user agents for eval phase?
  • Where is Opus's +2.3 coming from? What explains Opus's per-task wins and losses? Can the author provides some insights?
  1. Key claims contradict each other or needs more discussion.
  • The headline claim is inconsistent. The conclusion (lines 472–473) says agents "design libraries that help other agents, with Opus 5.5 scoring 4.9% above the production library, but they do so by reproducing the production library's design in eleven of fifteen tasks." If the designs mostly copy production, the paper needs to say where the gain comes from. It could come from the 4 tasks that don't converge, which are never named, or from streamlined copies, which aren't analyzed.
  • Gaming isn't discussed. A "do-all" function could reduce a solution to one line. The 2–3 problems shown to the designer are still scored. The agent-first prompt also tells the designer the metric, asks it to imagine 10–15 tasks, and pushes for one call per task. It's unclear whether the +2.3 reflects better design or fitting to the problem distribution.
  • Table 3 uses production libraries only. The effort and prompt findings are never tested on agent-designed libraries, where reading the docs likely matters more.
  • Results for RQ4 are thin: e.g., there is one designer (GPT-6 Astra, Codex) and one prompt bundle
Questions for authors
  1. What does the taxonomy look like when applied to production-library cells? Is “rigid or hard-to-use interfaces” also the main bottleneck for production library?
  2. Why using less capable user agents for eval phase?
  3. What is the construction process for the library? Why is using the library an appealing choice for the implementer?
  4. How complete is each spec compared with the production library's features? Were the specs written from the problems?
  5. How is implemeter prompted to use production library, agent-designed library, no library?
2 comments

Thank you for taking the time to review our work thoughtfully. We appreciate your highlighting the importance of the problem space and the rigor of both our dataset construction and experimental design.

W1a & W1c: We intentionally construct LDB so production libraries are the local maxima but not optimal. Production libraries have the clear advantage of being well represented in training corpora. Despite this, implementers achieve only 61.5±0.9 simplicity with them (Table 1) using the most prescriptive prompt (Listing 1). Agent-written libs also copy names and the overarching shape of the real libraries. However, an exact copy would only match the production library, while the best agent-written libraries exceed it (Table 1). While recall occurs, the best agent-written libraries outperform production libraries by streamlining common operations and including functionality production libraries skip.

W1b. We explicitly measure how much an agent's decisions help, not which library an agent picks. Giving agents a choice between an agent written and a production library would make this an unfair evaluation because of the distribution imbalance between the two. Further, LDB attempts to emulate the scenario where the library is for a specific use case and has no mature alternatives to pick from.

W1d/Q2. Weaker implementers expose design flaws that strong agents could mask. A great library should lift weaker agents and improve their capabilities. If we used stronger agents, the cost would be too high, and the agent could mitigate poor design itself. The rankings are stable across the three implementers (Table 2).

W1e: Opus 5.5's largest gains over production libraries come from where it departs from production designs. It packages multi-step idioms into one call and includes operations missing from production libraries. Its largest losses come from its own defects such as hardcoding a single traffic model. We added this discussion to Appendix F.

W2a: Agent-designed libraries outperform production libraries by copying the shape and names while streamlining operations. The majority of the gains (64%) come from the converged tasks because agents use the same names and abstractions but streamline the underlying approach. Opus, for example, uses Arg::count compared to clap's Arg::new(..).action(ArgAction::Count). Opus also outperforms production on the 4 non-convergent tasks (arazu, pyda, umami, wapi): +3.1 [1.7, 4.5] there vs. +2.0 [0.7, 3.3] on the converged tasks, so the gain does not come from just one group (Appendix F).

W2b: If a library can score higher with a single interface, then that is a valid result because the user agents are leveraging it fully. If an agent is able to come up with a "do-all" that user agents are easily able to use without writing excess code due to its rigidity, then it is either a case of training on the test set or a fundamental breakthrough in terms of agent-preferred design patterns. To mitigate the former, we follow best practices from frontier benchmarks (TerminalBench) and canary all task files with a unique string and phrase to identify contamination. The agent-first prompt instructs the author to generate examples to explore the design space and tells it that consumers are scored on how little code they write (Sec. 3.4), but it is never given the held-out problems.

W2c. Table 3 holds the library fixed to isolate prompting and effort from design. Implementers actually miss existing capabilities less often in agent libraries than in production ones (18% vs. 31% of excess-code cases, Q1).

RQ4 is now framed as an exploratory study. (See Sec. 1, 3, 3.4).

Q1: Hard-to-use interfaces do occur in production libraries, but rigidity is uniquely an agent problem. We added Table 10 for the production library failure taxonomies. We find that rigid interfaces are an agent-specific design failure (agents=35% vs prod=13%) while hard-to-use interfaces are an issue in both (agents=29% vs prod=33%). This further supports the claim that replicating production libraries is a local maximum.

Q3/Q4. Designers get a capability-level spec with at most 3 visible examples. We added the clirs task specification to Appendix D. The designer receives one-line feature areas, one to three one-sentence example usages, and packaging instructions. These specs list feature areas and require agents to infer lower-level functionality in order to elicit the design signal from the author agents. Experts verified that these specs covered the evaluation problems, so nothing was under-specified.

Q5. Agent and production libraries get the same prompt, with only the name changed, while the no-library arm gets a code-minimization prompt. Agent-designed and production setups get an identical prompt to write a "thin adapter" over the library, differing only in its name (Listing 1). The no-library arm is told it is judged on code size (Listing 2).

Thank you for addressing my concern. My review remains positive about the paper.

Reviewer_VTEzOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark for evaluating how well AI agents generate code libraries for other AI agents. How well they implement the design, and how useful it is for other agents

Critical strengths

The idea is interesting and important, and with the AI agent generating most of the business code, it is super useful. The experiments are extensive The examples are very easy to understand, and page 5 of the paper makes it realy to-the-point, I liked it very much.

Critical weaknesses

No critical weaknesses that support rejection The only thing that worries me a bit is the fact that the benchmark was generated with the models that were evaluated

Questions for authors
  1. How does this work scale to large libraries?
  2. Can you hypothesize why Fable results are less good than Opus?
  3. Can you explain better how you measure the code with respect to a human? Give one simple example.
  4. If you have only one argument to make with respect to the SOTA, what will be the most important one?
2 comments

Thank you for your review. We appreciate the recognition that this is an important and timely evaluation.

W1: Agents are used only to turn real-world library usage into a draft problem that is then validated by experts. The agents create a draft that undergoes significant expert modifications and revisions. Agents turn the seed repository into a minimal program and write a library-free equivalent with a shared test suite. Experts then rewrite the problem instruction to ensure it is solvable from the instructions alone, create optimized reference solutions with agent assistance, and ensure the tests are fair to the specification wording. A second expert audits this review for errors.

Q1: We test library scaling via benchmark construction. In our library selection process, we focused on libraries across a diverse range of sizes (Small: BeautifulSoup4/Clap/DOMPurify, Large: Rapier2D/Tantivy/UXSim). The implementation problems are also designed and selected to maximize coverage of the design space so that the optimal library needs well-designed abstractions that can scale.

Q2: Fable vs Opus. The gap is small, with overlapping intervals (47.5 [46.4, 48.5] vs 48.9 [48.0, 49.9]). Five tasks (arazu, canon, npr, clirs, pyda) differentiate the two, with a paired difference of +1.4 (95% CI 0.1-2.7). Opus's per-library cost is lower at $9.62 vs. Fable's $14.81. So Opus 5.5 is not clearly better, since removing any of arazu, canon, npr, or pyda would make the paired difference interval cross 0. The core finding is that Opus 5.5 edges out Fable 5.1 but at ~35% less cost.

Q3: Human Comparison. Our human comparison assumes the reference solution is the smallest possible program that solves the problem by exploiting the library's design. Experts write these and then validate them to ensure they are not code golfed. For clirs (underlying library Clap), experts used the derive macro to significantly simplify their command lines:

#[derive(Parser)]  
struct Cli {  
    target: String,  
    #[arg(short, action = ArgAction::Count)]  
    verbose: u8,  
    #[arg(long, default_value_t = 8080)]  
    port: u16,  
}  
let cli = Cli::parse();  

An implementor using the builder style API chosen by the majority of models would end up writing the same cli with:

let m = Command::new("inspect")  
    .arg(Arg::new("target").required(true))  
    .arg(Arg::new("verbose").short('v').action(ArgAction::Count))  
    .arg(Arg::new("port").long("port")  
        .value_parser(value_parser!(u16)).default_value("8080"))  
    .get_matches();  
let target = m.get_one::<String>("target").unwrap();  
let verbose = m.get_count("verbose");  
let port = *m.get_one::<u16>("port").unwrap();  

This version states arguments twice to access them, compared to the attribute access in the derive macro. It requires chained calls to set parsing, defaults, and validation. All of which is defined in one macro in the former. We then format both and then measure Source Lines of Code, Cyclomatic Complexity, Cognitive Complexity, and Halstead Volume. We then average the ratio for each metric, capped at 1, against the ground-truth solution to get our simplicity score.

In Figure 1 of the paper, for example, Luna's full 142-line clirs solution has ratios of 0.38, 0.62, 0.55 and 0.44 (simplicity about 0.50) and passes 64 of 72 tests, scoring (64/72)^2 x 0.50, about 39%.

Q4: LibraryDesignBench evaluates a library's quality through purely future usage rather than inspecting the library itself. Prior benchmarks grade a library either with hidden test suites, which do not provide a true usability signal, or against prescribed signatures, which remove design decisions from the agent. LibraryDesignBench is the first to grade a library based on how well future agents can build with it.

Thank you for your detailed answer. I raised my score accordingly. I really like the paper. BTW, you did not mention it but with all the blogs that are coming out from Anthropic and the like stating that most of the code is generate via Claude and alike it is natural to think that if we let an AI agent to write its own graph/libereies/pacages etc it will be better...

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.