- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces LibraryDesignBench, a two-phase benchmark in which an agent implements a library from a specification that defines required capabilities and potential use cases. The authors evaluated the library through the correctness and simplicity of programs written by other agents. They found the designed libraries reproduce the abstractions of their human-authored counterparts and also analyzed some failure modes.
Critical strengths
- The research question is important and timely. Designing libraries is a critical way to improve the usability and efficiency for AI and AI-human interaction in the long run.
- The benchmark and evaluation is thorough and run at scale, involving problems from different tasks, coding languages, and many evaluation trails.
- Some analyses are useful, for example, how and why the user agents underuse the libraries.
Critical weaknesses
I think the main issue of this paper is the soundness of its evaluation. The main score is given by the correctness simplicity. Later experiments show that the correctness (pass rate) using libraries is similar to that not using libraries. Therefore, the performance gain mainly comes from the improved simplicity. Although it is an important aspect, it alone cannot fully reflect the quality and functionality of the generated libraries. The evaluation therefore lacks direct and internal evaluation of these libraries, for example, whether the code is correct and simple, whether the design of API is reasonable and clear, which are important for future their future reuse.
Questions for authors
No questions.
2 comments
Thank you for your response. I will maintain my score.
Thank you for taking the time to write a thoughtful review of our work. We appreciate your highlighting the importance of measuring an agent’s ability to design a library as a crucial way to improve usability and efficiency. We are glad our analysis of how and why agents underuse libraries was useful. That analysis is, itself, the direct evaluation of the library, and we believe that clarifications and reframing of the manuscript will resolve the soundness concerns you raised.
Observing users on fixed tasks is the standard way to evaluate human preferences for API design, and we apply this approach to LDB to find agent design preferences. Software engineering evaluates API usability by observing developers solve fixed tasks, often under competing designs [1, 2, 3], because inspecting an API does not reveal where its users struggle [4, 5]. We follow this by viewing agents as the users, because agents will increasingly work with purely agent-designed code. Scoring the library directly requires injecting human biases through either prescribed testing interfaces or a rubric for what “good” design looks like. Instead, we use complexity and size metrics following existing analyses [6,7].
We do directly analyze the library directly through our failure taxonomy (Sec 3.2). We use our failure taxonomy to better understand the libraries without incorporating them into our scoring. Through these we can answer the following questions:
The same audit on production-library solutions found rigid interfaces cause 13% of excess code there vs. 35% for agent-written libraries. Library bugs cause 7% of test failures vs. 16% (new Table 10). Sec. 3.2 now frames the audit explicitly as a direct evaluation of the libraries.
Our library score penalizes incorrectness more than simplicity to punish library defects more. By design, every problem must be solvable without the library with the specification and supporting environment alone. (Sec 2.3). So LDB is measuring what the library adds rather than if the problem itself is possible. We updated section 3.1 to clarify this. Library defects can still reduce score through:
We updated the limitations section to highlight that code no downstream problem uses is not evaluated.
[1] B. Ellis, J. Stylos, and B. Myers. The Factory Pattern in API Design: A Usability Evaluation. ICSE, 2007.
[2] M. Piccioni, C. A. Furia, and B. Meyer. An Empirical Study of API Usability. ESEM, 2013.
[3] I. Rauf, E. Troubitsyna, and I. Porres. A Systematic Mapping Study of API Usability Evaluation Methods. Computer Science Review, 33:49–68, 2019.
[4] M. P. Robillard. What Makes APIs Hard to Learn? Answers from Developers. IEEE Software, 26(6):27–34, 2009.
[5] B. A. Myers and J. Stylos. Improving API Usability. Communications of the ACM, 59(6):62–69, 2016.
[6] C. Xie, X. Gu, Y. Shi, and B. Shen. Rethinking Code Complexity Through the Lens of Large Language Models. arXiv:2602.07882, 2026.
[7] S. Patel, B. L. Hou, A. Purohit, K. Xu, J. Pan, H. He, and V. Chen. Is Agent Code Less Maintainable Than Human Code? arXiv:2606.21804, 2026.