acceptodds
Under review as a conference paper at ICLR 2027

NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs

Abstract

Rapidly evolving software libraries require coding language models to use unfamiliar APIs. Yet failed solutions alone tell us little about what API knowledge models need or how to provide it. We present NovelAPIBench, an automated diagnostic framework that builds benchmarks around the API knowledge gaps of a target model. It identifies novel APIs, generates executable coding tasks with separately controllable knowledge components, and classifies failures into six categories. The framework can be rerun as models and libraries evolve. We study 21 libraries across five domains and six backbone models, with our primary benchmark comprising 1,670 tasks covering 856 APIs. Our experiments show that when supplied directly, usage examples provide the strongest standalone guidance for novel API acquisition, and combining them with signature descriptions approaches full-knowledge performance. Retrieving this knowledge from a pool of APIs yields lower accuracy than supplying only the target API's knowledge, even when retrieval finds the target. Fine-tuning on other APIs improves performance on unseen APIs mainly when external knowledge is supplied, helping models both select the target API and integrate it into the surrounding code. These findings highlight complementary roles for external API knowledge and the ability to apply it. By separating knowledge content, delivery, and use, NovelAPIBench helps diagnose what limits novel API use and guides adaptation to evolving libraries.

Primary area:
datasets and benchmarks
Keywords:
Evaluation Benchmark, Code LLMs, Knowledge Editting, API Knowledge, Model-Conditional Evaluation, Tool Use

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RxiFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

NovelAPIBench constructs model-specific API-use tasks and studies how signatures, examples, mechanism descriptions, and source code affect performance. It separates oracle knowledge from retrieved context and examines adaptation on held-out APIs. Execution records and a failure taxonomy support diagnosis beyond a single aggregate score.

Critical strengths

Separating knowledge content, delivery, and application produces useful diagnostic comparisons. The shared-task analyses across models and API-clustered confidence intervals strengthen the evaluation.

The authors investigate several construction concerns rather than leaving them implicit. Fresh-example tests, release-boundary analyses, human failure-label validation, and API-disjoint adaptation splits are valuable controls.

Critical weaknesses
  1. The reported pass@1 measures target-API use more directly than complete program correctness. Appendix B.3 checks reference call fingerprints or arguments, and some tasks require only call or subclass presence. Downstream outputs are not compared. A completion could therefore use the target API correctly while producing an incorrect overall result, while valid alternative usages may be rejected. An end-to-end audit on a representative subset would clarify this metric’s reliability.
  2. The knowledge-editing comparison primarily tests transfer to APIs absent from adaptation training. Those methods never receive the held-out APIs’ facts during editing, so their results say little about directly acquiring those facts. Appendix G.4 provides a useful same-API study for SFT; extending that control to the editing methods would support a more meaningful comparison of acquisition strategies.
  3. The retrieval analysis leaves two different causes mixed together. A retrieval hit requires only one target-API chunk, whereas the oracle receives the complete target bundle. The remaining performance gap may reflect incomplete evidence, distracting context, or both. The paper acknowledges these possibilities, but controlled context interventions would make the diagnosis substantially more actionable.
Questions for authors

How often do API-level passing completions fail independent end-to-end checks, and how often are valid alternatives rejected?

How do the editing methods perform on new tasks involving APIs that were actually edited?

Can you separately test partial target documentation and complete target documentation supplemented with distractors?

Reviewer_sz4YOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents NovelAPIBench, an automated benchmark for diagnosing how code LLMs acquire and use knowledge about novel APIs. The framework (1) identifies newly introduced or modified APIs, (2) constructs their knowledge bundles containing signatures, examples, mechanism descriptions, and implementation code, and (3) generates the corresponding executable programming tasks. It further introduces model-specific task filtering and different failure categories to analyze API usage. Experiments across multiple libraries and code LLMs evaluate the effectiveness of different knowledge components, knowledge delivery, and parametric adaptation. Results show that usage examples provide the strongest standalone guidance.

Critical strengths
  1. LLM's capabilities of using APIs is an interesting problem to study.
Critical weaknesses
  1. Unclear motivation for the proposed framework: The paper does not sufficiently justify why failure to use an unknown API without external documentation constitutes a meaningful model-specific knowledge gap. An API's name and usage constraints are externally defined information that cannot generally be inferred from a programming task, and in real software development, such information is also expected to be retrieved from documents instead of being inferred by human developers. Failure to use an unknown API without external knowledge is expected and does not necessarily indicate a meaningful model deficiency. Why is it necessary to benchmark a model for inferring an API that is not specified in the task description?

  2. The benchmark's check suffers from both false positive (FP) and false negative (FN) issues:

  • FP: The benchmark mainly checks whether generated code executes successfully and invokes the target API using calls that are consistent with the reference solution. Nevertheless, cases that incorrectly process the API's return value or omit required computations can also pass the check even if they use the API in the wrong way. Note that this is not a minor issue and simply acknowledging it in the limitation section is insufficient.

  • FN: Since the benchmark excludes the target API name from task descriptions (while requiring generated solutions to invoke that specific API), a functionally correct solution using an alternative API or implementation can be considered incorrect.

  1. RQ2 claims that retrieving the target API does not address the performance gap relative to oracle knowledge. However, the paper defines a successful retrieval as having at least one chunk from the target API among the top 5 results, without verifying whether the retrieved chunks contain sufficient information to complete the task. Moreover, Appendix D.3 shows that 81% of chunks under the Full condition exceed the embedding model's 512-token limit and are truncated during indexing, which potentially compromises the retrieval quality; as a result, the performance gap observed even on retrieval-hit tasks may simply reflect incomplete API documentation, not the model's inability to utilize retrieved knowledge.

  2. RQ3's comparison with knowledge editing methods is questionable because these methods are primarily designed to modify specific knowledge associations. In the setting of RQ3, however, the target APIs at evaluation time have never been included in the editing process, and it is not surprising that editing methods have poor performance. Why should this be treated as the ineffectiveness of novel API acquisition? A meaningful evaluation of knowledge editing should examine whether injecting knowledge about a target API improves its subsequent use on unseen tasks involving that API.

Questions for authors
  1. Why should failure to use an unfamiliar API (without external documentation) be considered as a meaningful model-specific knowledge gap?

  2. What are the FP and FN rates in the benchmark's checks?

  3. To what extent does the retrieval performance gap stem from incomplete API documentation vs. the model's inability to utilize retrieved knowledge?

  4. How do knowledge-editing methods perform on unseen tasks involving APIs whose knowledge was actually edited?

Reviewer_RLeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NovelAPIBench, a model-conditioned benchmark for evaluating how code LLMs acquire and use unfamiliar or modified APIs. The framework identifies novel APIs, generates executable coding tasks, and evaluates the effects of different knowledge components, including API signatures, usage examples, mechanism descriptions, and implementation code. The authors also investigate retrieval-based knowledge delivery and parameter adaptation methods such as SFT, RAFT, and knowledge editing. Experiments across six backbone models and multiple Python libraries suggest that usage examples provide the strongest standalone guidance, while fine-tuning improves the application of externally supplied API knowledge.

Critical strengths
  1. Important research problem and model-conditioned evaluation. The paper addresses a significant limitation of conventional code-generation evaluation metrics, which do not adequately distinguish a model's lack of API knowledge from its inability to apply available knowledge. The decision to construct a model-conditioned benchmark is particularly valuable, since API novelty is not universally defined. An API unfamiliar to an older model may already be represented in the training distribution of a newer model. This makes the proposed framework relevant for evaluating evolving coding models.

  2. Broad experimental coverage. The paper evaluates multiple backbone models with different sizes and architectures across a diverse set of Python libraries. This experimental coverage strengthens the empirical findings and makes the benchmark potentially useful to the broader code-generation research community.

Critical weaknesses
  1. Mismatch between evaluation criteria and the intended measurement of API knowledge. My primary concern is that the paper's objective of evaluating successful API use is not fully captured by its evaluation criterion. The execution-based checks primarily verify target-API invocation and agreement with reference-derived call records, rather than complete end-to-end functional correctness. Additionally, the empirical novelty criterion retains tasks when all three sampled completions fail. Although this is a reasonable filtering heuristic, it does not necessarily distinguish missing API knowledge from other causes of unsuccessful code generation, such as reasoning difficulties or implementation errors. These limitations raise concerns about whether the benchmark accurately measures the knowledge gaps it aims to diagnose.

  2. Potential bias from shared task and knowledge generation. Both the knowledge bundles and benchmark tasks are constructed using GPT-5-mini. Importantly, the task generator receives the API usage examples and descriptions that are subsequently evaluated as knowledge interventions. This creates a dependency between task construction and evaluation, potentially favoring the generated examples. Although the authors include a fresh-example robustness experiment, the observed performance decrease suggests that construction-example alignment may influence the reported results. Greater independence between task generation and evaluated knowledge would strengthen the conclusions.

Questions for authors

Evaluation validity: Since the current evaluation primarily relies on API-call agreement, have the authors assessed how often solutions that pass the benchmark also satisfy independent end-to-end functional correctness tests? Such evidence would help establish whether the evaluation adequately captures successful API use.

Empirical novelty: How do the authors distinguish a model's lack of API knowledge from failures caused by general code-generation or reasoning difficulties? Have they considered additional samples or independent API-knowledge probes to validate the three-sample novelty criterion?

Task-generation bias: Given that the task generator receives the same usage examples later evaluated as knowledge interventions, how much of the reported benefit of examples can be attributed to this overlap? Could the authors further validate their conclusions using tasks constructed independently of the evaluated examples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.