- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
NovelAPIBench constructs model-specific API-use tasks and studies how signatures, examples, mechanism descriptions, and source code affect performance. It separates oracle knowledge from retrieved context and examines adaptation on held-out APIs. Execution records and a failure taxonomy support diagnosis beyond a single aggregate score.
Critical strengths
Separating knowledge content, delivery, and application produces useful diagnostic comparisons. The shared-task analyses across models and API-clustered confidence intervals strengthen the evaluation.
The authors investigate several construction concerns rather than leaving them implicit. Fresh-example tests, release-boundary analyses, human failure-label validation, and API-disjoint adaptation splits are valuable controls.
Critical weaknesses
- The reported pass@1 measures target-API use more directly than complete program correctness. Appendix B.3 checks reference call fingerprints or arguments, and some tasks require only call or subclass presence. Downstream outputs are not compared. A completion could therefore use the target API correctly while producing an incorrect overall result, while valid alternative usages may be rejected. An end-to-end audit on a representative subset would clarify this metric’s reliability.
- The knowledge-editing comparison primarily tests transfer to APIs absent from adaptation training. Those methods never receive the held-out APIs’ facts during editing, so their results say little about directly acquiring those facts. Appendix G.4 provides a useful same-API study for SFT; extending that control to the editing methods would support a more meaningful comparison of acquisition strategies.
- The retrieval analysis leaves two different causes mixed together. A retrieval hit requires only one target-API chunk, whereas the oracle receives the complete target bundle. The remaining performance gap may reflect incomplete evidence, distracting context, or both. The paper acknowledges these possibilities, but controlled context interventions would make the diagnosis substantially more actionable.
Questions for authors
How often do API-level passing completions fail independent end-to-end checks, and how often are valid alternatives rejected?
How do the editing methods perform on new tasks involving APIs that were actually edited?
Can you separately test partial target documentation and complete target documentation supplemented with distractors?