- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes ICall-Bench, a novel benchmark which aims to evaluate the model's ability of interprocedural reasoning by recovering the possible function candidates for each indirect call site from binary files. To curate such benchmark, 50 real-world programs are selected from OSS-Fuzz which contains thousands of functions and indirect-call sites; such programs are compiled, stripped of static symbols and debug information, and augmented for better evaluation. Several state-of-the-art models are tested, which shows that the proposed benchmark proposes a challenge for those models, which can be greatly helped by tool use but not necessarily by Internet access.
Critical strengths
-
This paper studies a very interesting and timely question on the agent's capability of tracing function calls across a code project, which is very useful for today's long-horizon coding agent. Compared to single-function evaluaiton, such setting is closer to actual reverse engineering applications.
-
The case study on the reasoning failure of the models are particularly interesting; such results illustrates the pattern of reasoning deficiency of current state-of-the-art model, which helps iteration of these models. It is also quite interesting that Internet access does not necessarily help, which points out a direction for further optimization of these models.
Critical weaknesses
The evaluation results are insufficient.
a) As only one run is tested, the performance of models can be largely fluctuating due to LLM randomness (especially with default temperature). Considering the fact that the cost for the experiments is managable (Tab. 2), I would strongly recommend running more parallel trials (with segregated memory) and report standard deviation across the runs. Also, I would argue against the statement "We evaluate ... rather than pass@k or best-of-k, as selecting the best of k runs is impractical in real-world indirect-call recovery where ground truth is unavailable"; I think these metrics are still useful, as they reveal the potential of the model; a model with high pass@k or best-of-k may be better leveraged by techniques such as finetuning or majority vote.
b) I am a bit worried that the newer models, such as Opus 5.5 and GPT-6 Astra can already well-address the benchmark. While I understand that such models are published quite recently, these models are considered to be significantly stronger than the model tested in this benchmark such as Opus 5 and GPT-5.6 Sol. Opus 5 already achieves >60% performance on the large indirect call scale (Fig. 2(a)); the proposed benchmark may saturate quickly, which could limit the impact of this benchmark.
See "questions for authors" for more points.
Questions for authors
-
What is the relation between performance and the number of agent calls? Does the overall accuracy improves as the number of function calls grows due to self-reflection? Judging from Appendix B, the agent does not know the total budget. Will it discover cleverer strategy if it is budget-aware, e.g., reserve some calls for self-reflection?
-
Since PVSR is an "optimistic upper bound" (line 281) of the final performance, is there an estimation on how different PVSR is from actual performance? Such result might be indicated from precision/recall/F1 from the exact target set and difficulty distribution of such subset in the benchmark.
-
Since the model can only run in a limit of 3 hours, is the time proportion for waiting for response in normal range? If a model is accessed much more often (by other users) than other models during test time, much time might be wasted on waiting for the model to respond, and the quality of service might be degraded.
Thank you for your response. Most of my concerns has been addressed. Regarding my question 1, what I am asking for is not the relation of PVSR and number of function calls across different models, but within each model. Consider plotting a figure where the x-axis is the number of calls and y-axis is the PVSR. The question is, what is the area under the curve for each method? Did the PVSR grow quickly at the beginning of the run and then got stagnant, or did the model use 90% of its budget to understand the codebase before submitting anything? Can we learn useful insights from those results on the failing patterns, and how can we train a better model to address the task proposed?