acceptodds
Under review as a conference paper at ICLR 2027

ICall-Bench: Benchmarking LLM Interprocedural Reasoning in Binary Programs

Abstract

Recent benchmarks show that large language models (LLMs) achieve strong performance in both function-level binary understanding and end-to-end binary reverse engineering tasks, but whether LLMs can reason over interprocedural semantics in compiled binaries remains unclear. Function-level benchmarks offer limited visibility into cross-function relationships, while whole-program benchmarks typically evaluate final outcomes of tasks that may be solvable from local clues alone, making it difficult to disentangle interprocedural reasoning from exploration, tool use, execution, and retrieval. To address this limitation, we introduce ICALL-BENCH, a benchmark that directly evaluates whether LLMs and agents can understand and reason over interprocedural semantics in binary programs, using indirect-call target recovery as a measurable probe. ICALL-BENCH covers 50 complete real-world C/C++ programs spanning different complexity levels and multiple optimization configurations, with an average of 74,746 lines of code and 941 functions per program, enabling analysis of reasoning at whole-program scale and under diverse binary representations. We further introduce three analysis regimes spanning model-only reasoning, static tool-augmented agents, and Internet-enabled open-world agents to examine how intrinsic reasoning, active information acquisition, and external knowledge each contribute to interprocedural binary analysis. We evaluate five recent LLMs and their corresponding agents. Our results reveal a wide separation across current models and agents, with program-level verifiable success rates ranging from 23% to 81%. They also show that greater information access does not consistently lead to better results, with particularly limited gains on the hardest cases, where deeper interprocedural reasoning remains the primary bottleneck. Uneven performance across models and frequent verifiable failures even under an optimistic evaluation indicate that robust binary interprocedural reasoning remains an open challenge.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, LLM Agents, Program Reasoning, Program Understanding, Interprocedural Semantics, Benchmarking, AI for Code, Binary Analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QFKhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes ICall-Bench, a novel benchmark which aims to evaluate the model's ability of interprocedural reasoning by recovering the possible function candidates for each indirect call site from binary files. To curate such benchmark, 50 real-world programs are selected from OSS-Fuzz which contains thousands of functions and indirect-call sites; such programs are compiled, stripped of static symbols and debug information, and augmented for better evaluation. Several state-of-the-art models are tested, which shows that the proposed benchmark proposes a challenge for those models, which can be greatly helped by tool use but not necessarily by Internet access.

Critical strengths
  1. This paper studies a very interesting and timely question on the agent's capability of tracing function calls across a code project, which is very useful for today's long-horizon coding agent. Compared to single-function evaluaiton, such setting is closer to actual reverse engineering applications.

  2. The case study on the reasoning failure of the models are particularly interesting; such results illustrates the pattern of reasoning deficiency of current state-of-the-art model, which helps iteration of these models. It is also quite interesting that Internet access does not necessarily help, which points out a direction for further optimization of these models.

Critical weaknesses

The evaluation results are insufficient.

a) As only one run is tested, the performance of models can be largely fluctuating due to LLM randomness (especially with default temperature). Considering the fact that the cost for the experiments is managable (Tab. 2), I would strongly recommend running more parallel trials (with segregated memory) and report standard deviation across the runs. Also, I would argue against the statement "We evaluate ... rather than pass@k or best-of-k, as selecting the best of k runs is impractical in real-world indirect-call recovery where ground truth is unavailable"; I think these metrics are still useful, as they reveal the potential of the model; a model with high pass@k or best-of-k may be better leveraged by techniques such as finetuning or majority vote.

b) I am a bit worried that the newer models, such as Opus 5.5 and GPT-6 Astra can already well-address the benchmark. While I understand that such models are published quite recently, these models are considered to be significantly stronger than the model tested in this benchmark such as Opus 5 and GPT-5.6 Sol. Opus 5 already achieves >60% performance on the large indirect call scale (Fig. 2(a)); the proposed benchmark may saturate quickly, which could limit the impact of this benchmark.

See "questions for authors" for more points.

Questions for authors
  1. What is the relation between performance and the number of agent calls? Does the overall accuracy improves as the number of function calls grows due to self-reflection? Judging from Appendix B, the agent does not know the total budget. Will it discover cleverer strategy if it is budget-aware, e.g., reserve some calls for self-reflection?

  2. Since PVSR is an "optimistic upper bound" (line 281) of the final performance, is there an estimation on how different PVSR is from actual performance? Such result might be indicated from precision/recall/F1 from the exact target set and difficulty distribution of such subset in the benchmark.

  3. Since the model can only run in a limit of 3 hours, is the time proportion for waiting for response in normal range? If a model is accessed much more often (by other users) than other models during test time, much time might be wasted on waiting for the model to respond, and the quality of service might be degraded.

1 comment

Thank you for your response. Most of my concerns has been addressed. Regarding my question 1, what I am asking for is not the relation of PVSR and number of function calls across different models, but within each model. Consider plotting a figure where the x-axis is the number of calls and y-axis is the PVSR. The question is, what is the area under the curve for each method? Did the PVSR grow quickly at the beginning of the run and then got stagnant, or did the model use 90% of its budget to understand the codebase before submitting anything? Can we learn useful insights from those results on the failing patterns, and how can we train a better model to address the task proposed?

Reviewer_U4fEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ICall-Bench evaluates LLMs and agents on indirect-call target recovery as a diagnostic probe of interprocedural reasoning in binary programs. The benchmark comprises 100 stripped binaries compiled from 50 real-world C/C++ programs under two optimization settings. Ground truth combines dynamically observed and replay-verified call edges with injected callsites having exact target sets and known unreachable decoys. Five LLM-based systems are evaluated under model-only, static tool-augmented, and Internet-enabled open-world regimes. The primary metric, Program-Level Verifiable Success Rate (PVSR), measures whether a program contains any error detectable under the available ground truth. Results show substantial differences across systems, improvements from static tool augmentation, and heterogeneous effects of open-world access.

Critical strengths

S1. A concrete and potentially useful probe of binary program understanding.

The paper targets explicit callsite-to-target relations rather than relying exclusively on function-level reconstruction or final reverse-engineering outcomes. This formulation provides a useful basis for investigating cross-function pointer propagation and dispatch resolution. The assembly-level case studies further illustrate specific failure mechanisms, including incorrect pointer provenance, structure-field association, and control-flow reasoning.

S2. Execution-grounded evaluation with an explicit treatment of incomplete ground truth.

The benchmark verifies observed call edges through execution and replay, while exact-set injections and unreachable decoys provide additional checks against overprediction. Importantly, the paper acknowledges that PVSR is an optimistic bound rather than a guarantee of complete program-level correctness. This transparency and the use of mechanically checkable constraints are meaningful strengths, although the coverage and representativeness of those constraints require further validation.

Critical weaknesses

W1. Insufficient characterization of injected data and its impact on open-world evaluation.

While §3.2 describes three injection mechanisms and §5 reports 802 injected callsites with exact target sets, the manuscript does not provide a complete breakdown of natural versus injected callsites and edges, or evidence that the injected cases reasonably represent real-world dispatch and pointer-propagation patterns. Placing injections on executable paths and validating their targets establishes reachability, but not representativeness. Without distributional characterization and separate results for natural and injected relations, it remains unclear how strongly the benchmark’s difficulty and rankings depend on its synthetic construction.

The injections may also confound the interpretation of open-world results. Section 4.3 attributes several Claude Opus 5 failures to mismatches between retrieved source code and the analyzed binary, while §3.2 explicitly states that injected relations are absent from public program versions. However, the paper does not establish how many such failures are caused by these modifications. If performance losses concentrate on artificially introduced source–binary discrepancies, they may primarily reflect robustness to benchmark-induced inconsistencies rather than a general limitation in using external knowledge for real-world interprocedural analysis. A paired comparison of unmodified and augmented programs is needed to distinguish these explanations.

W2. Program-level scores do not sufficiently characterize callsite accuracy or validate the intended reasoning capability.

PVSR is informative as a strict program-level criterion, but it collapses substantially different outcomes—such as one missed target and widespread recovery failures—into the same unsuccessful result. Appendix C categorizes program failures, but does not provide a systematic account of prediction quality at individual callsites. Although exhaustive precision cannot be established for naturally occurring calls with incomplete ground truth, the paper could still report observed-target recall on a fixed set of covered callsites, exact-set accuracy and precision/recall on the closed-set subset, and consistent treatment of unresolved outputs. These measurements are necessary to understand what the reported program-level scores imply about actual recovery accuracy.

Furthermore, §§1 and 3.1 provide a plausible motivation for indirect-call recovery, but the benchmark does not quantify how many evaluated callsites actually require interprocedural rather than intraprocedural analysis. Whole-program context and the presence of indirect calls do not by themselves establish that successful predictions require cross-function reasoning. Without characterization of the necessary reasoning scope, the connection between PVSR and the claimed capability remains insufficiently validated, as does the rationale for selecting this probe over alternatives such as interprocedural data-flow reachability.

Questions for authors
  1. What are the proportions and structural characteristics of natural versus injected callsites and edges, and what evidence supports the representativeness of the injected patterns?

  2. How many open-world failures, including Claude Opus 5’s source–binary mismatches, are attributable to injected modifications, and do the conclusions persist on paired unmodified programs?

  3. Can you report observed-target recall at individual covered callsites and exact-set accuracy and precision/recall on the closed-set subset, with unresolved outputs handled consistently?

  4. Can you quantify the proportion of evaluated call sites that require interprocedural rather than intraprocedural data-flow analysis, and explain why indirect-call target recovery was chosen over interprocedural data-flow reachability as a probe of this capability?

Reviewer_ipeZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Call-Bench, a benchmark to evaluate the interprocedural reasoning ability of large language models on binary programs. This paper uses indirect-call target recovery as the evaluation task to measure whether models can reason about data-flow and control-flow relationships across different functions. It introduces three evaluation settings: model-only reasoning, static tool-augmented agents, and Internet-enabled open-world agents. This paper then evaluates five LLMs using this benchmark. The results show clear performance differences among systems on the indirect-call recovery task. Static tool augmentation consistently improves the success rate, while additional access to execution, tools, and external information does not always bring further improvements.

Critical strengths

1.This paper provides a meaningful benchmark for evaluating the interprocedural reasoning ability of large language models on binary programs. ICall-Bench requires LLMs to recover indirect-callsite-to-target relations from complete real-world binaries. Compared with end-to-end reverse engineering benchmarks, it provides a more focused evaluation task with verifiable correctness constraints.

Critical weaknesses

1.The novelty of this paper is limited compared with existing call-relation benchmarks. Previous benchmarks such as CG-Bench[1] and TraceEval[2] have already explored the use of call-relation recovery to evaluate LLMs' program reasoning ability. The main contribution of ICall-Bench is to extend this evaluation from source code to large-scale stripped binaries and introduce different analysis settings for LLM-based agents. Although these extensions are useful and introduce additional challenges, the underlying evaluation task is not fundamentally new. In addition, although the experiments provide quantitative results on different models and agent settings, many findings mainly confirm known challenges in indirect-call analysis and tool-assisted reasoning. The paper does not clearly explain what new insights in interprocedural reasoning that ICall-Bench can provide beyond existing benchmarks.

2.The soundness of this paper is also limited. PVSR is based on dynamically verified positive targets, injected exact-target callsites, and known hard negatives. Although this design avoids treating unobserved edges as incorrect by default, it does not fully penalize infeasible targets predicted at naturally occurring callsites. Therefore, a program may pass PVSR without precisely recovering the target sets of all indirect calls. Although this paper introduces exact-target callsites and hard negatives to reduce this limitation, the reported results do not fully show how well PVSR reflects overall recovery precision. In addition, the observed performance drop on larger programs may partly reflect error accumulation rather than increased reasoning complexity. Although the paper acknowledges this effect and provides a breakdown of failure categories, more fine-grained evaluation results are still needed.

[1]Yuan T, Zhang W, Chen D, et al. CG-Bench: Can Language Models Assist Call Graph Construction in the Real World?[C]//Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages. 2025: 12-20. [2]Li Y, Jiang J, Zhang T, et al. An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning[J]. arXiv preprint arXiv:2605.11006, 2026.

Questions for authors
  1. What is the main advance over existing call-relation reasoning benchmarks?
  2. How does ICall-Bench distinguish interprocedural reasoning from other binary-analysis capabilities?
  3. How well does PVSR reflect the actual correctness and precision of indirect-call target recovery?

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.