- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents BinTrap, a counterfactual benchmark for testing the robustness of LLM-based binary semantic recovery to misleading lexical cues. It creates variants of decompiled C/C++ functions by renaming variables, functions, and types, and by injecting topic-related strings, while aiming to preserve the main computation. The paper tests specialized tools and general-purpose LLMs on function naming, variable naming, function summarization, and source reconstruction. It evaluates both semantic accuracy and the tendency of predictions to follow misleading topics. The results suggest that lexical cues can redirect recovery outputs, and the paper includes pipeline checks, human annotations, and comparisons between LLM judges.
Critical strengths
The paired counterfactual design tests a relevant gap in standard binary-recovery benchmarks: models can rely on names and strings instead of code logic. The coverage of several recovery tasks and both specialized tools and general LLMs makes the finding potentially useful across systems.
The authors provide more than aggregate results. The compile-strip-decompile test, human labels, a second LLM judge, confidence intervals, and optimization-level checks help assess the observed effects. In particular, the pipeline test supports the practical relevance of string injection.
Critical weaknesses
The benchmark does not always reflect the stripped-binary setting it aims to study. Table 3 shows that renamed functions, variables, and types disappear after stripping in the tested samples, but these attacks still make up much of the evaluation and the combined attack. Appendix F.4 uses source code directly, so it does not show that these attacks work on decompiled stripped binaries.
The mislead rate may count copied words as errors even when the recovered program logic is correct, especially for T4 source outputs. The two judges also give very different results: under T2 variable renaming, several general LLMs score below 12% with Claude but around 70% with GPT (Tables 5 and 11). This makes it hard to know how much to trust the reported mislead rates.
String injection adds print calls, so it changes what the program outputs. A model that mentions these calls may simply be describing the code correctly. Without a control using neutral strings and the same number of print statements, it is hard to tell whether the effect comes from the misleading topics or just the added code.
Several systems already struggle to recover the original code without any attack, especially the specialized source-reconstruction tools in Table 7. Appendix F.2 looks only at functions recovered correctly, but very few T4 examples remain. This leaves limited evidence for the strongest claims about T4 robustness.
Questions for authors
-
How do the renaming attacks reflect the stripped-binary setting when Table 3 shows that these identifiers disappear after stripping?
-
For T4, how do you distinguish incorrect program logic from copied names or strings when measuring the mislead rate?
-
Do the string-injection effects persist when the same print statements contain neutral text instead of misleading topics?