- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes RuleCrafter, a framework that uses LLM agents to learn reusable symbolic rules for decompilation. Instead of generating C code directly from assembly, the agents write and refine Datalog rules that reconstruct program structures. The system uses execution feedback and provenance tracing to identify and fix incorrect rules. Once training is complete, the learned rules can decompile new programs without additional LLM calls. The authors evaluate RuleCrafter on HumanEval, MBPP, and ExeBench, comparing it with traditional and neural decompilers.
Critical strengths
-
I like the idea of having LLM agents learn reusable symbolic rules. Most existing LLM-based approaches handle each decompilation task separately, while RuleCrafter allows the knowledge acquired from previous tasks to be reused.
-
The combination of Datalog rules and LLM agents is interesting. The learned rules are interpretable and can be inspected or modified when errors occur. Provenance tracing also provides a reasonable way to identify which rules may be responsible for incorrect outputs.
-
The cross-dataset experiments are encouraging. The authors show that rules learned on one dataset can be applied to other datasets without modification.
Critical weaknesses
-
The performance of RuleCrafter varies significantly depending on the LLM used to construct the rules. For example, the DeepSeek-based rule base achieves only 3.79% and 7.57% re-execution accuracy on HumanEval and MBPP, despite having relatively high readability scores. This makes me question how reliable the rule learning process is.
-
The paper lacks sufficient ablation studies. RuleCrafter relies on several components, including predefined interfaces, provenance tracing, execution feedback, and MaxSMT-based selection. It would be useful to understand which components contribute most to the final performance. In particular, I am interested in how much of the decompilation capability is learned by the agents and how much comes from the predefined system architecture.
-
I have some concerns about the evaluation. Some baseline results are taken from prior work using different evaluation protocols, making direct comparisons difficult. Also, the experiments focus on relatively small benchmark programs. It would be interesting to evaluate the learned rules on more complex real-world programs to better understand their generalization.
Questions for authors
See Weaknesses
We thank the reviewer for the review and feedback. We address the weakness and questions below. Section, table and figure numbers refer to the revised PDF, a summary of the updated results is included under W2 for easier comparison.
W1.
The learner writes the decompiler, so its capability matters, as with any LLM-based method. What differs here is that the result is a fixed program: its correctness is measured by re-execution (R-E) and its readability by R2I, so a rule base’s quality is reflected in these numbers. The one case where R2I misleads is discussed next and in §4.4. DeepSeek's low R-E has a specific cause (§4.4): 74–79% of its learned code sits in the AST stage, concentrated on whole-function templates with placeholder fallbacks. Across 7,360 outputs there are no live loops, gotos or else branches. R2I scores such simple bodies highly, a limitation §6 now states; our correctness claims rest on R-E. With the two stronger learners the same process produces rule bases at the top of the comparison. The unified Opus 5.5 base ranks first in R-E on MBPP and ExeBench among all evaluated systems, neural ones included, and second on HumanEval, 1.5 points behind SK2Decompile. It exceeds every traditional decompiler on all three corpora, by 3 to 25 points over the best of them. The from-scratch GPT-6 Astra base is already on par with Hex-Rays (ahead on two corpora, behind on one).
W2.
All decompilation analysis is learned; the predefined architecture contains no analysis code. It supplies instruction decoding, the stage interfaces, C emitter, the Datalog engine, and the rule base starts empty. The interfaces only name the outputs each stage should produce. How to derive them is left to the workers, who wrote relations of their own in each rule base learned from scratch.
For the components of the rule base, new Table 3 (also included below) gives the share of learned code per reconstruction group. All four groups are populated in every rule base. Learned MaxSMT constraints appear only in the unified base (5.4%), so the single-corpus results do not depend on this component.
Provenance and execution feedback are used while the rules are being learned. They are not components of the finished decompiler, so removing either one would require a new learning run rather than a re-evaluation. §4.2 shows them at work in a recorded repair: a failing test and a KLEE counterexample expose the error, a provenance trace points to the rules responsible, and the worker fixes them in 26 edits. They are also in constant use: across all learners, 36.4% of tool calls read facts or trace provenance, and 17.7% run trial evaluations or read their results.
query_facts,trace_fact)read_c,read_observation,read_feedback)try_rule,promote,discard,edit_rule)claim_substeptry_emit,try_oracle,try_r2i)checkandrotate_batchW3.
Table 2 is now a matched comparison. We evaluate every system ourselves on the same validation sets at O0–O3 with one shared R2I pool. On complexity, ExeBench valid_real consists of real-world GitHub functions, not programming exercises. The unified base's R-E there (50.74) exceeds Hex-Rays (47.63) and every neural system. Like the neural baselines, RuleCrafter reconstructs one function at a time. These three corpora are the most widely used in this literature, which is what allows a matched comparison with prior systems such as LLM4Decompile, SK2Decompile, and Idioms.
We hope this addresses your concerns, and we will be happy to answer any further questions.