- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AIDAR, an AST-based method for detecting AI-generated code. The motivation is that existing methods mainly rely on textual features, such as comments and coding styles, which can be easily changed by LLMs. Instead, the paper focuses on the structure of code and uses subtree selection and prototype matching to distinguish human-written and AI-generated code.
In evaluation, the paper compares AIDAR with several existing detectors on code datasets and programming contest submissions. The proposed method generally outperforms the baselines. The paper also includes ablation studies to show the effectiveness of its components and a real-world case study to demonstrate its practical use in programming contests.
Critical strengths
- The paper is well-written and easy to follow. The proposed method is clearly described.
- The paper provides a comprehensive evaluation of the proposed method against several existing detectors, and the results show that the proposed method achieves state-of-the-art performance on the evaluated benchmarks.
- The paper also provides a detailed analysis of the proposed method, including a series of ablation study.
Critical weaknesses
- Design: Is AST alone sufficient for distinguishing human-written and AI-generated code?
- Dataset: Are the benchmarks sufficiently challenging and representative?
- Robustness: How does AIDAR perform under adversarial AST-level modifications?
- Detection Granularity: At what code level is AIDAR expected to work?
Questions for authors
1. Design: Is AST alone sufficient for distinguishing human-written and AI-generated code?
The paper mainly relies on AST structures to distinguish human-written and AI-generated code. However, I am not fully convinced that AST alone can provide sufficiently strong and reliable signals for this task.
AST mainly captures the syntactic structure of a program, such as how functions, statements, and expressions are organized. There might be some structural differences between human-written and AI-generated code. For example, human programmers may write code in a more casual or compact way, while LLMs may tend to produce more structured code with additional functions, blocks, or intermediate steps. However, these differences are not necessarily inherent to AI-generated code. An LLM could easily be instructed to mimic a human's coding style, simplify the implementation, or follow the structure of an existing human-written solution.
On the other hand, many other signals could also be useful for detection, such as naming conventions, comments, indentation, type annotations, and inconsistencies in coding style. Although these features can also be manipulated, they may still provide complementary information that AST alone cannot capture. Completely discarding them may unnecessarily limit the detector.
I have two main questions:
- Why is an AST-only design preferable to combining AST with textual or stylistic features? Is there any evidence that using AST alone is more robust, rather than simply discarding potentially useful signals?
- Can the authors compare the current AST-only detector with a combined detector using both AST and source-code features? This would help justify whether relying solely on AST is a reasonable design choice.
2. Dataset: Are the benchmarks sufficiently challenging and representative?
The paper mainly evaluates the proposed method on CodeNet100, which is derived from the relatively old Project CodeNet dataset, and C1, which is constructed from a single programming contest problem. Although the authors introduce polished human code and human-like AI-generated code to increase diversity, I am not sure whether these benchmarks are sufficiently challenging or representative of current AI-generated code detection scenarios.
For example, as shown in Table 1, several existing methods already achieve very high accuracy, especially on C1. GPTSniffer achieves 97.67% accuracy on C1, while the proposed method achieves 99.00%. This raises the question of whether the dataset contains relatively obvious differences between human and AI code, and whether the improvement would remain on more challenging datasets.
There are also more recent benchmarkes[1][2], which aim to cover more diverse human-written and AI-generated code. Such benchmarks could provide a more convincing evaluation of the proposed method.
My main questions are:
- How representative are CodeNet100 and C1 of real-world AI-generated code, especially given the rapid improvement of recent LLMs?
- Can the authors evaluate AIDAR on some more challenging and diverse benchmark[1][2], and compare it against the existing detectors? It would be interesting to see whether AST-based detection still provides an advantage when the differences between human and AI code become less obvious.
3. Robustness: How does AIDAR perform under adversarial AST-level modifications?
The paper argues that AST structures are more reliable than surface-level textual features, since an LLM can easily change comments, variable names, or formatting. However, AST structures themselves can also be modified, especially when the attacker knows that the detector relies on AST similarity.
For example, an attacker may have access to some human-written solutions and ask an LLM to generate code following similar structures. The attacker could also compare the AST of the generated code with human-written references and iteratively ask the LLM to modify the implementation to reduce their structural distance. Such modifications may involve changing loop structures, reorganizing conditional statements, merging or splitting functions, or replacing expressions with structurally different but functionally equivalent implementations.
More importantly, this type of attack seems particularly relevant to AIDAR, since its detection is based on comparing a query against human and AI reference distributions. If an attacker can make the generated code structurally similar to the human reference pool, the detector may become less effective.
My main questions are:
- How robust is AIDAR when AI-generated code is explicitly instructed to mimic the AST structures of human-written solutions, rather than merely mimic human coding style?
- Can the authors evaluate an adaptive attack where the attacker iteratively modifies AI-generated code to reduce its AST distance to human references, while preserving the program's functionality? This would provide stronger evidence for the claimed robustness of structure-based detection.
4. Detection Granularity: At what code level is AIDAR expected to work?
The paper evaluates the detector mainly on complete solutions to programming problems. However, it is unclear what the intended granularity of AI-generated code detection is.
For example, in real-world software development, AI-generated code may appear as a single function, a class, a file, a module, or even a small modification within an existing codebase. The structural information available at these levels can be very different.
My intuition is that AST-based detection may depend heavily on the size and complexity of the input code. For a very short function, there may be only a few reasonable ways to implement the same logic, so human-written and AI-generated code could have almost identical ASTs. In contrast, larger programs may contain more structural patterns that could help distinguish their origins. However, larger codebases can also contain a mixture of human-written and AI-generated components, which introduces another challenge.
My main questions are:
- What is the intended detection granularity of AIDAR? Is it designed for individual functions, classes, files, modules, or only complete programming solutions?
- How does detection performance change with code size and structural complexity? For example, can the authors report results separately for short functions, medium-sized programs, and larger code samples?
- Can AIDAR detect a small AI-generated function embedded within an otherwise human-written program, or does it require the entire input to be generated by a single source?
References
[1] Orel, Daniil, et al. "AICD Bench: A Challenging Benchmark for AI-Generated Code Detection." EACL 2026.
[2] Guo, Hanxi, et al. "Codemirage: A multi-lingual benchmark for detecting ai-generated and paraphrased source code from production-level llms." arXiv preprint 2025.