- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper describes experiments in detecting code that has some (even minimal) machine-involvement for generation. The authors do this at various granularities. They experiment with different datasets and have detailed results.
Critical strengths
Detailed experiments.
Critical weaknesses
Unfortunately, the authors do not explain well what is novel about their work (and this is not clear to me). They also do not motivate their method (again it is not clear which part is new). They also report too many numbers, but do not discuss what they mean and what we shoudl remember about this work. The list of contributions is quite complex, and I am lost what exactly is being claimed. The results are in the appendix as well as the related work, which makes it hard to follow the work (I also see this as an implicit page limit violation as critical results are in teh appendix, even though they are discussed in the main text, i.e., I see some main-text tables put beyond the page boundaries).
Detailed comments:
-
The list of contributions is somewhat unfocused -- the sentences are too long, which makes it hard to assess what exactly is being claimed. For example, here is how teh second contribution starts: "A mixed-authorship data construction protocol that synthesizes continuation and fill-in-the-middle edits with character-level ground truth and a controlled AI ratio ρ, which supplies within-file supervision for the line and block heads and test cells covering both tail and interior segment geometries." -- By the time I read line 2, I have forgotten what this was about. All three contributions are like that, and I am lost what exactly is being claimed. It is also unclear to me where novelty is, based on such unclear list list of contributions: is it a new task formulation, is it a new dataset, is it a new method? These over-specified descriptions make it extremely hard to tell.
-
Related Work is in the appendix rather than in the main text. I strongly advice against this. This is a critical part of the paper: without it, it is hard to position this work. Moreover, the related work for some research lines is outdated: very little work after 2024 (there are a couple of 2026 paprrs cited, by I find it insufficient).
-
091-093: "The file-level label is binary and AI-involved: y = 1 if any part of the file was generated by a language model, and y = 0 if the file is entirely human-written." -- I have a problem with this terminology. First, "The file-level label is binary and AI-involved" at first made me think this "and" meant "both" rather than "vs"; consider rephrasing. Second, "AI-involved" is not the same as "generated by a language model". Can you make it clear what the task is: is it to detect "AI-involved" or "LLM-generated" code? This is not the same thing. You need to pick one term and stick to it, but which one you choose has implications.
-
There are quite a few hyper-parameters set for the experiments. The authors need to explain how their values were set and to motivate these values.
-
Section 3 is badly written: It tells me what is done, but it does not explain why it is done that way. There is no motivation why things are done in a certain way. There is also no explanation what is novel vs. what follows previous work.
-
There is too much use of passive voice, which makes it hard to tell sometimes what the authors did vs. what was done by somebody else.
-
195: "Table 20 (Appendix A.8) summarizes the splits:" -- This is very important information. It cannot be in the appendix! (just like related work)
-
The text refers to table in the appendix, e.g., Tables 7, 18, 22, etc. I cannot read the paper's main text only to get the idea. I see this as an implicit violation of the page limit. I should be able to understand teh paper from the main text only, but critical information such as the results, and related work are in the appendix.
-
I am somewhat lost in the many experiments: there is so much detail on results, but not much discussion on what this means, why it matters, and what I should remember about this paper.
-
The conclusion is weak: it goes again into numbers etc. and doe snot tell us much about why this work matters and what we should remember about it. There are summary sentences like "The mechanisms are therefore complementary rather than competing.", but I find them too generic: having read the paper, I could not remembered anything about it. It reads like a list of experiments -- more like a documentation of what is done, without much effort to explain why this is done that way and what we learn from the results.
-
I would suggest adding some note on future work.
Minor comment:
- The main contributions of this paper are: --> "The main contributions of this paper are: as follows:" (do not break a clause in the middle by ":")
Questions for authors
What is new about this work? Which part is new? Can you motivate your method? Can you motivate your hyper-parameters?