- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses an LLM agent to develop AI-generated image detectors through iterative code changes and evaluation. Before each experiment, the agent states what it expects the change to achieve, then compares this prediction with results across image sources and processing conditions. It can revise feature extraction, training-image transformations, and backbone adaptation, while drawing on records of earlier experiments. The resulting detector achieves 94.34% mean balanced accuracy across 12 benchmarks, including 91.95% on four benchmarks excluded from search. Comparisons with the same backbone and training data, along with framework ablations, examine where the gains come from.
Critical strengths
- The framework gives detector search a useful experimental structure. Recording predictions before evaluation, restricting controlled edits, and retaining condition-level outcomes make revisions easier to interpret and revisit. The search traces provide concrete examples of how earlier results inform later designs.
- The empirical evidence goes beyond comparisons with published detectors. The paper includes baselines using the same backbone and training data, four search-excluded benchmarks, and three full search runs. The repeated runs retain gains over standard adaptation and the strongest transplanted recipe, supporting the usefulness of the overall framework. The final detector also runs without an LLM at inference.
Critical weaknesses
- Some controls are not evaluated using their highest-scoring programs. Table 15 lists a best observed J of 0.9585 for both Generic archive search and Unconstrained revision, but the evaluated programs have J values of 0.9577 and 0.9569, respectively. Full uses its highest-scoring program. The reason for this difference is unclear, and evaluating the best valid candidate from each control could change the reported detection gaps.
- The specific benefit of hypothesis registration is not fully separated from the rest of the protocol. “No hypothesis protocol” removes research cards, edit restrictions, and question continuation together. The other controls help assess parts of the framework, but do not isolate the research card itself. In addition, only Full is repeated across seeds, so the variability of the smaller gaps between search strategies remains unclear.
Questions for authors
- Why do the evaluated programs for Generic archive search and Unconstrained revision differ from their best observed programs in Table 15? What are the L2 results when each method uses its highest-J valid candidate?
- What happens if only the research card is removed while feedback, edit restrictions, and question continuation remain unchanged? Repeating this comparison and the Generic archive search control would help establish whether the gains are consistent.
- Does the advantage over Generic archive search remain under a matched token budget? Table 13 reports about 6.64M output tokens for Full and 2.35M for Generic, despite the same proposal allowance.
4 comments
Response (Part 2/2)
Q3: The advantage over Generic under a matched token budget (Q3)
A: Full remains ahead of Generic in J, overall and L2 bACC under both matched token budgets (Table I). We agree that output tokens differ: 6.64M for Full against 2.35M for Generic. Most of the difference comes from writing research cards and reading condition-level feedback, the parts of the method under test. We summed per-response usage within each proposal session and accumulated the session costs in candidate-completion order; these totals reproduce Table 13 after rounding. Table I retains the last completion within Generic's unrounded input or output budget (Appendix B.4, Table 14).
Table I: Full at Generic's token budget
| Budget matched to Generic's total | Last Full completion within budget | Program | Full's best J | All | L2 | Generic's final best J |
|---|---|---|---|---|---|---|
| Output tokens (2.35M) | 96 | R35/s0 | 0.9616 | 94.07 | 92.31 | 0.9585 |
| Input tokens (58.05M) | 158 | R17/s2 | 0.9620 | 94.22 | 91.43 | 0.9585 |
- Full overtakes Generic early. It first exceeds Generic's final J at completion 51 (J 0.9610), after using 50% of Generic's output tokens and 24% of its input tokens. That program (R21/s0) reaches 93.96 overall and 92.08 on L2, above Generic's final highest-J program (93.47 / 90.78).
- More tokens alone do not produce better programs. Archive restarts used twice Generic's output tokens (4.76M) and reached J 0.9542. No hypothesis protocol, Unconstrained revision and Independent proposals used 3.0M to 3.8M for J 0.9493 to 0.9585.
- Results at 30, 60 and 120 proposals are shown in Table D below. Each row uses the highest-J valid program available at that cutoff, counting completed proposals including failures.
Table D: Search stopped at n proposals
| n | Program | J | L0 | L1 | L2 | All | Best control J at n |
|---|---|---|---|---|---|---|---|
| 30 | R16/s0 | 0.9528 | 90.85 | 97.74 | 90.51 | 93.03 | 0.9495 |
| 60 | R21/s0 | 0.9610 | 92.03 | 97.78 | 92.08 | 93.96 | 0.9574 |
| 120 | R35/s0 | 0.9616 | 92.09 | 97.80 | 92.31 | 94.07 | 0.9577 |
| 180 | R17/s2 | 0.9620 | 93.18 | 98.05 | 91.43 | 94.22 | 0.9577 |
| 240 | R35/s1 | 0.9644 | 93.05 | 98.03 | 91.95 | 94.34 | 0.9633 |
Please let us know if anything remains unclear.
Thank you for the detailed response and additional experiments. The evaluation using the best-performing programs, the Research Card ablation, and the token-budget-matched comparisons have largely addressed my main concerns. Overall, I maintain a positive assessment of the paper.
Thank you for your thoughtful follow-up and positive assessment. We appreciate the time and care you have put into reviewing our work and the additional experiments, and we are glad that our responses have addressed your main concerns.
Response (Part 1/2)
Thank you for the careful review, and for checking the J values in Table 16 and the token counts in Table 13. In the following, we respond to each concern. Numbers are 12-benchmark bACC (%) with training-selected thresholds; L2 is the mean over the four benchmarks excluded from search, and brackets are 95% paired bootstrap CIs of Full minus the compared arm.
Q1: Evaluated versus best-observed programs (W1, Q1)
A: The difference you found comes from when the controls were evaluated. The rule (maximum valid J) was the same for every arm, but we evaluated the controls from a snapshot taken before three of their runs had finished (Generic at 161 of 240 outcomes, Unconstrained at 129, Restarts at 181), while Full was evaluated at the end of its run. This is why the evaluated programs of Generic and Unconstrained (J 0.9577 and 0.9569) differ from their best observed ones (0.9585). We have now evaluated every control's final highest-J valid program (Table E).
Table E: Snapshot programs and final highest-J programs
On L2:
At the completion counts of these three controls' snapshots, Full's best J was 0.9616 or 0.9620. Both values exceed the final best J of Generic archive search, Unconstrained revision and Archive restarts, whose maximum is 0.9585. In the revised manuscript, Table 2 reports the final results, Table 16 lists the selected programs and completed proposal counts, and Appendix C.1 states the selection rule.
Q2: Removing only the research card, and repeated runs (W2, Q2)
A: Following your valuable advice, we ran both requested experiments.
Card-only control (Table F). The new arm removes only the research card and keeps condition feedback, edit restrictions and question continuation. It reaches J 0.9505, 92.69 overall and 89.79 on L2, so removing the card costs 1.65 points overall and 2.16 on L2, about 60% of the effect of removing the whole protocol. The card therefore contributes on its own: keeping condition feedback, single-target edits and question continuation does not make up for removing it.
Table F: Removing only the research card
Repeated Generic runs (Table G). With two more seeds, each arm has three runs: Full reaches 94.10 ± 0.23 overall and 91.66 ± 0.38 on L2, Generic reaches 93.36 ± 0.28 overall and 90.94 ± 0.18 on L2. Every Generic seed stays below the weakest Full seed on J and on every benchmark aggregate, though only narrowly on L2. We report the gains as consistent in direction, significant on J and overall bACC.
Table G: Three seeds per arm (mean ± SD)
The card-only arm also remains below every Generic seed in overall bACC.