acceptodds
Under review as a conference paper at ICLR 2027

ForenEvolve: Hypothesis-Driven Program Evolution for AI-Generated Image Detection

Abstract

Developing an AI-generated image detector involves coupled choices: which forensic cues to extract, how to preserve them under image processing, and how to adapt pretrained features without overfitting to known generators. Because these choices interact, a revision's value becomes clear only after testing across image sources and processing conditions, so automation requires feedback that shows where a revision helps or fails and a record that carries this evidence forward. We introduce ForenEvolve, an agentic framework for hypothesis-driven evolution of detector programs. Its search space spans three nested capability tiers: frozen forensic feature views and readouts, training-image transformations, and low-rank backbone adaptation with programmable losses. Before each experiment, a language-model agent records a hypothesis and predicted outcome; the revision is then measured across image sources and processed copies of the same images, with controlled revisions compared against their parents. Persistent experimental lineages link each outcome to its prediction, so later revisions build on accumulated evidence. The discovered detector reaches 94.34% mean balanced accuracy across 12 public benchmarks, exceeding the strongest published detector by 9.12 percentage points; it ranks first on nine, including all four excluded from search, where it reduces mean balanced error by 55% relative to the best published result. With the same backbone and training data, it exceeds standard adaptation by 4.14 points overall and outperforms three transplanted state-of-the-art recipes on every benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI-generated image detection, LLM agents, program evolution, hypothesis-driven experimentation, image forensics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WuPXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses an LLM agent to develop AI-generated image detectors through iterative code changes and evaluation. Before each experiment, the agent states what it expects the change to achieve, then compares this prediction with results across image sources and processing conditions. It can revise feature extraction, training-image transformations, and backbone adaptation, while drawing on records of earlier experiments. The resulting detector achieves 94.34% mean balanced accuracy across 12 benchmarks, including 91.95% on four benchmarks excluded from search. Comparisons with the same backbone and training data, along with framework ablations, examine where the gains come from.

Critical strengths
  1. The framework gives detector search a useful experimental structure. Recording predictions before evaluation, restricting controlled edits, and retaining condition-level outcomes make revisions easier to interpret and revisit. The search traces provide concrete examples of how earlier results inform later designs.
  2. The empirical evidence goes beyond comparisons with published detectors. The paper includes baselines using the same backbone and training data, four search-excluded benchmarks, and three full search runs. The repeated runs retain gains over standard adaptation and the strongest transplanted recipe, supporting the usefulness of the overall framework. The final detector also runs without an LLM at inference.
Critical weaknesses
  1. Some controls are not evaluated using their highest-scoring programs. Table 15 lists a best observed J of 0.9585 for both Generic archive search and Unconstrained revision, but the evaluated programs have J values of 0.9577 and 0.9569, respectively. Full uses its highest-scoring program. The reason for this difference is unclear, and evaluating the best valid candidate from each control could change the reported detection gaps.
  2. The specific benefit of hypothesis registration is not fully separated from the rest of the protocol. “No hypothesis protocol” removes research cards, edit restrictions, and question continuation together. The other controls help assess parts of the framework, but do not isolate the research card itself. In addition, only Full is repeated across seeds, so the variability of the smaller gaps between search strategies remains unclear.
Questions for authors
  1. Why do the evaluated programs for Generic archive search and Unconstrained revision differ from their best observed programs in Table 15? What are the L2 results when each method uses its highest-J valid candidate?
  2. What happens if only the research card is removed while feedback, edit restrictions, and question continuation remain unchanged? Repeating this comparison and the Generic archive search control would help establish whether the gains are consistent.
  3. Does the advantage over Generic archive search remain under a matched token budget? Table 13 reports about 6.64M output tokens for Full and 2.35M for Generic, despite the same proposal allowance.
4 comments
AuthorsOpenReview ↗

Response (Part 1/2)

Thank you for the careful review, and for checking the J values in Table 16 and the token counts in Table 13. In the following, we respond to each concern. Numbers are 12-benchmark bACC (%) with training-selected thresholds; L2 is the mean over the four benchmarks excluded from search, and brackets are 95% paired bootstrap CIs of Full minus the compared arm.

Q1: Evaluated versus best-observed programs (W1, Q1)

A: The difference you found comes from when the controls were evaluated. The rule (maximum valid J) was the same for every arm, but we evaluated the controls from a snapshot taken before three of their runs had finished (Generic at 161 of 240 outcomes, Unconstrained at 129, Restarts at 181), while Full was evaluated at the end of its run. This is why the evaluated programs of Generic and Unconstrained (J 0.9577 and 0.9569) differ from their best observed ones (0.9585). We have now evaluated every control's final highest-J valid program (Table E).

Table E: Snapshot programs and final highest-J programs

ArmProgramJL0L1L2All
Full (main)R35/s10.964493.0598.0391.9594.34
Generic, snapshotR13/s20.957791.9097.7490.5593.39
Generic, final highest JR35/s00.958592.0397.6190.7893.47
Unconstrained, snapshotR13/s10.956992.3497.6487.7092.56
Unconstrained, final highest JR36/s20.958592.3197.5687.9692.61
Restarts, snapshotR29/s10.954191.0897.4487.7292.08
Restarts, final highest JR39/s00.954291.3597.3787.7792.16

On L2:

  • Generic moves from 90.55 to 90.78, Unconstrained from 87.70 to 87.96 and Restarts from 87.72 to 87.77, against 91.95 for Full.
  • Across all levels each program changes by at most 0.27 points, and Full's lead stays at 0.87 to 2.18 overall and 1.17 to 4.18 on L2.

At the completion counts of these three controls' snapshots, Full's best J was 0.9616 or 0.9620. Both values exceed the final best J of Generic archive search, Unconstrained revision and Archive restarts, whose maximum is 0.9585. In the revised manuscript, Table 2 reports the final results, Table 16 lists the selected programs and completed proposal counts, and Appendix C.1 states the selection rule.

Q2: Removing only the research card, and repeated runs (W2, Q2)

A: Following your valuable advice, we ran both requested experiments.

Card-only control (Table F). The new arm removes only the research card and keeps condition feedback, edit restrictions and question continuation. It reaches J 0.9505, 92.69 overall and 89.79 on L2, so removing the card costs 1.65 points overall and 2.16 on L2, about 60% of the effect of removing the whole protocol. The card therefore contributes on its own: keeping condition feedback, single-target edits and question continuation does not make up for removing it.

Table F: Removing only the research card

ArmJL0L1L2All
Full0.964493.0598.0391.9594.34
No research card (new)0.950590.8697.4389.7992.69
No hypothesis protocol0.949390.3096.8688.0991.75

Repeated Generic runs (Table G). With two more seeds, each arm has three runs: Full reaches 94.10 ± 0.23 overall and 91.66 ± 0.38 on L2, Generic reaches 93.36 ± 0.28 overall and 90.94 ± 0.18 on L2. Every Generic seed stays below the weakest Full seed on J and on every benchmark aggregate, though only narrowly on L2. We report the gains as consistent in direction, significant on J and overall bACC.

Table G: Three seeds per arm (mean ± SD)

ArmJL2All
Full0.9643 ± 0.001691.66 ± 0.3894.10 ± 0.23
Generic0.9585 ± 0.001690.94 ± 0.1893.36 ± 0.28

The card-only arm also remains below every Generic seed in overall bACC.

AuthorsOpenReview ↗

Response (Part 2/2)

Q3: The advantage over Generic under a matched token budget (Q3)

A: Full remains ahead of Generic in J, overall and L2 bACC under both matched token budgets (Table I). We agree that output tokens differ: 6.64M for Full against 2.35M for Generic. Most of the difference comes from writing research cards and reading condition-level feedback, the parts of the method under test. We summed per-response usage within each proposal session and accumulated the session costs in candidate-completion order; these totals reproduce Table 13 after rounding. Table I retains the last completion within Generic's unrounded input or output budget (Appendix B.4, Table 14).

Table I: Full at Generic's token budget

Budget matched to Generic's totalLast Full completion within budgetProgramFull's best JAllL2Generic's final best J
Output tokens (2.35M)96R35/s00.961694.0792.310.9585
Input tokens (58.05M)158R17/s20.962094.2291.430.9585
  • Full overtakes Generic early. It first exceeds Generic's final J at completion 51 (J 0.9610), after using 50% of Generic's output tokens and 24% of its input tokens. That program (R21/s0) reaches 93.96 overall and 92.08 on L2, above Generic's final highest-J program (93.47 / 90.78).
  • More tokens alone do not produce better programs. Archive restarts used twice Generic's output tokens (4.76M) and reached J 0.9542. No hypothesis protocol, Unconstrained revision and Independent proposals used 3.0M to 3.8M for J 0.9493 to 0.9585.
  • Results at 30, 60 and 120 proposals are shown in Table D below. Each row uses the highest-J valid program available at that cutoff, counting completed proposals including failures.

Table D: Search stopped at n proposals

nProgramJL0L1L2AllBest control J at n
30R16/s00.952890.8597.7490.5193.030.9495
60R21/s00.961092.0397.7892.0893.960.9574
120R35/s00.961692.0997.8092.3194.070.9577
180R17/s20.962093.1898.0591.4394.220.9577
240R35/s10.964493.0598.0391.9594.340.9633

Please let us know if anything remains unclear.

Thank you for the detailed response and additional experiments. The evaluation using the best-performing programs, the Research Card ablation, and the token-budget-matched comparisons have largely addressed my main concerns. Overall, I maintain a positive assessment of the paper.

Thank you for your thoughtful follow-up and positive assessment. We appreciate the time and care you have put into reviewing our work and the additional experiments, and we are glad that our responses have addressed your main concerns.

Reviewer_S1vbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ForenEvolve, an agentic framework for developing AI-generated image detectors through hypothesis-driven program evolution. Its search space covers frozen-feature views and readouts, training-image transformations, and backbone adaptation with programmable losses. The framework combines prospective research cards, controlled program revisions, condition-level feedback, and persistent experimental lineages.

Critical strengths

1、Strong detection results with meaningful controls for backbone and training data. 2、A concrete and inspectable approach to automated experimentation.

Critical weaknesses

1、The “No hypothesis protocol” ablation changes research cards, edit constraints, and question continuation simultaneously, leaving the independent contribution of hypothesis registration unclear. 2、Each framework control is evaluated with a single search run, and the aggregate-feedback arm has many missing-delivery outcomes, requiring clarification of both variability and failure causes. 3、Table 15 shows that some controls were not evaluated using their highest-objective candidates; the selection rule and its impact on the comparisons need clarification.

Questions for authors

1、What eligibility or selection rule explains the differences between the evaluated and best-observed programs in Table 15? Would evaluating each control’s highest-valid-JJ candidate change the conclusions, particularly on L2? 2、Can you provide a control that removes only research cards while retaining condition feedback, controlled edits, and question continuation? Repeated runs of Full and Generic archive search would also help assess the stability of their difference. 3、What caused the aggregate-feedback arm’s missing-delivery outcomes, and were these failures consequences of the intervention or unrelated execution problems?

6 comments
AuthorsOpenReview ↗

Response (Part 1/2)

Thank you for the constructive review and for the specific, checkable questions. In the following, we respond to each concern. Numbers are 12-benchmark bACC (%) with training-selected thresholds; L2 is the mean over the four benchmarks excluded from search, and brackets are 95% paired bootstrap CIs of Full minus the compared arm.

Q1: Selection rule in Table 16 (W3, Q1)

A: Thank you for checking this. Every arm used the same rule: the program with the maximum valid J, ties broken by name, as for Full and the public frameworks. What differed was the evaluation point. We took the control programs from a snapshot made before three control runs had finished (Generic at 161 of 240 outcomes, Unconstrained at 129, Restarts at 181), while Full was evaluated at the end of its run. The rule should have been applied at the end of every run, and we have now done so. Table E lists both programs for these three arms.

Table E: Snapshot programs and final highest-J programs

ArmProgramJL0L1L2All
Full (main)R35/s10.964493.0598.0391.9594.34
Generic, snapshotR13/s20.957791.9097.7490.5593.39
Generic, final highest JR35/s00.958592.0397.6190.7893.47
Unconstrained, snapshotR13/s10.956992.3497.6487.7092.56
Unconstrained, final highest JR36/s20.958592.3197.5687.9692.61
Restarts, snapshotR29/s10.954191.0897.4487.7292.08
Restarts, final highest JR39/s00.954291.3597.3787.7792.16

The conclusions are unchanged, including on L2. Each final program differs from its snapshot program by at most 0.27 points on every level. Full leads all of them by 0.87 to 2.18 points overall and 1.17 to 4.18 on L2.

At the completion counts of these three controls' snapshots, Full's best J was 0.9616 or 0.9620. Both values exceed the final best J of Generic archive search, Unconstrained revision and Archive restarts, whose maximum is 0.9585. In the revised manuscript, Table 2 reports the final results, Table 16 lists the selected programs and completed proposal counts, and Appendix C.1 states the selection rule.

Q2: Removing only the research card (W1, Q2)

A: We agree that No hypothesis protocol removes three elements together, so we ran the requested arm. No research card removes only the card: its question, hypothesis, predicted outcome and falsification criterion, together with the prediction-versus-outcome lines in observations. It keeps condition feedback, single-target controlled edits and question continuation, and matches Full in everything else.

Table F: Removing only the research card

ArmJL0L1L2All
Full0.964493.0598.0391.9594.34
No research card (new)0.950590.8697.4389.7992.69
No hypothesis protocol0.949390.3096.8688.0991.75

Removing only the card costs 1.65 points overall and 2.16 on L2, about 60% of the effect of removing the whole protocol. The arm is also below every Generic seed (Q3) and the weakest Full seed on J, overall and L2 bACC. The card therefore contributes on its own: keeping condition feedback, single-target edits and question continuation does not make up for removing it.

AuthorsOpenReview ↗

Response (Part 2/2)

Q3: Repeated runs of Full and Generic (W2, Q2)

A: Full has three searches (Tables 24 and 25). The weakest of them exceeds every single-run control in the paper on L0, L2 and overall bACC. We ran Generic archive search with two more seeds. Each seed is evaluated with its highest-J program after 240 proposals.

Table G: Three seeds per arm (mean ± SD)

ArmJL2All
Full0.9643 ± 0.001691.66 ± 0.3894.10 ± 0.23
Generic0.9585 ± 0.001690.94 ± 0.1893.36 ± 0.28

Every Generic seed (overall 93.47 / 93.03 / 93.56) stays below the weakest Full seed on J and on every benchmark aggregate.

Q4: Cause of the missing deliveries in the aggregate-feedback arm (W2, Q3)

A: Thank you for raising this. The missing deliveries came from a bug in our implementation, not from the intervention. To hide condition-level scores, this arm strips archived programs before the proposer reads them. Once an unparseable program entered the archive, this step raised an error and blocked the session instead of skipping that one file; this accounts for 151 of the 155 missing deliveries. We fixed the bug and re-ran the arm with the same budget, seed and protocol. The results are shown in Table H.

Table H: Aggregate feedback before and after the bug fix

ArmJL0L1L2All
Full0.964493.0598.0391.9594.34
Aggregate, as reported (with the bug)0.959291.8997.1588.8292.62
Aggregate, bug fixed (new)0.963392.5297.4090.5493.49

With the bug fixed, the arm still stays behind on the benchmarks. Full leads by 0.85 overall and 1.41 on L2, and the fixed arm is below every Full seed on both. Condition-level feedback improves held-out generalization in this comparison. In the revised manuscript, Appendix C.1 specifies the aggregate-feedback protocol, and Table 2 reports the re-run in place of the original arm.

Please let us know if anything remains unclear.

Thank you for the detailed responses and additional experiments. The rebuttal substantially addresses my main concerns and strengthens the empirical support for the framework. The consistent ordering across seeds supports the reported advantage, but the number of independent searches remains limited.

Reviewer_4LPKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents ForenEvolve, an agentic framework that searches over feature views, data transformations, adaptation strategies, and losses for AIGI detection. It records hypotheses and experimental outcomes during search. The final detector achieves strong results on 12 benchmarks. The experiments are extensive, but the paper mainly shows that a large task-specific search can find a strong detector. It does not sufficiently demonstrate transferable scientific insight or a fair advantage over carefully tuned detector baselines.

Critical strengths
  1. The paper is clearly written and provides extensive experimental details.
  2. The evaluation covers many benchmarks, backbones, training sets, and search baselines.
Critical weaknesses
  1. The search does not lead to clear, reusable insights. The paper mainly presents the final recipe and several search traces, without summarizing which findings generalize beyond this configuration.
  2. Transferability is unclear. Each new backbone or training set requires another full search, and the discovered recipes differ substantially.
  3. The detector comparison is asymmetric. ForenEvolve receives 240 search proposals, while the same-backbone baselines use fixed recipes. It also uses a searched three-view readout, whereas the baselines use a raw-view readout.
Questions for authors
  1. What concrete insights from the search can be directly reused on a new backbone or dataset without running another full search?
  2. How would the results change if Effort, SAFE, DDA-style training, and standard BCE were carefully tuned under the same experimental setting and a comparable development budget?
  3. Could the authors report L0/L1/L2 performance when search is stopped at smaller budgets, such as 30, 60, or 120 proposals?
6 comments
AuthorsOpenReview ↗

Response (Part 1/3)

Thank you for the detailed review. In the following, we respond to each concern. Numbers are 12-benchmark bACC (%) with training-selected thresholds; L2 is the mean over the four benchmarks excluded from search, and brackets are 95% paired bootstrap CIs of ForenEvolve minus the compared row.

Q1: Fair and comprehensive detector comparison (W3, Q2)

A: Thank you for highlighting comparison fairness. We clarify that the search-procedure comparisons in Table 2 use the same 240-proposal budget, backbone, data, program space, evaluator and objective. Generic archive search, OpenEvolve and ShinkaEvolve all remain below ForenEvolve in overall and L2 bACC.

For the detector comparison, we transplant Effort, SAFE and DDA-style training to the same backbone following their published training recipes, alongside standard BCE. We also tune all four recipes over their training hyperparameters (learning rate, epochs, LoRA rank and targets, dropout, batch size, and SAFE's augmentation strengths). Each recipe received 30 random-search trials, and configurations were selected by J under the same checks as the search candidates.

The three-view readout is one of the search's findings. It fuses raw, half-scale and resize224 features, and none of the published recipes uses it. To compare the adaptation recipes on equal terms, and to test whether this finding transfers, we also refit each recipe's readout with the three-view readout, keeping its adapter unchanged. Table A reports all four combinations.

Table A: Same-backbone recipes with default and tuned adapters, under both readouts (All / L2)

RecipeDefault, raw-view readoutDefault, three-view readoutTuned, raw-view readoutTuned, three-view readout
Standard BCE90.20 / 87.4492.55 / 90.1691.83 / 88.7892.86 / 90.61
Effort91.72 / 89.1392.69 / 90.7791.54 / 87.8192.56 / 90.13
SAFE90.00 / 88.0891.72 / 90.3591.75 / 88.9293.16 / 90.84
DDA-style88.09 / 86.9290.72 / 90.1690.68 / 90.0892.70 / 90.97
ForenEvolve (searched adapter)92.81 / 88.8594.34 / 91.95-/--/-

(Default and tuned differ only in the adapter's training hyperparameters. For the four baselines, the default raw-view column reproduces Table 1. ForenEvolve uses the searched adapter in both readout columns and is not re-tuned.)

The discovered readout is effective and transfers. On the default adapters, the three-view readout improves all four transplanted recipes by 0.97–2.63 points overall and 1.64–3.24 points on L2 (Table A).

ForenEvolve remains best overall after baseline tuning with the same three-view readout. It leads SAFE, the strongest baseline overall, by 1.18 points overall and 1.11 points on L2. On L2, ForenEvolve reaches 91.95%, compared with 90.97% for DDA-style, the strongest baseline on this metric.

ForenEvolve leads in overall and L2 bACC in both the 240-proposal search comparison (Table 2) and the tuned-recipe comparison with the same three-view readout (Table A). The component ablation in Q2 (Table B) tests the contributions of R35's readout, training-image resampling and auxiliary losses.

AuthorsOpenReview ↗

Response (Part 2/3)

Q2: Reusable insights, and their reuse without another search (W1, W2, Q1)

A: We identify one meta finding on self-evolution and three on detector generalization. Condition scores in R1–R3 are AUROC; the original search d-scores average AUROC and bACC.

Meta finding. It is practical to conduct hypothesis-driven self-evolution for AIGC detection.

Code and algorithm optimization commonly evaluate evolving programs by correctness and efficiency. In AIGC detection, a higher aggregate score can hide changes in how discrimination survives image processing. Known provenance lets us apply label-preserving transformations to the same images, measuring processing sensitivity with image identity and source fixed. Controlled edits and recorded predictions then let us test hypotheses about this sensitivity. Our JPEG trials raise the original joint d5 score while reducing J (Section 4.5). Removing condition-level feedback lowers L2 bACC by 1.41 points (Table 2), supporting its use in detector self-evolution.

Taking a deeper look into the evolved detector programs, we identify the following three findings about transferable authenticity discrimination.

R1. Transferable real/fake discrimination benefits from complementary image scales.

DDA and Aligned Datasets address differences between real and synthetic training images through data alignment. At inference, resizing and recompression preserve these labels but can change how well the same representation separates real and synthetic images. Across all five backbones, both frozen and selected programs have their two lowest condition AUROCs on downscaling with recompression (d4/d5, each compressed twice). Adding a half-scale view to R35's raw-only readout raises d5 AUROC by 0.046, compared with 0.002 on JPEG-only d2 (Appendix D.5, Table 30). With DDA-style's default adapter fixed, the three-view readout also improves L2 bACC by 3.24 points (Table A). Covering image scale at inference complements training-data alignment by improving transferable discrimination from the same learned representation.

R2. Local real/fake alignment can strengthen image-level authenticity discrimination.

DDA shows that bringing real images and their reconstructed synthetic counterparts closer in pixel and frequency domains can improve generalization by reducing dataset bias. R35 applies an alignment constraint within the representation: its cosine loss aligns real/fake class-mean directions at corresponding patch positions, alongside an image-level pairwise margin, without constructing paired reconstructions (Appendix E.3). Removing local alignment lowers L2 bACC from 91.95 to 90.27; removing the image-level margin lowers it to 91.00 (Table B). This feature-level result complements DDA's input-level finding: local real/fake similarity can support transferable discrimination at the image level.

R3. Cross-source discrimination and processing robustness are distinct dimensions of generalization.

UnivFD demonstrated the transferability of frozen CLIP features across generators. After BCE adaptation, CLIP-L's mean AUROC over the four development source collections rises from 0.774 to 0.885, while d5 AUROC falls from 0.895 to 0.832. DINOv2-L, under the same ViT-L/14 architecture and adaptation setting, reaches a similar source mean of 0.883 but a higher d5 of 0.894, reversing their frozen d5 ordering (CLIP-L 0.895 versus DINOv2-L 0.843). Transferring the complete R35 recipe to CLIP-L brings d5 AUROC to 0.901, showing that the reduction in processing robustness depends on the recipe (Appendix D.5, Table 30). Source discrimination can improve while processing robustness declines, even for a backbone with strong frozen-feature transfer.

Table B removes one component of the main run's fixed R35/s1 recipe at a time. Retraining with seed 0 reproduces the original program; seeds 1 and 2 measure retraining variation for R35 and standard BCE.

Table B: Component ablation of the fixed R35/s1 recipe (retraining seeds)

VariantJAllL2
R35, seed 0 / 1 / 20.9644 / 0.9608 / 0.964894.34 / 93.40 / 94.0591.95 / 91.17 / 91.85
− training-image resampling0.958593.3591.63
− PCL margin and patch geometry (BCE loss; resampling and readout kept)0.958193.6990.75
− patch geometry0.959093.7590.62
− inter-class cosine only0.950793.0490.27
− PCL margin only0.954693.3791.00
readout on raw + half-scale only0.950993.6889.54
readout on raw only0.944792.8188.85
Standard BCE, seed 0 / 1 / 20.9219 / 0.9151 / 0.917190.20 / 89.00 / 89.4587.44 / 86.21 / 87.09

The readout has the largest effect on L2: using only the raw view costs 1.53 bACC points overall and 3.10 on L2 (Table B).

AuthorsOpenReview ↗

Response (Part 3/3)

Q2 (continued): Reuse without another search. Table C applies the final recipe, unchanged, to the other backbones and training sets of Table 4.

Table C: R35's recipe without further search (All / L2)

SettingStandard BCER35 recipe, no searchSetting's own 240-proposal search
DINOv3-L85.36 / 81.2889.70 / 85.4790.28 / 86.49
DINOv2-L80.34 / 80.0184.17 / 81.5982.75 / 82.37
CLIP-L77.24 / 73.5382.92 / 78.9081.11 / 77.04
GenImage training81.41 / 74.9286.11 / 78.7892.60 / 89.68
Community Forensics training83.99 / 82.8586.27 / 84.3889.97 / 87.66

On new backbones the recipe transfers well. It significantly beats standard BCE on all three, overall and on L2 (all six 95% CIs exclude zero), and it exceeds the 240-proposal searches on DINOv2-L and CLIP-L in overall bACC. On DINOv3-L it recovers 88% of the overall gain from that backbone's search over BCE, and without further search it reaches a higher J than a separate 30-proposal, adapter-only search (0.9373 vs 0.9326).

A scale-aware RGB recipe improves on CLIP's high-pass recipe (W2).

CLIP-L's 15 highest-J programs all use high-pass views, compared with none of the other backbones' top 15. On the same CLIP backbone and training data, the unchanged R35 recipe uses raw, half-scale and resize224 views and raises overall/L2 bACC from 81.11/77.04 to 82.92/78.90 (Table C). It also lifts d5 AUROC from 0.831 to 0.901 and J from 0.8767 to 0.8935 (Appendix D.5, Table 30). Transferring the complete RGB recipe improves both bACC and robustness to downscaling with recompression over the selected high-pass recipe. Reuse succeeds across these different input designs.

On new training sets the transfer is partial. The recipe still significantly beats standard BCE (GenImage +4.70 overall [+3.74, +5.83]; Community Forensics +2.28 [+1.80, +2.79]), but recovers less than half of the overall gain from each setting's own search over BCE (42% and 38%). In these two settings, the benchmark that matches the training set is in-distribution. Appendix D.7 (Table 33) reports the transfer comparisons and their CIs.

Q3: Results when search stops earlier (Q3)

A: Table D reports the highest-J valid program available after n completed proposals, including failures, and its bACC on all 12 benchmarks. The last column gives the best valid J across the five controls in Table 17 and Aggregate feedback at the same completion count, using the aggregate re-run after the bug fix.

Table D: Search stopped at n proposals

nProgramJL0L1L2AllBest control J at n
30R16/s00.952890.8597.7490.5193.030.9495
60R21/s00.961092.0397.7892.0893.960.9574
120R35/s00.961692.0997.8092.3194.070.9577
180R17/s20.962093.1898.0591.4394.220.9577
240R35/s10.964493.0598.0391.9594.340.9633

Generic archive search provides the best control J at n = 30, 60, 120 and 180; the corrected aggregate-feedback run provides it at n = 240.

  • At 30 proposals the search already exceeds transplanted Effort (93.03 vs 91.72), and both public frameworks after their full 240 proposals (91.94 and 92.28).
  • From 60 proposals on it exceeds every 240-proposal control in Table 2, on both All and L2.

Near saturation, a higher development J does not guarantee higher external bACC.

From 60 to 240 proposals, J rises from 0.9610 to 0.9644, while L2 varies non-monotonically between 91.43 and 92.31 and overall bACC improves slowly (Table D; Appendix C.2, Table 18). Similar divergence appears in Effort's highest-J tuned configuration, which performs worse on the benchmarks than its default (Table A), and in Aggregate feedback, which reaches a J close to Full's (0.9633 vs 0.9644) but remains 1.41 points lower on L2 (Table H). Most of the later budget therefore refines the development objective, with smaller and non-monotonic changes in external bACC.

Table H: Aggregate feedback before and after the bug fix

ArmJL0L1L2All
Full0.964493.0598.0391.9594.34
Aggregate, as reported (with the bug)0.959291.8997.1588.8292.62
Aggregate, bug fixed (new)0.963392.5297.4090.5493.49

Thank you again for the constructive suggestions. Please let us know if any comparison should be set up differently.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback and suggestions, which have helped improve our work. Based on their comments, we have added experiments and revised the manuscript as follows:

  • Re-evaluated the framework controls using their final highest-J valid programs and clarified the selection protocol in Tables 2 and 16 and Appendix C.1, following S1vb and WuPX.
  • Added an ablation removing only the Research Card in Table 2 and Appendix C.1, following S1vb and WuPX.
  • Expanded the Full and Generic comparison to five independent searches per arm and reported run-level statistical tests in the follow-up discussion, following S1vb and WuPX.
  • Fixed the aggregate-feedback implementation bug and replaced the original results with a re-run in Table 2 and Appendix C.1, following S1vb.
  • Added hyperparameter tuning and comparisons under the same readout for the same-backbone baselines in Section 4.2 and Appendix D.2 (Table 23), following 4LPK.
  • Added results at 30, 60 and 120 proposals in Appendix C.2 (Table 18), following 4LPK, and token-budget-matched comparisons in Appendix B.4 (Table 14), following WuPX.
  • Added component ablations and transfer without further search to three backbones and two training sets in Appendix D.7 (Tables 32 and 33), following 4LPK.
  • Expanded the analysis of reusable findings in Section 4.5 and Appendix D.5 (Table 30), following 4LPK.

We provide detailed responses to each reviewer below and welcome further questions. All additional results and analyses reported during the discussion will be included in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.