- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ForensicCPN addresses image forgery detection and localization with a frozen MLLM (Qwen3.6-35B-A3B) and a frozen toolbox (11 detectors, 3 MLLM-based forensic models, 13 visual operators). Instead of updating parameters, it learns an explicit "skill graph" inspired by Colored Petri Nets: hypothesis-specific branches of tool transitions and evidence conditions. A harness executes the graph, enforces the prescribed actions, evaluates numerical conditions directly, and uses an MLLM Judge for semantic conditions. The skill is revised in two ways: (i) Offline skill learning: Refine/Merge agents propose revisions from labeled trajectories -> a revision is admitted only if it improves utility on a sampled 128-image labeled validation. (ii) Test-time skill evolving: UNSURE cases trigger exploration with the full toolbox, and a consolidation agent writes revisions back without labels, subject only to structural checks. On seven traditional benchmarks, the evolved skill reaches 88.4/88.2 weighted Acc/F1, versus 81.3/77.5 for majority voting over the toolbox and 84.0/80.5 for a dataset-level oracle single tool, and 63.1 weighted box IoU. Test-time evolving raises accuracy from 60.6 to 71.3 on COCOGlide and from 62.5 to 76.5 on AIGI-Bench.
Critical strengths
-
Strong aggregate detection, measured against the right reference points. The paper includes majority voting and a dataset-level oracle over its own toolbox, and the evolved skill beats both (+7.1 and +4.4 weighted acc). This makes the headline result more credible than a comparison against single detectors alone.
-
An explicit, inspectable procedure. The learned skill is a readable graph whose revisions can be traced, and stronger MLLMs can execute it without relearning. The authors are also candid about several limitations (e.g., unequalized budgets in Table 3).
Critical weaknesses
W1. No same-toolbox comparison with competing agentic approaches. The paper discusses tool-using forensic agents in related work (ForenAgent, EvoGuard, AIFo, AgentFoX, etc.) but does not evaluate them. Only ForgeryVCR appears, and it uses its own tools. No skill- or workflow-learning method (e.g., AWM, EvoSkill, SkillDAG, GraSP) is run with the same toolbox and data. The Table 3 variants are manual translations of the final learned CPN, so they compare execution formats, not learning methods. The benefit of the proposed skill-learning procedure over alternatives is therefore NEVER measured.
W2. Missing learned-cascade control and weak static-fusion baselines. Execution follows a satisfied edge or falls back to the next prescribed tool, and the Judge prefers "a definite terminal edge, then further evidence acquisition". The learned skill thus largely behaves as an early-exit cascade over tool outputs. The natural control is a learned cascade or tree model (e.g., GBDT) trained on the same data over all tool outputs, including map statistics. The LogReg and MLP baselines provided use only seven scalar scores and fall well below majority voting (67.1 and 72.7 vs. 81.3). LogReg reaches only 58.1 on Columbia, where TruFor alone reaches 94.8. This suggests these baselines are under-tuned, which weakens the claim that the gains "cannot be explained by prediction aggregation alone."
W3. The MLLM's contribution is unclear. With Qwen3.5-9B as executor, the same skill scores 66.8, below both majority voting (81.3) and TruFor alone (76.6). Much stronger executors add only 0.0–1.7 points over the default (Table B.2). The initial skill (68.3) also scores below its own scout tool, TruFor (76.6). The MLLM seems to matter mainly as a protocol follower. The paper does not analyze whether the 9B collapse reflects executor capacity or co-adaptation of the skill to the model it was learned with (Qwen3.6-35B-A3B). It also does not report a deterministic, MLLM-free execution of the learned graph, which would measure the MLLM's contribution directly.
W4. Possible metadata shortcuts. Fig. 5 shows a route conditioned on file format and resolution ("JPEG + size > 2600 → route R4"). In forensic and generated-image benchmarks, format, size, and compression are often dataset- or even class-specific (cf. the JPEG bias documented for GenImage by Grommelt et al., 2024). The paper does not normalize format or size. Neither the refine prompt nor the paired validation prevents such rules, and the full set of learned conditions and guards is not provided. ForensicCPN is also the only compared method that both adapts to labeled data and can encode explicit metadata rules, so I can't measure the possible impact of shortcuts.
W5. The two self-evolving modes are never evaluated in the same setting. Skill learning is evaluated only on traditional manipulations, and test-time evolving only on generative ones. Labeled generative data (SID-Set) was available but deliberately excluded from skill learning. As a result, label-free evolving is never compared with labeled skill learning on the same data.
W6. No exploration-only baseline. Fig. 4 compares the frozen CPN only against consolidated checkpoints. It does not report per-case exploration with the full toolbox without write-back. The value of consolidating revisions into the skill is therefore not isolated.
W7. Last but critical. Implausible values in Table 3. Codex and CPN report identical accuracies on five of seven datasets (92.0, 99.5, 93.0, 97.7, 77.5), even though their UNSURE rates (35.4% vs. 1.8%), round counts (18.9 vs. 8.2), and token counts (374k vs. 176k) differ sharply. Several values are also unattainable with the stated dataset sizes. For example, on the 200-image COVERAGE set accuracy must be a multiple of 0.5%, yet Table 3 reports 70.4% and 66.9%.
Minor (presentation).
- Figs. 1 and 2 list an "Unseen forgery" hypothesis that does not exist in the actual graph. Fig. 1 also omits the REAL branch and the paired-validation gate.
- Figs. 1 and 2 disagree on where hypothesis revision is triggered (evidence state vs. UNSURE), while the text defines UNSURE as terminal.
- More generally, Fig. 1's lower panel largely duplicates Fig. 2, with inconsistent details. Consider making Fig. 1 purely conceptual and keeping the method details in Fig. 2 only.
- The CPN formalism is largely nominal: there is a single token, execution is sequential, and there is no concurrency. In practice, the graph is a decision graph that carries state.
Questions for authors
- Under the same toolbox and labeled data, how does ForensicCPN compare with (a) existing tool-using forensic agents, (b) skill- or workflow-learning methods, and (c) a learned cascade or GBDT over all tool outputs, including map statistics?
- How does the learned graph perform when executed deterministically, without the MLLM (numerical conditions only)? Why does the 9B executor collapse (UNSURE rate, invalid calls)? Would a skill learned with the 9B model behave differently? In Table B.2, is the Judge also replaced?
- "JPEG + size > 2600" a hand-coded tool-eligibility guard or a learned rule? Please release all learned conditions and guards, report per-class format, size, and compression distributions for each dataset, and evaluate after re-encoding and resizing all images to a common format and size.
- For test-time evolving, please provide: (a) a comparison with labeled skill learning on the same generative data and with an exploration-only baseline; (b) Table 1 results after adaptation; (c) results on a held-out split of the target domain. (d) Was a single skill evolved across both datasets, or one per dataset?
- Please verify Table 3 (the identical Codex/CPN values, and the values that are unattainable on 200-image datasets) and the FakeVLM and GPT-5.6 baseline protocols. Are all results single runs, and how many images were evaluated per dataset?
- What is the variance across independent skill-learning runs, and what is the total cost (tokens and wall-clock time) of skill learning and of inference?
Response to Reviewer n46R
Thank you for taking time to view our response.
1. Format shortcuts. We share this valid concern and have accounted for it. We considered two options when designing the study: standardizing all images or retaining their native formats. While standardization could reduce potential format-related shortcuts, it would also change the forensic signals available to the system. For example, converting JPEG to PNG preserves decoded pixels but removes access to the original JPEG encoding parameters, which are useful for some forensic operators. Retaining native formats preserves these signals and better reflects practical scenarios, where images come from diverse sources and in different formats. We therefore chose to preserve the original formats while recognizing the need to examine potential shortcuts.
We first examined accuracy on 500 REAL images by format:
Notably, REAL PNG images are classified more accurately than REAL JPG images in both sets. A simple “PNG → FAKE” rule would instead misclassify every REAL PNG image. The difference may reflect image content or difficulty, but it does not follow the shortcut pattern.
We then converted every image in the AIGI-Bench held-out set and Columbia to PNG and reevaluated the same frozen skills:
After conversion, file format no longer distinguishes the labels, yet performance changes very little.
We also want to distinguish using image format to guide forensic analysis from using it as an authenticity shortcut. The agent can observe image formats, and we have seen format-aware tool proposals during skill learning and TTE exploration, such as using
jpeg_ghost_mapfor JPEG images. This is reasonable because different tools rely on different kinds of evidence. Importantly, we found no format-based routing conditions in the final learned skill, or conditions combining format with other evidence, or decisions directly associating format with FAKE or REAL. Instead, transitions depend on acquired evidence, its reliability, and its sufficiency for a verdict. Together with the results above, this suggests that the gains are not driven by treating image format as a proxy for authenticity.2. Codex vs. CPN. We checked the predictions image by image. Their binary decisions are identical on DSO-1, In-the-Wild, and Korus. They differ on only 11 CASIA v1 images and 2 Columbia images, while the reported accuracies still match after rounding. Thus, the repeated dataset-level numbers reflect highly similar predictions.
An important detail is that Codex does not learn this forensic skill independently: it executes a Markdown translation of the final skill learned by ForensicCPN. Similar predictions are therefore understandable.
The execution traces, however, differ substantially. On Columbia, for example, Codex produces 25 UNSURE outcomes versus 1 for CPN; our accuracy metric scores UNSURE by its binary leaning. In several traces, Codex continued to call low-level operators even after multiple expert tools provided consistent evidence, sometimes giving weak but explainable negative checks more weight than strongly supported expert prediction. Our harness instead constrains available actions and follows evidence-conditioned transitions. These results show that the learned forensic knowledge can transfer to another harness, while explicit execution control affects how efficiently and decisively it is used.
3. Forgetting after TTE. We agree that the existing source-domain labeled cohort could serve as a non-regression check, and we considered this option when designing TTE. We finally chose not to adopt it for two reasons. First, repeated paired validation would introduce considerable execution cost. Second, gains on generative attacks may not always preserve performance on conventional attacks. A strict source-domain validation gate could therefore reject useful adaptations to new attacks, creating a trade-off between knowledge retention and adaptation.
We are still interested in exploring ways to preserve acquired forensic knowledge. One possibility is to take advantage of the explicit, editable skill structure: routes can be adjusted directly with human or stronger-agent assistance, without retraining any underlying model.