- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MMSec-R1, a framework for Fake Image Reasoning (FIR) that jointly addresses image authenticity classification and manipulation localization. The authors first construct a small cold-start dataset of synthetic forensic chain-of-thought (CoT) demonstrations. They then apply reinforcement fine-tuning with DAPO, using a format reward and task-specific verifiable rewards, including a proposed Soft IoU score for bounding-box localization. A final DPO stage uses automatically judged preference pairs to improve consistency between generated rationales and final predictions. The approach is evaluated on in-domain and out-of-domain OpenMMSec splits and additional synthetic-image benchmarks. MMSec-R1-8B reports 90.34% average in-domain classification accuracy and 46.78 Soft IoU, together with 82.20% accuracy and 39.80 Soft IoU on the out-of-domain split.
Critical strengths
- Strong empirical performance on a broad forensic evaluation. The proposed training pipeline yields substantial improvements over the Qwen3-VL backbone on both classification and localization. 2.A useful empirical study of reinforcement fine-tuning components. The comparison among SFT-only, SFT+GRPO, SFT+DAPO, and the final DPO-aligned model helps identify where the reported gains arise.
Critical weaknesses
-
Limited methodological novelty and an insufficiently supported emphasis on test-time scaling. Using ground-truth bounding boxes as outcome rewards for multimodal reinforcement learning is not, by itself, a novel idea, either for spatial grounding more broadly or for forgery localization. The paper needs a clearer differentiation from prior localization-rewarded visual reasoning and forensic methods. DAPO and DPO are also existing optimization methods rather than new contributions of this work. In Table 4, SFT+GRPO already attains 85.80% ACC / 42.00 Soft IoU, while replacing GRPO with DAPO yields 88.87% ACC / 43.67 Soft IoU: an incremental gain of 3.07 accuracy percentage points and 1.67 Soft IoU points. Although useful, this does not establish a fundamentally new training method. Moreover, the title makes test-time scaling the principal contribution, but Table 3 shows a relatively weak compute–performance trade-off. For MMSec-R1-8B, increasing the sampling budget from N=1 to N=8 raises latency from 1.05 s to 7.75 s (approximately 7.38×), while ACC increases only from 90.34% to 91.83% (+1.49 percentage points) and Soft IoU from 46.78 to 48.20 (+1.42 points). The paper does not sufficiently explain why this should be considered an important test-time scaling result rather than a modest benefit of repeated sampling. I suggest reframing the title and central contribution around localization-rewarded reinforcement fine-tuning unless much stronger compute–performance evidence can be supplied.
-
The localization reward and generated CoT do not establish faithful visual reasoning; the reward formula itself requires correction or clarification. In DAPO, the reward primarily checks whether the predicted bounding boxes match the ground truth. A high localization reward does not verify whether the intermediate CoT correctly identifies the actual image artifacts, or whether the claimed forensic evidence is present at all. The later DPO stage targets logical consistency between textual reasoning and the final prediction, but such consistency is not equivalent to visual faithfulness. Consequently, the improved classification and grounding results do not establish the stronger claim that the model has learned reliable, causally grounded visual forensic reasoning. Eq. 3 appears unbounded as written because the two best-match IoU terms are summed, without normalizing by the numbers of ground-truth and predicted boxes. Let A = 1 denote the sum of best predicted-box IoUs for a single ground-truth box and B = 2 the sum of best ground-truth IoUs for two duplicate, perfectly overlapping predicted boxes. Then the stated expression yields Soft IoU = 2AB/(A+B+epsilon) ≈ 4/3 > 1. This raises concerns about duplicate predictions, false-positive penalties, reward hacking, and whether the implementation matches the formula. Since this is a core training reward and evaluation measure, it must be resolved. The cold-start CoT generation process explicitly provides the teacher model with ground-truth authenticity labels and, for locally manipulated images, ground-truth boxes (Section 3.1 and Appendix F). The teacher is then prompted to produce explanations invoking natural textures, lighting inconsistencies, and boundary artifacts. Such subtle forensic cues may not be reliably observable by a general-purpose VLM, especially at limited image resolution. The procedure could therefore produce post-hoc, label-conditioned rationalizations rather than evidence-based observations. The manuscript needs a more rigorous audit of teacher-generated explanations, and ideally should restrict them to visually verifiable cue types or report annotation/verification quality. Rule-based removal of conversational artifacts does not address this substantive hallucination risk.
Questions for authors
1.Prior work and novelty: Which specific aspect of the proposed bounding-box-rewarded RL formulation is new relative to existing spatial-grounding and image-forensic reinforcement learning approaches? Can the authors demonstrate a contribution beyond applying established DAPO/DPO recipes to an existing localization reward? 2.Soft IoU correctness: Is Eq. 3 identical to the metric and reward used in training and evaluation? How are multiple predicted boxes normalized, deduplicated, or penalized? 3.Test-time scaling protocol: How are the N independently sampled outputs in Table 3 combined or selected at inference time, when ground-truth labels and bounding boxes are unavailable? Can the authors report the trade-off against a fixed-compute baseline and disentangle gains from repeated sampling versus longer individual reasoning traces?
Rebuttal Summary for submission 43948
Dear Acs, We thank Reviewers SE5m, Ntdb, and Y4uq for their constructive feedback. We summarize the main additions and the remaining disagreement below.
Forensic supervision (Ntdb, Y4uq). We clarified why fully synthesized images (AIGC) receive image-level supervision and locally manipulated images (IMDL) receive box-level supervision. Adding pseudo-box supervision for AIGC increased false alarms and reduced localization accuracy on tampering tasks. Verifying individual local cues in fully synthesized images remains an open question.
Reasoning and localization (SE5m, Ntdb, Y4uq). We clarified Soft IoU normalization and duplicate-box handling. In a new experiment with the model held fixed, shifting the region described in the rationale shifted the predicted box toward that region. Reviewer Ntdb maintained their acceptance score (4) and raised their confidence to 4 (3->4).
Reporting corrections (SE5m). We clarified the training sequence and decoding settings and identified corrections to Tables 3 and 8. We also acknowledged that the reported human-audit percentages are inconsistent with the stated 300-sample protocol.
Contribution of TTS (SE5m, Y4uq). Both reviewers questioned whether TTS accounts for enough of the overall improvement to warrant its prominence. Our evidence for its incremental contribution comes from holding the RL-trained 8B model fixed. Increasing the sample count from one to eight improved accuracy by 1.49 points and Soft IoU by 1.42 points on clean inputs at roughly 7× the latency. On low-resolution inputs, the gains reached 3.34 and 4.25 points, respectively. We believe that identifying when additional inference compute benefits an RL-trained forensic model is a major contribution of this work. We respectfully leave its significance to the Area Chair and Program Chairs.