- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the lack of explicit confidence in MLLM-based tampered text detection. On a Qwen3-VL-8B backbone, it proposes a three-stage confidence-calibrated training framework that connects self-assessment with forensic learning.
Critical strengths
S1: This paper targets a real gap in forensic MLLMs: they output verdicts but no confidence that could decide between automatic acceptance and manual review.
S2: This paper evaluates all models without OCR Rectification.
Critical weaknesses
W1: This paper does not show that "lucky guesses" occur, and Sec. 3.4 admits its remedy, , does not measure reasoning correctness. CGAM supervises class labels rather than correctness, with no theoretical justification. This paper never isolates each stage.
W2: This paper is incremental over TextShield-R1, reusing its rewards, data split and annotations. CGAM attends over only two tokens, and replacing with a learnable query changes ECE by only 0.7 (Table 4). This paper omits the closest recent baselines: MLLM text-tampering methods (TextSleuth, Xu et al. ACM MM 2025, DocShield) and calibration methods (VL-Calibration, temperature scaling). It reports only ECE, without Brier, AUROC or risk–coverage results.
Questions for authors
Q1: Table 3, row 1 has no <conf> supervision yet reports an ECE of 36.8. How is its confidence obtained?
Q2: Can the authors report SFT+RL without forensic pretraining, and the full model without CGAM? Also, why does SFT on 25% of the data (Table 3, 89.3 Cls) exceed full-data SFT (Table 7, 87.9)?
Q3: Under teacher forcing, is the <conf> label computed on the TFR-annotated <think> or on a sampled one? Without a format reward, how are malformed outputs handled, and what is the parse-failure rate?
4 comments
Response to Reviewer sc9t (Part 2)
Response to W2: Is the Method Merely an Incremental Improvement over TextShield-R1?
We thank the reviewer for the candid assessment of our novelty and experimental completeness.
(1) Methodological differences from TextShield-R1.
Reusing TFR's data split and forensic rewards is a controlled variable: it makes the way confidence enters the learning process the only variable that changes, rather than inheriting TextShield-R1's methodological setup.
The core innovation of this work relative to TextShield-R1 is:
-
Methodologically, we are the first to propose that a calibration signal is not merely a scalar aligned with accuracy; it can serve as a learning signal for the forensic optimization process. As Table 3 of the main text shows, simply adding a confidence output on top of a forensic model improves calibration at the expense of forensic performance (ECE decreases by 10.1%, while OCR/Loc./Res. drop by 2.3%/2.6%/1.0%). With CGAM, designed according to our methodology, the confidence semantic representation participates in learning the forensic representation as an aggregation condition, and this trade-off disappears (ECE decreases by 2.7%, while Cls./OCR/Loc./Res. improve by 1.2%/3.1%/2.9%/3.7%). Hence, our methodological innovation is not the addition of a confidence output, but making the confidence signal an effective signal for forensic optimization.
-
In terms of practical value, our method can be used to build automated business workflows: when the model is uncertain about a prediction, the sample is routed to manual review, reducing the risk of misjudgment. Beyond the ECE reported in the paper, we further analyzed the ECE of the auto-accepted region: on the four test datasets, we retain only samples with as the auto-accepted set (results that can be accepted directly in an automated workflow) and recompute ECE within this set using the same 10-bin binning scheme. The results show that this threshold automatically accepts 48.0% of the test samples, while the ECE of the auto-accepted region is only 7.31%, far below the full-set value of 21.4%. In other words, our method can cut the manual review workload by nearly half, with a low risk of misjudgment for auto-accepted samples—an advantage that rests precisely on accurate confidence outputs.
In summary, the proposed method is not incremental over TextShield-R1, but a complete innovation built on a new methodology that can be deployed in practice.
(2) CGAM uses only two tokens. We emphasize that this is a deliberate choice, not a capability limitation: is the initial reasoning state and the conclusion state, corresponding respectively to the starting point of analysis and the endpoint of the verdict—the two most representative states in the forensic decision chain. We do not claim that the 0.7 ECE difference in Table 4 of the paper is a significant improvement; more important are the simultaneous gains in the forensic metrics (+0.4/+1.9/+1.0/+1.3), which show that CGAM achieves joint forensic–calibration optimization rather than calibration alone.
(3) Missing the closest baselines. TextSleuth and DocShield are not yet open-sourced. Due to time constraints, we will report the replication forensic and calibration metrics in the final revised version.
(4) Reporting only ECE. We accept this comment and have added Brier Score and AUROC (Table 4 (W2-4) below), as well as Risk–Coverage (Table 5 (W2-5) below).
Table 4 (W2-4): Comparison of calibration baselines (ECE from Table 2 of the paper; all other results are new)
| Method | ECE↓ | Brier↓ | AUROC↑ |
|---|---|---|---|
| Baseline | 26.7 | 18.3 | 41.0 |
| RLCR | 23.7 | 16.1 | 47.8 |
| DCPO | 22.9 | 16.1 | 47.9 |
| Ours | 21.4 | 15.2 | 55.0 |
Table 5 (W2-5): Risk–Coverage of Selective Prediction (Risk↓ in %, average over four test sets)
| Coverage | Baseline | RLCR | DCPO | Ours |
|---|---|---|---|---|
| 100% | 10.41 | 9.31 | 9.76 | 9.07 |
| 90% | 10.88 | 9.43 | 10.04 | 9.25 |
| 80% | 11.63 | 9.81 | 10.55 | 9.35 |
| 70% | 12.63 | 10.09 | 10.60 | 8.81 |
| 60% | 13.83 | 10.74 | 10.71 | 8.48 |
| 50% | 12.18 | 10.34 | 10.23 | 7.55 |
Response to Reviewer sc9t (Part 3)
Response to Q1: How is the ECE of 36.8 in the First Row of the Paper's Table 3 Obtained Without <conf> Supervision?
We thank the reviewer for pointing out this implementation detail. We acknowledge that the original manuscript did not explain it sufficiently. The actual situation is: the confidence in the first row comes from the inference prompt's direct request to the model.
Specifically:
- All configurations in this paper share the same inference prompt (Figure 6 of the main text), which explicitly asks the model to give a confidence in the 0–1 range inside the
<conf>tag. Thus “no<conf>supervision” means that the confidence value is not used as a supervision target during training, not that no confidence is produced at inference time. - The model in the first row learns only the content and format of
<think>/<answer>during SFT; the<conf>field has no label constraint. The value it outputs is purely prompt-induced verbalized confidence, which simulates a forensic detector that has not undergone any calibration (confidence) training. This is precisely the starting point our work aims to improve.
Revision: Sec. 4.4 will add a sentence explaining the source of the confidence in the first row (a prompt-induced verbalized output).
Response to Q2: SFT+RL Without Pre-train, the Full Model Without CGAM, and Why 25% SFT Outperforms Full-Data SFT?
We thank the reviewer for raising these issues. We have added more complete ablation results in Sec. 4.4. The two newly added configurations are tabulated in Table 3 (W1-5) of our response to W1.
(1) SFT + RL without Pre-train. Added: starting from the original Qwen3-VL-8B, skipping the first-stage forensic continued pre-training, and directly performing the same SFT (25% TFR) and GRPO training (same rewards, same number of steps). Given the challenge of joint forensic–calibration optimization, the Pre-train stage is crucial for building the model's basic forensic capability.
(2) Full model without CGAM. Added: we keep Pre-train, SFT, and GRPO (including and ) and remove only CGAM and . The no-CGAM row gives 90.2/55.8/51.0/54.6 and ECE = 22.3 (full model: 91.0/56.8/51.7/57.4, ECE = 21.4); both the forensic and calibration metrics degrade to some extent.
(3) Why 25% SFT outperforms full-data SFT. The cause is overfitting: the 25% SFT baseline is trained for 1,212 optimizer updates, whereas a single epoch over the 100% pool already reaches 1,567, so full-data SFT keeps optimizing well past the 25% budget and degrades on held-out data. We re-ran Stage 2 from the same Pre-train weight on the 100% pool with the identical recipe (CGAM removed) and evaluated checkpoints on the fixed 1,000-sample subset (Table 6 (Q2) below).
Table 6 (Q2): Full-data SFT as a function of optimizer updates (fixed 1,000-sample subset; four-test-set macro)
| Configuration | SFT data | Updates | Cls.↑ | OCR↑ | Loc.↑ | Res.↑ | ECE↓ | Brier↓ |
|---|---|---|---|---|---|---|---|---|
25% SFT + <conf>, two epochs | 25% | 1,212 | 90.10 | 51.41 | 46.99 | 52.00 | 26.58 | 0.1798 |
100% SFT + <conf>, matched updates | 100% | 1,200 | 90.20 | 53.80 | 48.96 | 53.52 | 24.77 | 0.1691 |
100% SFT + <conf>, mid-training | 100% | 1,500 | 88.30 | 52.70 | 47.42 | 50.51 | 26.90 | 0.1888 |
100% SFT + <conf>, one epoch | 100% | 1,567 | 87.50 | 50.86 | 46.33 | 50.10 | 26.67 | 0.1933 |
Two observations confirm this:
- At a matched update budget, more data is harmless (even slightly better). At ≈1,200 updates the 100% run beats 25% (Cls. 90.20 vs. 90.10, ECE 24.77 vs. 26.58); the two splits also share the same source subsets and class proportions, so the extra examples add few new patterns.
- The degradation comes from the extra updates. From step 1,200 to one epoch, held-out Cls. falls 90.20 → 87.50 and ECE rises 24.77 → 26.67 while training loss keeps decreasing (0.0490 → 0.0478)—a clear train/test divergence. The paper's 2-epoch 100% SFT is the same over-optimized regime, which our full four-test-set run reproduces (87.56/52.99/49.35/49.85, ECE 26.83 vs. paper's Table 7 row 2 87.9/53.2/48.9/50.3, ECE 26.3).
Response to Reviewer sc9t (Part 4)
Response to Q3: How Are <conf> Labels Generated Under Teacher Forcing, and How Are Format Errors Handled?
We thank the reviewer for raising this reproducibility issue.
(1) How <conf> labels are generated:
At each training step of Stage 2, the <conf> label is produced by a gradient-free forward pass with the current parameters over the entire TFR-annotated teacher-forced <think>…<answer>… sequence. We take the mean per-token cross-entropy over the annotated <think> span to obtain , then insert the tokens of <conf>c</conf> before the eos of that sequence and write them into the labels.
(2) How the format is enforced: the implementation includes a separate Format reward. The Format reward uses a regular expression to check whether the output strictly matches <think>…</think><answer>…</answer><conf>…</conf>, assigning 1 if valid and 0 otherwise. The revised manuscript will add a note on this implementation detail below Eq. (8); we thank the reviewer for pointing it out.
(3) Parse-failure rate: it has been measured and reported, and failed samples are not discarded. Table 7 (Q3) reports the format-error rates at each training stage (missing <think>/<answer>, missing or unparseable <conf>, and the overall failure rate), covering the post-SFT and post-GRPO checkpoints, with the Baseline as a reference.
Supporting Material
Table 7 (Q3): Parse-failure rates and two evaluation protocols
| Checkpoint | Missing <think> | Missing <answer> | Missing or Unparseable <conf> | Overall Parse-Failure Rate |
|---|---|---|---|---|
Baseline (no <conf> supervision) | 0.0076% | 1.1706% | 15.5643% | 16.7273% |
| After SFT | 0.0025% | 0.0025% | 0.0025% | 0.0025% |
| After GRPO (full model) | 0.0051% | 0.0051% | 0.0051% | 0.0051% |
Response to Reviewer sc9t (Part 1)
Response to W1: Lucky Guesses, CGAM Label Design, and Insufficient Isolation of Training Stages
We thank the reviewer for this critical comment.
(1) Lucky guesses do occur. Concretely, on the four test sets 7,613 of Stage-2's 19,869 tampered/generated predictions are "correct-but-flawed" predictions—the image is classified correctly, yet its forensic analysis is wrong in at least one respect (localization error, OCR error, or an evidence-inconsistent explanation). This amounts to 38.3% of all tampered/generated samples, confirming that lucky guesses do occur.
(2) The scope of Ethink. Ethink is an auxiliary signal about the certainty of the model's own reasoning trace. Through the CGAM-guided aggregation in Stage 2, optimizing it via hconf endows it with the ability to express the correctness of the forensic analysis process; in Stage 3 it is fused with Across and Acur to capture process and outcome correctness, respectively.
<think>sequence (Pearson ρ). As Ethink rises, ρ grows monotonically from 0.310 to 0.508 (+0.198) while accuracy climbs from 88.2% to 98.0%, indicating that Ethink tracks agreement with a correct analysis.(3) CGAM's supervision target and its design rationale. hconf needs to express the process correctness of the forensic analysis, whereas hts and hans need to express image authenticity; correctness itself carries no information about image authenticity and is therefore unsuitable as a supervision label. The image class label both expresses authenticity and is a necessary condition for verifying process correctness, so we use class labels as the auxiliary supervision for CGAM.
(4) Isolation of training stages. SFT and GRPO isolation is covered by Table 3 of the main text, and the Pre-train-only model is reported in Table 7 of the appendix. We also report the SFT+GRPO-only configuration to show the role of the Pre-train stage more completely, consolidating all results in Table 3 (W1-5) and Sec. 4.4 of the revised manuscript.
We will include the above experiments and analyses in the revised manuscript.
Table 2 (W1-2): ρ and classification accuracy across Ethink quintiles
Table 3 (W1-5): Stage isolation (average over four test sets)
<conf>(Stage2)<conf><conf>+ CGAM<conf>+ CGAM + Rfor<conf>+ CGAM + Rfor + Rconf (new)<conf>+ Rfor + Rconf (new)<conf>+ CGAM + Rfor + Rconf