acceptodds
Under review as a conference paper at ICLR 2027

Rationale Content Shapes What a Detector Learns: Controlled Supervision for Person-Centric AI-Image Forensics

Abstract

Detectors trained to generate rationales alongside authenticity decisions may learn different cues depending on the evidence they are taught to explain. This study focuses on consistency between gaze and attended targets in person-centric AI-generated images and analyzes how incorporating such relational gaze information into rationales changes detection performance and output behavior. We conduct experiments with Custom Gaze, 23,415 identity-preserving real/fake pairs that differ only in a generated eye band, and find that gaze-centered fine-tuning outperforms FakeVLM origin across three person-centric evaluation sets. The detector also maintains high balanced accuracy on eye edits from three held-out inpainters, and the largest point-estimate gain under person-count stratification appears in the stratum with two or more detected faces. It achieves the strongest aggregate metrics among the representative detectors compared, while paired-data training produces the same directional effect on the vision-only Effort detector. Finally, with the data, five-block structure, optimization, and checkpoint rule fixed, gaze-centered rationales outperform anatomy-centered and context-centered alternatives on COCOAI Person by 4.2–4.6 and 2.1–2.4 balanced-accuracy points, respectively. These results show that the evidence emphasized in a rationale is a substantive training factor that changes detector behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI-generated image detection, relational gaze supervision, social gaze consistency, person-centric image forensics, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q6qvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether rationale semantics affect AI-generated image detection beyond rationale structure. The authors construct the Custom Gaze benchmark and compare gaze-, anatomy-, and context-focused rationales with decision-only supervision. Experiments on Custom Gaze, COCOAI, held-out inpainting editors, and a second detector architecture show that gaze-focused supervision improves detection performance, particularly on Custom Gaze. However, it remains unclear whether these gains reflect semantic learning or sensitivity to synthetic eye artifacts.

Critical strengths
  1. Comparing different semantic content under a fixed rationale structure directly addresses whether rationale content matters.
  2. Tests across COCOAI, multiple generation methods, and a second architecture provide evidence beyond the custom benchmark.
Critical weaknesses
  1. Because all held-out inpainting examples still contain generated eyes, it is unclear whether the detector learns meaningful gaze-related cues or simply picks up on inpainting artifacts.
  2. Although fine-tuning changes the model's verbalized rationales, it is unclear whether these explanations reflect the actual basis for its predictions. The improvements on external datasets are also relatively modest.
Questions for authors
  1. Were the training data, optimization settings, checkpoint selection, and random seeds kept consistent across rationale conditions so that rationale content was the main difference?
  2. How do the authors distinguish learning meaningful gaze-related cues from picking up on artifacts introduced by eye inpainting?
  3. Do the reported confidence intervals account for both variation across training runs and uncertainty from test-set sampling?
  4. What evidence supports the interpretation that gaze-related output cards reflect the model's actual decision process rather than simply correlated verbalizations?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer q6qv

We have measured both of your weaknesses (W1, W2), and the answers are now part of the paper's result: the detector has no gaze-direction shortcut, and its verbalized evidence is not the basis of its decision. Tables R1 and R2 are in the General Response.

  • Q1. Were data, optimization, checkpoint selection, and seeds identical across conditions? Yes. Images, labels, mixture ratio, LoRA, optimizer, schedule, and evaluation step are identical, and only Blocks 2–5 of the caption differ (App. B.1). The new runs use the same three training seeds (101, 202, 303) for every variant.
  • W1, Q2. Gaze cues or inpainting artifacts? Evidence carried by the generated eyes: we find no evidence that gaze semantics contribute, and no claim of the paper depends on it. Four results answer this from different sides. (1) Supervision: a local eye-artifact rationale gives the same gain as the relational-gaze rationale (gaze − local = +0.14 [−0.15, +0.43], Table R1), and decision-only supervision already separates the pairs (99.97 on Custom Gaze). (2) No gaze-direction rule: authentic non-mutual-gaze images are not called fake more often than matched mutual-gaze images (7.8% vs. 10.4%). (3) Occlusion: with two or more faces, hiding the eye band changes gaze-card predictions no more than hiding a placebo band (+0.1 [−0.7, +0.9]). (4) The evidence is not a trivial artifact: it is not processing (0.1–0.4%, App. E.3), not FLUX-specific (93.0–95.3 BA on held-out inpainters vs. 68.3–73.8 for origin; as you note, these still contain generated eyes), and fake recall also rises on five full-image generators that involve no eye inpainting (Fig. 4(a)). The paper therefore claims the effect of describing the edit on the detector's decision (Table R1), not that the detector judges gaze. A direct test of gaze geometry needs a gaze-conditioned editor, since the gaze-preserving set we built did not manipulate gaze (recall 9.0% under both prompts, App. E.3); §5 states this open point.
  • Q3. Do the intervals include training-run variation and test-set sampling? Yes. We have run three independent training runs per variant, and the intervals in Table R1 combine training-run spread and test-image bootstrap: gaze − decision-only +0.70 [+0.38, +1.01] and local − decision-only +0.55 [+0.20, +0.91], positive in all three seeds.
  • W2, Q4. Do gaze cards reflect the decision process? No, and the paper now says so. We have added an intervention: occluding the eye band changes gaze-card predictions by +0.1 [−0.7, +0.9] more than occluding a placebo band when two or more faces are present (Table R2), and the multi-person premise appears on 11.9–15.1% of images with at most one face. We have rewritten L416–417 to state that the card is a verbalization measure.
  • W2 (second part). Modest gains on external sets. We have reported the effect at its measured size. The paired data add +0.6 BA on COCOAI Person for FakeVLM and +4.6 for Effort (Table 4), and rationale supervision adds +0.55 to +0.70 to the decision on the same data, three to four times the training-seed sd (0.17) and positive in all three seeds.
  • Presentation (rated 2). We have added to §4 a pointer to a one-table guide to what each experiment varies and holds fixed (Table E.1).

If a specific point remains open for your assessment, please let us know and we will address it.

Thanks for the response. Almost all of my concerns are addressed. Considering the novelty of this work, I would like to maintain my score.

AuthorsOpenReview ↗

Thank you; what is new, for the record

Thank you. We are glad that almost all of your concerns are addressed.

Since novelty was not among the points of the review, we state briefly what is new, for the record.

  1. A content-controlled design: images, labels, rationale structure, and training recipe are fixed and only the rationale content varies, now with three training seeds (your first strength).
  2. The result that rationale supervision moves three quantities differently: the decision improves (+0.55 to +0.70 BA in all three seeds), the emitted answer becomes less reliable, and the explanation is not the basis of the decision. We are not aware of a prior measurement of these three for explanation-generating detectors.
  3. Custom Gaze: 23,415 identity-preserving real/fake pairs that differ only inside the eye band.

If a specific prior work already covers these, we would be grateful for the reference.

Reviewer_2ZkPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the content of training rationales changes what a vision-language fake-image detector learns. The authors build Custom Gaze, 23,415 real/fake pairs from the mutual-gaze-positive part of OI Mutual Gaze, where the fake member has one participant's eye band regenerated by FLUX.1-Fill with the prompt "eyes looking to the side, avoiding eye contact". Each sample is paired with a templated five-block rationale about relational gaze. FakeVLM is LoRA-fine-tuned on Custom Gaze mixed with FakeClue and evaluated on the Custom Gaze test split, COCOAI Person and COCOAI Interaction. The fine-tuned model improves over the original FakeVLM, transfers to held-out inpainters, and, in a content-controlled ablation, gaze-centered rationales outperform anatomy- and context-centered rationales with the same five-block structure.

Critical strengths
  • The research question and motivation is interesting, and the fixed-structure content replacement is a reasonable experimental design.
  • The paired construction (same identity, scene and lighting, atomic splits) and the mask audit are careful.
  • The statistical reporting is more thorough than usual for this area (paired bootstrap, permutation tests, McNemar), and the limitations are stated honestly.
Critical weaknesses
  • The rationales carry no image-specific evidence, so the study does not test "evidence named in an explanation".

    • The decision sentence is label-deterministic, and the evidence and conclusion blocks are drawn from label-specific pools (L184-186). Every fake receives "implausible / inconsistent gaze" and every real receives "plausible / consistent gaze". The rationale is therefore a paraphrased function of the label, not a description of what is in each image.
    • The Scene block is meant to "establish when relational gaze reasoning applies" (L167-170), but all Custom Gaze samples come from mutual-gaze pairs with at least two people. The condition is never false during training, so the model cannot learn the claimed conditional behavior from this supervision.
    • The title and conclusion (L460-461) should be scoped accordingly: what is varied is the vocabulary of a label template, not the evidence a detector is taught to look for.
  • The content-controlled ablation compares a description that matches the manipulation with descriptions that do not.

    • The fake images are created by prompting the inpainter to make the eyes "look to the side, avoiding eye contact" (L152). The gaze rationale describes exactly this manipulation, whereas the context rationale (object layout, lighting, global coherence; L700-701) describes properties that the eye-band edit does not change.
    • Consistent with this, both alternatives perform below decision-only supervision on all three sets (Table 5: COCOAI Person 80.0 / 82.1 vs 83.2; Interaction 65.5 / 66.5 vs 68.8; Custom Gaze 98.3 / 98.1 vs 99.8). The headline 4.2–4.6 and 2.1–2.4 point gaps (abstract, L381) are therefore mostly the cost of a mismatched rationale. The gain of gaze over decision-only is 1.1 points on COCOAI Person.
    • The decisive control is missing: a rationale that is equally faithful to the edit but local rather than relational (e.g. iris texture, pupil shape, specular highlights, the cues of the prior ocular-forensics works cited in L90-92). Without it, "relational gaze" cannot be separated from "describing the eye region correctly".
  • Gaze semantics are not separated from generated-eye artifacts, and a gaze shortcut is not ruled out.

    • All fakes contain regenerated eyes with averted gaze, and all reals have mutual gaze. The detector can succeed either by detecting inpainting traces or by learning "averted gaze → fake". The authors acknowledge the first issue (L441-442), but it is central to the paper's claim.
    • The second is a deployment risk: real people frequently look away from each other. OI Mutual Gaze also contains negative (non-mutual) real images; evaluating real recall on them, and on fakes regenerated with gaze preserved (e.g. prompting "looking at the other person"), would directly test both explanations. The mirror control (E.3) does not do this, since it changes gaze by only 4.7° on median (L934-936).
    • Fig. 4(a) reports fake-recall gains of 4.4–9.6 pp on every COCOAI generator, while BA improves by only 1.2 pp (Table 3). This implies that real recall on COCOAI Person decreases by roughly 4 pp, which is consistent with a more fake-biased detector and contradicts L251-252 ("less class-asymmetric"). Real recall should be reported explicitly.
  • The reported variance does not reflect training variance.

    • The "three seeds" are decoding seeds applied to one frozen checkpoint per variant (L951). Each variant in Table 5 is trained once and selected at its own BA-best step (App. D.1). The ±0.1–0.3 standard deviations therefore measure sampling noise, not the training-run variability needed to support differences of ~1 BA point.
    • BA/F1/MCC are computed on each model's parseable subset (L728-730), so models with different parse failure rates are evaluated on different images. Parse failure rates are not reported. E.3 suggests they are not negligible: 2,201–2,212 real calls plus 3–9 fake calls (L930-931) from a pool of 2,324 real references (L948-949) would leave ~100 unparsed outputs. Note also that E.2 counts unknown predictions as incorrect (L909), a different rule from the main results.
    • COCOAI Interaction has 33 real images (L445); one real image changes real recall by 3 pp. The face-count stratification is also not significant across strata (overlapping intervals, L443-444), yet the abstract highlights it (L21-22).
  • Limited scope and comparisons.

    • All rationale experiments use a single detector family (FakeVLM). The general claim in the title would require at least one other rationale-generating detector (e.g. SIDA).
    • External baselines use released checkpoints without Custom Gaze training and a fixed 0.5 threshold (Table B.2), so the "strongest aggregate metrics" claim (L331) is largely expected. The LOKI comparison (L336-337) compares the authors' evaluation with a published number rather than re-evaluating FakeVLM origin under the same protocol.
    • Related work on learning from explanations and rationale supervision (e.g. "right for the right reasons", explanation-supervised training) is not discussed, although it is directly relevant to the central question.

Some minor issues:

  • Several numbers and statements are inconsistent:
    • L295-296 "ranging from 4.4 pp on SD2.1 to 9.6 pp on DALL-E 3" → Fig. 4(a) shows +9.6 pp on SD2.1 and +4.4 pp on DALL-E 3.
    • Table D.1 reports a COCOAI Person fake recall of 0.881 for the step-1,650 gaze checkpoint, whereas Fig. 4(a) shows fake recall of roughly 92–95% on every generator for the same checkpoint (Table B.2).
    • Table 1 lists the FakeVLM training data as "Custom Gaze 31% + FakeClue 69%; 151,173 images total". 31% of 151,173 ≈ 46,860, which matches the full Custom Gaze (46,830, including validation and test) rather than the 37,464-image training split given for Effort in the same table. Table B.1 (7,558 steps, batch 32, 2 epochs) implies ~120.9K images per epoch, i.e. 80% of the pool. Please state exactly which images were used for training and confirm that no Custom Gaze validation/test pairs were included.
    • E.2 states that the smaller gain in profile scenes reflects reduced eye visibility (L910-912), but the gain is also smaller for frontal faces (+0.8) than for semi-profile faces (+1.6) on COCOAI Person (Table E.2), which is not what a gaze-based mechanism would predict.
  • Table 7 reports "approx." BA values and "Variable" sample sizes. Exact numbers should be given.
  • The output audit (Table 6) counts keyword co-occurrence. Since the training captions contain these keywords by construction, the rate mainly measures how often the model reproduces the Custom Gaze template, not relational reasoning.
  • The authors should tone down the language:
    • L384-385 "This ablation directly supports the study's central hypothesis that the evidence named in an explanation helps shape the detector's decision criterion" → see the comments on label-deterministic templates and mismatched alternatives above.
    • L416-417 "the gaze card marks not only the surface reproduction of learned vocabulary but also a decision mode associated with relational scene reasoning" → this is a correlational observation on a keyword detector.
    • L823-824 "This result identifies the content of the rationale as the primary explanatory factor" → only one training run per variant.
  • Reference formatting is inconsistent (e.g. "K. H. Brodersen et al.", "B. Doosti et al." vs full author lists).
  • The dataset redistributes manipulated images of identifiable real people with altered gaze/interaction. The license restrictions are noted (L635-637), but the paper should discuss consent and misuse more explicitly.
Questions for authors
  • Each main argument above is followed by supporting evidence; please address each main argument and the evidence behind it.
  • Please add (i) a local eye-artifact rationale with the same five-block structure, (ii) real recall on non-mutual-gaze real images from OI Mutual Gaze and on gaze-preserving regenerated fakes, and (iii) results over multiple training seeds for Table 5.
  • Please report real recall and parse failure rates for all models, and clarify the exact training set composition of Table 1.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 2ZkP (1/2)

We have run the three experiments you specified, and the decisive control confirms your reading: relational and local rationales are indistinguishable, and both raise the detector's decision over decision-only. We follow your five arguments in order. Tables R1 and R2 are in the General Response.

Argument 1. Label-deterministic rationales; Scene condition never false; scope.

We have adopted this reading throughout the paper. We have rewritten §3.2 and §5 to describe the variants as label-conditioned rationale templates, and to state that rationale content here means the content of such templates, not image-specific evidence. We have scoped the conclusion (§6) to label-conditioned rationale templates and one detector family, and the camera-ready subtitle names this scope (General Response). We have also measured what the model does when the Scene premise is false at test time, since it is never false inside Custom Gaze: the multi-person phrase appears on 12.6–15.9% of zero-face and 23.6–26.3% of 2+-face COCOAI Person images, and on 11.9–15.1% of images with at most one detected face (Table R2). The premise is partly scene-conditioned but not reliable, and we have rewritten L167–170 to say so.

Argument 2. Matched vs. mismatched descriptions; the missing local control.

We have added the local control to Table 5, and it confirms your reading: gaze − local = +0.14 [−0.15, +0.43]. Both variants exceed decision-only in all three training seeds: gaze by +0.55 / +0.92 / +0.62 and local by +0.27 / +0.67 / +0.72 (Table R1). Blocks 2–5 of the local variant describe iris texture, pupil shape, specular highlights, and the eyelid–sclera boundary in the vocabulary of L90–92, with the same five-block structure, pool size (1,250 captions), images, recipe, and training seeds as the gaze variant (construction in App. B.1). We have stated every contrast relative to decision-only, and that the gain comes from describing the edit, whether relationally or locally, which is the formulation you proposed. Table 5 reports both read-outs. Under free generation gaze is statistically indistinguishable from decision-only (−0.50 [−1.02, +0.02]); the lower point estimate is the 0.65 BA formatting cost of unparseable five-sentence answers (Argument 4), a known cost of generating explanations (Camburu et al. 2018), and the read-out is 7 times more stable across seeds (sd 0.06 vs. 0.41). Anatomy and context were single runs and are reported as such in App. D; the claims now rest on the three-seed runs, and we have replaced the 4.2–4.6 / 2.1–2.4 headline (L027–028, L381) with Table R1.

Argument 3. Artifacts vs. gaze semantics; averted-gaze shortcut; real recall.

The detector reads evidence carried by the generated eyes, not a rule about where people look (7.8% vs. 10.4%), and the real-recall change comes with the paired data.

Real recall. It is 78.32 for FakeVLM origin and 75.00 for the gaze checkpoint on COCOAI Person, and we have removed L251–252. Decision-only has 74.08, so the drop comes with the paired data and not with the rationale. We have added real and fake recall for every model and set (App. B).

Shortcut. Authentic averted gaze is not called fake more often than authentic mutual gaze: the fake-call rate is 7.8% on 500 non-mutual-gaze images from OI Mutual Gaze, against 10.4% on 500 mutual-gaze images matched on face size, separation, and aspect ratio (|SMD| ≤ 0.08).

Gaze-preserving counterfactual. We built the set you proposed and report its recall in full: 9.0%, unchanged by the prompt, by the supervision, and by whether preserved gaze was verified. We regenerated 200 mutual-gaze sources under a gaze-preserving prompt and, with the same source, mask, and seed, under a gaze-averting prompt, as a manipulation check. (1) Prompt: 9.0% under both. (2) Supervision: decision-only gives 7.0 / 8.0%, and origin, whose fake recall on Custom Gaze is 72.2% without having seen it (App. B), falls to 14.0 / 14.0%. A model with no Custom Gaze training cannot have learned a gaze rule from it, so the drop follows the regeneration procedure (face-crop regeneration, guidance 30 vs. 25), not our rationale or training. (3) Verified gaze: on the 86 images verified as gaze-preserving, recall is 11.6%, not below the set-wide rate. The set does not contain the contrast you asked for: a condition-blind check found no pair in which the two prompts produced opposite gaze. We therefore attribute nothing to gaze, and App. E.3 states that our detection results hold for the procedures evaluated (Custom Gaze and three held-out inpainters). Separating gaze geometry from generated-eye texture requires a gaze-conditioned editor (§5). We have relabelled the mirror control (median 4.7°, L934–936) a processing control, as you note.

AuthorsOpenReview ↗

Reply to Reviewer 2ZkP (2/2)

Argument 4. Training variance; parseable subsets; parse rates; small strata.

All intervals now include training-run variance, and parse-failure rates and single-rule metrics are reported for every model and set.

Variance. We have run three independent fine-tuning runs per variant; the intervals in Table R1 combine training-run spread and test-image bootstrap, and the effect is three to four times the training-seed sd of decision-only (0.17). The runs use a 2,200-step schedule, past the steps at which the submitted runs peaked (1,500–1,650), and one fixed evaluation step, which removes the per-variant checkpoint selection you noted in D.1. We have relabelled the ±0.1–0.3 of the submission as decoding-seed sds (L951).

Parsing. We have added parse-failure rates and every metric under one rule (unparsed = incorrect, the rule McNemar and E.2 already used) for every model and set (App. B), and Table 5 uses this rule. Your estimate is reproduced: 103–120 unparsed outputs of 2,324 in E.3. On Person the three gaze models leave 75 of 15,720 outputs unparsed on average: 23.3 of 2,620 reals (0.89%) and 52.0 of 13,100 fakes (0.40%), against 0.01% for decision-only. This costs (0.89 + 0.40)/2 ≈ 0.65 BA, which offsets the +0.70 decision gain; the read-out does not carry this cost. On Custom Gaze the unparsed rate is 0.01 / 0.84 / 3.54% for decision-only / local / gaze, and 4–5% for gaze on the inpainter pools. Under the single rule the gaze checkpoint scores 99.17 on Custom Gaze and 93.0–95.3 on the pools (origin 85.85 and 68.3–73.8). We have corrected L297–298 to these values. Small strata. We have labelled COCOAI Interaction a supplementary view (33 reals), and the abstract now says that stratum-wise intervals overlap.

Argument 5. Single family; external baselines; LOKI; related work.

We have limited the scope to one detector family, re-evaluated LOKI under our protocol (84.39 vs. 84.26), and added the explanation-supervision literature. §1 and §6 now say "a rationale-generating detector (FakeVLM)"; a second family such as SIDA is the first follow-up. We have limited L331 to released checkpoints evaluated at their default threshold without Custom Gaze training, with Effort origin vs. Effort + Custom Gaze (Table 4) as the matched-data comparison. The LOKI paired difference is −0.14 [−1.22, +0.90]. We have added to §2 a paragraph on explanation supervision (Zaidan 2007; Ross 2017; Camburu 2018; Rajani 2019; Narang 2020; Wiegreffe & Marasović 2021; Hase & Bansal 2022).

Minor issues. All corrected.

ItemRevision made
L295–296 vs. Fig. 4(a)Corrected to +9.6 pp on SD2.1 and +4.4 pp on DALL-E 3.
Table D.1 (0.881) vs. Fig. 4(a)Corrected to 0.936 on all 13,100 Person fakes, as in Fig. 4(a); 0.881 came from a separate 999-image fake-only subset.
Table 1 compositionCorrected to the training set: Custom Gaze train split 37,464 + FakeClue 83,474 = 120,938. The training manifest of the submitted checkpoint is no longer available, so instead of asserting its split we have re-trained on a pair-disjoint split (zero overlap with the 2,342 test pairs). The clean models reach the submitted level on the Custom Gaze test set: decision-only 99.97, local 98.92, and gaze 95.55 with its 3.54% unparsed outputs counted as errors (submitted: decision-only 99.91, gaze 99.17). The comparison of Table 5 rests only on the clean runs. COCOAI and LOKI are separate datasets and do not depend on this split.
E.2 frontal vs. semi-profileL910–912 now reports the ordering without a mechanistic explanation.
Table 7Exact n and BA per seed added.
Table 6 and L416–417Relabelled a template-reproduction measure; occlusion test added (eye minus placebo +0.1 [−0.7, +0.9] with 2+ faces, Table R2); L416–417 now calls the card a verbalization measure.
L384–385, L823–824Rewritten to the Table R1 result.
ReferencesFull author lists.
Consent and misuseEthics Statement extended (licences, prohibited uses, release terms).
Presentation§4 now points to a one-table guide to what each experiment varies and holds fixed (Table E.1), and §4.3 reports every contrast against one reference point (decision-only).

Your three requests. (i) Local eye-artifact rationale with the same structure: Table R1. (ii) Non-mutual-gaze reals: Table R2, row 1; gaze-preserving fakes: Argument 3. (iii) Training seeds: three each for decision-only, gaze, and local (nine runs), on which every claim rests. Following your Argument 2, that mismatched alternatives cannot isolate the effect, we placed the seeds on the comparison you identified as decisive. Anatomy and context have not been re-run under this protocol: they remain labelled single runs in App. D, and the revision no longer ranks them against gaze.

If any point remains open for your assessment, please let us know and we will address it.

I thank the authors for the thorough response and for the candour with which they engaged with each point. The local-artifact control, the three training seeds, the parse-failure reporting and the shortcut test address my main methodological concerns, and I have raised my rating from 2 to 3. I suggest the following for the revised version:

Scope of the claim. Relational gaze and local eye-artifact rationales are indistinguishable. The gain appears only under the decision read-out, and under free generation gaze is below decision-only (83.20 vs. 83.70). The occlusion test also shows that the generated rationale is not the basis of the decision. The title and abstract should reflect this narrower finding.

Generalization. Detection depends strongly on the generation procedure: with the same FLUX inpainter and a different procedure, fake recall drops to 9%. Please temper the claim that the detector learns cues shared across generated eye regions.

I will discuss with other reviewers during the post-rebuttal phase.

AuthorsOpenReview ↗

Thank you; both suggestions are now in the revised PDF

Thank you for reading the response in such detail, for the generous words, and for updating your assessment. Your review defined the three experiments that now carry the paper, and the paper is better for it. We have made both changes you suggest and uploaded the revised PDF; the wording is given here so that it is on the record.

1. Scope of the claim: done in the abstract; the title follows in the final version. The abstract now states the three points you list:

  • relational-gaze and local eye-artifact rationales raise the decision by 0.70 and 0.55 BA in all three seeds, "and the two are statistically indistinguishable";
  • "the gain appears under a direct decision read-out and not under free generation, where more answers become unparseable (83.20 against 83.70 for decision-only)";
  • "an occlusion test shows that the generated rationale is not the basis of the decision".

Its closing sentence is limited to "label-conditioned rationale templates and one detector family" and now reads that rationale content changes "a detector's decision without becoming the evidence it states". The title cannot be edited during the discussion; in the final version it will read:

"Rationale Supervision Changes a Detector's Decision, Not Its Stated Evidence: A Controlled Study of Rationale Templates for Person-Centric AI-Image Forensics"

2. Generalization: tempered in the abstract, §1, §4.2, §5, and §6.

  • Transfer to the three held-out inpainters is now stated as holding "under the same regeneration procedure" (abstract, §1, §6).
  • §4.2 no longer speaks of cues recurring across generated eye regions. It now says the effect "holds under the regeneration procedure of Custom Gaze and should not be read as cues shared by generated eye regions in general".
  • §5 now reports the result in the main text: "with the same FLUX inpainter and a different procedure (face-crop regeneration, guidance 30), fake recall falls to 9.0%, so the transfer result is limited to the procedure evaluated".

Both changes narrow the wording to what the three-seed results and the counterfactual set support. No number has changed, and the main text remains within ten pages.

Thank you again for the care you put into this review and for taking the discussion forward.

Reviewer_TfbrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

VLM based methods for fakes detection output a decision together with an explanation concerning their decision. The paper poses the question whether the rationale provided to the model during direct supervised finetuning has an effect on model performance. Their study focuses on person centric images. To test their hypothesis they generate a dataset of paired real-fake images were fakes are modified in the eyes region. A five level set of captions is provided and the performance of a fakes detector VLM is ablated depending on the rationales provided during its finetuning.

Critical strengths
  • The experimental design is solid, Explicit differences in the eye region for all pairs, and different rationale while keeping all other variables unchanged is a clean way to pose the question whther the rationale has any effect on performance
  • Transfer to ood verifies results were not overft to specific model features
  • Limitations section is great in explaining what the method cannot achieve
Critical weaknesses
  • The method is limited to a specific domain of application, it works as a proff of concept but i cannot see how it can generally be useful as a general fake detection technique. The edit used to create the paired data is very specific and shouldnt scale well to cover additional cases, more precisely each new cue would require careful handcrafting the data generation and creates useful rationales.
  • The method is based on finding specific cues, however, SOTA generators often dont violate these cues, so the method should be sensitive to existing or new generators that completely bypass the mechanism by which an image is judged as real of fake. Furthermore, similar to Li et al. 2018 once a cue is known to the public generators could be finetuned to specifically avoid it.
  • Table 5. The results suggest that most of teh contribution comes from the visual data rather than the added rationales. Additionally added text for anatomy and context seem to hurt performance beyond decision only which suggests ablations will always be needed for inttroducing new designs for provided rationales of a more genral system
  • The result is not surprising, adding carefully designed data always helps explanations for rationalizing whther an image is fake gives valuable clues and points the model to a preknown way to detect fakes. As a result the gains are primarily due to data engineering rather than an insight that would be a valueable contribution to the research community.
  • Literature is missing, there are many works on fakes detection prior to 2020 which are not cited by this paper, just to name a few:
Questions for authors
  • All training and testing use generated eye regions. This doesnt exclude that the method could be detecting specific artifacts rather than unrealistic eye texture and gaze direction. How can you exclude generation artifacts from being used by your detector?
  • how can your method be used as part of a production fakes detection system? for example in fully generated images, which are a significant portion of fakes, the gain is modest at +1.2BA
  • How do you propose to use the findings of your study in followup works?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer Tfbr

The controlled design shows a separation that data alone cannot show: rationale supervision improves the detector's decision, degrades its emitted answer, and produces explanations that are not the basis of the decision. We have also stated in the paper, as you point out, how much of the gain comes with the paired data. Tables R1 and R2 are in the General Response.

W1. Domain-specific; each cue needs hand-crafted data. The domain is narrow by design, the specialization costs nothing outside it, and the protocol transfers. A controlled claim about supervision needs a localized intervention, which fixes the domain. Outside it, nothing is lost and three things carry over: the general benchmark LOKI is unchanged (84.39 vs. 84.26), five full-image generators with no eye edit improve (Fig. 4(a)), and the paired data transfer to a general vision-only detector (Effort, +4.6 BA, Table 4). Adding a second cue needed one 25-sentence caption pool with the same images and recipe (the new local variant).

W2. SOTA generators; adaptation once a cue is public. Fake recall rises on all five generators, DALL-E 3 and Midjourney v6 included (Fig. 4(a)). The class-wise table we have added shows that this comes with the paired data: 87.85 for FakeVLM origin, 92.56 for decision-only, 93.63 with the gaze rationale. On adaptation we agree, and we have cited this argument of Li et al. (2018) in §2 together with Rössler et al. (2019) and Afchar et al. (2018).

W3. Most of the contribution comes from the visual data. We have stated this directly in the revision. On COCOAI Person the paired data move decision-only from 83.09 to 83.70 (three training seeds), and Effort from 70.6 to 75.2 (Table 4). On the same data, a rationale that describes the edit adds +0.55 to +0.70 BA to the decision in all three seeds (Table R1). Anatomy and context are reported as single runs and the claims rest on the three-seed runs, which matches your point that each new rationale design needs its own ablation.

W4. Not surprising; data engineering rather than insight. The insight is a three-way separation that cannot be read off the data (Tables R1, R2). (1) The decision improves with rationale supervision. (2) The emitted answer becomes less reliable (0.48% vs. 0.01% unparseable on Person, 3.54% vs. 0.01% on Custom Gaze), which costs 0.65 BA and offsets the gain under free generation unless the decision is read out directly. (3) The explanation is not the basis of the decision (occlusion minus placebo +0.1 [−0.7, +0.9]). For anyone fine-tuning an explanation-generating detector, these are three different quantities to measure, and the paper gives the protocol for each.

W5. Pre-2020 literature. We have added it. §2 now includes Rössler et al. 2019 (FaceForensics++) and Afchar et al. 2018 (MesoNet), with the explanation-supervision line of work.

Q1. How do you exclude generation artifacts? The detector does read evidence carried by the generated eyes, and we have excluded the three trivial versions of it. Processing is excluded (0.1–0.4% fake calls when authentic eyes pass through the same pipeline, App. E.3), a FLUX-specific signature is excluded (93.0–95.3 BA on three held-out inpainters vs. 68.3–73.8 for origin), and a gaze-direction rule is excluded (7.8% vs. 10.4%, Table R2). No result indicates that gaze geometry adds to this evidence, and no claim of the paper depends on it; a direct test needs a gaze-conditioned editor (§5). The COCOAI images are full syntheses without an eye-band edit, so the gains there are not an inpainting signature.

Q2. Use in production. As a training complement, with the decision read out directly. The +1.2 BA on full synthesis is mostly a data effect: add the paired data, and read the decision from the read-out, not from the generated text (Table R1). The generated rationale should not be shown as evidence (Table R2).

Q3. Follow-up. Three directions. (1) The same protocol on a second rationale-generating family and a second cue. (2) Image-grounded rationales in place of templates. (3) Constrained decoding to remove the emission cost in statement 2.

Presentation. We have added to §4 a pointer to a one-table guide to what each experiment varies and holds fixed (Table E.1).

We hope the soundness assessment (3) stands, and that the separation of decision, emission, and verbalization adds to what a reader learns from the paper.

Reviewer_TfbrOpenReview ↗

Thank you for the detailed response and the additional experiments.

I appreciate the clarification that most of the gain comes from the paired data, and the added literature. The response, however, largely confirms my main concerns. The rationale’s benefit (+0.55 to +0.70 BA) is roughly offset by the emission cost under free generation, gaze geometry is not shown to contribute, and the anatomy and context ablations are single training runs. The new separation of decision, emission and verbalization is interesting, but it represents a significant change to the paper’s central claim and would need a revision built around it. For these reasons, I am keeping my current score.

AuthorsOpenReview ↗

Thank you; one factual point on the revision

Thank you for reading the response and for the candid assessment.

One factual point for the record: the revision built around the separation is the PDF we uploaded on 15 Nov. The abstract, §4.3 (Table 5), §5, and §6 state the result as decision, emission, and verbalization, within ten pages. The narrower scope is the one Reviewer 2ZkP asked for, and the abstract of the revised PDF now states it explicitly (see our reply to Reviewer 2ZkP).

On your three points we agree with the facts, and the revision states them itself: the gain is realized when the decision is read out directly (Table 5), the paper does not claim a contribution of gaze geometry (§5), and anatomy and context are labelled single runs on which no claim rests (App. D).

We appreciate that you find the separation interesting, and we thank you for the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank the reviewers and the AC for the careful reviews. We have run the controls the reviews asked for and have uploaded a revised PDF (changes in blue). The new results confirm the paper's central claim with three training seeds and sharpen it into three statements.

1. Decision. With images, labels, and recipe fixed, rationales that describe the eye-region edit raise the detector's decision over decision-only supervision by +0.55 to +0.70 BA on COCOAI Person in all three training seeds, three to four times the training-seed sd of decision-only (0.17). Relational-gaze and local eye-artifact rationales are indistinguishable, as Reviewer 2ZkP anticipated.

2. Emission. Under free generation gaze is statistically indistinguishable from decision-only (−0.50 [−1.02, +0.02]). Its point estimate is lower because five-sentence answers are sometimes unparseable (0.89% of reals, 0.40% of fakes; decision-only 0.01%), a formatting cost of 0.65 BA. Losing label accuracy when a model also generates an explanation is known from language tasks (Camburu et al. 2018); the read-out does not carry this cost and is 7 times more stable across seeds (sd 0.06 vs. 0.41).

3. Verbalization. The evidence the model states is not the evidence it uses.

Table R1. Decision vs. emission (COCOAI Person BA, mean ± sd over three training runs). Decision read-out: same input; after the fixed prefix "This is a", the larger of the "real" and "fake" logits is the prediction, so no output is unparseable. Free generation: first decision keyword, unparsed = incorrect.

SupervisionDecision read-outΔ vs. decision-only [95% CI]Free generationUnparsed (%), Person / Custom Gaze
Decision only83.67 ± 0.17083.70 ± 0.190.01 / 0.01
Five-block local eye-artifact84.23 ± 0.26+0.55 [+0.20, +0.91]83.80 ± 0.040.13 / 0.84
Five-block relational gaze84.37 ± 0.06+0.70 [+0.38, +1.01]83.20 ± 0.410.48 / 3.54
gaze − local+0.14 [−0.15, +0.43]

Table R2. What the detector says vs. what it uses (submitted gaze checkpoint).

TestResultReading
Fake calls on authentic non-mutual-gaze vs. matched mutual-gaze reals (500 each)7.8% vs. 10.4%no "averted gaze → fake" shortcut
Multi-person phrase with 0 vs. 2+ detected faces; with ≤1 face12.6–15.9% vs. 23.6–26.3%; 11.9–15.1%partly scene-conditioned, not reliable
Eye-band minus placebo occlusion, gaze-card outputs+1.1 [+0.2, +2.0] (1 face); +0.1 [−0.7, +0.9] (2+ faces)stated evidence is not the basis of the decision

Common questions.

CC1. Scope of the claim (2ZkP §1–§2, Tfbr W3–W4, q6qv). The central claim stands with three training seeds, and we have made its scope explicit. We have described the variants as label-conditioned rationale templates and stated every contrast against decision-only: the gain comes from describing the edited evidence, relationally or locally. Anatomy and context are labelled single runs (App. D); every claim rests on the three-seed runs, which replace the 4.2–4.6 / 2.1–2.4 headline.

CC2. Artifacts, shortcut, faithfulness (2ZkP §3, q6qv W1–W2, Tfbr Q1). The detector reads evidence carried by the generated eyes, not a rule about where people look. Each pair differs only inside the eye band, so the 99% separation must come from eye-band evidence. It is not gaze direction (Table R2, row 1), not processing (0.1–0.4% fake calls on the authentic-eye control), and not FLUX-specific (93.0–95.3 BA on three held-out inpainters vs. 68.3–73.8 for origin). No result indicates that gaze geometry adds to this evidence, and no claim depends on it (§5; counterfactual set in App. E.3).

CC3. Variance, parsing, split (2ZkP §4, q6qv Q3). All intervals now include training-run variance, under one evaluation rule and a verified split. We have run three independent fine-tuning runs per variant, evaluated at the same fixed final step (2,200, past the 1,500–1,650 steps at which the submitted runs peaked) so that no per-variant checkpoint selection enters, with unparsed = incorrect for every metric. All three-seed variants use a pair-disjoint split verifiable from pair identifiers, and their decision-only models match the submitted one (Person 83.70 vs. 83.32).

Revisions made. (1) Abstract, §4.3, §5, §6 state the result as Tables R1 and R2. (2) Table 5 has the local row and training seeds; App. B has parse-failure and recall tables. (3) L251–252 removed; Table 1, L295–296, D.1, Table 7 corrected. The paper's thesis is retained with its scope explicit: as Reviewer 2ZkP suggested, we have scoped the conclusion (§6); the title is frozen during discussion, and its camera-ready subtitle will read "A Controlled Study of Rationale-Template Supervision for Person-Centric AI-Image Forensics".

AuthorsOpenReview ↗

Summary of the discussion and committed changes

We thank all three reviewers for the discussion. For the Area Chair, the state of the record is as follows.

ReviewerOutcome of the discussion
2ZkP (confidence 3)Rating raised from 2 to 3: the local control, three training seeds, parse-failure reporting, and shortcut test "address my main methodological concerns". Two suggestions on wording (scope; generalization), both made in the revised PDF.
q6qv (confidence 2)"Almost all of my concerns are addressed." Rating maintained for novelty, which we have addressed in our reply.
Tfbr (confidence 2, Soundness 3)Finds the separation of decision, emission, and verbalization "interesting"; asks for a revision built around it, which is the PDF uploaded on 15 Nov.

Changes made after the discussion (wording only; no number changes): (1) the abstract states the narrower finding, and the final title will do the same (reply to 2ZkP); (2) transfer to held-out inpainters is stated as holding under the same regeneration procedure, with the 9.0% result in the main text (§5).

No reviewer questions the correctness of a reported result. What remains under discussion is how the scoped finding is worded, and that wording is now in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.