acceptodds
Withdrawn from ICLR 2027

ManiVid: Unified and Explainable Manipulation Analysis of Videos via MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Rapid advances in AI-generated video (AIGV) have increased the risks posed by deceptive video manipulation. Unlike fully synthetic videos, manipulated videos retain most source content and alter only localized regions, making forensic analysis particularly challenging. Existing video forgery research faces two limitations in both data and methodology: (1) High-quality datasets and benchmarks tailored for manipulated videos remain scarce. (2) Multimodal large language models (MLLMs) extend forgery analysis beyond binary classification but struggle to use low-level forensic cues and provide precise pixel-level grounding. Specifically, we introduce ManiVid, a unified forensic analysis task covering forgery detection, artifact grounding, and anomaly explanation for manipulated videos. We construct ManiVid-38K, the first dataset to combine paired, open-vocabulary localized manipulations of general videos with authenticity labels, forgery masks, and anomaly explanations. It comprises about 19K manually verified real-fake video pairs, mostly at 1080P resolution, generated under 2 paradigms with 15 powerful generation models. We sample 1K pairs for ManiVidBench, balanced across six manipulation types and generation models for fair evaluation. We further propose ManiVidLens, a unified framework for explainable video forgery analysis. Its Forensic Evidence Router supplies shared low-level forensic evidence for multimodal reasoning and video segmentation. Its Prompt Distill Module converts grounding states into semantic and geometric prompts and distills spatial priors for mask decoding and full-video propagation. ManiVidLens achieves relative gains over the strongest comparison methods in artifact grounding (+21.1% mIoU; +21.3% J&F) and anomaly explanation (+131.3% ROUGE-L; +9.9% CSS). Its forgery detection remains comparable to dedicated classifiers (0.914 Acc; 0.913 F1). Together, our dataset and framework enable accurate and interpretable forensic analysis of manipulated videos. The code, model, and dataset will be released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-generated Videos, Manipulation Detection, Artifact Grounding, Anomaly Explanation, MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SxGGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies local manipulation analysis in general videos, covering forgery detection, manipulated-region localization, and anomaly explanation. The authors construct ManiVid-38K, which contains about 19K original-manipulated video pairs with authenticity labels, region masks, and textual explanations. They also propose ManiVidLens, which combines AIDE, an MLLM, and SAM2. FER injects forensic features into the language and segmentation models, while PDM converts the MLLM's hidden states into segmentation prompts. The three tasks share model parameters and evidence but use separate task prompts.

Evaluation on ManiVidBench, a held-out set of 1K video pairs from ManiVid-38K, shows improvements in localization and explanation metrics over the compared methods.

Critical strengths
  1. The research problem has practical value. Answering whether a video was manipulated, where it was manipulated, and what anomalies are present provides more information for inspection than a binary authenticity decision alone.

  2. The dataset is a useful resource. It covers addition, removal, replacement, color alteration, material modification, and portrait editing, and provides paired original and manipulated videos together with region masks and anomaly explanations. These annotations support training and evaluation across the three tasks.

  3. The method shows a clear improvement in manipulation localization on ManiVidBench. Its overall mIoU is 0.407, compared with 0.336 for Sa2VA. The module ablations also show localization gains from FER and PDM, providing some experimental support for using forensic features and language-derived prompts to assist segmentation.

Critical weaknesses
  1. The method relies on a small number of sampled frames for forgery detection and anomaly explanation, but the paper does not establish whether these frames cover the period in which manipulation occurs. In particular, the experiments do not show whether the evidence supplied to the MLLM and SAM2 is sufficient when manipulation appears only briefly or late in a video. The frame-count ablation reports average performance but does not directly address this coverage problem.

  2. The MLLM hidden states at [BBOX1] and [BBOX2] are projected through MLPs into SAM2's prompt space. Whether the resulting vectors provide useful geometric information needs more direct validation. In particular, the supervision targets are corner embeddings averaged across multiple frames, and the same predicted prompts are then applied to each frame. It remains unclear whether this representation preserves the positional information needed for individual frames, especially when the target moves.

  3. The method's key designs involve organizing, injecting, and distilling embeddings, but the reasons for these specific choices are supported mainly by intuitive explanations. For example, averaging corner embeddings across frames is assumed to provide useful geometric guidance, yet this assumption is not tested separately. The overall FER and PDM ablations show how performance changes when entire modules are added, but do not adequately establish whether their internal components work as described. The connection between the specific designs and the problems they are intended to solve therefore needs stronger evidence.

  4. The dataset and experiments focus on short videos, with source videos limited to 10 seconds. Longer manipulated videos, including videos exceeding one minute, are also relevant application scenarios. The paper does not evaluate whether its conclusions extend to longer videos, leaving the dataset's coverage of these scenarios and the method's practical scope unclear.

  5. The method relies on AIDE features and several learned mappings: from AIDE to the MLLM, from AIDE to SAM2, and from the MLLM to SAM2. I am concerned about whether these features and mappings remain useful when the data distribution changes. Results grouped by generator and tests under video degradation provide some evidence, but do not establish transfer to generators excluded from training or to external datasets. The current evaluation does not sufficiently address this concern.

Questions for authors

Please refer to Weaknesses

8 comments
AuthorsOpenReview ↗

Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization

We thank Reviewer SxGG for carefully reading our paper and for the constructive feedback. We appreciate your comments on the value of evaluating detection, grounding, and explanation together and on the localization gains observed on ManiVidBench.

【Critical Weaknesses】

W1. Coverage of brief or late manipulations by the sampled frames.

(1) Task setting. ManiVid-38K studies edits to objects or local regions in short videos. These edits typically persist while the target region is visible. For targets that appear early and remain in view, early observations provide evidence of the edit; retaining these observations also helps when a target later leaves the scene. We therefore use a small set of temporally spaced frames and check whether it captures an annotated edit in each video.

(2) Sampling protocol. As described in Appendix E.1.2, we place 30 candidate frames uniformly along the full video timeline and use the first five decodable candidates as the model input. These frames retain the spacing of the candidate grid and generally span a longer interval than the first five consecutive frames of the original video. Training also uses five temporally ordered frames, with random temporal patterns and sampling strides of {4, 6, 8, 10}. Inference thus retains the five-frame, spaced-input format used during training. The ordered frames provide local temporal context for comparing the target's appearance and position across the observed time points. For grounding, these conditioning frames initialize SAM2, which propagates masks across all physical frames of the video.

Sampling a limited number of frames is common in video MLLMs. For example, Video-MME [1] evaluates Video-LLaVA with 8 frames and LLaVA-NeXT-Video with 32 frames. Sa2VA [14], which also combines an MLLM with SAM2, uses the first five consecutive video frames as its default MLLM input.

(3) Coverage on ManiVid-38K. We agree that Table 5 reports average performance at different frame counts and that temporal coverage calls for a direct check. For every manipulated video in ManiVid-38K, we compared the indices of the five conditioning frames actually supplied to the model with the frame-level ground-truth masks. We counted a video as covered if at least one of those five frames had a nonempty mask. Every manipulated video met this criterion (100% video-level coverage). Thus, in our experiments, the model saw at least one frame containing an annotated manipulation in every manipulated sample.

(4) Full-frame inspection. For brief or late edits that fall outside these five observations, the MLLM can also inspect all frames by processing the video in segments or sliding windows. The five-frame setting is the default used in our reported experiments. Full-frame inspection brings every frame into the MLLM at least once, at higher inference cost. This removes gaps caused by frame selection; detection accuracy still depends on whether the model recognizes the manipulation cues.

AuthorsOpenReview ↗

Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization

W2. Frame-specific geometric information in the predicted SAM2 prompts for moving targets.

The geometric prompts are designed to give SAM2 a coarse spatial prior about where the manipulated target is likely to appear across the conditioning frames. For the object- and region-level manipulations considered in our task, the aim is to focus segmentation on a relevant part of the scene within the full frame. This cue only needs to provide an approximate spatial reference; the mask decoder determines the target's precise location and boundaries in each frame using that frame's visual features.

As described in Appendix D.3, during training the frozen SAM2 prompt encoder maps the ground-truth bounding box in each valid conditioning frame to two embeddings, one for the top-left corner and one for the bottom-right. We average the top-left embeddings across valid frames and do the same for the bottom-right embeddings. These two averages serve as supervision targets for the projected MLLM hidden states at [BBOX1] and [BBOX2], respectively. The MLLM thus predicts one pair of geometric prompts per video. This aggregation provides a shared spatial reference derived from the target's locations in the sampled frames.

For a moving target, SAM2 combines this shared spatial prior with the semantic prompt and the visual features of each conditioning frame to predict its mask. The geometric prompts provide soft spatial guidance, while SAM2 retains access to the full frame's visual features. The decoder can therefore produce masks at different positions as the target moves. SAM2 then propagates the initialized target state to the remaining frames. Sharing the geometric prompts serves this coarse guidance role without requiring the MLLM to generate a separate box for every frame.

With bounding boxes derived from per-frame ground-truth masks, SAM2-Hiera-Large achieves 84.55% mIoU on the 1,000 manipulated videos in ManiVidBench. This shows that SAM2 can use box-derived geometric prompts to localize manipulated regions, supporting our choice of its prompt space as the target for geometric distillation.

W3. Evidence for the two forensic evidence-injection pathways in FER.

We evaluated FE2SAM and FE2VLM separately and together using the same Qwen3-VL-4B and SAM2 Hiera-L backbones, with PDM disabled and the training conditions held constant. FE2SAM injects forensic features into SAM2's spatial features for mask localization. FE2VLM supplies forensic evidence to the MLLM as additional visual tokens. The table reports the same metrics as the main paper.

Active FER pathwaysmIoUJ&FR-LCSSAccF1
Neither (Base)0.3360.3480.5900.7910.9150.914
FE2SAM only0.372‾\underline{0.372}0.383‾\underline{0.383}0.5920.7980.9030.902
FE2VLM only0.3670.3790.597‾\underline{0.597}0.807‾\underline{0.807}0.9060.905
Both (full FER)0.3780.3890.5980.8080.907‾\underline{0.907}0.906‾\underline{0.906}

FE2SAM alone reaches 0.372 mIoU and 0.383 J&F, compared with 0.336 and 0.348 for Base and 0.367 and 0.379 for FE2VLM alone. These results support injecting forensic evidence directly into SAM2's spatial features to improve localization. FE2VLM alone reaches 0.597 ROUGE-L and 0.807 CSS, compared with 0.590 and 0.791 for Base and 0.592 and 0.798 for FE2SAM alone. Supplying forensic evidence to the MLLM improves the agreement between generated explanations and the reference annotations.

Combining both pathways achieves 0.378 mIoU, 0.389 J&F, 0.598 ROUGE-L, and 0.808 CSS, the highest grounding and explanation scores among these four configurations. The component ablations therefore support FER's design of injecting forensic evidence into both the segmentation and language models.

AuthorsOpenReview ↗

Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization

W4. Applicability to videos longer than the 10-second source clips, including videos over one minute.

(1) Clip lengths in existing datasets. We limited source videos in ManiVid-38K to 10 seconds or less (Appendix C.1.1). Several video forensics datasets in Table 1 use clips of similar or shorter duration. DFDC [2] uses 10-second clips for its competition training, validation, and test sets. VideoSham [3] trims source videos to an average of 8 seconds before editing. Each clip in VideoFACT’s three public standard-manipulation subsets [4] is 1 second long (30 frames). FaceForensics++ [5] reports 509,914 frames across 1,000 original clips, or about 510 frames per clip. In the related task of instruction-guided video editing, the official OpenVE-3M dataset card [6] reports 65–129 frames per clip. Our focus on short clips is consistent with this practice.

(2) Computational cost of minute-long videos. Our two-stage training already uses eight NVIDIA RTX PRO 5000 GPUs with BF16 precision, a microbatch of two samples per GPU, and four steps of gradient accumulation (Appendix E). With the default five conditioning frames, inference across all three tasks takes an average of 15.98 seconds per video (Table 5). Even if we keep the MLLM input at five frames, extending a 10-second clip to one minute at the same frame rate gives SAM2 six times as many frames to process during mask propagation. Maintaining the same temporal sampling density would also require either more conditioning frames, which increases GPU memory use, or processing the video in segments, which increases runtime. Systematic training and evaluation on minute-long videos would exceed our current computing budget.

W5. Transfer of AIDE features and learned mappings to unseen generators and external datasets.

ManiVidLens uses a frozen AIDE [7] encoder to extract forensic features from each conditioning frame. For detection, FE2VLM projects these features into the MLLM input space, where the model uses them alongside the video’s visual tokens to judge authenticity. We evaluated the same ManiVidLens checkpoint trained on ManiVid-38K directly on VideoSham [3] and FaceForensics++ [5], without fine-tuning on either dataset. We tested standalone AIDE on the same videos.

(1) VideoSham. We evaluated all 352 real–fake pairs in the public release (704 videos), covering all six attack types. Results are video-level percentages. Each attack row reports the percentage of manipulated videos correctly detected.

MetricLiu et al. [8], via [3]AIDE [7]ManiVidLens
Real recall56.6697.1666.48‾\underline{66.48}
Attack 1: Add an entity or subject29.90‾\underline{29.90}2.9062.32
Attack 2: Remove an entity or subject52.585.0645.57‾\underline{45.57}
Attack 3: Modify the background or color29.63‾\underline{29.63}11.1162.96
Attack 4: Replace or add text24.44‾\underline{24.44}0.0042.22
Attack 5: Repeat, drop, or reverse frames, or change playback speed6.85‾\underline{6.85}1.6932.20
Attack 6: Replace or add audio12.77‾\underline{12.77}0.0036.96
Mean recall across the six attack types (equal weight per type)26.03‾\underline{26.03}3.4647.04
Fake recall (all manipulated videos)28.57‾\underline{28.57}3.6947.73
Overall accuracy42.6250.43‾\underline{50.43}57.10

AIDE and ManiVidLens both used RGB input only. Attack 6 therefore records their visual predictions on videos with edited audio. The Liu et al. column is derived from the counts in Table 4 of [3], which evaluated 826 videos.

Standalone AIDE correctly identified 97.16% of real videos but detected only 3.69% of forgeries. ManiVidLens detected 47.73% of forgeries while correctly identifying 66.48% of real videos. The gap between real and fake recall narrowed from 93.47 to 18.75 percentage points, and overall accuracy rose from 50.43% to 57.10%. The gains covered several visual edits, with recall above 62% for both entity additions and background or color changes.

Reviewer_Dh9TOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies local video manipulation and brings detection, localization, and anomaly explanation together into one framework. It introduces ManiVid-38K dataset and ManiVidBench benchmark, which provide mask and explanation annotations for six types of manipulation. The proposed model ManiVidLens uses FER and PDM to improve grounding, achieves higher grounding and explanation scores than compared methods on ManiVidBench.

Critical strengths
  1. The paper introduces ManiVid-38K and the ManiVidBench benchmark, providing about 19K original-edited video pairs with editing instructions, masks, explanation annotations. They support detection, localization, and explanation tasks, giving the dataset value beyond the proposed model.
  2. ManiVidLens improves performance on the ManiVidBench localization task.
Critical weaknesses
  1. There are serious concerns about the validity of the dataset’s manipulation categories and annotations. First, Table 12 defines operations, but one edit may fit several categories, creating ambiguity or overlap. The authors acknowledge confusion between operations in Figure 13 but do not further examine its causes. Second, 4 anomaly descriptions are required for each example, yet the examples include wrong descriptions. For example, D08545 in Figure 10 treats the lack of visible deformation of the stove grate under the kettle’s weight as an anomaly, which is not convincing.
  2. The method’s effectiveness is currently limited to the authors’ own dataset, with no experiments on other public datasets or tests of generalization to unseen generators. ManiVidLens is fine-tuned only on ManiVid-38K and evaluated only on ManiVidBench. Although Table 21 shows different generators’ results, they are also included in training. Therefore, additional evaluations are needed to assess how well the method generalizes.
Questions for authors
  1. Please explain how labels are assigned and evaluated when videos meet definitions of more than one of the six manipulation types.
  2. The dataset construction does not fully match the paper’s stated goals of local editing and low shortcut risk. The paper first states that unedited content is preserved, but C.1.5 shows that 46.2% of data uses Indirect I2V, editing the first frame and regenerates the full video. Although human screening is used, paper does not measure how well the other regions are preserved. Table 21 also shows that the generation pipeline affects detection. Please explain how this construction process supports the intended goals of the dataset.
  3. The explanation annotations in Figures 8, 9, 10 contain serious problems (ex. D08545 in Figure 10). Please explain why these annotations passed quality screening. Additionally, you may provide the absolute scores for each model in the human evaluation in E.3.4.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Dh9T on Dataset Quality and Cross-Dataset Generalization

We thank Reviewer Dh9T for carefully reading our paper and providing constructive feedback. We appreciate the opportunity to address these concerns. Several points appear to stem from factual misunderstandings of our work, which we would like to clarify below.

【Critical Weaknesses】

W1. We respectfully disagree with the assessment that there are serious concerns about the validity of our manipulation categories and annotations. We clarify the specific points below.

(1) Category overlap. As illustrated in Figure 2 and described in Section 2.2 (Lines 188–194), ManiVid-38K is constructed by first selecting a manipulation task for each source video and then generating an editing instruction conditioned on that task. If the initially selected task is unsuitable, it is replaced with a suitable one rather than combined with another task. Table 13 further requires exactly one final task label corresponding to the finalized instruction. Thus, each sample has a single task label grounded in its construction process. The label identifies the designated editing operation, rather than exhaustively enumerating every visual attribute that may change as a consequence. For example, replacing an object may also change its color, but this does not make it a separately assigned color-alteration task. We will make this distinction more explicit to avoid ambiguity.

(2) Interpretation of Figure 13. We believe there may be a misunderstanding about what Figure 13 shows. The matrix compares the model's predictions with the ground-truth manipulation types. It therefore reflects errors in model attribution, which do not by themselves indicate problems with the dataset annotations. In Section E.3.3, we ask a question beyond binary real/fake detection: can the model also identify how a video was manipulated? Since ManiVidLens achieves 91.4% detection accuracy, we explore whether it can provide this finer-grained information. Figure 13 shows that 37.3%–52.4% of samples in each manipulation category are correctly attributed. These rates exceed the 16.7% expected from uniformly guessing among six manipulation types, although this comparison is only illustrative because our evaluation also includes the Real class. The results show potential for attribution, while leaving considerable room for improvement. We also discuss the main confusion patterns in Section E.3.3 and explain that operations such as replacement, color alteration, and material modification can produce similar visual forensic cues. This experiment is an additional analysis beyond our main focus on detection, grounding, and explanation. We do not claim to have solved manipulation attribution, and a detailed investigation of its remaining challenges is a valuable direction for future research.

(3) Textual explanation annotation. We think three points below may help: (i) Among the explanation-providing works compared in Table 1, VidGuard-R1 [7] (ICLR 2026) and AEGIS [8] (ACM MM 2025) both generate textual explanations by conditioning an MLLM on video content and ground-truth authenticity labels. Our annotation pipeline follows this established practice and additionally uses masks of the manipulated objects to guide the analysis toward the relevant regions. (ii) VidGuard-R1 prompts its annotator to reason along four predefined dimensions: Motion Consistency, Lighting Consistency, Texture Artifacts, and Physics Violations. Our pipeline instead allows the annotator to select the four most applicable dimensions from eight candidate perspectives for each video (Table 17). This sample-specific selection is designed to reduce the pressure to provide irrelevant or unsupported explanations simply to cover the same fixed dimensions for every sample. We further conduct human verification of explanation fidelity (Section C.1.12), checking for incorrect or nonexistent manipulated objects, overly homogeneous descriptions, and claims inconsistent with the video content. Samples that fail verification are discarded. Neither VidGuard-R1 nor AEGIS explicitly reports a comparable human verification stage for the generated explanation annotations. These measures provide additional safeguards for annotation quality beyond label-conditioned generation alone. (iii) Taken together, our use of an established annotation approach, spatial guidance, adaptive dimension selection, and human verification provides a principled basis for constructing the explanation annotations. Individual examples warrant scrutiny, but they do not, by themselves, establish a systematic failure of the annotation pipeline or justify a broad conclusion that the dataset as a whole is unreliable.

AuthorsOpenReview ↗

Response to Reviewer Dh9T on Dataset Quality and Cross-Dataset Generalization

W2. Cross-dataset generalization in forgery detection.

To test whether the manipulation cues learned on ManiVid-38K transfer to other datasets, we evaluated the same ManiVidLens checkpoint on VideoSham [3] and FaceForensics++ [6] without fine-tuning on either dataset. Since ManiVidLens uses AIDE [2] to extract forensic features, we also evaluated standalone AIDE on the same test videos. On both datasets, ManiVidLens achieved higher overall accuracy and forgery recall than AIDE and narrowed the gap between real and fake recall.

(1) VideoSham. We evaluated all six attack types in the public release, which contains 352 pairs of real and manipulated videos. The table reports video-level percentages. Each attack row gives the percentage of videos of that type detected as fake.

MetricLiu et al. [1], via [3]AIDE [2]ManiVidLens
Real recall56.6697.1666.48‾\underline{66.48}
Attack 1: Add an entity or subject29.90‾\underline{29.90}2.9062.32
Attack 2: Remove an entity or subject52.585.0645.57‾\underline{45.57}
Attack 3: Modify the background or color29.63‾\underline{29.63}11.1162.96
Attack 4: Replace or add text24.44‾\underline{24.44}0.0042.22
Attack 5: Repeat, drop, or reverse frames, or change playback speed6.85‾\underline{6.85}1.6932.20
Attack 6: Replace or add audio12.77‾\underline{12.77}0.0036.96
Mean recall across the six attack types (equal weight per type)26.03‾\underline{26.03}3.4647.04
Fake recall (all manipulated videos)28.57‾\underline{28.57}3.6947.73
Overall accuracy42.6250.43‾\underline{50.43}57.10

AIDE showed a strong tendency to predict “real” on VideoSham. It correctly identified 97.16% of real videos but detected only 3.69% of forgeries. ManiVidLens detected 47.73% of forgeries while correctly identifying 66.48% of real videos. The gap between real and fake recall narrowed from 93.47 to 18.75 percentage points, and overall accuracy increased from 50.43% to 57.10%. The gains span several visual edits. For example, ManiVidLens detected over 62% of both entity additions and background or color changes. Both models received RGB input only, so the audio row records their visual predictions on videos with edited audio. We derived the Liu et al. percentages from the prediction counts in Table 4 of Mittal et al. [3]. That evaluation covered 826 videos, compared with 704 in our test.

(2) FaceForensics++. We used the official c23 (HQ) test split with 140 real videos and 140 videos from each of the four forgery categories defined in [6]. Since fake videos outnumber real videos four to one, we report both overall accuracy and binary balanced accuracy, which gives equal weight to real and fake recall. All values are percentages. A dash indicates that the cited study did not report the corresponding value.

MetricAttention-augmented hybrid model [4]ViT-B/16 [5]AIDE [2]ManiVidLens
Deepfakes——15.71‾\underline{15.71}83.57
Face2Face [17]——18.57‾\underline{18.57}75.00
FaceSwap——12.86‾\underline{12.86}75.00
NeuralTextures [18]——14.29‾\underline{14.29}77.86
Real——92.1441.43‾\underline{41.43}
Overall accuracy69.28‾\underline{69.28}38.7730.7170.57
Binary balanced accuracy——53.75‾\underline{53.75}59.64

ManiVidLens has the highest overall accuracy in the table at 70.57%, 1.29 percentage points above the 69.28% reported for the attention-augmented hybrid model [4]. The published results for that model and ViT-B/16 (Table 6 of [4] and Table 4 of [5]) use Celeb-DF v2 [16] for training and differ from our evaluation in test samples and protocol. Across the four forgery categories, ManiVidLens correctly classified 77.86% of manipulated videos as fake, compared with 15.36% for standalone AIDE. Real recall is 41.43% for ManiVidLens and 92.14% for AIDE. Giving the two classes equal weight yields a balanced accuracy of 59.64% for ManiVidLens and 53.75% for AIDE.

Without target-dataset fine-tuning, ManiVidLens has the highest overall accuracy listed in both tables (57.10% on VideoSham and 70.57% on FaceForensics++). In the direct comparison on identical test videos, ManiVidLens detected more forgeries than standalone AIDE on both datasets. These results support the transfer of the manipulation cues learned on ManiVid-38K to external videos.

【Questions For Authors】

Q1. Each manipulated video corresponds to exactly one manipulation type determined during dataset construction, so there is no ambiguity in label assignment. Please refer to W1 (1) Category overlap. under 【Critical Weaknesses】 for details.

AuthorsOpenReview ↗

Response to Reviewer Dh9T on Dataset Quality and Cross-Dataset Generalization

Q2. How do we evaluate the fidelity of unedited regions?

We already assess their preservation when selecting samples for ManiVid-38K. As described in Section 2.2, the MLLM assistant compares the source and manipulated videos against the editing instruction. It checks whether the intended edit succeeded, whether the result follows the instruction, and whether regions outside the intended edit retain their original content. Only samples that pass these checks proceed to annotation. During the final human review (Appendix C.1.12), we inspect each source–manipulated video pair for overall editing quality, including visible artifacts and changes beyond those specified by the instruction. Preservation of unedited content is thus an explicit criterion in automatic screening and part of the subsequent human review of editing quality.

To quantify how editing affects content outside the target region, we use the frame-level manipulation masks in ManiVid-38K to compare each manipulated video with its source at both the pixel and semantic levels.

(1) Pixel fidelity (MSE ↓). Following the FiVE-Bench [9] implementation, we normalize RGB values to [0, 1] and use the same mask to set the edited region to zero in each pair of corresponding source and manipulated frames. We compute the mean squared error over all pixels and channels of the full frame, then average the frame scores to obtain a video-level score. Only differences outside the mask contribute to the error. A lower MSE indicates smaller pixel changes in the unedited region.

(2) Semantic fidelity (CLIPsrc+\mathrm{CLIP}_{\mathrm{src}}^{+} ↑). We adapt the source faithfulness metric from Videoshop [10]. Videoshop uses CoTracker [11] to track the edited region across frames. We use the frame-level masks provided by ManiVid-38K. For each pair of corresponding source and manipulated frames, we mask out the same edited region, encode each masked frame with the CLIP [12] image encoder, and compute the cosine similarity between the two embeddings. We average these similarities across frames to obtain a video-level score. A higher score indicates that the unedited content is semantically more similar to the source.

The detailed evaluation results are shown below.

MetricAdditionRemovalReplacementColor AlterationMaterial ModificationPortrait EditingAvg.
(Pixel Fidelity) MSE ↓0.0190.0150.013‾\underline{0.013}0.0130.0140.0080.014
(Semantic Fidelity) CLIPsrc+\mathrm{CLIP}_{\mathrm{src}}^{+} ↑0.9400.9450.9450.9450.945‾\underline{0.945}0.9500.944

Across all six manipulation types, MSE is low (0.008–0.019) and CLIP similarity is high (0.940–0.950). These results indicate that the unedited regions in ManiVid-38K retain high fidelity to the source videos in both pixel values and semantic content.

We defined Direct V2V and Indirect I2V when constructing ManiVid-38K. In Table 21, we then evaluated the same model separately on videos produced by each paradigm. We assigned each source video to only one paradigm. The two groups therefore share no source videos and differ in the underlying video content as well as in the editing procedure. The observed difference in detection accuracy does not by itself establish that the generation pipeline caused it. To examine one possible shortcut, we compared the fidelity of unedited regions across the two groups. Direct V2V yielded a mean MSE of 0.014 and a mean CLIPsrc+\mathrm{CLIP}_{\mathrm{src}}^{+} of 0.944. For Indirect I2V, the corresponding values were 0.013 and 0.945. Both groups closely preserved the source background on these measures, providing no evidence of a systematic difference in background fidelity that the detector could exploit across paradigms.

AuthorsOpenReview ↗

Response to Reviewer Dh9T on Dataset Quality and Cross-Dataset Generalization

Q3. Please see (3) Textual explanation annotation in our response to W1 under 【Critical Weaknesses】. We believe examples cited do not establish a systematic problem with the explanation annotations. Section E.3.4 evaluates the explanations produced by the four models. For each video, four evaluators independently scored each explanation on a 1–5 scale for evidence faithfulness, forensic relevance, and explanation quality, with model identities hidden. We compute each model’s absolute score for a manipulation category by averaging these ratings across the four evaluators, the three dimensions, and the 50 videos sampled for that category.

ModelAdditionRemovalReplacementColor AlterationMaterial ModificationPortrait Editing
LEGION [13]3.773.72‾\underline{3.72}3.91‾\underline{3.91}3.84‾\underline{3.84}3.88‾\underline{3.88}3.87
Omni-Fake-R1 [14]3.81‾\underline{3.81}3.683.753.763.773.63
GPT-5.6 Sol [15]2.742.802.782.762.822.78
ManiVidLens (Ours)4.254.094.114.094.183.80‾\underline{3.80}

In the table above, ManiVidLens ranks first in mean explanation score for five of the six manipulation categories: Addition, Removal, Replacement, Color Alteration, and Material Modification.

【References】

  • [1] LIU Y, XIA C, ZHU X, et al. Two-Stage Copy-Move Forgery Detection With Self Deep Matching and Proposal SuperGlue[J]. IEEE Transactions on Image Processing, 2022, 31: 541-555. DOI: 10.1109/TIP.2021.3132828.
  • [2] YAN S, LI O, CAI J, et al. A Sanity Check for AI-Generated Image Detection[C]//International Conference on Learning Representations, 2025.
  • [3] MITTAL T, SINHA R, SWAMINATHAN V, et al. Video Manipulations Beyond Faces: A Dataset With Human-Machine Analysis[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. IEEE, 2023: 643-652. DOI: 10.1109/WACVW58289.2023.00071.
  • [4] SHIVAPRAKASH S J, H S, CHAUHAN A, et al. Attention-augmented hybrid framework with evolutionary optimization for robust deepfake detection[J]. Scientific Reports, 2026, 16: 24070. DOI: 10.1038/s41598-026-51284-6.
  • [5] HASAN S A, HERA M P, MIM A L, et al. Beyond Frame-Level Accuracy: Analyzing Video-Level Stability in Transformer-Based Deepfake Forensics[J]. International Journal of Intelligent Systems, 2026, 2026(1): 6801006. DOI: 10.1155/int/6801006.
  • [6] RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to Detect Manipulated Facial Images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2019: 1-11. DOI: 10.1109/ICCV.2019.00009.
  • [7] PARK K, YANG Y, YI J, et al. VidGuard-R1: AI-Generated Video Detection and Explanation via Reasoning MLLMs and RL[C]//International Conference on Learning Representations, 2026.
  • [8] LI J, ZHANG X, ZHOU J T. AEGIS: Authenticity Evaluation Benchmark for AI-Generated Video Sequences[C]//Proceedings of the 33rd ACM International Conference on Multimedia, 2025.
  • [9] LI M, XIE C, WU Y, et al. FiVE-Bench: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2025.
  • [10] FAN X, BHATTAD A, KRISHNA R. Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion[C]//European Conference on Computer Vision, 2024.
  • [11] KARAEV N, ROCCO I, GRAHAM B, et al. CoTracker: It is Better to Track Together[C]//European Conference on Computer Vision, 2024.
  • [12] RADFORD A, KIM J W, HALLACY C, et al. Learning Transferable Visual Models From Natural Language Supervision[C]//International Conference on Machine Learning, 2021: 8748-8763.
  • [13] KANG H, WEN S, WEN Z, et al. LEGION: Learning to Ground and Explain for Synthetic Image Detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2025: 18937-18947.
  • [14] LI T, HUANG Z, WEN H, et al. Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2026: 30299-30311.
  • [15] OpenAI. GPT-5.6 Sol[EB/OL]. OpenAI API model documentation, 2026.
  • [16] LI Y, YANG X, SUN P, et al. Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2020.
  • [17] THIES J, ZOLLHÖFER M, STAMMINGER M, et al. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016. DOI: 10.1109/CVPR.2016.262.
  • [18] THIES J, ZOLLHÖFER M, NIESSNER M. Deferred Neural Rendering: Image Synthesis Using Neural Textures[J]. ACM Transactions on Graphics, 2019. DOI: 10.1145/3306346.3323035.
Reviewer_6EqbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission introduces ManiVid, a unified task for forensic analysis of partially manipulated videos, covering forgery detection, artifact grounding, and anomaly explanation. It contributes ManiVid-38K, a large-scale paired dataset with open-vocabulary manipulation instructions, pixel-level forgery masks, and anomaly explanations, constructed under multiple editing paradigms with diverse editing models, together with ManiVidBench for evaluation and ManiVidLens, an end-to-end framework that couples a multimodal LLM with SAM2 through shared forensic evidence and structured grounding prompts. Experiments show state-of-the-art grounding and explanation on ManiVidBench, with detection performance comparable to dedicated video detectors.

Critical strengths
  • ManiVid-38K is the first video manipulation dataset that jointly provides paired open-vocabulary manipulations, pixel-level forgery masks, and anomaly explanations, supported by careful source-authenticity screening and human verification.
  • ManiVidLens couples a multimodal LLM with SAM2 through shared forensic evidence and distilled grounding prompts, achieving unified detection, pixel-level artifact grounding, and anomaly explanation with design choices supported by ablations.
Critical weaknesses
  • The three task protocols share a single adapter, and the component ablation shows that the grounding modules degrade explanation performance, leaving this cross-task interference unmitigated and unexplained.
  • The contribution of the frozen components, including the forensic encoder, the MLLM backbone, and SAM2, is not quantified by substitution or scale ablations, and the scope of the reported inference time remains undefined.
Questions for authors
  • Would task-specific adapters help reduce the cross-task interference of the shared LoRA?
  • How would the cross-task interference shown in Table 6 be mitigated?
  • Does the inference time in Table 5 cover all three generations and the SAM2 propagation?
  • Would component-level ablations on AIDE, the MLLM backbone, and SAM2 help quantify their individual contributions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6Eqb on Cross-Task Interference, Inference Time, and Component Ablations

We thank Reviewer 6Eqb for carefully reading our paper and for the constructive feedback. We appreciate the opportunity to address these concerns.

【Critical Weaknesses】

W1. Shared adapter and cross-task interference.

Please see our joint response to Q1–Q2 under 【Questions For Authors】 for details.

W2. Contributions of the frozen components and scope of inference timing.

Under 【Questions For Authors】, please see Q3 for the scope of inference timing and Q4 for the component ablations.

【Questions For Authors】

Q1–Q2. Task-specific adapters & mitigating the interference observed in Table 6.

(1) Task-specific adapters. As our title, ManiVid: Unified and Explainable Manipulation Analysis of Videos via MLLMs, emphasizes, unifying artifact grounding, anomaly explanation, and forgery detection is a central goal of our work. We use one architecture and train a single LoRA [1] adapter across all three tasks to keep the training procedure simple. With this shared adapter, ManiVidLens achieves the best overall results among the methods evaluated on ManiVidBench for both grounding and explanation (Tables 2 and 3). It also outperforms most comparison methods on detection, reaching 91.4% accuracy and 91.3% F1 (Table 4). Task-specific adapters could give each task more flexibility. At the same rank, however, they would require three separate sets of adapter parameters to train and maintain. Given the performance of the shared adapter across all three tasks, we chose it to keep the model compact and training straightforward.

(2) Mitigating the interference. We acknowledge the small cross-task trade-off in Table 6. Compared with Base, adding FER and PDM together improves grounding by 7.1 points in mIoU and 7.4 points in J&F [2]. The two explanation metrics drop by less than one point on average, while detection performance remains essentially unchanged. Actually, we designed FER and PDM primarily to improve grounding. The gains on both grounding metrics show that the modules serve this purpose, while the full model retains nearly all of the baseline performance on explanation and detection. Given the size of the grounding gains relative to these small changes, we retain both modules in the unified model.

Q3. Inference timing in Table 5.

Table 5 reports the total time per video to run grounding, explanation, and detection. This includes all three task-specific generation calls and SAM2 [3] mask propagation. Timing starts with video decoding and ends when the results are written. It includes preprocessing, model generation, mask propagation, postprocessing, and result output. Model loading, warm-up, and evaluation metric computation are excluded.

SettingGroundingExplanationDetectionTotal Inference Time
3 Frames7.1074.3222.40113.83
5 Frames7.858‾\underline{7.858}5.297‾\underline{5.297}2.825‾\underline{2.825}15.98‾\underline{15.98}
7 Frames8.4486.0873.18517.72
9 Frames8.9566.7753.46919.20

Q4. Component ablations for AIDE, the MLLM backbone, and SAM2.

We kept AIDE [4] fixed and evaluated two sizes of the Qwen3-VL [5] backbone (2B and 4B parameters) and two SAM2 variants (Small and Large). All four configurations retain FER and PDM and use the same data split, training budget, and optimization settings.

Qwen3-VLSAM2GroundingGroundingExplanationExplanationDetectionDetection
ParametersVariantmIoUJ&FR-LCSSAccF1
4BL0.4070.4220.5830.7800.9140.913
2BL0.404‾\underline{0.404}0.421‾\underline{0.421}0.5760.7640.9050.902
4BS0.4010.4150.580‾\underline{0.580}0.775‾\underline{0.775}0.911‾\underline{0.911}0.909‾\underline{0.909}
2BS0.4030.4180.5740.7610.9080.905

Our main configuration, Qwen3-VL-4B with SAM2-L, achieves the highest scores on all six metrics. For grounding, SAM2-L improves both mIoU and J&F over SAM2-S with either Qwen3-VL backbone. With the 4B backbone, the scores are 0.407 mIoU and 0.422 J&F for SAM2-L, compared with 0.401 and 0.415 for SAM2-S.

The 4B backbone improves ROUGE-L [6] and CSS [7] over the 2B backbone with either SAM2 variant. The 4B + S configuration also exceeds 2B + L on both explanation metrics, reaching 0.580 ROUGE-L and 0.775 CSS, compared with 0.576 and 0.764. Detection accuracy and F1 are higher with the 4B backbone for both SAM2 variants. Across the four configurations, accuracy ranges from 0.905 to 0.914, with 4B + L achieving 0.914 Acc and 0.913 F1.

AuthorsOpenReview ↗

Response to Reviewer 6Eqb on Cross-Task Interference, Inference Time, and Component Ablations

【References】

  • [1] HU E J, SHEN Y, WALLIS P, et al. LoRA: Low-Rank Adaptation of Large Language Models[C]//International Conference on Learning Representations. 2022.
  • [2] YUAN H, LI X, ZHANG T, et al. Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos[R]. arXiv:2501.04001, 2025.
  • [3] RAVI N, GABEUR V, HU Y T, et al. SAM 2: Segment Anything in Images and Videos[C]//International Conference on Learning Representations. 2025.
  • [4] YAN S, LI O, CAI J, et al. A Sanity Check for AI-Generated Image Detection[C]//International Conference on Learning Representations. 2025.
  • [5] BAI S, CAI Y, CHEN R, et al. Qwen3-VL Technical Report[R]. arXiv:2511.21631, 2025.
  • [6] LIN C Y. ROUGE: A Package for Automatic Evaluation of Summaries[C]//Text Summarization Branches Out. Association for Computational Linguistics, 2004: 74-81.
  • [7] XU Z, ZHANG X, LI R, et al. FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models[C]//International Conference on Learning Representations. 2025.

Thanks for the response. I will keep my scores.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A gentle reminder to read our rebuttal

Dear Reviewers,

Thank you for your time and effort in reviewing our submission. We are extremely sorry for being late in responding to your comments. As you can see, we had to conduct several experiments, which took quite some time. This also made our responses very lengthy, and we are extremely sorry for that.

With the discussion period drawing to a close, we kindly invite you to review our responses. We have conducted additional experiments you requested. Please let us know if you have any remaining questions or concerns.

If our responses have adequately addressed your feedback, we would be very grateful if you would consider raising your score.

--

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.