- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies local manipulation analysis in general videos, covering forgery detection, manipulated-region localization, and anomaly explanation. The authors construct ManiVid-38K, which contains about 19K original-manipulated video pairs with authenticity labels, region masks, and textual explanations. They also propose ManiVidLens, which combines AIDE, an MLLM, and SAM2. FER injects forensic features into the language and segmentation models, while PDM converts the MLLM's hidden states into segmentation prompts. The three tasks share model parameters and evidence but use separate task prompts.
Evaluation on ManiVidBench, a held-out set of 1K video pairs from ManiVid-38K, shows improvements in localization and explanation metrics over the compared methods.
Critical strengths
-
The research problem has practical value. Answering whether a video was manipulated, where it was manipulated, and what anomalies are present provides more information for inspection than a binary authenticity decision alone.
-
The dataset is a useful resource. It covers addition, removal, replacement, color alteration, material modification, and portrait editing, and provides paired original and manipulated videos together with region masks and anomaly explanations. These annotations support training and evaluation across the three tasks.
-
The method shows a clear improvement in manipulation localization on ManiVidBench. Its overall mIoU is 0.407, compared with 0.336 for Sa2VA. The module ablations also show localization gains from FER and PDM, providing some experimental support for using forensic features and language-derived prompts to assist segmentation.
Critical weaknesses
-
The method relies on a small number of sampled frames for forgery detection and anomaly explanation, but the paper does not establish whether these frames cover the period in which manipulation occurs. In particular, the experiments do not show whether the evidence supplied to the MLLM and SAM2 is sufficient when manipulation appears only briefly or late in a video. The frame-count ablation reports average performance but does not directly address this coverage problem.
-
The MLLM hidden states at [BBOX1] and [BBOX2] are projected through MLPs into SAM2's prompt space. Whether the resulting vectors provide useful geometric information needs more direct validation. In particular, the supervision targets are corner embeddings averaged across multiple frames, and the same predicted prompts are then applied to each frame. It remains unclear whether this representation preserves the positional information needed for individual frames, especially when the target moves.
-
The method's key designs involve organizing, injecting, and distilling embeddings, but the reasons for these specific choices are supported mainly by intuitive explanations. For example, averaging corner embeddings across frames is assumed to provide useful geometric guidance, yet this assumption is not tested separately. The overall FER and PDM ablations show how performance changes when entire modules are added, but do not adequately establish whether their internal components work as described. The connection between the specific designs and the problems they are intended to solve therefore needs stronger evidence.
-
The dataset and experiments focus on short videos, with source videos limited to 10 seconds. Longer manipulated videos, including videos exceeding one minute, are also relevant application scenarios. The paper does not evaluate whether its conclusions extend to longer videos, leaving the dataset's coverage of these scenarios and the method's practical scope unclear.
-
The method relies on AIDE features and several learned mappings: from AIDE to the MLLM, from AIDE to SAM2, and from the MLLM to SAM2. I am concerned about whether these features and mappings remain useful when the data distribution changes. Results grouped by generator and tests under video degradation provide some evidence, but do not establish transfer to generators excluded from training or to external datasets. The current evaluation does not sufficiently address this concern.
Questions for authors
Please refer to Weaknesses
8 comments
Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization
W2. Frame-specific geometric information in the predicted SAM2 prompts for moving targets.
The geometric prompts are designed to give SAM2 a coarse spatial prior about where the manipulated target is likely to appear across the conditioning frames. For the object- and region-level manipulations considered in our task, the aim is to focus segmentation on a relevant part of the scene within the full frame. This cue only needs to provide an approximate spatial reference; the mask decoder determines the target's precise location and boundaries in each frame using that frame's visual features.
As described in Appendix D.3, during training the frozen SAM2 prompt encoder maps the ground-truth bounding box in each valid conditioning frame to two embeddings, one for the top-left corner and one for the bottom-right. We average the top-left embeddings across valid frames and do the same for the bottom-right embeddings. These two averages serve as supervision targets for the projected MLLM hidden states at [BBOX1] and [BBOX2], respectively. The MLLM thus predicts one pair of geometric prompts per video. This aggregation provides a shared spatial reference derived from the target's locations in the sampled frames.
For a moving target, SAM2 combines this shared spatial prior with the semantic prompt and the visual features of each conditioning frame to predict its mask. The geometric prompts provide soft spatial guidance, while SAM2 retains access to the full frame's visual features. The decoder can therefore produce masks at different positions as the target moves. SAM2 then propagates the initialized target state to the remaining frames. Sharing the geometric prompts serves this coarse guidance role without requiring the MLLM to generate a separate box for every frame.
With bounding boxes derived from per-frame ground-truth masks, SAM2-Hiera-Large achieves 84.55% mIoU on the 1,000 manipulated videos in ManiVidBench. This shows that SAM2 can use box-derived geometric prompts to localize manipulated regions, supporting our choice of its prompt space as the target for geometric distillation.
W3. Evidence for the two forensic evidence-injection pathways in FER.
We evaluated FE2SAM and FE2VLM separately and together using the same Qwen3-VL-4B and SAM2 Hiera-L backbones, with PDM disabled and the training conditions held constant. FE2SAM injects forensic features into SAM2's spatial features for mask localization. FE2VLM supplies forensic evidence to the MLLM as additional visual tokens. The table reports the same metrics as the main paper.
| Active FER pathways | mIoU | J&F | R-L | CSS | Acc | F1 |
|---|---|---|---|---|---|---|
| Neither (Base) | 0.336 | 0.348 | 0.590 | 0.791 | 0.915 | 0.914 |
| FE2SAM only | 0.592 | 0.798 | 0.903 | 0.902 | ||
| FE2VLM only | 0.367 | 0.379 | 0.906 | 0.905 | ||
| Both (full FER) | 0.378 | 0.389 | 0.598 | 0.808 |
FE2SAM alone reaches 0.372 mIoU and 0.383 J&F, compared with 0.336 and 0.348 for Base and 0.367 and 0.379 for FE2VLM alone. These results support injecting forensic evidence directly into SAM2's spatial features to improve localization. FE2VLM alone reaches 0.597 ROUGE-L and 0.807 CSS, compared with 0.590 and 0.791 for Base and 0.592 and 0.798 for FE2SAM alone. Supplying forensic evidence to the MLLM improves the agreement between generated explanations and the reference annotations.
Combining both pathways achieves 0.378 mIoU, 0.389 J&F, 0.598 ROUGE-L, and 0.808 CSS, the highest grounding and explanation scores among these four configurations. The component ablations therefore support FER's design of injecting forensic evidence into both the segmentation and language models.
Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization
W4. Applicability to videos longer than the 10-second source clips, including videos over one minute.
(1) Clip lengths in existing datasets. We limited source videos in ManiVid-38K to 10 seconds or less (Appendix C.1.1). Several video forensics datasets in Table 1 use clips of similar or shorter duration. DFDC [2] uses 10-second clips for its competition training, validation, and test sets. VideoSham [3] trims source videos to an average of 8 seconds before editing. Each clip in VideoFACT’s three public standard-manipulation subsets [4] is 1 second long (30 frames). FaceForensics++ [5] reports 509,914 frames across 1,000 original clips, or about 510 frames per clip. In the related task of instruction-guided video editing, the official OpenVE-3M dataset card [6] reports 65–129 frames per clip. Our focus on short clips is consistent with this practice.
(2) Computational cost of minute-long videos. Our two-stage training already uses eight NVIDIA RTX PRO 5000 GPUs with BF16 precision, a microbatch of two samples per GPU, and four steps of gradient accumulation (Appendix E). With the default five conditioning frames, inference across all three tasks takes an average of 15.98 seconds per video (Table 5). Even if we keep the MLLM input at five frames, extending a 10-second clip to one minute at the same frame rate gives SAM2 six times as many frames to process during mask propagation. Maintaining the same temporal sampling density would also require either more conditioning frames, which increases GPU memory use, or processing the video in segments, which increases runtime. Systematic training and evaluation on minute-long videos would exceed our current computing budget.
W5. Transfer of AIDE features and learned mappings to unseen generators and external datasets.
ManiVidLens uses a frozen AIDE [7] encoder to extract forensic features from each conditioning frame. For detection, FE2VLM projects these features into the MLLM input space, where the model uses them alongside the video’s visual tokens to judge authenticity. We evaluated the same ManiVidLens checkpoint trained on ManiVid-38K directly on VideoSham [3] and FaceForensics++ [5], without fine-tuning on either dataset. We tested standalone AIDE on the same videos.
(1) VideoSham. We evaluated all 352 real–fake pairs in the public release (704 videos), covering all six attack types. Results are video-level percentages. Each attack row reports the percentage of manipulated videos correctly detected.
| Metric | Liu et al. [8], via [3] | AIDE [7] | ManiVidLens |
|---|---|---|---|
| Real recall | 56.66 | 97.16 | |
| Attack 1: Add an entity or subject | 2.90 | 62.32 | |
| Attack 2: Remove an entity or subject | 52.58 | 5.06 | |
| Attack 3: Modify the background or color | 11.11 | 62.96 | |
| Attack 4: Replace or add text | 0.00 | 42.22 | |
| Attack 5: Repeat, drop, or reverse frames, or change playback speed | 1.69 | 32.20 | |
| Attack 6: Replace or add audio | 0.00 | 36.96 | |
| Mean recall across the six attack types (equal weight per type) | 3.46 | 47.04 | |
| Fake recall (all manipulated videos) | 3.69 | 47.73 | |
| Overall accuracy | 42.62 | 57.10 |
AIDE and ManiVidLens both used RGB input only. Attack 6 therefore records their visual predictions on videos with edited audio. The Liu et al. column is derived from the counts in Table 4 of [3], which evaluated 826 videos.
Standalone AIDE correctly identified 97.16% of real videos but detected only 3.69% of forgeries. ManiVidLens detected 47.73% of forgeries while correctly identifying 66.48% of real videos. The gap between real and fake recall narrowed from 93.47 to 18.75 percentage points, and overall accuracy rose from 50.43% to 57.10%. The gains covered several visual edits, with recall above 62% for both entity additions and background or color changes.
Response to Reviewer SxGG on Temporal Coverage, Model Design, and Cross-Dataset Generalization
We thank Reviewer SxGG for carefully reading our paper and for the constructive feedback. We appreciate your comments on the value of evaluating detection, grounding, and explanation together and on the localization gains observed on ManiVidBench.
【Critical Weaknesses】
W1. Coverage of brief or late manipulations by the sampled frames.
(1) Task setting. ManiVid-38K studies edits to objects or local regions in short videos. These edits typically persist while the target region is visible. For targets that appear early and remain in view, early observations provide evidence of the edit; retaining these observations also helps when a target later leaves the scene. We therefore use a small set of temporally spaced frames and check whether it captures an annotated edit in each video.
(2) Sampling protocol. As described in Appendix E.1.2, we place 30 candidate frames uniformly along the full video timeline and use the first five decodable candidates as the model input. These frames retain the spacing of the candidate grid and generally span a longer interval than the first five consecutive frames of the original video. Training also uses five temporally ordered frames, with random temporal patterns and sampling strides of {4, 6, 8, 10}. Inference thus retains the five-frame, spaced-input format used during training. The ordered frames provide local temporal context for comparing the target's appearance and position across the observed time points. For grounding, these conditioning frames initialize SAM2, which propagates masks across all physical frames of the video.
Sampling a limited number of frames is common in video MLLMs. For example, Video-MME [1] evaluates Video-LLaVA with 8 frames and LLaVA-NeXT-Video with 32 frames. Sa2VA [14], which also combines an MLLM with SAM2, uses the first five consecutive video frames as its default MLLM input.
(3) Coverage on ManiVid-38K. We agree that Table 5 reports average performance at different frame counts and that temporal coverage calls for a direct check. For every manipulated video in ManiVid-38K, we compared the indices of the five conditioning frames actually supplied to the model with the frame-level ground-truth masks. We counted a video as covered if at least one of those five frames had a nonempty mask. Every manipulated video met this criterion (100% video-level coverage). Thus, in our experiments, the model saw at least one frame containing an annotated manipulation in every manipulated sample.
(4) Full-frame inspection. For brief or late edits that fall outside these five observations, the MLLM can also inspect all frames by processing the video in segments or sliding windows. The five-frame setting is the default used in our reported experiments. Full-frame inspection brings every frame into the MLLM at least once, at higher inference cost. This removes gaps caused by frame selection; detection accuracy still depends on whether the model recognizes the manipulation cues.