acceptodds
Desk rejected at ICLR 2027

Measuring and Mitigating Automated Personal Information Extraction with Video Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Brief appearances of names, addresses, license plates, and account identifiers in public videos are easy for uploaders and human reviewers to miss, yet a video language model (VLM) can repeatedly search for and recover them. We investigate how effectively current VLMs can turn such fleeting exposures into verifiable personal-information records. To do so, we construct Video-PIE, which defines ten video-observable information types and contains 170 timestamp-annotated YouTube and TikTok videos, together with a precision-oriented audit of 5,346 additional public videos. We evaluate three proprietary and open-source VLM families using direct prompting and a multi-stage procedure that separates temporal screening, type localization, and detail recovery. The decomposition matters most when temporal search is difficult: on 20–40 minute videos, it raises recall from 0.13 to 0.63 for Qwen2.5-VL and from 0.15 to 0.57 for InternVL-2.5. We further introduce AutoPurify, which converts the localized audit trace into spatial masks and optimizes bounded perturbations jointly over an ensemble of pretrained visual backbones. Rerunning the extractors on the protected videos shows a 55–72% reduction in residual recall under the tested perturbation scopes. These results show that fleeting exposure is not a reliable form of privacy and that mitigation should be evaluated by what a temporally aware extractor can still recover after protection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Video Large Language Models, Personal Information Extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L6rkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether video language models can recover personal information briefly exposed in public videos. It introduces VIDEO-PIE, with 170 annotated videos and a precision audit of 5,346 additional videos, and compares direct prompting with multi-stage extraction. It also proposes AutoPurify, which uses localized exposures to guide adversarial perturbations. Evaluation covers three model families and two platforms, showing substantial extraction gains from task decomposition and reduced extraction after protection. Visual distortion is evaluated on 100 frames.

Critical strengths
  1. A useful empirical privacy study. Timestamped annotations and human verification provide concrete evidence of recoverable information across models and platforms. The distinction between the annotated benchmark and the larger precision audit makes the scope of the evidence clear.

  2. Substantial gains from multi-stage extraction. For 20–40 minute videos, Qwen recall increases from 0.13 to 0.63 under matched video access. This is a consequential finding for assessing exposure in longer videos.

Critical weaknesses
  1. The protection results need reconciliation. Table 5 implies different original baselines for the paired Frame and ROI evaluations. For Qwen Direct, the reported recall and changes imply original recalls of 0.13 and 0.18, respectively. The claimed 55–72% recall reduction also needs an explicit aggregation rule: the Qwen Multi/Frame entry implies approximately 15%. These discrepancies affect a central quantitative claim.

  2. Essential protection settings are unspecified. Appendix D provides pseudocode but omits the exact backbone ensemble, representation distance, perturbation budget, step size, and iteration count. These choices directly affect protection efficacy and visual distortion, making Tables 5–6 difficult to reproduce from the paper.

Questions for authors
  1. Can you provide the original baselines underlying Table 5, reconcile the Frame/ROI discrepancies, and explain how the 55–72% reduction is calculated?

  2. What exact backbones, distance function, perturbation budget, step size, and iteration count were used? Please also specify the evaluation videos and aggregation procedure for Tables 5–6.

Reviewer_ch9KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Brief appearances of personal information in public videos are often overlooked by uploaders, yet Video LLMs can repeatedly search for and recover them. This paper introduces VIDEO-PIE to measure this privacy risk and proposes a multi-stage extraction pipeline that separates temporal screening, type localization, and detail recovery. It further introduces AUTOPURIFY, which applies localized perturbations to reduce information recovery. Experiments show that the extraction pipeline improves recall on long videos, while AUTOPURIFY reduces residual recall across the tested models.

Critical strengths

1 A concrete privacy problem. This paper moves beyond general video understanding and examines whether briefly exposed personal information can be recovered automatically, providing a more direct assessment of privacy risks in public videos.

2 Fine-grained evaluation. VIDEO-PIE evaluates exposure timing, information type, and recovered content, making it possible to examine where an extraction system succeeds or fails.

3 Effective extraction and mitigation. Experiments show that decomposing the extraction task substantially improves long-video recall, and that the proposed perturbations reduce information recovery by the tested extractors.

Critical weaknesses

1 The extraction comparison is not sufficiently controlled. The multi-stage pipeline makes several specialized calls and revisits candidate clips, whereas direct prompting produces one answer. This makes it difficult to determine whether the improvement comes from task decomposition or simply from a larger inference budget.

2 The robustness of AUTOPURIFY is insufficiently evaluated. Small perturbations may lose their protective effect after compression, resizing, or denoising. The current results do not establish whether the protection survives these common transformations or transfers to independent OCR systems and adaptive extractors.

3 The utility evaluation does not establish human privacy protection. FLIP measures visual distortion rather than the readability of personal information. A perturbation may reduce model extraction while leaving a name or address readable to a person, so lower residual recall alone does not establish that the sensitive content has been concealed.

4 No comparison against existing protection baselines. VPA-Guard is benchmarked only against its own no-retrieval-style variants and opaque masking. Established visual-desensitization / privacy-cloak lines (adversarial perturbations, cloaking, learned blur/inpainting meant to defeat downstream models) are neither reproduced as external baselines nor compared,

Questions for authors

1 Could the authors compare multi-stage extraction with direct prompting and sliding-window baselines under the same total inference budget? 2 Does AUTOPURIFY remain effective after standard video compression and resizing, and against an independent OCR system or an adaptive extractor? Can humans still read the protected information? 3 Could the authors report annotation agreement and separate exact-content recovery from carrier-description recovery? 4 How does VPA-Guard compare against published visual-desensitization / privacy-cloak defenses

Reviewer_QKEFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a multi-stage pipeline and VIDEO-PIE benchmark for extracting briefly exposed personal information from public videos with Video LLMs. Decomposing extraction into screening, localization, and recovery substantially improves long-video recall—for example, Qwen2.5-VL rises from 0.13 to 0.63. It also proposes AUTOPURIFY, which protects sensitive regions with bounded adversarial perturbations, reducing residual extraction recall by 55–72%, though without a formal privacy guarantee.

Critical strengths
  1. The paper clearly motivates the risk that automated Video LLMs can systematically search long videos for briefly exposed personal information.
  2. The multi-stage pipeline substantially improves recall and precision on long videos, providing convincing evidence for improved practical extractability.
  3. Extractions must be temporally localized, typed, and grounded rather than merely flagging possible private information.
  4. AUTOPURIFY is assessed by rerunning extraction on protected videos and measuring residual extraction performance.
Critical weaknesses
  1. The multi-stage extraction pipeline is effective but largely relies on hierarchical decomposition rather than a fundamentally new Video LLM method. The paper would be stronger if framed around the benchmark, privacy threat model, and empirical impact of temporal decomposition.
  2. As Video LLMs continue to evolve, this limitation may be addressed implicitly by stronger models, potentially rendering the paper’s multi-stage extraction pipeline obsolete.
  3. Only 170 videos have exhaustive annotations, and the paper lacks inter-annotator agreement statistics for exposure presence, category, and temporal boundaries. This is especially important given the subjective definition of unintended exposure.
  4. The proposed TCS metric requires clearer motivation, sensitivity analysis for its parameters, comparison with standard metrics such as temporal IoU or boundary error, and justification for its dependence on video duration.
  5. The paper shows promising residual-risk reductions in AUTOPURIFY generalization, but its effectiveness depends on the evaluated extractors, backbones, perturbation settings, and transferability. Evaluation on unseen extraction pipelines and inference configurations is needed to establish broader robustness.
Questions for authors
  1. How many ground-truth exposure events exist per category, platform, duration stratum, and carrier type? What are pre-adjudication agreement values for presence, category, transcription, and temporal boundaries?
  2. What value of α is used in TCS? Is TCS clipped to [0,1], and how are duplicate/multiple intervals matched?
  3. What aggregate calculation yields the stated 55–72% residual-recall reduction?
  4. Does protection survive recompression, resizing, crop-and-zoom, a new OCR engine, and a held-out VLM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.