acceptodds
Withdrawn from ICLR 2027

HuRF: Human-Centric Representation Learning for Generalizable Digital Human Video Forensics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Native digital humans, whose appearance, motion, and identity are synthesized or reconstructed end-to-end, pose growing challenges to media authenticity. Existing AI-generated video and face-centric deepfake detectors often rely on generator-specific visual artifacts, temporal irregularities, or editing traces, which do not transfer reliably across heterogeneous digital human generation mechanisms. We therefore seek transferable forensic evidence from the human subject itself, which provides a shared representational basis across different generation mechanisms. Based on this insight, we propose HuRF, a human-centric representation learning framework for generalizable digital human video forensics. HuRF leverages intermediate representations from a pose-oriented human foundation model, retaining dense RGB appearance alongside fine-grained awareness of human structure. It further distills region-level semantic relations from a human parsing teacher to suppress background interference, and adaptively calibrates temporal evidence across generation mechanisms, reducing reliance on unreliable motion-naturalness cues. In this way, the detector focuses on transferable evidence from the human subject rather than generator- or background-specific shortcuts. To systematically evaluate this problem, we construct HuDigivid, containing over 35K videos from 19 known generators, together with nearly 1K collected in-the-wild digital human videos with unknown generators. Extensive experiments reveal distinct failure modes of existing detector families and show that HuRF achieves the strongest overall performance under cross-generator, cross-paradigm, and in-the-wild evaluations while remaining robust to common video degradations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Synthetic Human Video Detection, Human-Centric Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vwu6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HuRF for detecting synthetic digital-human videos using pose-oriented representations, human-part relation distillation, and a gated temporal adapter. It also introduces HuDigivid, a dataset covering 19 generators across three paradigms, and reports strong cross-paradigm and in-the-wild results.

Critical strengths
  1. The evaluation covers different generation mechanisms. HuDigivid includes implicit-driven, explicitly guided, and 3D-rendered digital humans, as well as two collected in-the-wild subsets. Training only on implicit generation provides a useful test of transfer to unseen paradigms. HuRF achieves a mean AUC of 0.9160 in this setting, compared with 0.7624 for the strongest reported baseline.
  2. Several experiments support the complete framework. The representation comparisons, component ablations, external DigiFakeAV evaluation, and degradation tests examine different aspects of the method. The full model substantially improves over the same pose-oriented classification baseline. Its in-the-wild results also suggest that the gains extend beyond controlled generation settings.
Critical weaknesses
  1. The evaluation leaves authenticity confounded with source and identity. Table A3 lists JD as the source of all controlled-test real videos, while synthetic videos use reference sources including HDTF, TikTok, DFDC, and official avatar assets. Holding out generators and real-video platforms tests distribution shift, but does not rule out reliance on source-dependent appearance, composition, or processing cues.

    The paper does not clearly establish that identities and source videos are disjoint across training and testing, nor does it provide source-matched real/fake controls. The JD in-the-wild subset provides some platform overlap between classes, and the external evaluation is useful, but neither fully resolves this concern. Attention visualizations alone cannot show that background shortcuts have been removed. This is particularly relevant because reducing such shortcuts is a central motivation for HuRF.

  2. The ablations do not isolate the benefit of adaptive temporal calibration. TAC extends an existing spatiotemporal adapter, but the ablation removes the entire TAC module. This tests temporal adaptation as a whole, rather than the learned gate specifically. A comparison with the underlying adapter using a fixed gate g=1g=1, with the same backbone and two-stage training schedule, would help establish what the gate adds.

    The large gap between end-to-end and two-stage training is also difficult to interpret without matched optimization budgets and repeated runs. No uncertainty estimates are reported. These are especially relevant for the smaller gains in cross-generator evaluation, where AUC increases from 0.9140 to 0.9164, and on DigiFakeAV, where it increases from 0.8519 to 0.8594.

Questions for authors
  1. Are identities, original recordings, reference images, and driving videos disjoint across training and testing? Please describe the validation split and explain how checkpoints, representation layers, and hyperparameters were selected without using the Shared Test Set.
  2. Do the gains persist in source-matched real/fake comparisons or after controlled foreground/background replacement?
  3. How does HuRF compare with the underlying spatiotemporal adapter using a fixed gate g=1g=1, under the same backbone and training schedule? Do the learned gates vary across generation paradigms as the motivation predicts?
  4. Please report the training budget for each stage and for the end-to-end comparison, along with variability across training seeds. Total trainable parameters and inference cost would also help assess the cost of the temporal modules.
Reviewer_yDULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HuRF addresses generalizable detection of natively generated digital human videos. It adapts intermediate representations from Sapiens2-Pose, introduces human-aware semantic relation distillation (HSRD/HARW), and employs temporal adaptive calibration (TAC) to regulate motion evidence. The authors also construct HuDigivid, containing 35,599 videos across 19 controlled generators and two in-the-wild sources. Experiments evaluate cross-generator, cross-paradigm, and external-dataset generalization, reporting strong performance across heterogeneous generation mechanisms.

Critical strengths
  • Well-motivated human-centric formulation. The paper identifies meaningful differences between native digital human generation, conventional facial manipulation, and general AI-generated video detection. Leveraging pose-oriented representations to retain appearance and structural information is a technically reasonable direction for cross-paradigm forensic learning.

  • Broad evaluation with informative analyses. HuDigivid covers 19 generators across three generation paradigms, including independently collected in-the-wild samples. The cross-paradigm results are particularly promising, and the representation comparisons, component ablations, and degradation experiments provide useful empirical evidence for the proposed framework.

Critical weaknesses
  • Insufficient control of dataset-source confounding in the generalization protocols. Appendix Tables A1–A3 show that real test videos primarily originate from JD Platform Real, whereas generated videos use heterogeneous reference sources, including TikTok, HDTF, CelebV-HQ, and official avatar assets. Consequently, generator-disjoint evaluation does not eliminate differences in identity, content, background, or acquisition pipelines that may correlate with authenticity labels. The in-the-wild BaiduAvatar evaluation likewise lacks matched real videos from the same platform. Without source-matched comparisons or foreground/background controls, the observed cross-paradigm performance cannot be confidently attributed to transferable human-specific forensic evidence rather than residual dataset shortcuts. This directly affects the paper's central generalization claim.

  • Incomplete attribution of the proposed human-centric and temporal mechanisms. The ablations show strong improvements for the complete framework, but do not fully isolate the claimed mechanisms. HSRD without human-aware weighting provides essentially no gain (0.8115 vs. 0.8118 AUC), while TAC introduces substantial spatiotemporal modeling capacity. The paper lacks matched-capacity comparisons against standard temporal adapters with fixed gating, as well as direct evidence that the learned temporal gates adapt appropriately across generation paradigms. Similarly, foreground-focused or background-suppressed baselines are needed to substantiate the proposed human-centric explanation. The results establish the effectiveness of the combined architecture more convincingly than the specific causal interpretation of its components.

Questions for authors
  • Can the authors evaluate source- and identity-matched real/generated videos, particularly using the original driving or reference videos? How does performance change under human-only, background-only, or background-swapped evaluation?

  • Does TAC outperform an otherwise identical spatiotemporal adapter with a fixed temporal gate and matched parameter count? Can the authors report the learned gate distributions across implicit-driven, explicit-guided, and 3D avatar generation to verify the proposed adaptive behavior?

  • How does HuRF compare against matched-capacity non-pose visual backbones and simpler foreground-masking or human-cropping strategies? These controls would clarify the specific contribution of pose-oriented representations and semantic relation distillation.

  • Can the authors provide confidence intervals or multi-seed variability for the main comparisons? In particular, the cross-generator AUC improvement over DGS-Net is only 0.0024, and the external DigiFakeAV improvement over Effort is 0.0075. Are these differences statistically reliable?

Reviewer_mn1nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HuRF targets forensics for native digital humans, videos whose appearance and motion are synthesized end to end rather than locally edited. It builds on a pose-oriented human foundation model (Sapiens2-Pose), distills human-part semantic relations from a parsing teacher (HSRD with human-aware relation weighting), and adds a temporal adaptive calibration module that gates the temporal branch per generation mechanism. It also introduces HuDigivid, a 35K-video dataset spanning 19 generators across three paradigms plus two in-the-wild test sets. HuRF reports the strongest cross-generator, cross-paradigm, and in-the-wild results against 11 baselines.

Critical strengths
  • The dataset is a real contribution. Covering 2D implicit, 2D explicit, and 3D avatar-rendered paradigms with in-the-wild test sets is broader than prior digital-human benchmarks, and the cross-paradigm protocol is a sensible stress test.
  • The appendix zero-shot analysis showing family- and paradigm-dependent transfer rather than uniform failure is honest and useful.
  • The two-stage training ablation (two-stage 0.9160 vs end-to-end 0.7889) is a clear, non-trivial finding, and the component ablation attributes gains to both HSRD+HARW and TAC.
Critical weaknesses
  • The method is an assembly of existing components (a frozen foundation model, LoRA, relational distillation, and a borrowed spatiotemporal adapter), and the central claim that human-centric features are the source of generalization is asserted more than isolated.
  • The strong baselines are competitive under the cross-generator protocol (DGS-Net 0.914, DeMamba 0.909 vs HuRF 0.916), so the headline advantage rests mainly on the harder cross-paradigm and in-the-wild settings, where a dataset the authors built and curated could favor their own design.
  • There is no variance or significance reporting on any table despite one-epoch training.
  • The in-the-wild sets (JDAvatar, BaiduAvatar) are single-platform and of unknown provenance, so the real-world claim rests on two narrow sources.
  • The attribution to human-centric structure is not cleanly supported, since a strong RGB-only Sapiens2-Pose block-32 baseline already reaches 0.8118 and much of the remaining gain comes from TAC, a generic temporal adapter, rather than from the human-semantic distillation that the paper foregrounds.
Questions for authors
  • How much of the generalization advantage is specifically human-centric rather than a property of a strong frozen video representation plus a temporal adapter, and can you isolate this with a non-human foundation backbone under the same pipeline?
  • Can you report variance or significance across runs given one-epoch training?
  • What evidence rules out that the cross-paradigm and in-the-wild advantages reflect curation choices in a dataset you constructed?
  • What are the provenance and generator diversity of JDAvatar and BaiduAvatar, and do results hold on a third independent real-world source?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.