acceptodds
Under review as a conference paper at ICLR 2027

FRAME: A Fully Automated Pipeline for Scalable Multimodal Body Language Datasets

Abstract

Large-scale multimodal datasets are essential for training representations that capture non-verbal communication, yet most are limited by the cost of manual labeling and remain too small to support large-scale models. We introduce FRAME (Fully-automated Retrieval and Annotation of Motion and Expression), a fully automated pipeline for constructing large-scale multimodal datasets from natural dialogue videos, and use it to release POSTURE (Pose, Speech, and Talking-face Utterances for Representation Evaluation), a dataset of 156,020 movie clips with synchronized subtitles, facial expression frames, and body pose sequences. From segmentation to pose and face extraction, every stage of the pipeline runs without human input, unlike most existing multimodal datasets, which rely on manual annotation and consequently reach only a fraction of this scale. We show that this scale translates directly into better representations. Averaged across a zoo of 15 loss functions and pose encoder architectures, models pretrained on POSTURE achieve 6.8× the cross-dataset transfer R@1 of models pretrained on any other single comparable dataset (CAER, MELD, CMU-MOSEI, DFEW) at full size. This advantage does not come at the cost of quality: at matched dataset size, the advantage falls to near-parity (0.98×), confirming that scale, not source-specific quality, drives POSTURE’s advantage. Because construction requires no manual annotation, the same pipeline can be given more source footage to produce proportionally larger datasets at low marginal cost, making POSTURE inherently scalable. We release the dataset, the full extraction pipeline, and evaluation code to support future work on scalable multimodal representation learning. Code for the dataset pipeline can be found through this link: here

Primary area:
datasets and benchmarks
Keywords:
data pipeline, multimodal contrastive learning, pose estimation, representation learning, dataset, contrastive loss, modality gap, emotion recognition, evaluation metrics, InfoNCE

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bRseOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FRAME, an automated pipeline that takes movie videos and timestamped subtitles as input and produces paired text, face-image sequences, and body landmarks.

The resulting POSTURE dataset contains 156,020 clips. The authors train contrastive encoders and compare cross-dataset retrieval with models trained on existing datasets.

The experiments examine whether increasing automatically collected data improves representation learning.

Critical strengths
  1. The pipeline addresses a practical data bottleneck. Collecting text, facial observations, and body landmarks automatically could provide useful data for multimodal representation learning.
  2. The full-size and equal-size comparisons help distinguish the benefit of more data from differences in individual sample quality.
  3. The evaluation covers multiple objectives and encoders, with additional movie-disjoint, compute-matched, and pose-confidence controls.
Critical weaknesses
  1. The value of the pipeline design could be clearer. Could the authors discuss which design choices improve data quality or efficiency beyond straightforward use of existing tools? Comparisons with pipelines designed by different coding agents on the same source videos would help clarify advantages in extraction accuracy, processing speed, or recall of usable clips.
Questions for authors

Please refer to the weakness above.

1 comment

Thank you for the positive assessment and constructive feedback.

  1. Pipeline design and contribution: We agree that the paper should better explain the design choices behind FRAME. Our contribution is not the development of new face or pose extraction algorithms, but the integration of existing tools into a fully automated pipeline that produces synchronized, clip level text, face, and pose data at scale. We will expand the methodology to clarify the rationale behind our key design choices, including subtitle based segmentation, temporal alignment, pose extraction at 6 FPS, and sampling 16 face frames per clip. We will also better explain how these choices balance data quality and computational cost.
  2. Pipeline quality and efficiency: We will provide additional information about the pipeline's extraction quality and computational efficiency, including processing time and the success rate of producing usable multimodal clips, where available. We agree that these measurements would make the practical value of FRAME clearer. We appreciate the suggestion to compare pipelines designed by different coding agents. However, we believe controlled measurements and ablations of our pipeline's design choices would provide a more direct assessment of its effectiveness than comparisons that may depend heavily on agent prompts and implementation choices.
Reviewer_ukx3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors proposed a large-scale body and speech dataset, namely, FRAME. It consists of short video clips from YouTube movies. Then annotated with MediaPipe and speech transcription. The authors then introduced POSTURE as a learning representation framework, trained on the introduced dataset and evaluated on external datasets.

Critical strengths
  • The scale and the nature of the dataset are timely and appreciated.
  • The idea of using self-supervised pre-training on the dataset and using how this helps on external ones is also good.
Critical weaknesses
  • It is not clear to me what the contribution of the paper is beyond gathering 150K video clips. This dataset is not properly validated, and it is not clear to me what kind of bodily behaviors it is targeting.

  • Why and how does the paper propose POSTURE (the representational learning framework)? This contribution is described vaguely, without proper description and motivation

  • What does POSTURE improve on the external dataset? This is not properly quantified or measured.

Questions for authors
  • Please check the three main substantive concerns I raised, which mean that the paper should properly write and introduce the contribution, validate the dataset, as well as report what is actually being measured or investigated.

  • How will the dataset be released and shared?

1 comment

Thank you for the constructive feedback. We appreciate the opportunity to revise our paper.

  1. We agree that the contribution and scope of our dataset POSTURE need to be clarified. Beyond collecting 156K clips, our work introduces a fully automated pipeline for extracting synchronized text, facial expressions, and body pose from natural dialogue videos. POSTURE is intended to support general body-language representation learning rather than target specific predefined behaviors. Regarding dataset validation, our equal-size experiments (Table 3) show that using POSTURE for training achieves comparable cross-dataset retrieval performance to using existing manually annotated datasets. Our pose-confidence filtering experiments further demonstrate the usefulness of the extracted data. We will clarify these contributions and better explain the intended use of the dataset.

  2. We apologize that the motivation and methodology were not sufficiently clear. We aim to use FRAME, the automated dataset construction pipeline, to create large-scale multimodal datasets for contrastive learning like POSTURE. Our representation learning experiments investigate whether large-scale multimodal data can improve cross-dataset transfer through contrastive alignment of text, face, and pose representations. Rather than proposing a new contrastive learning architecture, our focus is on evaluating the dataset across 15 different combinations of loss functions and encoder architectures. We will clarify this distinction and provide a better explanation of our motivation and methodology.

  3. Our experiments quantify POSTURE's effectiveness through cross-dataset retrieval performance. In the full-size setting (Table 2), POSTURE achieves the best R@1 score (except for the values on the main diagonal, which are not a part of cross-dataset transfer because these values are produced by using the same dataset for retrieval and training) in every cross-dataset transfer setting. However, this advantage largely disappears under equal-size conditions (0.98× on average), suggesting that scale is the primary source of improvement. We will conduct additional experiments to properly attribute the improvement in more aspects.

For the dataset, we will release it on google drive.

Reviewer_WUzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FRAME, an automated pipeline for extracting subtitles, face images and body pose sequences from movie clips. The authors use FRAME to build POSTURE dataset, then test its value as training data through cross-dataset retrieval. They train contrastive models on POSTURE, CAER, MELD, CMU-MOSEI and DFEW. In the default setup, the text and face encoders are frozen, while the projection heads and pose encoder are trained. The experiments compare full datasets and equal-sized subsets using different losses and encoders. Further controls use movie-disjoint splits and matched training steps; a separate experiment tests pose-confidence filtering. The results suggest that automatically collected multimodal data are useful for training and that increasing the dataset size can improve transfer.

Critical strengths

(1) The paper addresses a practical problem and offers a valuable multimodal resource at scale: Training models on text, facial expressions and body pose requires large paired datasets. Collecting and aligning these data by hand takes considerable effort. By constructing 156,020 multimodal clips from movie dialogue, the paper demonstrates a feasible route to scaling such resources through automation. The dataset could reduce the data preparation burden for subsequent multimodal representation research. The relevance of the problem and the resource's potential utility are important reasons in favor of acceptance.

(2) The data are useful for training, with a clear alignment method and relatively modest resource requirements: The authors combine existing text and face representations with projection heads, a pose encoder, and contrastive objectives. Dataset construction requires no manual labeling, and the reported training runs were completed on a single workstation. These features suggest relatively modest requirements for annotation effort and training hardware. The approach is easy to understand, and the results at full training-set size show favorable cross-dataset retrieval performance. The matched-size results also suggest that the automatically constructed data can provide useful training signals. These results provide initial evidence that the dataset is useful for training and that a straightforward alignment method can achieve meaningful transfer.

(3) The controls target relevant alternative explanations and offer useful observations about scale and filtering: The authors investigate factors beyond dataset size through matched-size comparisons, alternative pretraining configurations, disjoint-movie splits, and matched optimization steps. The confidence-filtering experiments also explore the trade-off between reducing noise and retaining diversity. These controls help assess how dataset size and filtering affect transfer.

Critical weaknesses

(1) Methodological novelty is limited, and the dataset's distinctive value needs clearer explanation: The data pipeline mainly combines subtitle-based segmentation with existing face and pose extraction tools. The model uses pretrained encoders, projection heads and established contrastive objectives. The overall methodological contribution is incremental. The paper needs to make a clearer case for POSTURE as a distinct resource: which data gaps it fills, which research problems call for dialogue text, faces and body pose together, and what these data enable beyond the usual benefits of scale. The current experiments mainly show retrieval gains from more training data, while these specific research uses remain insufficiently explained.

(2) Important validation experiments are missing or insufficiently reported, limiting attribution of the gains and confidence in the conclusions. The experiments demonstrate overall gains, but three aspects remain insufficiently validated:

The contributions of the trainable components are unclear. The default setup trains a pose encoder in addition to the projection heads, but the trainable parameter counts, initialization, and architectural details are incompletely specified. Although the paper mentions a random-initialization baseline with R@1 of approximately 0.0003, it does not clearly provide a task-matched comparison before and after training under fixed pretrained backbones, or controlled ablations of the projection heads and pose encoder. This makes the incremental effects of the individual trainable components difficult to assess. Using frozen backbones is reasonable; the concern is that these incremental contributions have not been sufficiently quantified.

The incremental contributions of the modalities lack direct evidence. Results are not sufficiently broken down into text–face, text–pose, and face–pose retrieval, and the paper does not report controlled modality-removal experiments involving retraining and evaluation on the same retrieval task. Replacing encoders or changing objectives does not directly establish whether each modality contributes additional information, particularly whether body pose provides the intended incremental signal.

The matched-size comparison covers only one dataset size. Table 3 compares all training sources at 9,222 clips, whereas the scaling experiment varies only POSTURE's size. It remains unclear whether the datasets offer comparable utility across multiple matched sizes. Learning curves at shared sizes, for example from 1,000 to 5,000 clips alongside the existing 9,222-clip result, would help assess this in representative configurations. Fixed evaluation conditions, explicit sampling rules, and uncertainty estimates across repeated runs would help determine whether the conclusion depends on a particular size or sampled subset.

Together, these gaps limit the central interpretation: which components and modalities produce the gains, how stable those gains are, and whether the automatically constructed data offer comparable utility at matched sizes.

Questions for authors

Have the authors conducted any of the component ablations, modality-contribution experiments, or comparisons across multiple matched dataset sizes discussed above? If these experiments were omitted for space, please provide their key settings, numerical results, and variability across repeated runs. If they have not been conducted, please clarify the scope of the conclusions supported by the current evidence and consider validation in representative configurations. Please also state whether Transfer includes DFEW and explain the discrepancies between the stated means and the reported values in Tables 2–3.

The paper would be much stronger for acceptance if the authors clarify and revise the evaluation protocol, provide targeted ablation studies and results from repeated runs, and verify the matched-size conclusion under comparable experimental conditions.

1 comment

Thank you for the careful review. Changes will appear in the revised PDF.

  1. Transfer score, DFEW, and Tables 2–3: Thank you for catching this. The captions are wrong and will be corrected. Transfer is the mean R@1 over the other three of the four 3,500-clip held-out sets (POSTURE, CAER, MELD, CMU-MOSEI). DFEW's held-out set has 2,341 clips and is not comparable, so it is excluded from every mean except DFEW's own row (e.g., POSTURE: (0.0411+0.0292+0.0133)/3 = 0.0279). Our headline advantage is POSTURE's transfer score divided by the average transfer score of CAER, MELD, and CMU-MOSEI, so our abstract and Section 4.2 wording ("any single dataset," "best comparison dataset") will be corrected. We will also state the figures with DFEW as a fourth comparison dataset: 5.6× at full size and 0.78× at matched size. We will fix the captions and Section 4.1 and make the abstract and introduction consistent on which datasets the headline ratio covers.

  2. Matched size at one size only: We agree that one matched size limits the conclusion. We will scope our claims of comparable quality to the size tested (9,222 clips) and state explicitly that other matched sizes were not evaluated.

  3. Modality contributions: We will state in the text that the retrieval signal is concentrated in face–pose. When transferred to other datasets, face→pose retrieval is far above chance (about 288× for the full-size POSTURE model), while text pairs are only about 1–5× above chance for every dataset, including MELD and CMU-MOSEI, which have human transcripts, so we claim no text benefit. We did not run modality-removal retraining, and the revision will say so.

  4. Trainable components: We will add the pose encoder's architecture (a two-layer transformer trained from scratch) and initialization. We will also describe the lack of projection-head ablations as a limitation.

  5. Novelty and distinct value: Our contributions are the FRAME pipeline and the POSTURE resource it produces, along with a controlled evaluation of what drives transfer. FRAME is, to our knowledge, the first pipeline to produce synchronized dialogue text, face, and body pose at clip level with no manual labeling at any stage, which is what lets POSTURE reach 156,020 clips (Table 1). The evaluation is also new: cross-dataset transfer at full and matched size, with movie-disjoint and compute-matched controls, shows that the advantage comes mainly from scale. At matched size, POSTURE is on par with the average of CAER, MELD, and CMU-MOSEI (0.98×) but below DFEW (0.78× when DFEW is included). Curated data is worth more per clip, and automation's advantage is scale. We will revise the abstract, introduction, and contribution list to say this.

  6. Scope and variability: Main results use three seeds (two for the compute-matched control). Pooled advantage ratios have 95% intervals of 5.8–7.9 (full size) and 0.91–1.06 (equal size), though these treat runs sharing datasets and held-out sets as independent and are likely optimistic. We will state this in the paper and narrow our conclusions to: transfer gains are driven mainly by scale, and at the matched size tested, POSTURE's per-clip utility is on par with the average of three existing datasets and below DFEW's.

We also found the other suggestions valuable, including matched-size evaluation at additional sizes, modality-removal ablations, and component-level controls. These are not part of this revision, but they would strengthen the work, and we plan to pursue them as we continue to improve the paper and the dataset going forward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.