acceptodds
Withdrawn from ICLR 2027

HeroFrame-Bench: Reference-Anchored Evaluation via Rubric–Ranking Co-Evolution for Movie Hero Frame Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hero frames are in-film stills used as source imagery for theatrical posters, streaming cover art, film database listings, and other promotional placements. As the first visual entry point, they shape audiences' initial impressions of the movie and their subsequent willingness to watch it. Selecting these frames, a task we term *hero frame selection*, requires balancing content relevance with aesthetic appeal. A related task is keyframe selection, yet its benchmarks prioritize relevance over aesthetics, using either finite annotations that exclude valid alternatives or VideoQA that entangles selection quality with downstream model capability. We therefore introduce **HeroFrame-Bench**, built through a scalable *VLM-as-a-Judge* framework. We construct multimodal contexts from diverse metadata to ground a VLM judge that scores selected frames using our *Reference-anchored Percentile*. The percentile is obtained by inserting each frame into reusable, pre-ranked reference chains, enabling direct, extensible, and reliable evaluation. To reduce ambiguity and improve consistency in these subjective judgements, we further propose *Rubric–Ranking Co-Evolution*, which generates movie-specific rubrics to condition the VLM judge and refines rubrics jointly with the resulting rankings. Within this process, we introduce several verifiable signals, most notably the *Inverted Rubric Attack*, to select robust rubrics. Finally, **HeroFrame-Bench** is instantiated over 204 movies with 2,031 reference chains and 1,970 learned rubrics. We build an annotation interface for human-alignment studies which show that our construction design improves VLM agreement with human from 77.56% to 83.78%. Evaluation on multiple methods show that hero frame selection remains challenging.

Primary area:
datasets and benchmarks
Keywords:
VLM-as-a-Judge, Rubric Generation, Hero Frame, Keyframe Selection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_qxt9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces HeroFrame-Bench, a benchmark for hero frame selection: choosing a single in-movie still that can serve as a movie's cover, thumbnail, or promotional entry point. The benchmark covers 204 movies built from LSMDC clips (frames; 140.8 hours), IMDb stills (anchors), and IMDb/TMDB/MovieBench metadata. For each movie, the pipeline constructs about 10 six-image reference chains (one human-selected IMDb still + five frames sampled from LSMDC), ranked by pairwise VLM judgments (GPT-5.6-Sol, two placements per pair) aggregated with Bradley-Terry, and learns a movie-level rubric set (1,970 rubrics in total) via Rubric-Ranking Co-Evolution: rubrics proposed from most-tied and least-tied pairs must pass an order-preservation check, an Inverted Rubric Attack (the winner must flip when the rubric's preference direction is inverted), and a cross-chain generalization check. Evaluation uses a Reference-anchored Percentile: a candidate frame returned by a tested method is inserted into each of the movie's frozen chains by a local Qwen3.6-27B judge conditioned on the final rubrics; the insertion position is converted to a percentile and averaged over chains, frames, and movies. Validation includes a 450-pair human study on 30 movies (judge-human agreement improves from 77.56% without rubrics to 83.78% with the full pipeline; 82.00% when the open-weight Qwen3.6-27B replaces GPT-5.6-Sol), a ranking-fidelity check of direct insertion (Spearman rho = 0.973 vs. 0.988 for pairwise recovery), and evaluations showing that all automatic selectors (best: largest-face heuristic, 0.557; prior method WFS-SB: 0.263) fall well below human-selected IMDb stills (0.722).

Critical strengths

S1. First dedicated benchmark for a practically meaningful, previously unmeasured task. Hero frame selection matters for streaming platforms, posters, and media databases, yet prior keyframe-selection benchmarks emphasize content relevance over aesthetic/promotional quality. The paper operationalizes the task with a reusable measurement instrument: once the chains and rubrics are frozen, new frames are scored with zero additional human effort (unlike Elo-style leaderboards that require new human votes for every incoming frame), and the human-selected IMDb still provides a calibrated quality anchor (0.722) against which all automatic selectors fall short. This is consequential to my assessment because it establishes an evaluation substrate for a task that previously had none.

S2. Verifiable rubric admission is a genuinely novel mechanism. The Inverted Rubric Attack -- requiring the pairwise winner to flip when a rubric's preference direction is inverted while the images and all other context are held fixed -- provides behavioral evidence that the judge actually conditions on each rubric rather than following its internal prior, and the ablation shows the necessary-condition checks matter (agreement drops from 80.67% to 78.44% when they are removed). This is consequential because it distinguishes the paper from prior rubric-generation work that filters criteria by external task success signals.

Critical weaknesses

W1. The effectiveness of the benchmark's two central artifacts -- the learned rubrics and the chains' score resolution -- is not established at evaluation time. (a) Each rubric is strongly conditioned ("Only when <condition> ..."), but the paper never measures how often rubrics actually fire on test frames. The construction-time generalization check cannot detect rubrics that are inert outside their source pair (such rubrics pass vacuously by leaving other chains unchanged), and Table 3(b) tests cross-movie mismatch but not firing coverage. If rubrics rarely fire on the frames being scored, their contribution is doubtful -- consistent with the modest gains in Table 2(b) (all components together: +6.22 points; the rubric-related components: +3.11). (b) Each chain provides only 7 discrete percentile slots, and nothing enforces discriminative spacing between adjacent frames; frames selected by different methods can crowd into the same slot, and no tie or resolution statistics are reported.

W2. Missing comparisons against the most relevant alternative, and insufficient evaluation breadth. (a) The pipeline relies on GPT-5.6-Sol throughout construction (median 6,778 judge calls per movie), while human involvement is limited to dataset-internal artifacts (IMDb stills) and a 450-pair validation study. A simpler alternative -- sampling about 10 diverse frame clusters per movie and having humans pick 5+1 frames per cluster to build chains, without any rubric -- is affordable at this scale and might even yield better discriminative spacing between chain frames; the paper never compares its VLM-built chains against such human-built chains, so the necessity of the entire co-evolution pipeline relative to direct human annotation remains unquantified. (b) Table 4 evaluates only one prior selection method (WFS-SB) plus generic heuristics; no general-purpose VLM-based selectors are evaluated, and the appendix provides no qualitative side-by-side examples of the frames actually selected by the tested methods. As a result, the benchmark's discriminative power on real models is largely untested.

Questions for authors

Q1. [Clarification] Were the 30 six-image study groups used in the 450-pair human study identical to frozen reference chains of those movies, or independently constructed groups? If they are the same chains, the most-tied/least-tied pairs from which rubrics were proposed coincide with the pairs used to measure agreement; in that case, please report the agreement of Table 2(b) excluding the pairs from which rubrics were proposed. This directly affects the interpretation of the component gains in Table 2(b).

Q2. [Clarification / experiment request, relates to W1(a)] Please report statistics on how often the learned rubrics actually fire at evaluation time: e.g., the fraction of insertion calls in which each rubric's condition is applicable, the distribution of per-rubric firing rates, and how many of the 1,970 rubrics are effectively inert outside their source pairs. If a large fraction of rubrics never fire on test frames, the contribution claimed for Rubric-Ranking Co-Evolution would need to be reinterpreted.

Q3. [Experiment request, relates to W1(b)] Please report resolution and tie statistics of the percentile scores: the distribution of insertion positions, how often frames from different methods receive identical scores, and an ablation over the number of reference chains per movie (e.g., 3 / 5 / 10) showing how score granularity and method separation change. This would also justify the choice of ~10 chains per movie.

Q4. [Experiment request, relates to W2(a)] Please compare the proposed VLM-built chains against human-built chains on a held-out subset of movies: using the same sampling protocol, have human annotators build 5+1-frame chains from ~10 diverse clusters per movie (optionally with human-written rubrics), freeze both instruments, score the same methods under both, and compare agreement with human preferences, discrimination between IMDb stills and random frames, and test-retest stability. Given the median 6,778 GPT calls per movie reported in Appendix E, the human alternative appears affordable, and this comparison is necessary to quantify the necessity of the co-evolution pipeline.

Q5. [Experiment request, relates to W2(b)] Please evaluate at least one or two general-purpose VLM-based frame selectors on the benchmark, and provide qualitative side-by-side examples (in the appendix) of the frames selected by the tested methods together with their percentile scores. Without these, the benchmark's discriminative power on realistic selectors cannot be assessed from Table 4 alone.

Reviewer_mZ58OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper the authors introduce HeroFrame-Bench, a benchmark for selecting movie frames that balance relevance to the film with visual appeal, which combines movie context, learned evaluation rubrics, and a vision-language model (VLM) judge to assess frames as potential covers or promotional images. The main contributions are a reusable reference-based metric and a procedure that jointly refines rubrics and rankings. Human preference comparisons, benchmark construction ablations, and evaluations of existing frame selectors support the design and indicate room are claimed for better selection methods.

Critical strengths
  1. The task is useful and different from selecting frames for video QA. Frozen chains and explicit rubrics make this subjective evaluation easier to inspect and reuse. The human ablations provide evidence that the rubrics help.

  2. The distinction between selecting informative frames and selecting a movie's promotional image is clear. The motivation makes a useful case that video QA and annotation overlap miss part of this task. A reusable benchmark for this purpose could be potentially valuable even without a new selection algorithm.

Critical weaknesses
  1. The final score needs stronger human validation. The metric averages s(F,C)=(N+1−pˉ(F,C))/Ns(F,C)=(N+1-\bar p(F,C))/N into SS (Eqs. 2–3), but Table 3(a)'s ρ=0.973\rho=0.973 only measures recovery of the constructed chain order, not agreement with human preferences. Appendix C reports one annotation per pair across 30 six-image groups, without directly validating SS on selector outputs. This leaves the reliability of the final method rankings unclear.

  2. I notice the authors claim they propose a "scalable VLM-as-a-Judge framework" in line 98, but line 473 says, "Current VLM context capacity also limits each reference chain to six images," which is confusing and requires explanation.

  3. The presentation is poor, especially the inconsistent bibliographic details across references. Meanwhile, Appendix E (“Construction Cost”) is only a brief call-count summary; actual costs and runtime are missing. Even this subsection itself is making the scalability claim difficult to assess and confusing the focus and information this academic manuscript aims to demonstrate.

Questions for authors
  1. Were the human labels used to select the co-evolution rounds or other settings?
  2. Can SS be validated on held-out selector outputs with multiple raters?
  3. What aspect of the framework is claimed to scale, and why is six images the current limit for a chain? Please provide a cost–fidelity analysis and separate the actual cost of benchmark construction from recurring evaluation, including token usage, monetary expenditure, local hardware, and runtime.
Reviewer_t4H5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HeroFrame-Bench for evaluating movie hero-frame selection based on content relevance and aesthetic appeal. It constructs movie-specific rubrics and reference rankings through VLM-based Rubric–Ranking Co-Evolution, then scores selected frames using a Reference-anchored Percentile. The benchmark covers 204 movies, 2,031 reference chains, and 1,970 learned rubrics. Experiments examine human agreement, construction ablations, insertion fidelity, and several frame-selection baselines.

Critical strengths
  1. A useful benchmark. The paper provides a concrete evaluation framework for balancing movie-specific relevance and aesthetic appeal. Reusing frozen reference chains makes evaluation extensible.

  2. Experiments support rubric condition. The reported human study supports the usefulness of the learned rubrics. The comparison between matched and mismatched movie rubrics also provides evidence that movie-specific conditioning contributes beyond generic guidelines.

Critical weaknesses
  1. Human validation is insufficient to support benchmark reliability. The human study contains 450 comparisons from only 30 six-image groups, with one annotation per pair. This design does not establish inter-rater reliability for a subjective task, and no uncertainty estimates are reported. Furthermore, the insertion experiment measures recovery of constructed reference rankings rather than agreement with humans on outputs from competing selectors. The reliability of the final method ranking therefore remains insufficiently validated.

  2. IMDb-based filtering complicates the interpretation of IMDb superiority. Reference groups are retained only when their IMDb anchor ranks in the upper half, and anchors ranked fourth or below are removed during final freezing. The paper subsequently interprets the superior score of IMDb stills as confirmation of their quality prior. Because construction explicitly favors this source, this comparison is not an independent validation of the prior. Although the human top-three rate provides supporting evidence, an ablation without anchor-rank filtering is needed to assess its effect on benchmark scores and method rankings.

  3. The contribution of the Inverted Rubric Attack is not isolated. The Inverted Rubric Attack verifies that reversing a rubric changes the judge’s preference, demonstrating instruction sensitivity without independently establishing that the original preference is appropriate. Table 2 removes all necessary-condition checks together and therefore does not isolate the inversion test. Its specific contribution to human agreement needs clearer evidence.

Questions for authors
  1. Have you considered directly prompting a VLM to select hero frames as an additional baseline, using the same inputs available to the other selection methods?

  2. Appendix E reports a median of 6,778 judge calls per movie. How much does each construction stage contribute to this cost, and what human agreement can be retained with a smaller call budget, for example by using one co-evolution round or fewer candidate groups? This would clarify the practical scalability of the framework.

Reviewer_TXR3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HEROFRAME-BENCH, a benchmark for movie hero-frame selection that jointly considers narrative relevance and visual appeal. It covers 204 films and includes 2,031 ranked reference chains and 1,970 movie-specific evaluation rubrics. The authors build the reference chains through pairwise VLM comparisons and Bradley–Terry ranking, then use “Rubric–Ranking Co-Evolution” to generate and screen evaluation rubrics. At evaluation time, a candidate frame is inserted into a reference chain and scored by its relative percentile. In the human preference study, adding rubrics increases agreement between the VLM and individual human annotations from 77.56% to 83.78%. Baseline results show that most automatic selection methods score below random sampling.

Critical strengths
  1. Directly evaluates both goals of hero-frame selection. The paper combines movie-specific relevance with visual appeal, rather than using downstream VideoQA or overlap with a limited set of reference frames as proxies for selection quality. This makes the benchmark more closely aligned with real cover-art and promotional-image selection.
  2. A systematic evaluation pipeline with initial empirical validation. Reference chains, pairwise judgments, rubric screening, and insertion-based scoring form a scalable evaluation process. Ablations show that rubric co-evolution improves agreement with human preferences, and some of the gain remains when using a different judge model.
Critical weaknesses
  1. The human-alignment study does not fully support strong reliability claims. It reports results on 450 image pairs from 30 films, with only one human annotation per pair. It does not report inter-annotator agreement or uncertainty estimates. Moreover, 83.78% is the highest result among several ablations, and the paper does not clarify whether it was measured on an independent human evaluation set.
  2. The data coverage limits the generality of the conclusions. The benchmark relies on LSMDC movie clips from films released between 1943 and 2014, and candidate frames are drawn from those clips rather than from complete movies. The paper acknowledges the data-access limitation, but the current results do not establish how well the benchmark generalizes to a broader range of films or full-length movies.
Questions for authors
  1. Were the image groups from the 30-film human study held out from reference-chain construction and rubric learning? Was the reported 83.78% agreement measured on data that were not used to select among the ablations?
  2. Each image pair received only one human annotation. How many annotators participated, and was any subset independently annotated more than once? If not, could you report uncertainty around the agreement rate and clarify whether it reflects agreement with individual annotators or with a broader human preference consensus?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.