acceptodds
Withdrawn from ICLR 2027

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce **MFVSR**, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: **(1) Large-scale and diverse data**, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; **(2) Controlled subtitle diversity**, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; **(3) Fine-grained evaluation**, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods. Data and code will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
video subtitle removal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wcDHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MFVSR, a benchmark for video subtitle removal comprising 3,000 paired videos, 16 languages, and 24 subtitle configurations. Experiments on six methods show that good subtitle restoration does not necessarily imply faithful background preservation or complete video outputs.

Critical strengths

The dataset construction is systematic. This paper examines subtitle regions, their bounding boxes, and full frames separately. The results show residual text, incorrect background reconstruction, and unwanted changes outside subtitle regions. Some methods also return incomplete videos.

Critical weaknesses
  1. Limited additional insight. Relative to existing diagnostic evaluation approaches, the contribution primarily consists of expanding the dataset, organizing subtitle variations, and integrating evaluation protocols. The main paper focuses on aggregate metrics and stratified comparisons, without a detailed analysis of individual failure cases. Qualitative failure examples are confined to Appendix E, where they illustrate errors but do not systematically explain why existing methods fail or establish new insights that could guide method development.
  2. Limited temporal and real-world validation. The human study provides limited support for the temporal component. The paper also reports no separate results on the 953 naturally subtitled videos. It therefore remains unclear how well the benchmark’s findings transfer to real-world subtitle removal.
  3. Incomplete evaluation of complex subtitle styles. Audit300 excludes subtitles with background boxes. Their separate evaluation omits preservation and BBox temporal metrics because the annotations may not cover the full subtitle area. This limits the benchmark’s coverage of complex styles.
Questions for authors

See weaknesses.

Reviewer_yLgkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

MFVSR is a benchmark for video subtitle removal, comprising 3,000 clean videos (Real-HQ, Real-LQ, and AI-generated) with subtitles in 16 languages. It covers controlled variations in layout, animation, style, and language mode, using the original videos as ground truth. Since full-frame metrics can obscure meaningful differences, the authors evaluate blind and mask-guided methods separately. They divide each frame into subtitle, surrounding, and outside regions, then assess restoration, preservation, and temporal consistency separately, reporting output completeness independently from quality. Across six models, no method excels in every area; strengths are split across restoration, preservation, and temporal consistency. A human study supports the spatial metrics, but temporal alignment remains unvalidated.

Critical strengths
  • Principled benchmark design with pixel-accurate ground truth. Rendering controlled subtitles over clean videos yields pixel-accurate ground truth, glyph masks, and bounding boxes. The 24-cell factorial design (layout, animation, style, and language mode) enables stratified performance analysis across subtitle conditions.

  • Evaluation that separates what full-frame metrics conflate. Reporting blind and mask-guided methods separately and dividing frames into subtitle, surrounding, and outside regions reveal real trade-offs (e.g., subtitle restoration versus outside preservation) that conventional metrics obscure.

Critical weaknesses
  • The central claim is only partially supported. The case for decoupled evaluation depends on metrics aligning with human judgment, but the human study validates only the spatial metrics. Temporal judgments remain largely inconclusive because inter-annotator agreement is low (the authors acknowledge that temporal validation remains open). The findings also show associations, not causation: source and resolution are confounded, and the >30 s slice includes only 10 videos. These limitations matter because the paper’s main contribution is its evaluation protocol, and unvalidated components weaken its core claim.

  • Delivery and reliability results are difficult to interpret because key evidence is limited. Complete/partial rates largely reflect each model’s output-length contract rather than failures, which limits the table’s use as a reliability measure. Shared-ID comparisons do not guarantee equal frame coverage. The main quality results are based on Audit300 (300 videos, up to 48 frames each). The blind setting includes only one method (CLEAR), and transfer from synthetic to real subtitles has not been validated because the 953 real videos have no ground truth.

  • The paper is poorly organized and written, making it difficult to interpret. It introduces many overlapping constructs (three fidelity dimensions, five views, Lite3000/Audit300, complete/partial/failed categories, shared-ID versus native sets, and SR-Q) without a clear hierarchy. Several results depend on decoding terms that compress complex ideas into single words, such as “delivery” (whether a model returns a complete output), “verified prefix” (evaluating only the leading frames returned by a model), and “native” (each model’s own evaluable set). Their definitions, including the complete/partial/failed categories used in Table 5, appear only in the appendix. Paragraphs also tend to list facts rather than lead with the claim (e.g., the evaluation-region paragraph introduces five “views” before explaining the three-region design).

  • The experiments show that metrics disagree but offer little insight into why. Here, six models are compared, and results are reported by indicating which method leads on each metric, with at most a one-sentence explanation of the causes. A practitioner cannot easily tell which method to use under which conditions.

Questions for authors
  • What is the distribution of missing frames across partial outputs? Are near-complete outputs (e.g., 1–2 missing frames) also labeled “partial”? If so, does the complete rate in Table 5 understate a model’s practical usability?

  • B.2 notes that shared IDs do not guarantee equal frame coverage. How are frame indices aligned across models within a video for scoring? Could the authors report results on the common frame intersection and how much the rankings change?

  • Can the authors show that rankings on synthetic subtitles transfer to real subtitled videos, for example, through a small human study of the 953 in-the-wild videos?

Reviewer_er2MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces MFVSR, a large-scale benchmark for video subtitle removal designed to overcome the limited coverage and coarse evaluation of existing datasets. It provides paired videos with diverse sources and subtitle conditions, along with in-the-wild videos for broader real-world coverage. MFVSR separately evaluates blind and mask-guided methods across restoration, preservation, temporal fidelity, and output reliability. Experiments on six representative methods reveal substantial trade-offs among these dimensions, showing that conventional full-frame metrics can obscure local restoration errors, unintended changes, temporal instability, and incomplete outputs.

Critical strengths
  1. MFVSR introduces an evaluation protocol specifically designed for video subtitle removal.

Existing benchmarks often rely on generic video inpainting metrics or evaluate the entire frame, which can make it difficult to distinguish errors within subtitle regions from unintended changes to the surrounding content. MFVSR addresses this by evaluating restoration quality, preservation of unaffected regions, temporal consistency, and output reliability separately. I find this decomposition useful because it allows for a more diagnostic comparison of subtitle-removal and video-inpainting methods, making it easier to understand where each method succeeds or fails.

  1. The benchmark also covers a broad range of subtitle conditions in a controlled manner.

MFVSR varies spatial layout, temporal behavior, visual style, and language mode, while also including multilingual subtitles. Rather than evaluating performance only on an aggregate test set, this setup allows model behavior to be examined under specific subtitle conditions. I expect this to be useful for future work, particularly for identifying failure cases and understanding which types of subtitle variation remain difficult for subtitle-removal and related video-inpainting models.

Critical weaknesses
  1. The coverage of real-world subtitle structures in the controlled benchmark is not fully clear.

The quantitative benchmark is constructed with a controlled rendering pipeline based on multilingual text corpora and a predefined taxonomy covering spatial layout, temporal behavior, visual style, and language mode. This setup supports systematic analysis, but it is less clear how well it captures subtitle structures beyond conventional text rendering. Social-media captions, for example, often include emojis, icons, or mixed text-symbol elements, and the paper does not clearly state whether the rendering pipeline supports such cases. This limitation is more important because the auxiliary in-the-wild videos are not used for paired quantitative evaluation. It is therefore difficult to judge how well conclusions from the controlled benchmark extend to these types of real-world subtitles. Clarifying what the rendering pipeline can and cannot represent, or discussing these cases explicitly as a limitation, would make the scope of MFVSR clearer.

  1. The perceptual validity of the metric-induced method rankings remains somewhat unclear.

The paper includes a held-out human study and reports pairwise agreement between the proposed metrics and human judgments, including Kendall's τb. The agreement, however, is noticeably weaker for Temporal Fidelity than for Restoration and Preservation. I also found it difficult to tell whether these pairwise results translate into similar rankings of the evaluated methods. Since MFVSR is intended to compare subtitle-removal methods, agreement at the method-ranking level seems directly relevant. A comparison between the rankings induced by the proposed metrics and those obtained from human judgments would make the perceptual validity of each benchmark dimension easier to assess.

Questions for authors
  1. Coverage of non-text subtitle elements. Does the controlled rendering pipeline support subtitle elements such as emojis or mixed text-symbol captions? If not, could the authors discuss how this may limit the real-world coverage of MFVSR?

  2. Agreement between metric and human rankings. Do the method-level rankings induced by Restoration, Preservation, and Temporal Fidelity agree with rankings derived from the human study?

  3. Readability of Figures 1-3. Figures 1-3 contain important information about the benchmark design, but several labels are difficult to read at normal manuscript scale. Could the authors enlarge the text or simplify the figures so that their main components can be understood without substantial zooming?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.