acceptodds
Under review as a conference paper at ICLR 2027

What, When, and How: Audio Description as Constrained Global Optimization

Abstract

Audio Description (AD) makes movies accessible to blind and visually impaired audiences by narrating visual information in gaps between dialogue. Existing automatic AD systems largely treat generation as a local video-to-text problem, assuming that the content to describe and its temporal location are already provided. Realistic AD instead requires coupled decisions about what visual information is narratively important, when it can be spoken without interfering with dialogue, and how it should be formulated to fit within the available time. We formalize AD generation as a constrained optimization problem over these three decisions. Our hybrid system uses large language models to propose and ground visual elements, estimate their salience to the narrative, and generate compressed realizations. A mixed-integer linear program then jointly selects and schedules descriptions across a scene subject to temporal constraints. When evaluated on REFRAMED, a benchmark for realistic AD of movies, our approach makes better decisions than prompted LLMs about what to describe and when to describe it, establishing a new SOTA on narrative QA and temporally grounded metrics. Ablations show that explicit temporal constraints drive gains in placement, while salience estimation controls how much narratively useful content is retained. Improvements are concentrated on temporal and narrative measures rather than n-gram overlap, although a significant gap to professional describers remains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
movies, narrative, audio description, linear programming, ilp, milp

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_tgkpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper formulates automatic Audio Description (AD) generation as a constrained global optimization problem that jointly determines what visual information to describe, when to deliver the descriptions, and how to formulate them within limited narration time. The proposed framework employs MLLMs to generate candidate descriptions, estimate their narrative salience, identify temporal spans, and produce compressed variants. A mixed-integer linear programming (MILP) solver then selects and schedules descriptions while satisfying temporal constraints. Experiments on REFRAMED demonstrate improvements over existing AD systems and LLM-based schedulers, particularly in temporal alignment and narrative QA metrics. Ablation studies further investigate the contributions of temporal grounding, compression, and salience estimation.

Critical strengths
  1. Meaningful problem formulation. The paper focuses on an important limitation of existing AD systems, which typically assume predefined narration timestamps. Jointly considering content selection, temporal placement, and linguistic realization provides a more realistic formulation of automatic AD generation.

  2. Principled optimization framework. The combination of LLM-generated candidates and MILP-based optimization is intuitive and well motivated. Explicitly enforcing temporal constraints offers greater control over narration scheduling than relying solely on LLM generation.

  3. Promising improvements in narrative understanding. The proposed method achieves strong performance on temporally grounded and QA-based metrics, demonstrating the potential benefits of global optimization beyond conventional text-overlap evaluation.

Critical weaknesses
  1. Insufficient validation of LLM-generated descriptions. The framework heavily relies on LLMs for event generation, temporal grounding, salience estimation, and description compression. However, the reliability of these intermediate outputs is not sufficiently evaluated. In particular, compression may remove critical details or introduce semantic inconsistencies, potentially affecting the final AD quality. A systematic evaluation of factual accuracy, grounding quality, and semantic preservation would strengthen the paper.

  2. Lack of human evaluation. The experiments rely primarily on automatic metrics, which may not adequately capture the fluency, coherence, and practical usability of generated audio descriptions. Human evaluation, ideally involving professional describers or visually impaired users, would provide stronger evidence of the method's practical effectiveness.

  3. Limited comparisons with simpler optimization strategies. Although the MILP solver outperforms LLM-based schedulers, the experiments do not establish whether global optimization is necessary compared with simpler constrained approaches, such as greedy selection or heuristic scheduling. Including these baselines would better isolate the benefits of the proposed optimization framework.

Questions for authors

Please see weaknesses. If the authors adequately address the above concerns, I would consider raising my score.

6 comments
AuthorsOpenReview ↗

Response [1/2]

  • Insufficient validation of LLM-generated descriptions.

    We agree that the quality of intermediate LLM outputs affects the final audio descriptions. Our ablations provide evidence of their contribution to downstream performance: removing temporal grounding substantially reduces temporal alignment, while removing compressed variants reduces QEval from 48.4 to 43.4 and QEval-T from 28.2 to 22.7. We also investigate the effect of different salience estimates and find that oracle salience does not necessarily improve temporally grounded metrics, highlighting the interaction between content selection and scheduling.

    These experiments establish the usefulness of the intermediate components but do not directly measure their factual accuracy or semantic fidelity. In particular, compressed variants may omit important details or introduce inconsistencies, and our current formulation does not explicitly verify that they preserve the meaning of the original event. An advantage of our modular framework is that improved generation, grounding, and compression methods can be incorporated without changing the optimisation formulation. We will clarify these limitations and distinguish component-level effectiveness from the factual and linguistic quality of the resulting descriptions.

  • Lack of human evaluation.

    We agree that human evaluation is essential to establish whether improvements in automatic metrics translate into better narrative comprehension, accessibility, and listening experiences. We would like to clarify the scope of our current contribution and outline our plans for user-centred evaluation.

    Scope of the current work. Our contribution is a framework for selecting, compressing, and scheduling visual descriptions, rather than a complete, deployment-ready audio description (AD) system. In particular, we have not yet incorporated speech synthesis, which will need to be tailored to the delivery conventions and stylistic preferences of AD audiences. Consequently, our current evaluation focuses on the quality of content selection and temporal placement, rather than the end-to-end listening experience.

    Planned user-centred evaluation. We have already taken steps towards recruiting blind and visually impaired AD users, including contacting national-level organisations representing these users, and are in discussions with professional audio describers experienced in working with AD users. We envisage an evaluation involving three complementary participant groups:

    • Blind and visually impaired AD users will assess narrative comprehension, engagement, and listening preferences. Participants will answer questions probing their understanding of the plot, characters, and events, and provide ranked preferences for alternative AD versions.

    • Professional audio describers will assess the relevance and completeness of selected visual information, the coherence and clarity of descriptions, and adherence to AD conventions, including narration timing and pacing.

    • Sighted participants will assess the correspondence between descriptions and visual events, including whether important information is omitted, whether descriptions are appropriately timed, and whether they interfere with dialogue or other meaningful audio.

    Experimental design. We plan to compare automatically generated AD with professional descriptions and appropriate controls, including listening to the movie audio without AD. These comparisons will allow us to distinguish improvements in narrative comprehension from preferences for particular narration styles.

    A rigorous evaluation across these groups requires careful experimental design, participant recruitment, and appropriate control groups, and we feel this is beyond the scope of the present, methods-focused study. Nevertheless, are are already taking concrete steps towards assessing the practical validity of the framework. We will clarify our evaluation plan and the limitations of our current automatic metrics in the revised paper.

AuthorsOpenReview ↗

Response [2/2]

  • Limited comparisons with simpler optimization strategies.

    We thank the reviewer for this suggestion. We implemented a constraint-aware greedy baseline that processes candidate events in decreasing order of salience. At each iteration, the algorithm attempts to place the candidate and all previously selected events at their earliest feasible times in scene description order. The algorithm accepts the candidate if a feasible schedule is found and then proceeds to the next most salient candidate. The results below compare this approach with MILP under three temporal grounding settings.

    SystemCIDErMETEORSODA-MSODA-TQEvalQEval-T
    Qwen MILP12.97.77.354.948.428.2
    Qwen Greedy
    - Δmax⁡=3\Delta_{\max}=312.56.67.049.644.022.1
    - Δmax⁡=10\Delta_{\max}=1011.47.17.249.345.923.8
    - w/o grounding6.95.77.538.440.514.9

    MILP outperforms the greedy approach on nearly all metrics, with particularly clear improvements in temporal alignment and narrative QA. For example, at Δmax⁡=10\Delta_{\max}=10, SODA-T increases from 49.3 to 54.9 and QEval-T from 23.8 to 28.2. These results demonstrate the benefit of global optimisation: locally selecting and placing salient events can prevent better combinations of descriptions from being scheduled. MILP jointly considers these trade-offs and also allows additional objectives and constraints to be incorporated. We will include this comparison in the revised paper.

Thank you for the rebuttal and the constraint-aware greedy baseline, which strengthen the optimization comparison. I also appreciate the methods-focused scope and the detailed plan for user evaluation. My remaining concern is whether the selected compressions preserve core narrative information and form coherent descriptions in context. The ablations establish downstream usefulness, but, as acknowledged in the response, they do not directly validate compression fidelity. This matters because event-level salience is reused across variants, while duration weighting does not directly assess which information is retained.

A small sampled manual audit of selected compressions for key-fact retention and semantic errors, together with a lightweight blinded human comparison of contextual clarity and coherence against the fully instructed LLM scheduler, would help address these concerns using the existing textual outputs. The small and mixed narrative-QA differences relative to that LLM scheduler make these quality trade-offs particularly relevant, while the MILP’s explicit constraint guarantees remain a meaningful strength. I would be open to raising my score if the results substantively address the remaining concerns.

Reviewer_HnyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates automatic audio description (AD) as constrained global optimization over what to describe, when to deliver it, and how to phrase it, instead of captioning clips whose timing is inherited from reference AD. An LLM (Qwen 3.5 27B or Gemini 3.1 Flash-Lite) writes a scene description that is segmented into eventualities. A second prompt grounds each eventuality in time, produces five compressed variants, and assigns a salience score. A per-scene MILP then selects eventualities and variants and schedules their delivery to maximize duration-weighted salience. Its constraints are dialogue-gap containment, a 10 s proximity window, and non-overlap. On the REFRAMED challenge set (10 films), Gemini MILP achieves the best QEval and QEval-T among automatic systems. Given the same inputs, it beats an LLM scheduler on all metrics except CIDEr, with larger margins on harder scenes. Ablations cover grounding, compression, salience sources, and screenplay-based descriptions; a large gap to expert AD remains.

Critical strengths

The main strength is the problem formulation. Most AD work inherits timing, and implicitly content selection, from the reference AD. This paper instead makes what/when/how explicit decisions and satisfies the hard constraints of real narration (gap containment, speaking rate, non-overlap) by construction. Prompted LLMs demonstrably violate these constraints: 28.9% (Qwen) and 16.7% (Gemini) of their descriptions exceed 300 WPM, and filtering them out significantly lowers every metric. The decomposition is also interpretable and modular, so better salience or grounding models can be plugged in as inputs. This matters because it moves automatic AD from a captioning proxy toward the actual task, with consistent temporal-placement gains across both backbones.

Critical weaknesses
  1. Narrative gains are small and uneven. Qwen MILP beats the scheduler by only 0.3 on QEval/QEval-T and trails unfiltered prompted Qwen on QEval (43.6 vs. 43.9); Gemini MILP gains 3.6 over prompted Gemini. Random AD already scores 34.1 QEval, so the best system covers only about a third of the random-to-expert range, and CIDEr does not improve. The gains concentrate in the temporally thresholded metrics questioned above.
  2. Coherence is unmodeled, and there is no human evaluation. Eventualities are selected independently, so antecedents can be lost. The worked example (App. A) keeps e10 “It contains a black, leather motorcycle jacket” but drops e8, which introduces the garment bag. Outputs also include fragments such as “Rises from seat.” and “Positional shift made.” For an accessibility tool consumed by ear, a study with blind/low-vision listeners or professional describers is essential.
  3. Omission is driven by available time, not narrative judgment. Salience is positive and speaking has no cost, so the objective rewards every additional description that fits. Selection reaches 91.8% when over 80 s is free (Gemini), and outputs add trivia where both references are silent (“Occupies vacant seat.”, “Bends ahead gently.”). Oracle salience even lowers QEval-T (Table 3: 23.8/24.2 vs. 28.2). The “what” decision, framed as the central challenge, is the least supported of the three.
Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response [1/2]

  • Narrative gains are small and uneven.

    We agree that temporal control is the strongest contribution of our current system. As discussed in our response to Reviewer mctj27, an LLM scheduler given the full optimisation problem can approach MILP performance, but requires substantially more reasoning tokens (over 11K for Gemini and 16K for Qwen on average). In contrast, MILP solves the constrained optimisation problem directly, guaranteeing that the resulting schedule satisfies the specified temporal constraints without lengthy LLM reasoning. Its explicit formulation also allows additional objectives and constraints to be incorporated, providing a foundation for more complex scheduling problems.

    Regarding narrative understanding, our experiments with screenplay-derived descriptions provide evidence of the framework's potential: Qwen achieves a QEval score of 53.1, substantially higher than with automatically generated event descriptions. Although screenplay-based inputs introduce temporal grounding difficulties, this result suggests that narrative performance is strongly influenced by the quality of the candidate descriptions, rather than being limited by the optimisation framework itself.

  • Coherence is unmodeled, and there is no human evaluation.

    We agree that our current formulation does not explicitly model coherence between selected descriptions. As the reviewer notes, independently selecting events can omit antecedents or produce descriptions that are difficult to interpret in isolation. One advantage of our optimisation framework is that such dependencies can be incorporated as additional constraints, for example requiring an antecedent to be included whenever an anaphoric description refers to it. A complementary approach would be to use an LLM to improve the fluency and coherence of the selected descriptions after scheduling. We view these as promising extensions and will clarify this limitation in the paper.

    We agree that human evaluation is essential to establish whether improvements in automatic metrics translate into better narrative comprehension, accessibility, and listening experiences. We would like to clarify the scope of our current contribution and outline our plans for user-centred evaluation.

    Scope of the current work. Our contribution is a framework for selecting, compressing, and scheduling visual descriptions, rather than a complete, deployment-ready audio description (AD) system. In particular, we have not yet incorporated speech synthesis, which will need to be tailored to the delivery conventions and stylistic preferences of AD audiences. Consequently, our current evaluation focuses on the quality of content selection and temporal placement, rather than the end-to-end listening experience.

    Planned user-centred evaluation. We have already taken steps towards recruiting blind and visually impaired AD users, including contacting national-level organisations representing these users, and are in discussions with professional audio describers experienced in working with AD users. We envisage an evaluation involving three complementary participant groups:

    • Blind and visually impaired AD users will assess narrative comprehension, engagement, and listening preferences. Participants will answer questions probing their understanding of the plot, characters, and events, and provide ranked preferences for alternative AD versions.

    • Professional audio describers will assess the relevance and completeness of selected visual information, the coherence and clarity of descriptions, and adherence to AD conventions, including narration timing and pacing.

    • Sighted participants will assess the correspondence between descriptions and visual events, including whether important information is omitted, whether descriptions are appropriately timed, and whether they interfere with dialogue or other meaningful audio.

    Experimental design. We plan to compare automatically generated AD with professional descriptions and appropriate controls, including listening to the movie audio without AD. These comparisons will allow us to distinguish improvements in narrative comprehension from preferences for particular narration styles.

    A rigorous evaluation across these groups requires careful experimental design, participant recruitment, and appropriate control groups, and we feel this is beyond the scope of the present, methods-focused study. Nevertheless, are are already taking concrete steps towards assessing the practical validity of the framework. We will clarify our evaluation plan and the limitations of our current automatic metrics in the revised paper.

AuthorsOpenReview ↗

Response [2/2]

  • Omission is driven by available time, not narrative judgment.

    We thank the reviewer for this observation. Our current objective assigns positive value to every description, potentially including low-salience events whenever narration time is available. To investigate this, we introduced a simple modification that subtracts a constant cc from each salience score, effectively rewarding silence and allowing the model to omit events even when they could be accommodated.

    The results show that selective omission can improve performance at moderate thresholds. For Δmax⁡=3\Delta_{\max}=3, CIDEr increases from 14.0 to 14.9, while QEval-T improves from 22.4 to 24.7. For Δmax⁡=10\Delta_{\max}=10, CIDEr increases from 12.9 to 13.7. Larger penalties eventually reduce performance as too much information is omitted. These findings demonstrate that our framework can explicitly trade off narrative coverage against selective silence, although the quality of these decisions still depends on the underlying salience estimates. We will include this analysis in the revised paper.

    ccCIDErMETEORSODA-MSODA-TQEvalQEval-T
    Δmax⁡=3s\Delta_{\max} = 3\mathrm{s}
    014.07.27.150.044.722.4
    0.314.27.27.149.246.222.3
    0.514.97.17.145.644.124.7
    0.79.24.46.334.037.817.6
    1.00.00.00.00.034.60.0
    Δmax⁡=10s\Delta_{\max} = 10\mathrm{s}
    012.97.77.354.948.428.2
    0.313.27.67.353.047.727.0
    0.513.77.37.350.043.124.6
    0.78.94.76.539.337.119.0
    1.00.00.00.00.034.60.0

    Regarding oracle salience, the oracle optimises content importance rather than the complete scheduling problem. Consequently, better salience estimates do not necessarily improve temporally grounded metrics, since the selected events must also be accommodated within the available dialogue gaps. This highlights the importance of jointly optimising content selection and temporal placement.

Hello, we hope that our response may have addressed your concerns. We have also added a follow-up to reviewer tgkp, in which we give results of a small-scale human study. Thank you for your time!

Reviewer_mctjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates movie audio description as joint content selection, compression, and scheduling. An LLM supplies candidate visual events, temporal spans, salience scores, and alternative wordings. A mixed-integer program selects and schedules these candidates around dialogue. Evaluation uses ten REFRAMED movies, with component analyses on eight movies with well-matched screenplays. The strongest system improves temporal alignment and reference-based question answering, although lexical overlap does not consistently improve and performance remains below professional descriptions.

Critical strengths
  1. Most audio-description systems are evaluated on clips whose timing has already been chosen. This paper tackles the harder practical problem of deciding what to describe and where to put it. The distinction between an event's occurrence and its narration time is particularly important here. Sections 3 and 4 turn these decisions into a clear formulation that can be used with existing models.
  2. The component analysis helps explain the results. Table 1 compares the MILP with LLM schedulers given the same upstream candidates, while Table 3 examines grounding, compression, and salience separately. With Gemini inputs, SODA-T rises from 45.8 to 55.7 and QEval-T from 19.9 to 25.5. Removing grounding or compression produces distinct losses. Reporting solver timeouts and optimality gaps is also welcome.
Critical weaknesses
  1. I am less convinced by the comparison with the LLM scheduler. Section 4 says the scheduler relies on its knowledge of audio description rather than being given the MILP's constraints. The prompt in Appendix E does not specify gap containment, non-overlap, the ten-second grounding limit, or the salience objective. The two methods therefore receive similar inputs but different task specifications. Compression is also unclear: the main text says all variants are supplied, but the prompt shows one description and duration per element, without variant grouping or a rule against selecting several variants of the same event. Filtering changes scheduler scores by at most 0.4 points, so infeasible outputs alone do not explain the difference. Still, the comparison needs a fuller account, particularly for Qwen, where QEval improves only from 43.3 to 43.6 and QEval-T from 21.8 to 22.1. A deterministic scheduler using the same candidates would be a useful further comparison.
  2. The objective does not distinguish good compression from loss of important information. Equation 1a multiplies salience by narration duration and gives every variant of an event the same salience. A longer wording is thus preferred even if a shorter version says everything that matters; conversely, a shortened version can retain its score after losing the relevant person or action. Figure 2 illustrates this problem. Figure 3 also contains fragments with unclear subjects. Higher QA scores show that more reference information is retained, but they do not establish that the resulting narration is more coherent or easier to follow. The claim about narrative usefulness should be qualified.
Questions for authors
  1. Is Appendix E the full prompt used for Table 1? Please clarify how compression variants were represented, how multiple variants of one event were excluded, and which constraints and objective were given to the scheduler.
  2. Was the essential information in each compressed variant checked, or was the shared event-level salience score the only quality signal? What can the existing analysis tell us about the quality of the selected wordings?
  3. Are the MILP-versus-scheduler gains consistent across the ten movies? Per-movie differences would help interpret the small Qwen QA gains, since Appendix B uses sequences as the permutation unit and sequences from the same film may be related.
3 comments
AuthorsOpenReview ↗

Response [1/2]

  • Is Appendix E the full prompt used for Table 1? Please clarify how compression variants were represented, how multiple variants of one event were excluded, and which constraints and objective were given to the scheduler.

    Yes, Appendix E contains the full prompt used for the LLM scheduler in Table 1. The scheduler receives the same candidate descriptions, salience scores, temporal grounding and dialogue gaps as the MILP, but is not explicitly given its optimisation objective or scheduling constraints. Although the prompt limits each candidate ID to a single selection, it does not explicitly group compressed variants or enforce selection of at most one variant per underlying event.

    To address this concern, we conducted a new experiment in which the LLM receives the complete optimisation problem in natural language, including the duration-weighted salience objective, explicit grouping of compressed variants, and all temporal constraints. This establishes a substantially stronger comparison in which both approaches are given the same problem specification.

    The table below compares the fully instructed LLM scheduler with MILP across six evaluation metrics, including an alternative temporal displacement threshold requested by Reviewer 5HVL03. With sufficient reasoning, the LLM approaches MILP performance, but reducing or disabling reasoning substantially degrades its results.

    ModelThinkingCIDErMETEORSODA-MSODA-TQEvalQEval-T
    GeminiMinimal12.96.77.145.043.821.8
    High15.18.27.753.445.524.8
    MILP13.68.48.055.745.925.5
    - Δmax⁡=3\Delta_{\max}=315.68.57.855.445.924.8
    QwenOff8.95.06.536.941.217.9
    On11.97.77.652.243.921.8
    MILP11.47.97.753.643.622.1
    - Δmax⁡=3\Delta_{\max}=313.37.77.650.443.822.7

    This performance comes at considerable computational cost: compared with the original scheduler, average reasoning tokens per scene increase from 5.0K to 11.1K for Gemini and from 9.3K to 16.6K for Qwen, with increases across all ten films. Thus, while reasoning LLMs can approximate the constrained optimisation, MILP solves the problem directly, guarantees that returned schedules satisfy the specified constraints, and avoids costly LLM reasoning. The following table shows mean the mean number of reasoning tokens per scene for the original (Old) and fully verbalized MILP (New) scheduler prompts, with their difference (Δ\Delta).

    MovieGemini OldNewΔ\DeltaQwen OldNewΔ\Delta
    Barbie (2023)5.2k9.2k+4.0k9.0k13.4k+4.4k
    Charlie St. Cloud (2010)4.9k11.3k+6.3k9.4k16.7k+7.3k
    Legion (2010)5.2k12.4k+7.3k9.8k17.2k+7.4k
    Dune: Part Two (2024)4.8k12.1k+7.3k9.3k18.8k+9.4k
    Les Misérables (2012)5.3k10.6k+5.4k9.3k17.0k+7.7k
    The Roommate (2011)4.8k11.6k+6.8k9.5k16.8k+7.2k
    Battle: Los Angeles (2011)5.1k10.9k+5.8k9.2k15.5k+6.2k
    Harry Potter & Goblet of Fire (2005)5.0k11.6k+6.6k9.3k18.5k+9.2k
    How Do You Know (2010)5.1k12.0k+6.9k9.6k18.0k+8.4k
    Priscilla (2023)4.7k10.5k+5.7k8.9k15.2k+6.3k
    Mean5.0k11.1k+6.1k9.3k16.6k+7.3k
  • Was the essential information in each compressed variant checked, or was the shared event-level salience score the only quality signal? What can the existing analysis tell us about the quality of the selected wordings?

    In our current experiments, variants share an event-level salience score, and their semantic fidelity is not evaluated separately. The objective favours fuller descriptions when time permits, reflecting the assumption that they preserve more information. Compression nevertheless plays an important role: removing compressed variants reduces QEval from 48.4 to 43.4 and QEval-T from 28.2 to 22.7, suggesting that shorter variants allow more narrative information to be included within the available narration time. However, these results do not establish that individual compressions preserve all essential details. We acknowledge that some selected descriptions are terse or syntactically incomplete, and that important information may be lost during compression. Explicitly assessing semantic preservation and coherence, and incorporating these measures into variant selection, is an important direction for future work.

AuthorsOpenReview ↗

Response [2/2]

  • Are the MILP-versus-scheduler gains consistent across the ten movies? Per-movie differences would help interpret the small Qwen QA gains, since Appendix B uses sequences as the permutation unit and sequences from the same film may be related.

    We thank the reviewer for raising this issue. The table below reports per-movie differences in SODA-T and QEval-T. Improvements in SODA-T are consistent across all ten films, whereas QEval-T gains are smaller and vary across films.

    Re-examining our significance testing also revealed an implementation error in how QEval and QEval-T scores were prepared for permutation testing on the challenge set. After correcting this error, the small Qwen QEval improvement is no longer statistically significant. We will correct this claim in the revised paper. This does not affect the consistent improvements in temporal alignment, which provide consistent evidence in favor of our approach.

    MovieSODA-T SchedulerMILPΔ\DeltaQEval-T SchedulerMILPΔ\Delta
    Barbie39.847.2+7.522.024.4+2.3
    Battle: Los Angeles45.952.8+6.914.014.00.0
    Charlie St. Cloud52.557.8+5.326.524.9-1.6
    Dune: Part Two45.648.9+3.315.017.0+1.9
    Harry Potter and the Goblet of Fire47.555.2+7.723.522.1-1.4
    How Do You Know41.851.2+9.421.123.4+2.3
    Legion40.745.7+5.017.517.4-0.2
    Les Misérables40.947.9+7.019.119.6+0.5
    Priscilla64.569.4+4.929.629.8+0.2
    The Roommate56.660.1+3.529.828.6-1.2
    All47.653.6+6.121.822.1+0.3

Hello, we hope that our response may have addressed your concerns. We have also added a follow-up to reviewer tgkp, in which we give results of a small-scale human study. Thank you for your time!

Reviewer_5HVLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for automatic movie audio description (AD) that jointly optimizes what, when, and how to describe visual events. It combines MLLM-based event extraction, salience estimation, and description compression with mixed-integer linear programming (MILP) for temporal scheduling. Experiments demonstrate improvements over LLM-based baselines on temporal alignment and narrative comprehension metrics.

Critical strengths
  1. Meaningful problem formulation. Jointly optimizing content selection, narration timing, and compression addresses an important challenge in automatic AD generation. The MILP formulation is interpretable and practically motivated.

  2. Promising empirical results. The improvements over LLM-based scheduling and the component ablations provide useful evidence for the effectiveness of explicit temporal optimization.

Critical weaknesses
  1. Limited justification of AD design assumptions. The fixed narration rate (200 WPM), maximum temporal displacement (10 seconds), and emphasis on concise descriptions may not accommodate diverse user preferences. Moreover, the framework focuses primarily on visual narrative events, potentially overlooking cinematographic expression and audio-visual interactions.

c2. Limited evaluation of practical effectiveness and generalization. Although automatic metrics demonstrate improvements, the absence of subjective evaluations involving blind or visually impaired audiences and professional describers makes it difficult to assess real-world accessibility. The relatively small challenge set and limited analysis of model variability also raise questions about generalizability.

Questions for authors
  1. Problem motivation: The Introduction argues that previous MLLM approaches bypass what and when to describe. Could the authors clarify this distinction? Moreover, how does the system account for artistically meaningful visual details that may not be central to the plot?

  2. Narration constraints: What empirical or HCI evidence supports the fixed 200 WPM narration rate? Some blind users routinely consume speech at 2–4 times conventional speeds. Could adaptive narration speed accommodate more information while preserving important content? Similarly, could the authors provide a sensitivity analysis of Δmax⁡\Delta_{\max} beyond 10 seconds, given that professional narration conventions are not necessarily optimal?

  3. Optimization formulation: What motivates MILP over alternative scheduling algorithms? How are the finite event candidates and compressed variants constructed? Could the authors also justify weighting salience by narration duration, which may favor longer descriptions?

  4. Audiovisual understanding: How does the framework capture events requiring joint audiovisual reasoning, such as a character turning off a radio while the background music simultaneously stops? Could explicit audio understanding improve event selection?

  5. Dataset and reproducibility: Could the authors clarify the separation between the 206-film corpus, eight-film validation set, and ten-film challenge set, including potential overlaps and hyperparameter selection? How diverse are the evaluation films in language, genre, and cultural background? How are variations across repeated Qwen and Gemini calls controlled?

  6. Evaluation: Could the authors consider subjective evaluations with blind audiences and professional describers, alongside complementary LLM-as-a-judge assessments? In particular, does the generated AD preserve both narrative information and cinematographic expression?

  7. Ethical considerations: The character-identification pipeline uses IMDb actor portraits. Could the authors clarify the licensing and permitted use of these images, particularly regarding likeness rights and potential redistribution?

6 comments
AuthorsOpenReview ↗

Response [1/2]

We thank the reviewer for their thorough reading and questions.

  • Problem motivation

    Previous MLLM approaches generate a one sentence description for one clip lasting a few seconds. The clip is what a human annotator determines is the part of the movie described in the reference human audio description. These approaches thus bypass "what" by only captioning the clip corresponding to the reference AD, and bypass "when" by never placing the generated description sentence in any dialogue gap. Our more realistic framework tackles both deciding which of the many visual details need to be described, and when they can be placed.

    Audio description of movies does not require describing details not central to the plot. This is, in part, because the available time for narration within dialogue gaps is so limited. Most audio description guidelines stipulate that the descriptions remain in the movie world, and cinematic details like camera position are rarely included (Vercauteren, 2007, 2016). Our system is designed with this in mind, and thus does not generate such descriptions.

  • Narration constraints

    Audio description guidelines suggest a speaking rate of 160 WPM (American Council of the Blind, Audio Description Project, 2010). Using the PTB tokenizer used in our evaluations, reference human audio description is narrated at speaking rates tightly centered around 200 WPM (based on the REFRAMED validation set). There is some variability between two human references, showing subjective preferences, but all videos have AD in 150-250 WPM. Future extensions of our framework could certainly add an adaptive speaking rate. Current human-created audio description accommodates the average audience member, but technology such as ours can enable more personalized audio description, meaning that an audience member that prefers more information spoken faster could specify this preference and so influence the system's output.

    We completed the requested sensitivity analysis of Delta max. Results for Qwen 3.5 27B on the ablation set with well-matched screenplays are as follows:

    Δmax⁡\Delta_{\max}CIDErMETEORSODA-MSODA-TQEvalQEval-T
    010.54.25.831.938.319.3
    112.45.97.241.343.424.7
    212.66.67.346.046.524.5
    314.07.27.150.044.722.4
    513.47.47.253.144.524.2
    1012.97.77.354.948.428.2
    1512.87.57.453.543.821.2
    3010.07.07.348.843.418.6

    The temporal metrics peak at Δmax⁡=10\Delta_{\max}=10 seconds, consistent with their design. More interestingly, QEval also peaks at this threshold, suggesting that moderate flexibility in narration placement benefits narrative understanding, while larger displacements can be detrimental. In contrast, CIDEr favours smaller displacement thresholds, likely because reference-based lexical evaluation rewards descriptions closely aligned with local visual events. This provides a possible explanation for the weaker CIDEr results in our original experiments. We therefore also evaluated Δmax⁡=3\Delta_{\max}=3 seconds on the full challenge set. This improves CIDEr for both Qwen (11.4 to 13.3) and Gemini (13.6 to 15.6), bringing performance closer to or above the LLM scheduler, with relatively small changes in the remaining metrics.

    CIDErMETEORSODA-MSODA-TQEvalQEval-T
    Qwen MILP11.47.97.753.643.622.1
    - Δmax⁡=3\Delta_{\max}=313.37.77.650.443.822.7
    Gemini MILP13.68.48.055.745.925.5
    - Δmax⁡=3\Delta_{\max}=315.68.57.855.445.924.8
AuthorsOpenReview ↗

Response [2/2]

  • Optimization formulation

    Our formulation requires joint content selection and scheduling (i.e., it does not at any point decide scheduling on its own), over potentially large numbers of candidate visual events, under explicit constraints. MILP is a natural way to encode and solve such a problem, but any solver can be used in our framework. We get guarantees about the output (e.g. no overlap with dialogue), which is essential for audio description. Mature and efficient general-purpose MILP solvers also exist, which we are able to take advantage of.

    The finite event candidates are constructed by segmenting a prose scene description into the events it describes (see Appendix C for more details). The compressed variants are generated by an LLM under explicit instructions (see Appendix D for more details).

    Weighting salience by narration duration assigns a value to each second of narration according to the importance of the event being described. This serves two purposes. Within an event, it favours fuller descriptions over compressed variants when time permits; without duration weighting, the solver would be indifferent between them. Across events, it prioritises narration time devoted to more salient content: several short, highly salient descriptions can displace a longer, less salient one. Thus, duration weighting allows the solver to balance content selection and compression under the available narration budget. We acknowledge that this assumes longer variants preserve more information, an assumption that could be refined by scoring the informativeness of individual variants.

  • Audiovisual understanding

    We agree that explicit audio understanding could improve our framework. Currently, event candidates are derived from visual information, without explicitly modelling its interaction with the soundtrack. Incorporating audio could improve event selection by identifying audiovisual events, such as a character switching off a radio as the music stops. It could also inform scheduling, for example by avoiding narration that duplicates information already conveyed through sound or interferes with important auditory cues. Our framework can accommodate these extensions through improved event candidates and additional scheduling constraints, although developing audiovisual understanding is beyond the scope of the present work.

  • Dataset and reproducibility

    The eight-film validation set was drawn from the training and validation splits of the 206-film corpus and used exclusively for framework development and hyperparameter selection. These films were selected based on the quality of their alignment with the corresponding screenplays, rather than genre diversity. The ten-film challenge set was held out for evaluation and covers a range of genres, reflecting the diversity of the REFRAMED dataset. All films and audio descriptions are in English, as this is the language for which suitable audio-description resources are most readily available.

    To assess sensitivity to stochastic LLM generation, we repeated the Gemini MILP pipeline five times on the full ten-film challenge set, independently regenerating the LLM-produced inputs. The table reports the mean and standard deviation across runs. Performance is stable across all six metrics, with standard deviations below 0.6 points. The results are also consistent with those reported in the paper, suggesting that our conclusions are not sensitive to variation in individual model calls.

    MetricOriginal paperMean ±\pm SD (5 runs)
    CIDEr13.613.37±0.4413.37 \pm 0.44
    METEOR8.48.35±0.068.35 \pm 0.06
    SODA-M8.07.92±0.037.92 \pm 0.03
    SODA-T55.756.05±0.6056.05 \pm 0.60
    QEval45.945.39±0.5945.39 \pm 0.59
    QEval-T25.524.52±0.4124.52 \pm 0.41
  • Evaluation

    We agree that evaluation with professional audio describers and blind or visually impaired audiences is essential for establishing the practical usefulness of generated audio descriptions. Our current study focuses on evaluating the proposed content selection and scheduling framework, rather than a complete, deployment-ready audio description system. We therefore use automatic metrics to assess temporal alignment and the preservation of narrative information, while recognising that these do not adequately capture fluency, coherence, or the listening experience. Human evaluation is an important next step, particularly once the quality of the underlying descriptions improves, and we will make this limitation clearer in the paper.

  • Ethical considerations

    We will not be distributing the IMDb actor portraits used in our face recognition pipeline. However, we will make the URLs of the image files available, so that other researchers can fetch them and use them for their work. This is permitted under Fair Use and research exemptions to copyright in most jurisdictions.

Thank you for the detailed rebuttal and additional experiments. I appreciate the sensitivity analysis of Δmax⁡\Delta_{\max}, the clarification of the MILP formulation, and the repeated-run evaluation, which address several of my technical concerns.

However, my main concern remains the lack of subjective evaluation with blind or visually impaired users and professional audio describers. While the automatic metrics demonstrate improvements, it is unclear whether these translate into better narrative comprehension, accessibility, or listening experiences. Additionally, the assumptions regarding fixed narration speed and duration-weighted salience remain insufficiently validated.

I acknowledge the technical contributions and improvements presented in the rebuttal. If the authors could provide further information on any existing subjective evaluation results or a concrete plan for user-centred evaluation, including the proposed methodology and evaluation criteria, I would be open to reconsidering and potentially increasing my rating.

Comments on the paper

2 comments

Dear Reviewer mctj and HnyZ,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best, AC

AuthorsOpenReview ↗

Summary of author response

We thank the reviewers for their constructive feedback. We have responded to all reviews and conducted several new experiments, which we believe strengthen the paper:

  • Stronger LLM baselines: We evaluated LLM schedulers given the full optimization objective and constraints. With sufficient reasoning, they approach MILP performance, but at substantially higher computational cost and without guarantees that constraints are satisfied.

  • Global optimization matters: A new constraint-aware greedy baseline performs consistently worse than MILP, particularly on temporal alignment and narrative QA, demonstrating the benefits of joint global optimization.

  • Human evaluation: In response to reviewer concerns, we conducted a small-scale blinded human study. MILP descriptions make substantially more narrative questions answerable than the fully instructed LLM scheduler (43.6% vs. 22.8%) and achieve higher coherence ratings (4.3 vs. 3.7). A separate compression analysis confirms high retention of key narrative information (4.4/5) with virtually no semantic contradictions.

  • Additional robustness analyses: We evaluated sensitivity to temporal constraints, selective omission of low-salience events, consistency across films, and variability across repeated runs. Results support the robustness of our main findings, particularly for temporal alignment.

  • Limitations and future work: We clarified the scope of our methods-focused contribution and outlined a concrete plan for evaluation with blind and visually impaired users and professional audio describers. We also addressed a statistical significance issue identified during rebuttal.

Reviewer feedback has been encouraging: Reviewer tgkp explicitly confirmed that their concerns were addressed and increased their rating; reviewer 5HVL also confirmed a weak accept. We hope the additional evidence will help resolve the remaining concerns during the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.