- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper formulates automatic Audio Description (AD) generation as a constrained global optimization problem that jointly determines what visual information to describe, when to deliver the descriptions, and how to formulate them within limited narration time. The proposed framework employs MLLMs to generate candidate descriptions, estimate their narrative salience, identify temporal spans, and produce compressed variants. A mixed-integer linear programming (MILP) solver then selects and schedules descriptions while satisfying temporal constraints. Experiments on REFRAMED demonstrate improvements over existing AD systems and LLM-based schedulers, particularly in temporal alignment and narrative QA metrics. Ablation studies further investigate the contributions of temporal grounding, compression, and salience estimation.
Critical strengths
-
Meaningful problem formulation. The paper focuses on an important limitation of existing AD systems, which typically assume predefined narration timestamps. Jointly considering content selection, temporal placement, and linguistic realization provides a more realistic formulation of automatic AD generation.
-
Principled optimization framework. The combination of LLM-generated candidates and MILP-based optimization is intuitive and well motivated. Explicitly enforcing temporal constraints offers greater control over narration scheduling than relying solely on LLM generation.
-
Promising improvements in narrative understanding. The proposed method achieves strong performance on temporally grounded and QA-based metrics, demonstrating the potential benefits of global optimization beyond conventional text-overlap evaluation.
Critical weaknesses
-
Insufficient validation of LLM-generated descriptions. The framework heavily relies on LLMs for event generation, temporal grounding, salience estimation, and description compression. However, the reliability of these intermediate outputs is not sufficiently evaluated. In particular, compression may remove critical details or introduce semantic inconsistencies, potentially affecting the final AD quality. A systematic evaluation of factual accuracy, grounding quality, and semantic preservation would strengthen the paper.
-
Lack of human evaluation. The experiments rely primarily on automatic metrics, which may not adequately capture the fluency, coherence, and practical usability of generated audio descriptions. Human evaluation, ideally involving professional describers or visually impaired users, would provide stronger evidence of the method's practical effectiveness.
-
Limited comparisons with simpler optimization strategies. Although the MILP solver outperforms LLM-based schedulers, the experiments do not establish whether global optimization is necessary compared with simpler constrained approaches, such as greedy selection or heuristic scheduling. Including these baselines would better isolate the benefits of the proposed optimization framework.
Questions for authors
Please see weaknesses. If the authors adequately address the above concerns, I would consider raising my score.
6 comments
Response [2/2]
-
Limited comparisons with simpler optimization strategies.
We thank the reviewer for this suggestion. We implemented a constraint-aware greedy baseline that processes candidate events in decreasing order of salience. At each iteration, the algorithm attempts to place the candidate and all previously selected events at their earliest feasible times in scene description order. The algorithm accepts the candidate if a feasible schedule is found and then proceeds to the next most salient candidate. The results below compare this approach with MILP under three temporal grounding settings.
System CIDEr METEOR SODA-M SODA-T QEval QEval-T Qwen MILP 12.9 7.7 7.3 54.9 48.4 28.2 Qwen Greedy - 12.5 6.6 7.0 49.6 44.0 22.1 - 11.4 7.1 7.2 49.3 45.9 23.8 - w/o grounding 6.9 5.7 7.5 38.4 40.5 14.9 MILP outperforms the greedy approach on nearly all metrics, with particularly clear improvements in temporal alignment and narrative QA. For example, at , SODA-T increases from 49.3 to 54.9 and QEval-T from 23.8 to 28.2. These results demonstrate the benefit of global optimisation: locally selecting and placing salient events can prevent better combinations of descriptions from being scheduled. MILP jointly considers these trade-offs and also allows additional objectives and constraints to be incorporated. We will include this comparison in the revised paper.
Thank you for the rebuttal and the constraint-aware greedy baseline, which strengthen the optimization comparison. I also appreciate the methods-focused scope and the detailed plan for user evaluation. My remaining concern is whether the selected compressions preserve core narrative information and form coherent descriptions in context. The ablations establish downstream usefulness, but, as acknowledged in the response, they do not directly validate compression fidelity. This matters because event-level salience is reused across variants, while duration weighting does not directly assess which information is retained.
A small sampled manual audit of selected compressions for key-fact retention and semantic errors, together with a lightweight blinded human comparison of contextual clarity and coherence against the fully instructed LLM scheduler, would help address these concerns using the existing textual outputs. The small and mixed narrative-QA differences relative to that LLM scheduler make these quality trade-offs particularly relevant, while the MILP’s explicit constraint guarantees remain a meaningful strength. I would be open to raising my score if the results substantively address the remaining concerns.
Response [1/2]
Insufficient validation of LLM-generated descriptions.
We agree that the quality of intermediate LLM outputs affects the final audio descriptions. Our ablations provide evidence of their contribution to downstream performance: removing temporal grounding substantially reduces temporal alignment, while removing compressed variants reduces QEval from 48.4 to 43.4 and QEval-T from 28.2 to 22.7. We also investigate the effect of different salience estimates and find that oracle salience does not necessarily improve temporally grounded metrics, highlighting the interaction between content selection and scheduling.
These experiments establish the usefulness of the intermediate components but do not directly measure their factual accuracy or semantic fidelity. In particular, compressed variants may omit important details or introduce inconsistencies, and our current formulation does not explicitly verify that they preserve the meaning of the original event. An advantage of our modular framework is that improved generation, grounding, and compression methods can be incorporated without changing the optimisation formulation. We will clarify these limitations and distinguish component-level effectiveness from the factual and linguistic quality of the resulting descriptions.
Lack of human evaluation.
We agree that human evaluation is essential to establish whether improvements in automatic metrics translate into better narrative comprehension, accessibility, and listening experiences. We would like to clarify the scope of our current contribution and outline our plans for user-centred evaluation.
Scope of the current work. Our contribution is a framework for selecting, compressing, and scheduling visual descriptions, rather than a complete, deployment-ready audio description (AD) system. In particular, we have not yet incorporated speech synthesis, which will need to be tailored to the delivery conventions and stylistic preferences of AD audiences. Consequently, our current evaluation focuses on the quality of content selection and temporal placement, rather than the end-to-end listening experience.
Planned user-centred evaluation. We have already taken steps towards recruiting blind and visually impaired AD users, including contacting national-level organisations representing these users, and are in discussions with professional audio describers experienced in working with AD users. We envisage an evaluation involving three complementary participant groups:
Blind and visually impaired AD users will assess narrative comprehension, engagement, and listening preferences. Participants will answer questions probing their understanding of the plot, characters, and events, and provide ranked preferences for alternative AD versions.
Professional audio describers will assess the relevance and completeness of selected visual information, the coherence and clarity of descriptions, and adherence to AD conventions, including narration timing and pacing.
Sighted participants will assess the correspondence between descriptions and visual events, including whether important information is omitted, whether descriptions are appropriately timed, and whether they interfere with dialogue or other meaningful audio.
Experimental design. We plan to compare automatically generated AD with professional descriptions and appropriate controls, including listening to the movie audio without AD. These comparisons will allow us to distinguish improvements in narrative comprehension from preferences for particular narration styles.
A rigorous evaluation across these groups requires careful experimental design, participant recruitment, and appropriate control groups, and we feel this is beyond the scope of the present, methods-focused study. Nevertheless, are are already taking concrete steps towards assessing the practical validity of the framework. We will clarify our evaluation plan and the limitations of our current automatic metrics in the revised paper.