acceptodds
Under review as a conference paper at ICLR 2027

SoundSense: Evaluating Visual Sound Understanding in Comics and Webtoons

Abstract

Vision and sound provide complementary evidence for understanding physical events. While generative models increasingly produce audiovisual content and vision–language models (VLMs) are widely evaluated on image captioning and visual question answering, their ability to infer auditory events from static imagery remains less explored. We introduce SoundSense, an evaluation framework and annotation resource for studying whether depicted moments warrant sound effects. Comics provide a useful testbed because artists explicitly represent selected sounds through onomatopoeia. On eight Manga109-s volumes, we use these annotations as a proxy for sound placement, remove the lettering through local inpainting, and analyze VLM predictions across 6,238 panel-level decisions. Complementing this proxy, we provide 450 author-confirmed sound-effect regions across four vertical-scroll webtoons, including an audio-backed core of 359 moments with 421 reference clips. Among 444 regions with lettering metadata, 65.5% contain no sound-effect lettering. We benchmark five VLMs, including two hosted open-weight model families, on a frozen scope of 808 webtoon segments. We report localization, output-format coverage, cohort differences and clustered bootstrap intervals. On 750 shared parse-valid segments, localization F1 ranges from 0.159 to 0.680. An output audit reveals that coordinate-format compliance and spatial matching materially affect these scores, highlighting the need to distinguish spatial output failures from sound judgments. SoundSense provides a reproducible basis for evaluating visual sound understanding, analyzing prediction errors, and developing sound-aware systems for visual narratives. We release the webtoon annotations, reference audio, and source links.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, multimodal understanding, visual sound grounding, benchmark, comics, webtoons

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eHnsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

SoundSense studies sound-effect placement in comics and webtoons. The manga study removes artist-written onomatopoeia and uses its original presence as a panel label. The webtoon benchmark uses regions, descriptions, and audio clips curated by an author. VLMs propose sound regions and descriptions, but the main evaluation scores region placement. The paper also examines presentation settings, coordinate formats, and matching rules.

Critical strengths
  1. I like the application. Comics contain plausible sound moments without visible lettering. Regions, descriptions, and reference clips could be useful for sound design in visual narratives.
  2. The paper reports coordinate-format failures, matching sensitivity, clustered uncertainty, and content-free baselines. It also discloses AI-assisted annotation and detectable inpainting. These details help readers understand the limits of the results.
Critical weaknesses
  1. I find it very hard to understand the paper itself. The wording used is a bit weird to me, which makes me feel like I'm reading an AI-generated paper.
  2. The manga labels include non-auditory mimetics, while absent lettering does not mean silence. I need a semantic audit across works before treating agreement with these labels as evidence of sound understanding.
  3. There is no independent agreement study, so missed events and alternative valid placements can become model errors. Reviewing selected model predictions with the same annotator does not establish reference completeness.
  4. The curator reads continuous chapters, while models receive individual segments. Some reference placements may rely on information the model never receives. This needs checking with annotators given the same input as the models.
  5. No independent listeners check whether the clips and descriptions fit the depicted events. The clips are also unused by the localization scorer, so they do not resolve the model evaluation's semantic gap.
  6. The main scorer ignores description meaning. A bird chirp could match a region annotated for a door impact. Unlike category-aware audio-visual event localization (https://arxiv.org/abs/2411.11278), this leaves sound identity unchecked.
Questions for authors

See Weaknesses.

Reviewer_mk4POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SoundSense studies sound-effect placement in static comics and webtoons. It combines two reference protocols: a manga analysis using artist-drawn onomatopoeia as a proxy, with the lettering removed through local inpainting, and a webtoon resource curated by one paper author with model assistance. The latter contains 450 reference regions across four works, including 359 moments associated with 421 reference audio clips.

The webtoon benchmark evaluates five VLMs on 808 requested segments, with the main comparison using 750 segments that have parse-valid responses from all models. The paper reports localization, segment-level sound presence, coordinate compliance, and several sensitivity analyses. Its main observations are that many curated placements lack sound-effect lettering, and that output-format and localization failures substantially affect the apparent differences between models.

Critical strengths

1.The move beyond lettering-based supervision is useful. In this corpus, 65.5% of the 444 regions with lettering metadata have no sound-effect lettering. This gives a concrete reason to study sound placement beyond recovering written onomatopoeia. Compared with the onomatopoeia-based scene-description task in MangaUB, the focus on free placement and unlettered moments provides a meaningful extension.

2.The evaluation audit is a substantial part of the contribution. The paper reports coordinate violations, alternative matching rules, parse coverage, simple baselines, and episode-clustered uncertainty. Separating segment-level proposals from localization makes the results more informative. These checks help identify what the benchmark scores actually measure, and the manuscript is appropriately cautious about interpreting them.

Critical weaknesses

1.The reliability of the reference set remains insufficiently established for a benchmark contribution. One paper author reviews model-generated proposals and makes the final placement decisions, without independent annotation or an estimate of agreement. This matters especially for the zero-reference segments: an unmatched proposal may be an unnecessary addition, a missed reference event, or another reasonable sound-design choice. The second-pass review is useful, but it involves the same annotator and only 20 selected candidates; it cannot establish the completeness of the references. The author also reads continuous chapters while models receive individual segments, leaving uncertainty about whether all reference decisions are supported by the available input. The paper acknowledges these issues, but the current experiments do not show how much they affect precision or model comparisons.

2.The main score provides limited evidence about the sound interpretation itself. Matching evaluates location without checking whether the predicted sound describes the reference event. A proposal can therefore receive credit at the right location while naming an inappropriate sound. The reference audio makes the curator’s intended interpretation more concrete, but that interpretation is not independently validated or used in scoring. Meanwhile, coordinate conventions have large effects on some models, and the content-free fixed-centre baseline reaches F1 = 0.298. The reported diagnostics are valuable, but a semantic audit of matched predictions is still needed to establish how well localization performance tracks sound understanding, even within the stated placement task.

Questions for authors

1.Is there any independent assessment of reference completeness and placement acceptability? A review of a stratified sample, including zero-reference segments, by annotators seeing only the model’s input and no candidate proposals would materially affect my assessment.

2.Among predictions counted as localization true positives, how often does the sound description identify an appropriate event? Does requiring semantic agreement substantially change the scores or model ordering?

3.Which model or models generated the initial annotation proposals for the original corpus and the GM extension? Clarifying their relationship to the evaluated models would help assess possible preference toward particular proposal patterns.

Reviewer_spshOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SoundSense evaluates where static comic images should receive sound effects. It uses two protocols: a manga analysis where onomatopoeia provides panel labels and is removed through inpainting, and a webtoon benchmark with sound-effect regions curated by one paper author, including a subset with reference audio.

Critical strengths
  1. The webtoon annotations include sound moments without lettering and segments with no selected sound. These are useful additions for evaluating when a model should add a sound effect.

  2. Separating segment presence from localization is also useful to interpret failures that could otherwise be interpreted as poor sound judgments.

Critical weaknesses

1.. The writing for the method section and the experiments looks sloppy and not well written. Much of the writing almost everything in the appendix especially seems entirely generated slop. To clarify I don't mind AI generated works but I really mind slop content.

  1. There is very little data in the benchmark, it contains only 450 annotated sound-effect regions from four works across 12 installments. There is no novelty in the dataset, analysis, or the insights from benchmarking models on such datasets.

  2. The webtoon references come from one author reviewing model proposals, with no independent annotation. Since several sound placements can be reasonable, it is unclear how much the reported low precision reflects inappropriate proposals, valid alternatives or missed references.

  3. The matching rule in Section 4.2 checks location but ignores the predicted sound description. A proposal can therefore count as correct even when its sound is incompatible with the depicted event.

  4. Section 6 states that the annotator reads continuous chapters, while the models receive isolated segments. Some reference placements may therefore depend on information unavailable in the evaluated image. The labels should be checked using the exact model inputs.

  5. Section 2 should discuss MangaUB’s Onomatopoeia Scene task explicitly. It already uses COO annotations to construct scene-description questions and evaluates a condition with the lettering removed?

Questions for authors
  1. Which VLMs generated the original webtoon annotation proposals and the proposals in Section 3.2? Were any of them also evaluated in Table 1?

  2. Can the webtoon models be evaluated on sound-placement decisions for a fixed set of regions, including negative regions, without generating coordinates?

  3. How many positive manga panels contain only nonauditory mimetics, and how do the results change when those cases are separated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.