acceptodds
Under review as a conference paper at ICLR 2027

Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models

Abstract

Puns represent a typical linguistic phenomenon that exploits polysemy and phonetic ambiguity to generate humour, posing unique challenges to the understanding of natural languages. Compared with text and images, spoken language plays a more central role in human communication. However, datasets and systematic resources for audio puns remain scarce, leaving this important modality largely underexplored. In this paper, we present APUN-Bench, the first benchmark dedicated to evaluating large audio language models (LALMs) on audio pun understanding. Our benchmark contains 4,447 audio samples annotated across three stages: pun recognition, pun word location and pun meaning inference. We conduct a deep analysis of APUN-Bench by systematically evaluating 10 state-of-the-art LALMs, uncovering substantial performance gaps in recognizing, localizing, and interpreting audio puns. Motivated by our analyses of positional biases in pun localization and error patterns in pun meaning inference, we further propose a prompt-based strategy and a fine-tuning framework to improve audio pun understanding. Our findings provide actionable insights toward advancing humour-aware audio intelligence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
pun understanding, reasoning, knowledge graph

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fLdKOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
3
Presentation
2(−1)
Soundness
1(−1)
Read review
Paper summary

This paper introduces APUN-Bench for evaluating spoken pun recognition, localization, and meaning inference using synthetic speech and human recordings. It compares audio-language models, ASR–LLM pipelines, and transcript inputs, analyzes characteristic errors, and explores position-aware prompting and SFT/DPO adaptation. The principal contribution is the benchmark and its diagnostic evaluation.

Critical strengths
  1. APUN-Bench provides a useful resource for studying spoken wordplay, testing whether models recover alternative interpretations rather than merely recognize an utterance.

  2. The recognition, localization, and inference tasks provide informative diagnostics. Section 4.4 particularly highlights explanations that recover only one of a pun's meanings.

  3. The audio-model, ASR–LLM, and error-free-transcript comparisons help separate speech-processing difficulties from downstream reasoning. The benchmark also includes human verification of audio and annotations.

Critical weaknesses
  1. The relationship between audio puns and textual wordplay deserves clearer framing. Established pun definitions already include phonological similarity alongside lexical ambiguity [1]; a reader can recognize homophonic wordplay in writing. Could the authors distinguish evaluating puns presented through speech from evaluating interpretations that depend on a particular acoustic realization? Section 3's linguistic categories do not establish that distinction by themselves. This does not require a separate, strictly audio-specific category. It would also help explain how exact-match evaluation treats identical-sounding alternatives when spelling choice and understanding need not coincide.

  2. The acoustic-mechanism interpretation could be more cautious. The existing ASR baselines, transcripts, and WER analysis are useful. However, Section 4.2's “pronunciation cues that are unavailable in text” overlooks pronunciation knowledge accessible to text models. The small MiniCPM audio advantage does not isolate an acoustic cause, and Appendix F does not specify whether its two speakers read matched content. Could the authors illustrate when hearing the utterance changes its interpretation, rather than only its transcription? A narrower interpretation of these results would also address this concern.

  3. The speech-source mixture needs clearer characterization. Appendix B combines TTS-rendered written puns, competition recordings, and participant readings of website texts. The filtering and expert checks in Section 3 and Appendix E are reassuring, including verification of pun presence and alternative meanings. Reporting the competition/read-aloud proportions would nevertheless help readers understand the “real-world” subset. A brief discussion of how these delivery conditions may affect the findings would clarify the benchmark's coverage; matched synthetic/human recordings would be a useful future extension.

[1] Miller, T., Hempelmann, C., & Gurevych, I. (2017). SemEval-2017 Task 7: Detection and interpretation of English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) (pp. 58–68). Association for Computational Linguistics. doi:10.18653/v1/S17-2005.

Minor Issues

  • Table 2's synthetic MiniCPM “Pun (%)” is below the true-positive fraction implied by its recall and Table 1's class counts. Please reconcile these entries.
  • Appendix G should identify the similarity function, explain how its threshold was selected, and specify that matching covers both distinct gold meanings.
Questions for authors
  1. Audio presentation versus acoustic dependence. Beyond the linguistic categories already defined in Section 3, how do the authors distinguish puns presented through speech from interpretations that depend on a particular acoustic realization? Could examples across the existing categories illustrate when listening changes the interpretation rather than only the transcription, taking into account pronunciation knowledge available to text models?

  2. Interpreting the model comparisons. Given the competitive ASR–LLM pipelines and the error-free-transcript comparisons, what do the results establish about speech perception versus downstream reasoning as bottlenecks? In particular, what supports attributing MiniCPM's homophonic-inference advantage to acoustic information, and did the two speakers in Appendix F read matched content?

  3. Preserving ambiguity in a hybrid pipeline. Beyond the existing ASR–LLM baselines, have the authors considered using an audio-language model to produce a transcript or several plausible word/transcription alternatives, followed by a larger text model for reasoning? Could retaining alternatives help avoid committing prematurely to one interpretation of a pun? ( I raise this as a possible extension and discussion point, not an acceptance requirement. )

  4. Audio-source coverage. Appendix B reports the overall synthetic/real-world counts and the random assignment of each synthetic text to one of 34 voices. Within the real-world subset, what proportion comes from competition recordings versus participant readings, and is there any textual overlap with the synthetic subset? A brief account of these conditions and their possible effect on the findings would clarify the benchmark's coverage; matched human/synthetic or multi-voice recordings would be optional extensions.

  5. Meaning-matching evaluation. In Appendix G, what similarity function is used, how was the threshold selected ?

7 comments

We sincerely appreciate your comments.

Q1 & W1

The goal of APUN-Bench is to evaluate the complete process of audio pun understanding, from speech perception to pun detection, word localization, and meaning inference, rather than to claim that every sample requires acoustic cues beyond textual information. Our benchmark includes samples with varying degrees of acoustic dependence.

For example, in the homographic pun “A bank manager who was also a high jumper spent most of his time in the vault,” the two meanings of vault, a bank vault and a movement in high jumping, are already preserved in the ground-truth transcript. Thus, this example primarily evaluates semantic reasoning.

For homophonic puns, the phonological relationship is central to the wordplay, but interpreting the pun does not necessarily require a particular acoustic realization. For instance, in “Two construction workers had a stairing contest,” stairing and staring share the same pronunciation, allowing a text-based model with phonological knowledge to recognize the pun. Indeed, both SenseVoice and Whisper transcribed the annotated word stairing as staring, illustrating that the audio alone does not uniquely determine the intended spelling.

In contrast, some examples are sensitive to connected speech, pauses, and word-boundary interpretation. Consider the real-world example “I'm so purrfect that whenever I meet a pretty girl, I whisker away,” which involves the alternative interpretations whisker away and whisk her away. Whisper transcribed the relevant phrase as whisk her away, illustrating how similar spoken sequences can lead to different lexical interpretations depending on word segmentation and connected-speech processing. In such cases, acoustic perception can influence not only transcription but also the interpretations considered by the model.

Q2 & W2

This conclusion is supported by audio–transcript comparisons conducted on the same samples. With ground-truth transcripts, the pun-localization accuracies of GPT-4o, Gemini, and MiniCPM improve by 42.95%, 34.19%, and 40.11%, respectively. This substantial gap indicates that accurately recovering ambiguous lexical forms from speech remains a major bottleneck for current LALMs. The competitive localization performance of the ASR–LLM pipelines, together with the higher transcription error rates on real-world speech reported in Appendix C, further supports this observation.

However, eliminating transcription errors does not fully resolve the task. Even with ground-truth transcripts, MiniCPM-Text achieves only 47.53% and 40.22% accuracy on heterographic and homographic meaning inference, respectively, demonstrating persistent challenges in lexical interpretation, contextual integration, and dual-meaning reasoning. Thus, our three-stage evaluation reveals two interconnected bottlenecks: speech perception, reflected in the substantial audio–text localization gap, and downstream semantic reasoning, reflected in the remaining inference errors under ground-truth transcription.

Regarding Appendix F, we clarify that the two speakers recorded matched textual content, and we analyzed 276 samples to investigate the effects of acoustic variation on model performance.

Q3

We thank the reviewer for this valuable suggestion. We agree that our current ASR–LLM baselines rely on a single best transcription, which may prematurely resolve lexical ambiguities before downstream reasoning. We primarily include these cascaded pipelines as baselines to disentangle the contributions of speech transcription and language reasoning, rather than as a definitive solution to audio pun understanding. Our research focus remains on end-to-end multimodal models capable of jointly perceiving speech and reasoning about its multiple meanings.

Our error analysis further illustrates the limitations of single-best transcription. For example, best cellars was transcribed as best sellers by both ASR systems, while whisker away could be transcribed as whisk her away. In such cases, committing to a single spelling or word segmentation may discard plausible lexical alternatives, making it more difficult for downstream language models to recover the intended wordplay.

A promising extension would be a hybrid pipeline that preserves multiple hypotheses through N-best transcripts, or lexical, word-segmentation, and phoneme-level candidates with associated confidence scores. A stronger text-based LLM could then jointly reason over these alternatives using contextual information. For instance, a pun-specific lexical knowledge base could provide candidate pairs such as cellars/sellers and whisker/whisk her, rather than requiring the ASR system to select a single interpretation before reasoning begins. We view this as a valuable direction for future work that complements our primary focus on end-to-end audio-language understanding.

Q4 & W4

Among the 899 samples in the Real-world Dataset, approximately 10% (78 samples) are excerpts from the competition recordings, while the remaining samples cover other real-world speech sources, including participant readings, providing natural speech characteristics and recording conditions distinct from synthetic audio. We further conducted a text-overlap check and found no exact textual duplicates between the Real-world and Synthetic subsets. We acknowledge that differences in speaking styles and recording conditions across sources may introduce variability. Therefore, we do not treat these samples as a strictly controlled acoustic comparison. Instead, they are intended to complement the synthetic data by increasing the diversity of speech conditions.

First Minor Issue

We re-examined the original prediction records and the process used to compile Table 2. We found that some of the MiniCPM-o-2.6 statistics were inconsistent across versions of the compiled results. In particular, the Pun(%) entry retained an outdated value from an intermediate evaluation output, and other statistics also required updating.

The Synthetic Dataset contains 3,548 samples, including 2,396 puns and 1,152 non-puns. After recomputing the metrics from the sample-level predictions, the MiniCPM-o-2.6 confusion matrix is:

Predicted Non-punPredicted Pun
Actual Non-punTN = 624FP = 528
Actual PunFN = 777TP = 1,619

The corresponding corrected metrics are:

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
MiniCPM-o-2.663.2275.4167.5771.2760.51

The corrected Accuracy, Precision, Recall, F1, and Pun(%) are directly verifiable from this confusion matrix.

We further extended the audit to all models on both the Synthetic and Real-World Datasets:

Synthetic Dataset

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
Qwen2-Audio-Instruct45.9767.7838.1148.7837.97
SALMONN51.4468.4852.0559.1451.32
Audio-Flamingo-337.0654.6739.8646.1049.24
Qwen-2.5-Omni64.0975.7168.9572.1761.50
MiniCPM-o-2.663.2275.4167.5771.2760.51
Omni-R165.2274.3973.9674.1767.14
MERaLiON271.6572.3993.7881.7187.49
Gemini 2.0 Flash80.5089.6380.4384.7860.60
GPT4o-Audio75.7375.6394.5384.0384.41
SenseVoiceSmall + GPT4o79.4382.2488.6985.3472.83
SenseVoiceSmall + Gemini78.9781.8588.4885.0473.00
Whisper-Large + GPT4o80.8685.4386.3985.9168.29
Whisper-Large + Gemini78.5282.2486.9884.5471.42

Real-World Dataset

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
Qwen2-Audio-Instruct57.8695.1136.8353.1025.08
SALMONN44.9357.0760.4158.7068.56
Audio-Reasoner68.1289.9757.1469.8941.14
Audio-Flamingo-363.6697.4045.0961.6529.99
Qwen-2.5-Omni71.6899.1056.8072.2137.12
MiniCPM-o-2.674.3698.0861.6275.6940.69
Omni-R183.5097.3776.5985.7450.95
MERaLiON288.1887.0596.0491.3371.46
Gemini 2.0 Flash86.2984.2896.9090.1574.47
GPT4o-Audio84.6286.8689.8588.3267.00
SenseVoiceSmall + GPT4o84.9596.4679.6987.2853.51
SenseVoiceSmall + Gemini83.7292.5681.4186.6356.97
Whisper-Large + GPT4o85.0698.2778.3187.1651.62
Whisper-Large + Gemini83.0595.3577.6285.5852.73

Although some numerical results have been corrected, the principal observations and conclusions regarding pun recognition remain substantively unchanged. Strong proprietary systems and several cascaded systems still generally perform well on pun recognition, whereas fine-grained pun localization and dual-meaning inference remain challenging. This supports our central observation that detecting a pun does not necessarily imply the ability to localize the punning word accurately or infer both meanings.

Q5 & Second Minor Issue

As clarification of the scoring details in Appendix G, we use sentence-transformers/all-MiniLM-L6-v2 embeddings and cosine similarity for predicted and gold meaning phrases. A Hungarian one-to-one assignment maximizes the sum of matched similarities, and a response is accepted only when both matched meaning pairs exceed the fixed threshold τ = 0.30. During evaluation-protocol development, we empirically selected this threshold by examining cosine-similarity scores and manually reviewing the matching outcomes on a small random sample of homographic puns. To assess whether our main conclusions depend strongly on τ = 0.30, we held all original model predictions fixed and reran the scoring pipeline on the same 1,298 homographic samples under five thresholds:

τ∈{0.20,0.25,0.30,0.35,0.40}.\tau\in\{0.20,0.25,0.30,0.35,0.40\}.

At every threshold, we used the same configurations. Missing or unparseable outputs were counted as incorrect. The resulting accuracies (%) are:

Modelτ = 0.20τ = 0.25τ = 0.30τ = 0.35τ = 0.40
Qwen2-Audio42.5328.1217.0310.406.86
Audio-Flamingo-359.6343.9929.5118.8011.40
Qwen2.5-Omni63.0246.2232.2022.4214.79
MiniCPM-o-2.665.7249.0837.1326.7319.26
Omni-R166.4953.2438.7529.8220.96
MERaLiON-259.6341.9929.2820.1814.41
GPT-4o76.8965.7256.7848.2339.75

As τ\tau increases, the correctness criterion becomes stricter, and the absolute accuracy of every model decreases monotonically. To assess the stability of model comparisons, we also computed Spearman rank correlations against the ranking at τ=0.30\tau=0.30 across these nine models:

Threshold τ\tauSpearman correlation with τ=0.30\tau=0.30
0.200.996
0.251.000
0.301.000
0.350.950
0.400.950

The rank correlations remain consistently high (ρ≥0.950\rho \geq 0.950), indicating that model rankings are largely stable across the evaluated thresholds. These results suggest that, although absolute accuracy varies with the similarity threshold, our main comparative findings remain robust within the evaluated range.

Once again, we thank you for your time and valuable feedback.

Reviewer_JwSQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose APUN-Bench, which is claimed to be the first benchmark for evaluating large audio-language models (LALM) on audio pun understanding.

The benchmark contains synthetic clips and real-world clips. The synthetic clips come from the SemEval-2017 pun text corpus and are rendered with Parler-TTS across 34 synthetic speakers. The real-world clips come from the O. Henry Pun-Off videos, pun websites, and recordings by 10 volunteers.

The authors evaluate 10 LALMs and 4 cascaded ASR+LLM systems, and also compare text-only with audio-only inputs. They find that models are weak at fine-grained location and at meaning inference, and that cascaded systems beat LALMs on location and inference. Motivated by two analyses, they propose two improvements. The first is PunPAP, a prompt that tells the model to attend to the latter half of the sentence, motivated by a positional bias in predicted pun locations. The second is PunAlign, an SFT+DPO pipeline built on an error taxonomy for pun inference.

Critical strengths
  • S1: Mix of synthetic and real-world data, where the former provides scale and control when the latter adds some natural accents and noise.
  • S2: The evaluation spans 10 open and proprietary LALMs plus 4 cascaded systems as broad model coverage.
  • S3: The comparison of transcript-only and audio inputs and the WER analysis help separate perception errors from reasoning errors.
  • S4: The positional analysis motivates PunPAP when the error taxonomy motivates the DPO preference construction in PunAlign.
Critical weaknesses
  • W1: Synthetic clips are TTS renderings of text puns. Parler-TTS pronounces the written word with no pun-specific prosody, and for homophonic puns the audio is identical to the un-punned sentence. Hence, the audio could carry little pun-specific signal in most of the data.
  • W2: Verification is weak and under-specified. The 6% correction rate is reported only for the homographic synthetic annotations when the independent verification covers only a random sample of 300 items.
  • W3: The improvement methods are modest and weakly validated.
  • W4: There could be potential model hacking. In the synthetic set the negatives are much shorter than the puns. In the real-world set negatives come from different sources than positives, and positives are partly re-recorded by volunteers. Models could exploit length, source and recording condition.
Questions for authors
  • Q1: How much of the benchmark requires acoustic information at all? For the synthetic set, what is the accuracy of a text-only model on the ground-truth transcripts, and how many items are solvable from text alone, compared with items where audio is essential?
  • Q2: Real-world labels were produced with Whisper and Claude-Opus-4-1. What is the label error rate on a human-checked sample, broken down by pun type?
  • Q3: For homographic inference, which similarity function SS is used, and why τ\tau = 0.3? Please give the sensitivity of rankings and accuracies to τ, and the false-positive rate of this criterion on generic or one-sided answers.
  • Q4: How does PunPAP perform on the minority of puns that occur early in the sentence?
6 comments

Thank you for the valuable suggestion!

W1 Q1

We agree that synthetic speech may lack prosodic cues specifically designed to signal puns. In particular, for homophonic puns, the punning word and its alternative may have identical pronunciations [1]. However, the absence of explicit pun-specific prosodic cues does not necessarily imply that acoustic information is irrelevant to audio pun understanding. Homophonic, near-homophonic, and lexical ambiguities in spoken language still pose significant challenges for understanding puns from audio [2].

As shown in Table 2, end-to-end audio-language models consistently outperform cascaded models, particularly in pun recognition, on both synthetic and real-world datasets. As shown in Appendix Table 10, we further compare the inference efficiency of end-to-end and cascaded approaches. On average, end-to-end models achieve a 1.6× inference speedup while requiring only 0.87× as many parameters as cascaded models. These results suggest that directly processing acoustic input may offer advantages over relying on intermediate transcriptions. Furthermore, in Section 4.2, we report the performance of a text-only model using ground-truth transcripts from the synthetic dataset, providing an additional reference for understanding the role of transcription in audio pun comprehension.

Motivated by these observations, APUN-Bench aims to evaluate whether models can directly extract linguistic information from raw acoustic signals and integrate it with contextual reasoning to recognize and interpret potential dual meanings. To further investigate the challenges associated with transcription-based approaches, we extend the transcription error analysis in Appendix Table 9 by examining the frequency and overlap of errors made by two automatic speech recognition (ASR) systems across different pun types. This analysis helps quantify the limitations of ASR-based pipelines and assess the potential benefits of directly processing acoustic input. The detailed results are presented below:

DatasetTypeNSenseVoiceSmall errorsWhisper-Large errorsBoth incorrect
SyntheticHeg.794619 (77.96%)499 (62.85%)477 (60.08%)
SyntheticHog.1,298819 (63.10%)623 (48.00%)598 (46.07%)
SyntheticHop.304271 (89.14%)249 (81.91%)241 (79.28%)
Synthetic total—2,3961,709 (71.33%)1,371 (57.22%)1,316 (54.92%)
Real-worldHeg.223196 (87.89%)179 (80.27%)170 (76.23%)
Real-worldHog.285210 (73.68%)151 (52.98%)137 (48.07%)
Real-worldHop.7368 (93.15%)67 (91.78%)65 (89.04%)
Real-world total—583474 (81.58%)397 (68.33%)372 (64.03%)

On the synthetic pun samples, both systems made transcription errors on 1,316 of 2,396 items (54.92%); on the real-world pun samples, the corresponding rate was 64.03%. For homophonic puns, the two-system error overlap was particularly high (79.28% for synthetic and 89.04% for real-world speech). These observations highlight the difficulty of faithfully recovering ambiguous spoken expressions through the tested ASR systems. Consequently, a pipeline that relies only on their generated transcripts may lose information relevant to subsequent pun interpretation. End-to-end audio-language models can instead reason directly over the original acoustic input without committing to a single intermediate transcript. As shown in Appendix Table 10, we further compare the inference efficiency of end-to-end and cascaded approaches. On average, end-to-end models achieve a 1.6× inference speedup while requiring only 0.87× as many parameters as cascaded models.

Overall, these findings highlight the importance of evaluating pun understanding in end-to-end audio-language models. Unlike cascaded approaches, end-to-end models can directly process acoustic signals, potentially preserving information lost during transcription and avoiding error propagation from ASR systems. Together with their superior empirical performance and inference efficiency, these advantages motivate the evaluation of end-to-end models in APUN-Bench to better understand their ability to recognize and interpret puns directly from speech.

For the synthetic set, we report the accuracy of a text-only model on the ground-truth transcripts in section 4.2.

[1] Attardo, S. (2020). The linguistics of humor: An introduction. Oxford university press.

[2]. Miller, T., Hempelmann, C. F., & Gurevych, I. (2017, August). SemEval-2017 Task 7: Detection and interpretation of English puns. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017) (pp. 58-68).

W3

We agree that the improvements from PunPAP are modest in absolute terms. However, PunPAP is deliberately designed as a lightweight, training-free intervention. It improves both localization metrics for both evaluated models: for Qwen2.5-Omni, coarse- and fine-grained accuracy increase by 2.51 and 1.55 percentage points, respectively, while MiniCPM-o-2.6 gains 2.38 and 1.80 points. These consistent gains provide initial evidence that the positional analysis can inform a useful prompting strategy.

The gains from PunAlign are more substantial:

ModelHeg.+Hop.Hog.
Qwen2.5-Omni42.27 → 58.64 (+16.37 pp)32.38 → 49.22 (+16.84 pp)
MiniCPM-o-2.646.12 → 55.25 (+9.13 pp)33.07 → 53.31 (+20.24 pp)

Table 5 further compares the vanilla baseline, DPO-only, SFT-only, and SFT+DPO settings. The combined method achieves the best results for both models and both task groups, supporting the complementary contributions of supervised fine-tuning and error-aware preference optimization.

W4

We agree that sentence length, audio duration, source, and recording conditions can introduce shortcut cues for pun recognition. To assess how much of the observed performance can be attributed to length and duration, we constructed matched evaluation subsets. Each non-pun sample was paired with a pun sample having similar word count and audio duration, without reusing samples and excluding pairs with excessive discrepancies.

DatasetMatched samplesWords: Pun / Non-punDuration: Pun / Non-pun
Synthetic1,7249.20 / 9.123.02 / 2.99 s
Real-world3749.52 / 9.525.78 / 5.85 s

We then trained a simple classifier using only word count and audio duration as input. Its balanced accuracy on the original synthetic and real-world datasets was 67.13% and 67.53%, respectively, confirming that these features carry some label-associated information in the original benchmark. On the matched subsets, performance dropped to 47.27% and 45.72%, respectively, close to chance. Thus, these two features alone no longer provide effective discrimination on the matched samples.

In contrast, the evaluated audio-language models retained broadly comparable performance after matching:

ModelOriginal BAcc.Matched BAcc.Change
MiniCPM-o-2.6 (Synthetic)60.8758.53-2.34
MiniCPM-o-2.6 (Real-world)79.7078.34-1.36
Omni-R1 (Real-world)86.4085.56-0.84
Qwen2.5-Omni (Real-world)77.9277.54-0.38
GPT-4o-Audio (Real-world)80.9882.55+1.57
Gemini-2.0-Flash (Real-world)81.8479.14-2.69

These results suggest that word count and audio duration are potential shortcuts in the original data, but are insufficient by themselves to explain the models' overall performance.

We also acknowledge that Appendix E.4 did not describe the collection process for real-world negative examples clearly enough. Negative samples were also recorded by volunteers, and some were collected from Pun Off. We will revise the appendix to explicitly document the sources and recording procedures for both classes, rather than leaving the impression that positive and negative examples necessarily followed different collection pipelines.

Q3

As clarification of the scoring details in Appendix G, we use sentence-transformers/all-MiniLM-L6-v2 embeddings and cosine similarity for predicted and gold meaning phrases. A Hungarian one-to-one assignment maximizes the sum of matched similarities, and a response is accepted only when both matched meaning pairs exceed the fixed threshold τ = 0.30. During evaluation-protocol development, we empirically selected this threshold by examining cosine-similarity scores and manually reviewing the matching outcomes on a small random sample of homographic puns. To assess whether our main conclusions depend strongly on τ = 0.30, we held all original model predictions fixed and reran the scoring pipeline on the same 1,298 homographic samples under five thresholds:

τ∈{0.20,0.25,0.30,0.35,0.40}.\tau\in\{0.20,0.25,0.30,0.35,0.40\}.

At every threshold, we used the same configurations. Missing or unparseable outputs were counted as incorrect. The resulting accuracies (%) are:

Modelτ = 0.20τ = 0.25τ = 0.30τ = 0.35τ = 0.40
Qwen2-Audio42.5328.1217.0310.406.86
Audio-Flamingo-359.6343.9929.5118.8011.40
Qwen2.5-Omni63.0246.2232.2022.4214.79
MiniCPM-o-2.665.7249.0837.1326.7319.26
Omni-R166.4953.2438.7529.8220.96
MERaLiON-259.6341.9929.2820.1814.41
GPT-4o76.8965.7256.7848.2339.75

As τ increases, the acceptance criterion becomes stricter and each model's absolute accuracy decreases monotonically. To assess the stability of model comparisons, we computed Spearman rank correlations relative to the ranking at τ = 0.30 across the nine models:

Threshold τSpearman correlation with τ = 0.30
0.200.996
0.251.000
0.301.000
0.350.950
0.400.950

The correlations remain high (ρ ≥ 0.950), indicating that model rankings are largely stable throughout the tested threshold range. Although absolute accuracy depends on the threshold, our main comparative findings remain robust within this range.

We also evaluated false acceptances using two kinds of negative controls. For each of the 1,298 homographic samples, we constructed five fixed, instance-independent generic answer pairs:

(1) literal meaning, figurative meaning

(2) ordinary interpretation, alternative interpretation

(3) direct meaning, humorous meaning

(4) one possible meaning, another possible meaning

(5) a common usage, a wordplay interpretation.

This yielded 6,490 generic negative controls, only 6 of which were incorrectly accepted at τ = 0.30 (0.09% false-acceptance rate).

As a more demanding oracle stress test of one-sided interpretations, we constructed two controls per sample by duplicating either gold meaning in two predicted meanings, namely (m1, m1) and (m2, m2), for 2,596 controls in total. The evaluation metric incorrectly accepted 15.02% of these deliberately adversarial examples. This figure does not represent the error rate on natural model outputs. Instead, it shows that the metric can occasionally mistake one correct meaning repeated twice for an answer that covers both meanings.

Q4: How does PunPAP perform on the minority of puns that occur early in the sentence?

We thank the reviewer for drawing attention to a potential positional trade-off. We further analyzed the precise position of each punning word. For a sentence of L words with the punning word at zero-based index i, we define its normalized position as:

r=i+1L.r=\frac{i+1}{L}.

Among the 2,396 synthetic pun samples, 1,965 (82.01%) have the punning word in the final 20% of the sentence, 2,288 (95.49%) in the second half, and only 108 (4.51%) in the first half. Thus, PunPAP exploits a pronounced sentence-final prior, consistent with the common tendency for punchlines to appear near the end of an utterance. We re-evaluated fine-grained localization on a consistent set of 2,396 samples, stratified by whether the punning word appears in the first or second half:

ModelPun positionVanillaPunPAPChange
Qwen2.5-OmniFirst half (4.51%)32.4126.85-5.56
Qwen2.5-OmniSecond half (95.49%)49.1750.79+1.62
MiniCPM-o-2.6First half (4.51%)20.3720.37+0.00
MiniCPM-o-2.6Second half (95.49%)45.8947.51+1.62

For Qwen2.5-Omni, the decline on first-half puns contributes approximately -0.25 percentage points to overall accuracy (4.51% × -5.56), whereas the improvement on second-half puns contributes approximately +1.55 points (95.49% × 1.62). The net gain is therefore approximately +1.29 points, from 48.41% to 49.71%. For MiniCPM-o-2.6, first-half performance is unchanged, while the second-half improvement yields an overall gain of approximately +1.54 points, from 44.74% to 46.29%.

These results show that PunPAP's benefit primarily arises from the dominant second-half subset, while also identifying its limitations: it does not guarantee improvements for early-position puns and can reduce performance on that minority subset for Qwen2.5-Omni. We will therefore frame PunPAP more precisely as a training-free strategy that leverages a strong sentence-final prior, rather than as a position-independent improvement.

We sincerely thank the reviewer for the valuable suggestions and constructive feedback. We will carefully revise the manuscript, improve the wording, and thoroughly verify the reported statistics. We hope that our responses have addressed your concerns, and we would greatly appreciate your reconsideration of our evaluation.

Reviewer_HwKnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces APUN-Bench, a benchmark for evaluating large audio language models’ understanding of spoken puns. The dataset comprises 4,447 synthetic and real-world audio samples and organizes evaluation into three stages: pun detection, pun-word localization, and meaning inference. The paper compares 10 audio language models and four cascaded systems, analyzes positional biases and inference errors, and proposes PunPAP, a position-aware prompting strategy, and PunAlign, an alignment approach based on SFT and DPO.

Critical strengths

The staged benchmark design offers diagnostic value. Separating pun detection, word localization, and meaning recovery helps distinguish failures at different levels. The broad comparison of models and cascaded systems, together with text-based controls, human verification, and training ablations, also makes the benchmark a useful resource for future research.

Critical weaknesses

W1. The reporting conventions for pun detection metrics are unclear. If recall is computed for the positive class on the same set of samples, MiniCPM’s recall implies that true positives account for approximately 44.42% of all samples, exceeding the 27.92% predicted-positive rate reported in Table 2. The paper does not clarify whether different aggregation methods are used, making these metrics difficult to reconcile.

W2. Key details of the test split and meaning-inference scoring are missing. The source and size of the independent test set, as well as its relationship to the 80/20 training/validation split, are unclear. The scoring of homographic pun meanings also lacks sufficient detail on the similarity model, sense-matching rules, and the calibration basis for the threshold τ = 0.3, limiting the reproducibility of the results.

Questions for authors

Q1. Are all metrics in Table 2 based on the same samples and predictions? Please clarify whether the metrics are positive-class or macro-averaged, specify the denominator for Pun(%), and provide confusion matrices to explain the inconsistency in MiniCPM’s results.

Q2. Please specify the sizes of the training, validation, and test sets and the splitting procedure, clarify the relationship between the test sets used in Tables 2 and 5, and provide text-level deduplication and cross-split overlap checks.

3 comments

We sincerely appreciate your comments. We provide the following explanations and supplementary information regarding your questions.

W1 and Q1 Please clarify whether all metrics use the same samples and predictions, specify positive-class versus macro-averaging and the definition of Pun%, and provide confusion matrices to resolve the inconsistency.

1. Definition and computation of evaluation metrics

We first clarify our evaluation conventions for pun recognition. Precision, Recall, and F1 are computed for the positive class. Accuracy measures classification accuracy over all evaluated samples, while Pun(%) denotes the proportion of all evaluated samples predicted as puns:

Pun=TP+FPN×100.\mathrm{Pun}=\frac{TP+FP}{N}\times100.

Here, TPTP and FPFP denote true positives and false positives, respectively, and NN is the total number of evaluation samples.

For each model, all five metrics are calculated from the same sample-level predictions and their corresponding confusion matrix, using a consistent evaluation pipeline.

2. Identification and correction of reporting inconsistencies

We re-examined the original prediction records and the process used to compile Table 2. We found that some of the MiniCPM-o-2.6 statistics were inconsistent across versions of the compiled results. In particular, the Pun(%) entry retained an outdated value from an intermediate evaluation output, and other statistics also required updating.

The Synthetic Dataset contains 3,548 samples, including 2,396 puns and 1,152 non-puns. After recomputing the metrics from the sample-level predictions, the MiniCPM-o-2.6 confusion matrix is:

Predicted Non-punPredicted Pun
Actual Non-punTN = 624FP = 528
Actual PunFN = 777TP = 1,619

The corresponding corrected metrics are:

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
MiniCPM-o-2.663.2275.4167.5771.2760.51

The corrected Accuracy, Precision, Recall, F1, and Pun(%) are directly verifiable from this confusion matrix.

We further extended the audit to all models on both the Synthetic and Real-World Datasets:

Synthetic Dataset

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
Qwen2-Audio-Instruct45.9767.7838.1148.7837.97
SALMONN51.4468.4852.0559.1451.32
Audio-Flamingo-337.0654.6739.8646.1049.24
Qwen-2.5-Omni64.0975.7168.9572.1761.50
MiniCPM-o-2.663.2275.4167.5771.2760.51
Omni-R165.2274.3973.9674.1767.14
MERaLiON271.6572.3993.7881.7187.49
Gemini 2.0 Flash80.5089.6380.4384.7860.60
GPT4o-Audio75.7375.6394.5384.0384.41
SenseVoiceSmall + GPT4o79.4382.2488.6985.3472.83
SenseVoiceSmall + Gemini78.9781.8588.4885.0473.00
Whisper-Large + GPT4o80.8685.4386.3985.9168.29
Whisper-Large + Gemini78.5282.2486.9884.5471.42

Real-World Dataset

ModelAcc. (↑)Pre. (↑)Rec. (↑)F1 (↑)Pun (%)
Qwen2-Audio-Instruct57.8695.1136.8353.1025.08
SALMONN44.9357.0760.4158.7068.56
Audio-Reasoner68.1289.9757.1469.8941.14
Audio-Flamingo-363.6697.4045.0961.6529.99
Qwen-2.5-Omni71.6899.1056.8072.2137.12
MiniCPM-o-2.674.3698.0861.6275.6940.69
Omni-R183.5097.3776.5985.7450.95
MERaLiON288.1887.0596.0491.3371.46
Gemini 2.0 Flash86.2984.2896.9090.1574.47
GPT4o-Audio84.6286.8689.8588.3267.00
SenseVoiceSmall + GPT4o84.9596.4679.6987.2853.51
SenseVoiceSmall + Gemini83.7292.5681.4186.6356.97
Whisper-Large + GPT4o85.0698.2778.3187.1651.62
Whisper-Large + Gemini83.0595.3577.6285.5852.73

Although some numerical results have been corrected, the principal observations and conclusions regarding pun recognition remain substantively unchanged. Strong proprietary systems and several cascaded systems still generally perform well on pun recognition, whereas fine-grained pun localization and dual-meaning inference remain challenging. This supports our central observation that detecting a pun does not necessarily imply the ability to localize the punning word accurately or infer both meanings.

W2 and Q2

1. Dataset splitting and the relationship between Tables 2 and 5

We clarify that the “separate held-out test set” in Section 4.4.1 refers to the 20% held-out portion of the fixed 80/20 split, which we will consistently term the “held-out evaluation split.” As also clarified to Reviewer fLdK, PunAlign uses only positive samples: 2,384 for training and 595 for held-out evaluation.

  • Table 2 (Baseline Evaluation): Baseline models are evaluated separately on the complete Synthetic and Real-World Datasets to characterize their pun recognition, localization, and meaning-inference performance.
  • Table 5 (PunAlign Evaluation): The Synthetic and Real-World Datasets are combined and partitioned using a fixed 80/20 split. The reserved 20% is used to report the effectiveness of the SFT and DPO fine-tuning strategies.

Thus, Tables 2 and 5 differ in evaluation purpose and sample coverage rather than relying on two unspecified independent test sets. Table 2 characterizes performance on the complete benchmark, while Table 5 evaluates improvements on data held out from PunAlign training.

2. The scoring of homographic pun meanings also lacks sufficient detail on the similarity model, sense-matching rules, and the calibration basis for the threshold τ = 0.3.

As described in Appendix G (Eq. 5), homographic pun inference is evaluated by matching the predicted meanings against two gold meanings, with both matched pairs required to exceed the similarity threshold. We provide the following implementation details.

Similarity model. We use sentence-transformers/all-MiniLM-L6-v2 to encode predicted and gold meaning phrases into sentence embeddings. Their semantic similarity is measured using cosine similarity between the corresponding embeddings.

Sense matching. We compute pairwise similarities between the two predicted meanings and the two gold meanings, forming a 2×22\times2 similarity matrix. The Hungarian algorithm is then used to find the optimal one-to-one assignment that maximizes the total similarity. This ensures order-invariant matching and prevents multiple predicted meanings from being assigned to the same gold meaning. As specified in Eq. (5), a prediction is correct only if both matched similarities strictly exceed τ\tau.

To assess whether our main conclusions depend strongly on τ=0.30\tau=0.30, we held all original model predictions fixed and reran the full scoring pipeline on the same 1,298 homographic samples under five thresholds:

τ∈{0.20,0.25,0.30,0.35,0.40}.\tau\in\{0.20,0.25,0.30,0.35,0.40\}.

At every threshold, we used the same configurations. Missing or unparseable outputs were consistently counted as incorrect. The resulting accuracies (%) are:

Modelτ=0.20\tau=0.20τ=0.25\tau=0.25τ=0.30\tau=0.30τ=0.35\tau=0.35τ=0.40\tau=0.40
Qwen2-Audio42.5328.1217.0310.406.86
Audio-Flamingo-359.6343.9929.5118.8011.40
Qwen2.5-Omni63.0246.2232.2022.4214.79
MiniCPM-o-2.665.7249.0837.1326.7319.26
Omni-R166.4953.2438.7529.8220.96
MERaLiON-259.6341.9929.2820.1814.41
GPT-4o76.8965.7256.7848.2339.75

As τ\tau increases, the correctness criterion becomes stricter, and the absolute accuracy of every model decreases monotonically. To assess the stability of model comparisons, we also computed Spearman rank correlations against the ranking at τ=0.30\tau=0.30 across these nine models:

Threshold τ\tauSpearman correlation with τ=0.30\tau=0.30
0.200.996
0.251.000
0.301.000
0.350.950
0.400.950

The rank correlations remain consistently high (ρ≥0.950\rho \geq 0.950), indicating that model rankings are largely stable across the evaluated thresholds. These results suggest that, although absolute accuracy varies with the similarity threshold, our main comparative findings remain robust within the evaluated range.

3. Provide text-level deduplication and cross-split overlap checks.

We additionally performed text-level leakage checks on the complete set of 3,295 samples, dividing the dataset into 2,636 training samples and 659 held-out evaluation samples following an 80/20 split.

Data subsetTraining samplesHeld-out samplesWithin-dataset duplicate groupsCross-split exact duplicate groups
Synthetic + Real-world2,636659224

Among the 659 held-out samples, only four samples (0.61%) had normalized text identical to at least one training sample. This indicates that exact text overlap between the training and held-out evaluation splits is limited, suggesting a relatively low risk of evaluation contamination due to exact duplicates.

We again thank the reviewer for these constructive comments and respectfully ask you to kindly reconsider your score in light of these clarifications.

Dear Reviewer HwKn,

Thank you once again for your valuable comments and constructive feedback on our submission. As the discussion phase is coming to an end, we would like to kindly ask whether there are any remaining questions or concerns that we could further clarify, at least in part. If you find our responses satisfactory, we would greatly appreciate your reconsideration of the overall evaluation and score in light of the clarifications and additional analyses we have provided. We sincerely thank you for your time and thoughtful review, and we look forward to your feedback.


Best regards,

The Authors

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We thank Reviewers HwKn, JwSQ, and fLdK again for their time and effort in reviewing our submission. We introduce APUN-Bench, the first benchmark for audio pun understanding, comprising 4,447 samples across three stages: pun recognition, localization, and meaning inference. Through evaluating 10 LALMs and 4 cascaded systems, we identify key challenges in audio pun understanding and propose two improvement strategies, PunPAP and PunAlign, demonstrating their effectiveness in localization and inference tasks.

We post a general response and a point-by-point reply to each reviewer:

  1. Evaluation Consistency: Clarification of pun recognition metrics, correction of reporting inconsistencies, and verification of results using confusion matrices across models (HwKn, fLdK).

  2. Dataset Splits and Leakage: Clarification of the training and evaluation protocols, including the relationship between Tables 2 and 5, and additional text-level deduplication and cross-split overlap checks (HwKn, fLdK).

  3. Meaning-Inference Evaluation: Detailed explanation of semantic similarity scoring and sense matching, threshold sensitivity analysis across five settings, and false-acceptance tests using negative controls (HwKn, JwSQ, fLdK).

  4. Acoustic and Semantic Analysis: Further analysis of ASR transcription errors, comparisons with ground-truth transcripts, and clarification of the respective roles of speech perception and semantic reasoning in audio pun understanding (JwSQ, fLdK).

  5. Dataset Bias Analysis: Controlled evaluations matching pun and non-pun samples by sentence length and audio duration, alongside an analysis of potential shortcut cues (JwSQ).

  6. Improvement Strategy Analysis: Additional discussion of PunPAP and PunAlign improvements, including position-stratified evaluations revealing PunPAP's benefits and limitations for early- and late-position puns (JwSQ).

  7. Dataset Transparency: Clarification of synthetic and real-world audio sources, recording conditions, annotation procedures, and the scope of acoustic dependence in audio pun understanding (JwSQ, fLdK).

We believe these additions strengthen the empirical support for APUN-Bench with derived methods and improve the transparency of our evaluation. As the discussion period is ending soon, we would be grateful if the reviewers could let us know whether our responses have addressed their concerns. We are happy to provide any further clarification.


Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.