- Rating
- 3
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2(−1)
- Soundness
- 1(−1)
Read review
Paper summary
This paper introduces APUN-Bench for evaluating spoken pun recognition, localization, and meaning inference using synthetic speech and human recordings. It compares audio-language models, ASR–LLM pipelines, and transcript inputs, analyzes characteristic errors, and explores position-aware prompting and SFT/DPO adaptation. The principal contribution is the benchmark and its diagnostic evaluation.
Critical strengths
-
APUN-Bench provides a useful resource for studying spoken wordplay, testing whether models recover alternative interpretations rather than merely recognize an utterance.
-
The recognition, localization, and inference tasks provide informative diagnostics. Section 4.4 particularly highlights explanations that recover only one of a pun's meanings.
-
The audio-model, ASR–LLM, and error-free-transcript comparisons help separate speech-processing difficulties from downstream reasoning. The benchmark also includes human verification of audio and annotations.
Critical weaknesses
-
The relationship between audio puns and textual wordplay deserves clearer framing. Established pun definitions already include phonological similarity alongside lexical ambiguity [1]; a reader can recognize homophonic wordplay in writing. Could the authors distinguish evaluating puns presented through speech from evaluating interpretations that depend on a particular acoustic realization? Section 3's linguistic categories do not establish that distinction by themselves. This does not require a separate, strictly audio-specific category. It would also help explain how exact-match evaluation treats identical-sounding alternatives when spelling choice and understanding need not coincide.
-
The acoustic-mechanism interpretation could be more cautious. The existing ASR baselines, transcripts, and WER analysis are useful. However, Section 4.2's “pronunciation cues that are unavailable in text” overlooks pronunciation knowledge accessible to text models. The small MiniCPM audio advantage does not isolate an acoustic cause, and Appendix F does not specify whether its two speakers read matched content. Could the authors illustrate when hearing the utterance changes its interpretation, rather than only its transcription? A narrower interpretation of these results would also address this concern.
-
The speech-source mixture needs clearer characterization. Appendix B combines TTS-rendered written puns, competition recordings, and participant readings of website texts. The filtering and expert checks in Section 3 and Appendix E are reassuring, including verification of pun presence and alternative meanings. Reporting the competition/read-aloud proportions would nevertheless help readers understand the “real-world” subset. A brief discussion of how these delivery conditions may affect the findings would clarify the benchmark's coverage; matched synthetic/human recordings would be a useful future extension.
[1] Miller, T., Hempelmann, C., & Gurevych, I. (2017). SemEval-2017 Task 7: Detection and interpretation of English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) (pp. 58–68). Association for Computational Linguistics. doi:10.18653/v1/S17-2005.
Minor Issues
- Table 2's synthetic MiniCPM “Pun (%)” is below the true-positive fraction implied by its recall and Table 1's class counts. Please reconcile these entries.
- Appendix G should identify the similarity function, explain how its threshold was selected, and specify that matching covers both distinct gold meanings.
Questions for authors
-
Audio presentation versus acoustic dependence. Beyond the linguistic categories already defined in Section 3, how do the authors distinguish puns presented through speech from interpretations that depend on a particular acoustic realization? Could examples across the existing categories illustrate when listening changes the interpretation rather than only the transcription, taking into account pronunciation knowledge available to text models?
-
Interpreting the model comparisons. Given the competitive ASR–LLM pipelines and the error-free-transcript comparisons, what do the results establish about speech perception versus downstream reasoning as bottlenecks? In particular, what supports attributing MiniCPM's homophonic-inference advantage to acoustic information, and did the two speakers in Appendix F read matched content?
-
Preserving ambiguity in a hybrid pipeline. Beyond the existing ASR–LLM baselines, have the authors considered using an audio-language model to produce a transcript or several plausible word/transcription alternatives, followed by a larger text model for reasoning? Could retaining alternatives help avoid committing prematurely to one interpretation of a pun? ( I raise this as a possible extension and discussion point, not an acceptance requirement. )
-
Audio-source coverage. Appendix B reports the overall synthetic/real-world counts and the random assignment of each synthetic text to one of 34 voices. Within the real-world subset, what proportion comes from competition recordings versus participant readings, and is there any textual overlap with the synthetic subset? A brief account of these conditions and their possible effect on the findings would clarify the benchmark's coverage; matched human/synthetic or multi-voice recordings would be optional extensions.
-
Meaning-matching evaluation. In Appendix G, what similarity function is used, how was the threshold selected ?
7 comments
Q4 & W4
Among the 899 samples in the Real-world Dataset, approximately 10% (78 samples) are excerpts from the competition recordings, while the remaining samples cover other real-world speech sources, including participant readings, providing natural speech characteristics and recording conditions distinct from synthetic audio. We further conducted a text-overlap check and found no exact textual duplicates between the Real-world and Synthetic subsets. We acknowledge that differences in speaking styles and recording conditions across sources may introduce variability. Therefore, we do not treat these samples as a strictly controlled acoustic comparison. Instead, they are intended to complement the synthetic data by increasing the diversity of speech conditions.
First Minor Issue
We re-examined the original prediction records and the process used to compile Table 2. We found that some of the MiniCPM-o-2.6 statistics were inconsistent across versions of the compiled results. In particular, the Pun(%) entry retained an outdated value from an intermediate evaluation output, and other statistics also required updating.
The Synthetic Dataset contains 3,548 samples, including 2,396 puns and 1,152 non-puns. After recomputing the metrics from the sample-level predictions, the MiniCPM-o-2.6 confusion matrix is:
| Predicted Non-pun | Predicted Pun | |
|---|---|---|
| Actual Non-pun | TN = 624 | FP = 528 |
| Actual Pun | FN = 777 | TP = 1,619 |
The corresponding corrected metrics are:
| Model | Acc. (↑) | Pre. (↑) | Rec. (↑) | F1 (↑) | Pun (%) |
|---|---|---|---|---|---|
| MiniCPM-o-2.6 | 63.22 | 75.41 | 67.57 | 71.27 | 60.51 |
The corrected Accuracy, Precision, Recall, F1, and Pun(%) are directly verifiable from this confusion matrix.
We further extended the audit to all models on both the Synthetic and Real-World Datasets:
Synthetic Dataset
| Model | Acc. (↑) | Pre. (↑) | Rec. (↑) | F1 (↑) | Pun (%) |
|---|---|---|---|---|---|
| Qwen2-Audio-Instruct | 45.97 | 67.78 | 38.11 | 48.78 | 37.97 |
| SALMONN | 51.44 | 68.48 | 52.05 | 59.14 | 51.32 |
| Audio-Flamingo-3 | 37.06 | 54.67 | 39.86 | 46.10 | 49.24 |
| Qwen-2.5-Omni | 64.09 | 75.71 | 68.95 | 72.17 | 61.50 |
| MiniCPM-o-2.6 | 63.22 | 75.41 | 67.57 | 71.27 | 60.51 |
| Omni-R1 | 65.22 | 74.39 | 73.96 | 74.17 | 67.14 |
| MERaLiON2 | 71.65 | 72.39 | 93.78 | 81.71 | 87.49 |
| Gemini 2.0 Flash | 80.50 | 89.63 | 80.43 | 84.78 | 60.60 |
| GPT4o-Audio | 75.73 | 75.63 | 94.53 | 84.03 | 84.41 |
| SenseVoiceSmall + GPT4o | 79.43 | 82.24 | 88.69 | 85.34 | 72.83 |
| SenseVoiceSmall + Gemini | 78.97 | 81.85 | 88.48 | 85.04 | 73.00 |
| Whisper-Large + GPT4o | 80.86 | 85.43 | 86.39 | 85.91 | 68.29 |
| Whisper-Large + Gemini | 78.52 | 82.24 | 86.98 | 84.54 | 71.42 |
Real-World Dataset
| Model | Acc. (↑) | Pre. (↑) | Rec. (↑) | F1 (↑) | Pun (%) |
|---|---|---|---|---|---|
| Qwen2-Audio-Instruct | 57.86 | 95.11 | 36.83 | 53.10 | 25.08 |
| SALMONN | 44.93 | 57.07 | 60.41 | 58.70 | 68.56 |
| Audio-Reasoner | 68.12 | 89.97 | 57.14 | 69.89 | 41.14 |
| Audio-Flamingo-3 | 63.66 | 97.40 | 45.09 | 61.65 | 29.99 |
| Qwen-2.5-Omni | 71.68 | 99.10 | 56.80 | 72.21 | 37.12 |
| MiniCPM-o-2.6 | 74.36 | 98.08 | 61.62 | 75.69 | 40.69 |
| Omni-R1 | 83.50 | 97.37 | 76.59 | 85.74 | 50.95 |
| MERaLiON2 | 88.18 | 87.05 | 96.04 | 91.33 | 71.46 |
| Gemini 2.0 Flash | 86.29 | 84.28 | 96.90 | 90.15 | 74.47 |
| GPT4o-Audio | 84.62 | 86.86 | 89.85 | 88.32 | 67.00 |
| SenseVoiceSmall + GPT4o | 84.95 | 96.46 | 79.69 | 87.28 | 53.51 |
| SenseVoiceSmall + Gemini | 83.72 | 92.56 | 81.41 | 86.63 | 56.97 |
| Whisper-Large + GPT4o | 85.06 | 98.27 | 78.31 | 87.16 | 51.62 |
| Whisper-Large + Gemini | 83.05 | 95.35 | 77.62 | 85.58 | 52.73 |
Although some numerical results have been corrected, the principal observations and conclusions regarding pun recognition remain substantively unchanged. Strong proprietary systems and several cascaded systems still generally perform well on pun recognition, whereas fine-grained pun localization and dual-meaning inference remain challenging. This supports our central observation that detecting a pun does not necessarily imply the ability to localize the punning word accurately or infer both meanings.
Q5 & Second Minor Issue
As clarification of the scoring details in Appendix G, we use sentence-transformers/all-MiniLM-L6-v2 embeddings and cosine similarity for predicted and gold meaning phrases. A Hungarian one-to-one assignment maximizes the sum of matched similarities, and a response is accepted only when both matched meaning pairs exceed the fixed threshold τ = 0.30. During evaluation-protocol development, we empirically selected this threshold by examining cosine-similarity scores and manually reviewing the matching outcomes on a small random sample of homographic puns. To assess whether our main conclusions depend strongly on τ = 0.30, we held all original model predictions fixed and reran the scoring pipeline on the same 1,298 homographic samples under five thresholds:
At every threshold, we used the same configurations. Missing or unparseable outputs were counted as incorrect. The resulting accuracies (%) are:
| Model | τ = 0.20 | τ = 0.25 | τ = 0.30 | τ = 0.35 | τ = 0.40 |
|---|---|---|---|---|---|
| Qwen2-Audio | 42.53 | 28.12 | 17.03 | 10.40 | 6.86 |
| Audio-Flamingo-3 | 59.63 | 43.99 | 29.51 | 18.80 | 11.40 |
| Qwen2.5-Omni | 63.02 | 46.22 | 32.20 | 22.42 | 14.79 |
| MiniCPM-o-2.6 | 65.72 | 49.08 | 37.13 | 26.73 | 19.26 |
| Omni-R1 | 66.49 | 53.24 | 38.75 | 29.82 | 20.96 |
| MERaLiON-2 | 59.63 | 41.99 | 29.28 | 20.18 | 14.41 |
| GPT-4o | 76.89 | 65.72 | 56.78 | 48.23 | 39.75 |
As increases, the correctness criterion becomes stricter, and the absolute accuracy of every model decreases monotonically. To assess the stability of model comparisons, we also computed Spearman rank correlations against the ranking at across these nine models:
| Threshold | Spearman correlation with |
|---|---|
| 0.20 | 0.996 |
| 0.25 | 1.000 |
| 0.30 | 1.000 |
| 0.35 | 0.950 |
| 0.40 | 0.950 |
The rank correlations remain consistently high (), indicating that model rankings are largely stable across the evaluated thresholds. These results suggest that, although absolute accuracy varies with the similarity threshold, our main comparative findings remain robust within the evaluated range.
Once again, we thank you for your time and valuable feedback.
We sincerely appreciate your comments.
The goal of APUN-Bench is to evaluate the complete process of audio pun understanding, from speech perception to pun detection, word localization, and meaning inference, rather than to claim that every sample requires acoustic cues beyond textual information. Our benchmark includes samples with varying degrees of acoustic dependence.
For example, in the homographic pun “A bank manager who was also a high jumper spent most of his time in the vault,” the two meanings of vault, a bank vault and a movement in high jumping, are already preserved in the ground-truth transcript. Thus, this example primarily evaluates semantic reasoning.
For homophonic puns, the phonological relationship is central to the wordplay, but interpreting the pun does not necessarily require a particular acoustic realization. For instance, in “Two construction workers had a stairing contest,” stairing and staring share the same pronunciation, allowing a text-based model with phonological knowledge to recognize the pun. Indeed, both SenseVoice and Whisper transcribed the annotated word stairing as staring, illustrating that the audio alone does not uniquely determine the intended spelling.
In contrast, some examples are sensitive to connected speech, pauses, and word-boundary interpretation. Consider the real-world example “I'm so purrfect that whenever I meet a pretty girl, I whisker away,” which involves the alternative interpretations whisker away and whisk her away. Whisper transcribed the relevant phrase as whisk her away, illustrating how similar spoken sequences can lead to different lexical interpretations depending on word segmentation and connected-speech processing. In such cases, acoustic perception can influence not only transcription but also the interpretations considered by the model.
This conclusion is supported by audio–transcript comparisons conducted on the same samples. With ground-truth transcripts, the pun-localization accuracies of GPT-4o, Gemini, and MiniCPM improve by 42.95%, 34.19%, and 40.11%, respectively. This substantial gap indicates that accurately recovering ambiguous lexical forms from speech remains a major bottleneck for current LALMs. The competitive localization performance of the ASR–LLM pipelines, together with the higher transcription error rates on real-world speech reported in Appendix C, further supports this observation.
However, eliminating transcription errors does not fully resolve the task. Even with ground-truth transcripts, MiniCPM-Text achieves only 47.53% and 40.22% accuracy on heterographic and homographic meaning inference, respectively, demonstrating persistent challenges in lexical interpretation, contextual integration, and dual-meaning reasoning. Thus, our three-stage evaluation reveals two interconnected bottlenecks: speech perception, reflected in the substantial audio–text localization gap, and downstream semantic reasoning, reflected in the remaining inference errors under ground-truth transcription.
Regarding Appendix F, we clarify that the two speakers recorded matched textual content, and we analyzed 276 samples to investigate the effects of acoustic variation on model performance.
We thank the reviewer for this valuable suggestion. We agree that our current ASR–LLM baselines rely on a single best transcription, which may prematurely resolve lexical ambiguities before downstream reasoning. We primarily include these cascaded pipelines as baselines to disentangle the contributions of speech transcription and language reasoning, rather than as a definitive solution to audio pun understanding. Our research focus remains on end-to-end multimodal models capable of jointly perceiving speech and reasoning about its multiple meanings.
Our error analysis further illustrates the limitations of single-best transcription. For example, best cellars was transcribed as best sellers by both ASR systems, while whisker away could be transcribed as whisk her away. In such cases, committing to a single spelling or word segmentation may discard plausible lexical alternatives, making it more difficult for downstream language models to recover the intended wordplay.
A promising extension would be a hybrid pipeline that preserves multiple hypotheses through N-best transcripts, or lexical, word-segmentation, and phoneme-level candidates with associated confidence scores. A stronger text-based LLM could then jointly reason over these alternatives using contextual information. For instance, a pun-specific lexical knowledge base could provide candidate pairs such as cellars/sellers and whisker/whisk her, rather than requiring the ASR system to select a single interpretation before reasoning begins. We view this as a valuable direction for future work that complements our primary focus on end-to-end audio-language understanding.