- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies how spoken dialogue models can adapt their responses to vocal cues while avoiding unsupported assumptions. The authors construct ToneFoil, a dataset containing acoustic evidence descriptions and reviewed response relations, and propose VoxTact, which combines ToneTrace to preserve temporal acoustic information with Appropriateness-Relative Policy Optimization (ARPO) to learn contextually appropriate responses under quality and relation constraints. Experiments and ablation studies show that VoxTact improves response suitability over the evaluated baselines while maintaining response quality.
Critical strengths
-
The motivation is clear, and the proposed methodology is straightforward and effective, as supported by the experimental results and ablation studies.
-
The authors construct ToneFoil, a dataset with acoustic evidence descriptions and response relation annotations, which could serve as a valuable resource for future research on acoustically appropriate response generation.
Critical weaknesses
-
Dataset validity. My biggest concern is the validity and reliability of the dataset annotations. The acoustic evidence descriptions, response relation labels, and quality and suitability judgments appear to rely heavily on LLM-based annotation and evaluation. Although the supplementary material provides the corresponding prompts, it remains unclear to what extent these annotations have been validated by humans. A systematic human validation study, particularly for the response relations and suitability judgments, would be important to establish the reliability of the dataset.
-
Interpretability of ToneTrace. As the authors acknowledge in Lines 195–196, ToneTrace captures multifaceted information, including lexical content, speaker characteristics, and affective cues. Therefore, it is unclear to what extent the learned representations (Z) specifically capture vocal nuances rather than other potentially confounding information. Additional representation analyses, such as clustering, dimensionality reduction, or probing, could provide insights into what information is encoded in Z and whether the model relies on the intended acoustic cues.
-
Design of temporal pooling. The choice of eight temporal summaries in ToneTrace appears somewhat arbitrary and is not sufficiently justified. It is also unclear how well this fixed configuration generalizes across utterances of different lengths, as the temporal resolution varies with input duration. An ablation over the number of pooling segments, ideally across different utterance lengths, would help establish the robustness of this design choice.
Questions for authors
My main question concerns the dataset construction process, as several annotation steps and their validation remain insufficiently documented. I would appreciate a more detailed description of how the annotations were generated, reviewed, and validated, particularly the acoustic evidence descriptions, response relation labels, and quality and suitability judgments.
2 comments
Thanks for providing additional results. I think 3 is a fair score for the work, so I will keep it.
Response to 4EB6: annotation agreement and acoustic interventions
Thank you for separating the accuracy of the acoustic descriptions from the policy's use of acoustic information. The observer heard one recording at a time, with its source emotion label hidden. In the human audit, 80% of 200 descriptions were judged accurate (95% CI 74–86%). Assigning a response relation then requires judging candidate replies. Two listeners agreed on 38/48 families (κ = 0.688); six of ten disagreements involved compatible versus uncertain. Without a preference reversal, compatibility still requires an identifiable reply that fits both recordings (Section 3; Appendix B).
To assess bilateral success, three raters evaluated both policies on 100 families. Success requires both replies to pass quality review and each to be preferred in its own context. Agreement across the 200 decisions was Fleiss κ = 0.61 [0.50, 0.72]; human and automated decisions agreed in 78% of cases [72%, 84%] (κ = 0.52 [0.40, 0.64]). In the separate comparison of systems on the same recording, 68 of 128 judgments favored VoxTact and 48 favored sample GRPO.
We tested use of the summaries by changing only those inputs at inference. With the 9B policy frozen, original order yielded 190/300 successes; reversal and two permutations yielded 185, 187, and 186 (mean difference −1.33 points [−3.5, +0.9]). Constant replacement yielded 167 (−7.67 [−11.0, −4.3]). The evidence is clearer for recording-specific values than for their order. As your proposed probe suggests, it does not identify how much of the contribution reflects affect, speaker identity, or recording conditions. Identical wording controls lexical content, but these acoustic factors may still vary together.
Fixed-resolution pooling does give longer utterances coarser coverage. With eight summaries, BRS gains over sample GRPO were 14.0, 13.5, and 12.7 points for recordings under 4 s, 4–8 s, and over 8 s (180, 760, and 260 families). Across durations, four, eight, and sixteen summaries yielded 60.8%, 63.3%, and 63.9% BRS. These results motivate eight summaries in this setting; they do not determine the best resolution for each duration. That question requires jointly varying duration and summary count (Appendices C and E.3).