acceptodds
Under review as a conference paper at ICLR 2027

VoxTact: Preserving Vocal Nuance for Acoustically Appropriate Responses

Abstract

The same words can call for different responses when spoken with different affect, yet responding distinctively can also introduce unsupported assumptions. We present VoxTact, a spoken-dialogue framework that learns when vocal nuance should change a reply while preserving factual and role consistency. ToneTrace retains temporally ordered acoustic summaries alongside the existing speech representation. Appropriateness-Relative Policy Optimization (ARPO) evaluates each candidate reply across paired contexts and rewards supported specificity under quality and relation constraints. ToneFoil provides 100,000 training recordings with reviewed response relations; ToneFoil-Bench evaluates strictly matched wording and distinguishes justified adaptation from appropriate shared responses. ToneTrace preserves the cues; ARPO learns their relevance to response suitability; ToneFoil tests the distinction. In matched response-learning comparisons, VoxTact achieves 63.34% bilateral response success, compared with 49.94% for sample-level policy optimization, while attaining 94.10% quality acceptance and 2.90% unsupported assertions. Representation and objective controls distinguish the contribution of acoustic input from additional capacity and expose the cost of rewarding specificity without quality constraints. The results support selective acoustic conditioning rather than indiscriminate emotional differentiation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spoken language models, acoustic grounding, paralinguistics, counterfactual learning, spoken dialogue

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4EB6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how spoken dialogue models can adapt their responses to vocal cues while avoiding unsupported assumptions. The authors construct ToneFoil, a dataset containing acoustic evidence descriptions and reviewed response relations, and propose VoxTact, which combines ToneTrace to preserve temporal acoustic information with Appropriateness-Relative Policy Optimization (ARPO) to learn contextually appropriate responses under quality and relation constraints. Experiments and ablation studies show that VoxTact improves response suitability over the evaluated baselines while maintaining response quality.

Critical strengths
  1. The motivation is clear, and the proposed methodology is straightforward and effective, as supported by the experimental results and ablation studies.

  2. The authors construct ToneFoil, a dataset with acoustic evidence descriptions and response relation annotations, which could serve as a valuable resource for future research on acoustically appropriate response generation.

Critical weaknesses
  1. Dataset validity. My biggest concern is the validity and reliability of the dataset annotations. The acoustic evidence descriptions, response relation labels, and quality and suitability judgments appear to rely heavily on LLM-based annotation and evaluation. Although the supplementary material provides the corresponding prompts, it remains unclear to what extent these annotations have been validated by humans. A systematic human validation study, particularly for the response relations and suitability judgments, would be important to establish the reliability of the dataset.

  2. Interpretability of ToneTrace. As the authors acknowledge in Lines 195–196, ToneTrace captures multifaceted information, including lexical content, speaker characteristics, and affective cues. Therefore, it is unclear to what extent the learned representations (Z) specifically capture vocal nuances rather than other potentially confounding information. Additional representation analyses, such as clustering, dimensionality reduction, or probing, could provide insights into what information is encoded in Z and whether the model relies on the intended acoustic cues.

  3. Design of temporal pooling. The choice of eight temporal summaries in ToneTrace appears somewhat arbitrary and is not sufficiently justified. It is also unclear how well this fixed configuration generalizes across utterances of different lengths, as the temporal resolution varies with input duration. An ablation over the number of pooling segments, ideally across different utterance lengths, would help establish the robustness of this design choice.

Questions for authors

My main question concerns the dataset construction process, as several annotation steps and their validation remain insufficiently documented. I would appreciate a more detailed description of how the annotations were generated, reviewed, and validated, particularly the acoustic evidence descriptions, response relation labels, and quality and suitability judgments.

2 comments
AuthorsOpenReview ↗

Response to 4EB6: annotation agreement and acoustic interventions

Thank you for separating the accuracy of the acoustic descriptions from the policy's use of acoustic information. The observer heard one recording at a time, with its source emotion label hidden. In the human audit, 80% of 200 descriptions were judged accurate (95% CI 74–86%). Assigning a response relation then requires judging candidate replies. Two listeners agreed on 38/48 families (κ = 0.688); six of ten disagreements involved compatible versus uncertain. Without a preference reversal, compatibility still requires an identifiable reply that fits both recordings (Section 3; Appendix B).

To assess bilateral success, three raters evaluated both policies on 100 families. Success requires both replies to pass quality review and each to be preferred in its own context. Agreement across the 200 decisions was Fleiss κ = 0.61 [0.50, 0.72]; human and automated decisions agreed in 78% of cases [72%, 84%] (κ = 0.52 [0.40, 0.64]). In the separate comparison of systems on the same recording, 68 of 128 judgments favored VoxTact and 48 favored sample GRPO.

We tested use of the summaries by changing only those inputs at inference. With the 9B policy frozen, original order yielded 190/300 successes; reversal and two permutations yielded 185, 187, and 186 (mean difference −1.33 points [−3.5, +0.9]). Constant replacement yielded 167 (−7.67 [−11.0, −4.3]). The evidence is clearer for recording-specific values than for their order. As your proposed probe suggests, it does not identify how much of the contribution reflects affect, speaker identity, or recording conditions. Identical wording controls lexical content, but these acoustic factors may still vary together.

Fixed-resolution pooling does give longer utterances coarser coverage. With eight summaries, BRS gains over sample GRPO were 14.0, 13.5, and 12.7 points for recordings under 4 s, 4–8 s, and over 8 s (180, 760, and 260 families). Across durations, four, eight, and sixteen summaries yielded 60.8%, 63.3%, and 63.9% BRS. These results motivate eight summaries in this setting; they do not determine the best resolution for each duration. That question requires jointly varying duration and summary count (Appendices C and E.3).

Thanks for providing additional results. I think 3 is a fair score for the work, so I will keep it.

Reviewer_ot59OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VoxTact, a spoken-dialogue framework designed to generate acoustically appropriate responses while avoiding unsupported assumptions. It combines ToneTrace, which preserves temporally ordered acoustic features, with Appropriateness-Relative Policy Optimization (ARPO), which extends GRPO using quality-constrained, cross-context suitability rewards. This paper also introduces ToneFoil, a dataset of 100,000 training recordings, and ToneFoil-Bench, which evaluates response preferences under identical wording.

Critical strengths
  • Distinguishing appropriate acoustic adaptation from unsupported emotional assumptions is important for spoken dialogue. The distinction between contrastive, compatible, and uncertain response relations is conceptually useful, particularly because acoustically different utterances do not necessarily require different responses.
Critical weaknesses
  • This paper lacks novelty at ICLR 2027. Essentially, ToneTrace is temporal pooling followed by an MLP projection and prefix concatenation. ARPO adds a handcrafted, quality-gated relational reward to GRPO. The underlying idea closely resembles CRPO, and the paper fails to establish a substantial methodological advance beyond existing work.

  • The validity of ToneFoil-Bench is questionable. The benchmark is derived from LIME-440K, which contains substantial synthetic emotional speech. The paper does not report the synthetic/human-recorded composition of ToneFoil-Bench or validate BRS on an independent, speaker-disjoint natural-speech test set. The reported gains may therefore partly reflect dataset-specific acoustic patterns, not genuine conversational appropriateness.

  • The main evaluation lacks sufficient reliability evidence. BRS depends on subjective judgments of response appropriateness, yet the provided manuscript does not establish adequate inter-annotator agreement, independent human validation, or independence between training and evaluation judgments. Optimizing a related suitability scorer may inflate improvements on the proposed benchmark.

  • The baseline comparisons are insufficient. The primary comparisons focus on generic SFT, DPO, and GRPO, not an adapted CRPO-style objective or recent affect-aware spoken-dialogue methods such as CogAudio-LLM. The additional cross-context scoring cost is also not matched in the main optimization comparisons.

  • The claimed benefits of temporal vocal modeling and spoken-response quality are overstated. The ablations do not isolate temporal ordering from additional segment-level acoustic information. Moreover, human-rated vocal congruence improves only from 3.77 to 3.81, without sufficient statistical evidence to establish a meaningful improvement in expressive speech.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to ot59: relational supervision, natural speech, and scoring cost

Thank you for suggesting a closer comparison with CRPO. Sample GRPO scores each reply on its own. A dialogue control that already rewards the relation between two replies lets us ask what ARPO adds beyond paired supervision. The revised introduction opens with the same sentence delivered in two different ways. Section 5.1 now makes the success criterion explicit: both replies from a single policy must pass quality review, and each must be preferred in its own context.

W1. For “The meeting is tomorrow,” help with preparation may suit an apprehensive delivery better than a relieved one, while the date remains unchanged. The control retains CRPO's correctness checks and reward for the relation between replies sampled for the two inputs. It does not add ARPO's conditions for positive contextual credit: the candidate-quality gate, reviewed contrastive-family mask, and own-context suitability threshold. ARPO also scores each fixed reply under both recordings. At a reported cost index of 1.00 for each method, BRS was 63.3% for ARPO and 60.9% for the CRPO-inspired control (+2.4 percentage points, 95% CI [+0.2, +4.6]). This supports the complete ARPO reward over this matched adaptation, without isolating any one mechanism. Scoring against the wrong context reduces BRS to 51.0%; removing the quality gate raises unsupported assertions from 2.90% to 13.85% (Section 5; Appendix A.3).

W2. Of the 4,000 ToneFoil-Bench recordings, 30% are human-recorded and 70% are synthetic. The benchmark speaker-ID audit found no overlap with training. On a separate panel of 100 natural recordings, 56% of 300 judgments favored VoxTact, 38% favored sample GRPO, and 6% were ties (net preference +18 percentage points, 95% CI [+4, +32]). For BRS, each of three raters assessed 200 policy decisions, checking that both replies from the same policy passed quality review and each was preferred in its own context. The BRS difference was +6.0 percentage points (95% CI [−3, +15]). The speaker-ID audit covered ToneFoil-Bench only.

W3. Qwen3-32B scored training candidates. The primary BRS evaluator, Qwen3-Omni-30B-A3B-Thinking, also generated the acoustic observations. In the human audit, three raters assessed 200 BRS decisions from 100 families (Fleiss κ = 0.61 [0.50, 0.72]). Human and automated decisions agreed in 78% of cases [72%, 84%] (κ = 0.52 [0.40, 0.64]). We also rescored fixed replies on 300 families. MiniCPM-o 2.6 gave weighted ARPO gains over CRPO-inspired of +1.9, +2.4, and +2.6 percentage points across three repeats, with a combined estimate of +2.3 [0.0, +4.6]. Qwen2.5-Omni-7B gave +2.4 [+0.1, +4.7]. Both evaluators favored ARPO, with the MiniCPM-o interval reaching zero (Appendix B.2).

W4. CogAudio achieved 54.0% BRS zero-shot without a reference emotion label, 56.5% with the label, and 58.8% after fine-tuning on the ToneFoil training split. The scorer-work measurements compare ARPO with two sample-GRPO variants. ARPO and work-matched sample GRPO each used eight requests for four candidates, with equal token volume and relative scoring times of 1.00 and 0.98, respectively. Own-context-only GRPO used four requests, 0.52 times the tokens, and 0.51 times ARPO’s scoring time (Appendices A.3 and D.1).

W5. The frozen 9B intervention separates summary values from their order. Original order yielded 190/300 successes; reversal and two permutations yielded 185, 187, and 186 (mean difference −1.33 points [−3.5, +0.9]). Constant replacement yielded 167 (−7.67 [−11.0, −4.3]). This supports the contribution of recording-specific values, while the effect of order remains uncertain. Human-rated speech differences from sample GRPO were +0.04 for congruence [−0.04, +0.12] and 0.00 for naturalness [−0.04, +0.04], both on five-point scales. These intervals leave the effect on vocal expression unresolved. On reply selection, the intervention gives a clearer answer: keeping the acoustic detail of each recording helps the assistant choose a reply that fits the delivery.

Reviewer_uQWAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when vocal delivery should change a dialogue assistant's reply. ToneFoil provides a large collection of lexically matched or semantically equivalent recording pairs annotated with reply-level relations (contrastive, compatible, and uncertain). VoxTact consists of three parts: ToneTrace inserts 8 temporally ordered acoustic summaries as a prefix between the speech encoder and the response model to preserve vocal cues; ARPO builds on GRPO by scoring the same candidate reply across paired contexts, with a quality gate and a relation mask constraining the differentiation reward; ToneFoil-Bench is a strictly same-word, EN/ZH-balanced test set. Experiments validate each component through matched-budget comparisons, a 2×2 factorial design, and several ablations, and test the relevant capabilities on multiple external benchmarks.

Critical strengths

1.Despite being constrained to a single source corpus, ToneFoil still provides relatively large-scale supervision with 100k recordings and 40k relation-annotated pairs, together with an independent EN/ZH-balanced test set, offering usable infrastructure for follow-up work in this direction.

2.The temporally ordered design of ToneTrace is innovative. Compared with a single utterance-level average, keeping 8 summaries in temporal order can capture how delivery changes over the recording, and the repeated global summary control also verifies the contribution of the temporal structure itself.

3.The ablations are extensive and verify the contribution of each component. The 2×2 factorial design separates the effects of the representation and the reward, and controls such as constant input, shuffled families, and removing the quality gate all have clear purposes.

4.One strength of the paper is the novel problem perspective: the constructed dataset does not assume that "different delivery should lead to different replies", but instead focuses on the relation between replies, dividing them into contrastive, compatible, and uncertain categories, explicitly acknowledging that "an appropriate shared reply" is a valid and often correct outcome.

Critical weaknesses

1.In the construction of ToneFoil, almost all of the reviewing, generation, pairing, and selection steps are done by models, with no human consistency validation, making it hard to verify the reliability of the data grouping. 2.Although the paper divides the data into contrastive, compatible, and uncertain relations, there is no positive evaluation metric designed for categories other than contrastive — on compatible and uncertain data there are only quality-check metrics (Q, UA). As a result, whether the method can correctly distinguish the other two cases is not verified.

Questions for authors

1.The specific model used in each step of the ToneFoil construction pipeline (observation, reply generation, quality review, relation review) is not clearly stated in the paper. It would be helpful if the authors could clarify this. 2.The paper says "use eight summaries in temporal order" to preserve the changing information in the audio. I am curious about the reason for choosing 8 — was this choice validated by any ablation? I hope the authors can discuss the rationale for this parameter. 3.When evaluating on EchoMind, the paper only uses ROUGE-L as the metric for the open-ended response task, but ROUGE-L only measures lexical overlap and is not friendly to replies that are "worded differently but equally appropriate". EchoMind has more metrics, and the authors could try some LLM-scored metrics to evaluate the empathetic quality of the outputs. 4.The dataset contains both English and Chinese, but this is not reflected in the experimental results. I suggest the authors report the results for the two languages separately to see whether language has any effect. 5.This work uses only a single backbone (Qwen3.5-9B + Qwen2-Audio), so its generalizability is not verified.

1 comment
AuthorsOpenReview ↗

Response to uQWA: shared replies, language coverage, and backbone transfer

Thank you for asking how we recognize an appropriate unchanged answer. On compatible families, VoxTact finds a reply suitable for both recordings in 82.5% of cases, versus 78.5% for sample GRPO, and forces a distinction less often (15.0% versus 19.0%). On uncertain families, appropriate replies increase from 70.0% to 76.7%, and unsupported attributions decrease from 7.5% to 5.0% (Section 5.2; Table A5).

The relation audit clarifies this boundary. Two bilingual listeners agreed on 38/48 families (κ = 0.688); 72/96 labels matched the reference. Half the reference disagreements involved compatible versus uncertain: without a preference reversal, a compatible label still requires a specific reply that fits both contexts. Of 200 acoustic descriptions, 80% were judged accurate [74%, 86%]. Table A3 assigns observation to Qwen3-Omni-30B-A3B-Thinking, reply construction and relation review to Qwen3-32B, and quality review to Qwen3.6-27B-FP8.

ROUGE-L measures word overlap and can miss a valid paraphrase. Your EchoMind example motivated us to examine the ratings of the replies themselves. On five-point scales, gains over sample GRPO were +0.27 for speech-information relevance, +0.18 for context fit, +0.03 for text naturalness, and +0.02 for colloquial expression. ToneFoil BRS gains were 14.28 points in English and 12.50 in Chinese, over three evaluations of the same 600 families per language (Appendices C and D).

Four, eight, and sixteen summaries yield 60.8%, 63.3%, and 63.9% BRS; sixteen raises the reported cost index by 30% relative to eight. ToneTrace needs access to encoder states, and ARPO scores candidate replies across contexts. Both could be implemented on a non-Qwen backbone. We evaluated ARPO on Qwen3.5-9B; the supervised 4B study tested acoustic sensitivity within Qwen, so transfer of the reward gain remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.