- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: RubriVox breaks down the evaluation of spoken responses into critic rules covering both semantics and paralinguistics. Each rule is then directed to the evidence source best suited to verify it, and the verified judgments at the rule level are combined into a single scalar reward. The resulting RubriVox-Verifier is trained using contrastive verifier GRPO; afterwards it is frozen and supplies scalar rewards to the downstream spoken-policy GRPO.
Contributions: This work develops a contrastive spoken-verifier data pipeline that combines speech-suitable prompt filtering, difficulty-aware response selection, and Contrastive rule-tree search to construct discriminative rule-level verifier traces; proposes RubriVox-Verifier and contrastive verifier GRPO, where RubriVox-Verifier produces evidence-routed rule-level reward traces while contrastive verifier GRPO optimizes verifier rollouts with format, calibration, stability, and discrimination rewards; and builds RubriVox, a full verifier-guided RL framework for spoken dialogue models, showing that structured RubriVox-Verifier rewards outperform Gemini factorized and holistic rewards in downstream policy optimization.
Critical strengths
-
The paper addresses a topic with a very worthwhile focus: reward modeling for spoken dialogue indeed needs to cover both semantic and paralinguistic dimensions at the same time, and existing approaches that use LALMs as judges tend to produce holistic, uninterpretable scores.
-
The reward design is reasonably structural. Instead of having the model produce a single overall score, it first generates three weighted atomic rules with at least one semantic rule and one paralinguistic rule, then scores each rule separately and aggregates the scores according to their weights. The resulting reward is therefore decomposable and traceable: when a failure occurs, one can pinpoint which rule and which type of evidence are responsible. Relative to a black-box holistic reward, this constitutes a substantive design advance.
-
Using RubriVox, both the reward model and the downstream policy improve over the base model and over Gemini-based holistic/decoupled rewards. In addition, the human evaluation also shows that the generated rules are highly discriminative, which adds a certain degree of persuasiveness.
Critical weaknesses
-
The claim on "interpretability or credit assignment" is not validated. The paper repeatedly asserts that the rule-level reward is "decomposable and traceable" and provides "precise credit assignment," yet after Eq. 3, GRPO sees only the scalar ; there is no per-rule advantage, no token-level credit, and no rule-level ablation demonstrating that decomposition actually improves credit assignment. "Interpretable" is supported only by a case study (Figure 4) and by rule discriminability; there is no faithfulness test (e.g., rule swap, rule deletion, counterfactual audio, or whether humans can predict the verifier's score from the trace).
-
The RubriVox-Verifier first sees the response and then generates a rubric for it, rather than scoring all responses with a fixed rubric. The method feeds the full interaction (user speech, 1s silence, assistant response) into the verifier, so rule generation already knows the candidate response, unlike the response-blind SFT teacher. In policy RL, each response to the same question induces its own rules and its own scalar reward, yet GRPO compares these rewards as if they came from a common scale. They do not: one response may score higher because it elicits vaguer, easier rules or weights the paralinguistic dimension it excels at, while another may be penalized for triggering stricter rules. The policy may thus learn to make the verifier hand it easy questions rather than to answer better, which is structural reward hacking. This also contradicts the paper's own requirement that evaluation stay consistent under the same question: verifier RL uses shared-trajectory scoring, but policy RL does not. Consequently, Table 1 gains cannot be cleanly attributed to response quality, since within-group reward differences conflate better responses with looser rules, and re-scoring under fixed rules could flip the ranking.
-
A further concern is data quality: the target responses here are synthesized with TTS models such as CosyVoice3, and the positive and negative samples most likely come from TTS style conditions or spoken-dialogue model outputs, with the paper not clearly explaining how the negatives were obtained. The verifier may therefore learn "traces of the TTS style condition" rather than the emotion, speaking rate, and tone of real human speech. VStyle and VoiceBench are also mainly read-aloud / single-turn QA rather than natural conversation. For a paper motivated by "spoken dialogue", this is a construct-validity problem: what it measures as "paralinguistic quality" may not equal paralinguistic quality in real interaction. In my view, the authors should disclose an analysis and quality check of the constructed data.
Questions for authors
-
Section 3.2 generates rules from the full interaction including the candidate response, while Section 3.1 requires evaluation to be consistent across responses to the same question. In policy RL (Eq. 14–15), does each rollout use its own response-conditioned rule set, or is one rule set shared across the GRPO group? If per-response, how do you ensure the rewards are comparable and prevent the policy from inducing lenient rubrics? A query-only vs. per-response ablation, plus rule-set overlap/variance, would materially affect my assessment.
-
What are the sizes, sources, and annotation protocols of the held-out semantic/paralinguistic test sets? Are the labels human, TTS style conditions, or model-generated? What is the inter-annotator agreement, and how was contamination with training prompts controlled? Is Table 2 accuracy per-response 3-way or pairwise, and what is the chance level?
7 comments
Response to Reviewer DQXe(2/3)
W2 & Q1. Response-conditioned rubrics: comparability within GRPO groups and risk of lenient rubrics
Direct answer. Yes. In downstream policy GRPO (§3.5), each rollout receives its own rubric from the frozen verifier. We agree that this makes cross-response comparability essential. It is an explicit training objective of the verifier (§§3.3–3.4), and we verified it empirically with the analyses below.
How consistency is trained. In SFT, the teacher generates rules with all candidate responses hidden, and the selected rubric (rules and weights) is shared by the traces of every positive and negative response in the pool; only the scores differ, each from that response's own evidence. The verifier is thus trained to keep the rubric fixed when only the response changes. Verifier RL then adds two terms: (Eq. 9) rewards rubrics whose criteria match, in content and type, those generated from the opposite response, and (Eq. 10) applies each rubric to both responses — criteria, weights, routes, and instructions fixed — paying a bonus only if the positive response scores strictly higher. A lenient rubric that gives both responses full marks therefore forfeits . Removing lowers held-out discrimination from 80.8% to 76.4% (semantic) and 78.5% to 75.3% (paralinguistic) (Table 2).
Does rescoring under a shared rubric change the ranking? We took 200 held-out positive–negative pairs and 200 four-response GRPO groups from policy training. An LLM judge (GPT-6-Astra) matched rules by requirement and criterion type, allowing differences in wording and order; for GRPO groups it also required equal strictness, so a more lenient rule counted as a mismatch. Whenever rubrics differed, we rescored all responses in the case under one shared rubric. No preference was reversed:
| Setting | Cases | Full rubric match | Rescored (mismatch) | Reversals |
|---|---|---|---|---|
| Held-out pairs | 200 | 72% | 56 | 0 |
| Policy GRPO groups | 200 | 78% | 44 | 0 |
Why we control for shared rubrics rather than query-only rubrics. You suggest a query-only condition. We want to explain why we report a shared-rubric control instead. The verifier is trained throughout (§§3.3–3.4) on the full interaction , so feeding it the user turn alone is out of distribution: its rule generation, weighting, and routing have never been optimized for that input, and the resulting degradation would measure the input mismatch rather than the effect of rubric sharing. We therefore target your underlying concern directly — that within a GRPO group, responses are compared under different rubrics — while keeping the verifier in distribution.
Do the downstream gains persist when rubrics are shared by construction? We retrained VITA-Audio with a within-group control. For each four-response group, the frozen verifier generated rules and routes from the first response, and we then applied that single rubric to all four responses, scoring each from its own evidence. Within a group, every response is now compared under identical criteria, weights, and routes, so a response can no longer gain by inducing a more lenient rubric. This is the condition under which your concern predicts the gains should disappear.
| VITA-Audio | VStyle Avg ↑ | Alpaca ↑ | OBQA ↑ | IFEval ↑ | Adv ↑ |
|---|---|---|---|---|---|
| Base policy | 2.55 | 3.83 | 74.3 | 26.2 | 94.2 |
| RubriVox, per-response rubrics | 2.89 | 4.14 | 77.1 | 30.6 | 98.3 |
| RubriVox, shared rubrics | 2.83 | 4.09 | 78.2 | 28.6 | 97.3 |
With shared rubrics, VITA-Audio still improves over the base policy on all five metrics (VStyle 2.55→2.83, OBQA 74.3→78.2, IFEval 26.2→28.6) and stays close to the per-response setting, so the gains do not depend on within-group rubric variation.
Why reward hacking would be visible. We agree a frozen verifier can in principle be gamed, so it is worth stating what that failure would look like. If the policy learned to elicit lenient rubrics rather than to answer better, its verifier reward would rise while externally scored quality stayed flat or fell — every metric in Table 1 comes from VoiceBench and VStyle evaluators that are independent of RubriVox and never see its rubrics. We observe the opposite: the gains appear on those external metrics for all three backbones, and on human-recorded sets (W3). Two structural constraints also limit the room for such a strategy. Each rubric must contain at least one semantic and one paralinguistic rule with weights summing to 100 (§3.2), so a policy can shift weight but cannot drop a dimension it is weak on; and the verifier is frozen, so leniency it does not already exhibit cannot be induced over training.
Response to Reviewer DQXe(3/3)
W3. Synthetic speech, label construction, and transfer to human speech
Semantic pairs. Labels come from the response model's own errors: Qwen2.5-Omni samples eight answers per query, a teacher checks their transcripts against the task references, and we keep queries with both correct and incorrect answers, each four-response pool containing at least one positive and one negative (§3.3). Pools mix spoken-dialogue model outputs with speech from CosyVoice3, IndexTTS 2.5, and CosyVoice2 (Appendix B.3), so semantic labels depend on content, not on the speech source. To verify the references themselves, GPT-5 received each original question, its spoken rewrite, and the dataset's answer key or reference response, and flagged ambiguous questions, unsupported answers, and rewrites that changed the facts the answer depends on. A human reviewer examined every flagged item and a random sample of accepted items per task type.
Paralinguistic pairs cover eight attributes (accent, age, emotion, gender, language, pitch, speaking rate, volume). GPT-5 writes a user turn and two replies for a conversational setting in which one attribute matters. For explicit understanding, one reply describes the relevant attribute of the user's voice correctly and the other incorrectly. For explicit generation, both replies answer the request and the same TTS engine renders one with the requested delivery and the other with a deliberately mismatched delivery — so engine-specific artifacts cannot separate a pair; only the targeted delivery differs. For implicit cases, the label reflects whether the reply's tone fits the user's speech and the situation. We also label replies actually produced by spoken-dialogue models with the same criteria.
Quality checks. Scripted clips are discarded if their Whisper-large-v3 WER exceeds 5%; emotion-conditioned clips must be recognized by emotion2vec as the intended emotion with score ≥ 0.5; and Gemini 2.5 Pro checks that the audio and dialogue context support each label. Five experts then independently audited 600 paralinguistic training samples covering all eight attributes. All 600 labels received majority support: 579 (96.5%) unanimously, 10 by four of five, and 11 by three of five.
Transfer to human speech. In Table 1, all three backbones improve on VoiceBench's human-recorded subsets (CommonEval, WildVoice, SD-QA). For example, SD-QA rises by 2.0, 1.9, and 1.8 points for VITA-Audio, KimiAudio, and Step-Audio-2-mini. To probe paralinguistic quality more directly, we also evaluated the policies on the human-recorded EchoMind-Human set [4]. Its open-ended task scores context fit (CtxFit), use of the speaker's vocal cues (SpeechRel), and vocal empathy of the reply (VES) on 5-point scales (before → after RubriVox):
| Backbone | CtxFit | SpeechRel | VES |
|---|---|---|---|
| VITA-Audio | 4.00 → 4.12 | 3.00 → 3.08 | 2.13 → 2.40 |
| KimiAudio | 3.43 → 3.50 | 2.54 → 2.69 | 3.23 → 3.24 |
| Step-Audio-2-mini | 3.88 → 4.02 | 3.21 → 3.40 | 3.14 → 3.20 |
All three policies improve in context fit and in using the speaker's vocal cues, and VITA-Audio gains most in vocal empathy (+0.27).
Q2. Held-out data, annotation, and chance levels
Test-set size, sources, and split. During data construction, we held out 203 semantic pairs and 279 paralinguistic pairs, for a total of 482 positive–negative pairs. The semantic pairs were drawn from the reference-checked response pools described in §3.3, and the paralinguistic pairs were drawn from the spoken tasks described in W3. We split by query so that responses to the same query stayed together. We also checked the held-out prompts against those used for verifier SFT, verifier RL, and downstream policy RL and removed detected duplicates.
Label provenance and agreement. Please ref to W3. We have performed the same quality check method on all data in the test set.
Table 2 metrics. As defined at the start of §4.2, Accuracy is per-response binary accuracy. A prediction is correct when the model’s judgment of whether the response meets the relevant requirement matches its ground-truth label. Uniform random guessing gives 50% expected accuracy. Discrimination is pairwise. A pair is counted as correct when the positive response receives a strictly higher score than the negative response. A tie receives zero credit. A uniformly random tie-free ordering gives 50% expected discrimination.
References
[1] Checklists Are Better Than Reward Models For Aligning Language Models.
[2] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.
[3] Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning.
[4] EchoMind: An Interrelated Multi-Level Benchmark for Evaluating Empathetic Speech Language Models.
Response to Reviewer DQXe(1/3)
Thank you for your detailed and constructive review. We appreciate your recognition that spoken-dialogue reward modeling must cover semantic and paralinguistic quality jointly. We are also grateful that you view our decomposable, traceable rule-level reward as a substantive design advance over black-box holistic rewards. We address your concerns below; revisions in the PDF are marked in blue.
We agree. Policy GRPO uses one advantage per response (Eq. 15), and RubriVox does not provide per-rule or token-level advantages. What the trace provides is an explicit decomposition of each response's reward. Under a given rubric, a one-point drop on a speaking-rate rule weighted at 30% lowers the reward by exactly 0.30 (Eq. 3), and the trace records the evidence behind that score. Rubric-based RL methods use the same response-level aggregation of checklist or rubric scores while keeping a scalar policy advantage [1–3]. We have replaced "precise credit assignment" with "explicit rule-level contributions to the response reward" in the paper.
Are the rule-level scores faithful to the response? The reward is a deterministic function of the trace (Eq. 3), so the remaining question is whether each rule-level score is correct. We tested this with human experts. RubriVox evaluated 200 spoken responses to 100 questions from the held-out set, with three rules per response, yielding 600 response–rule items. Five experts independently scored every item on the verifier's 1–5 scale. They saw the question, the response audio, the rule, and any tool measurements, but not the verifier's own scores or analyses . We averaged the expert scores for each rule and recombined them using the verifier's weights:
sˉji=51∑h=15sji(h), Rjhuman=∑i=13100wji,sˉji.
Under the same rules and weights, replacing the experts' scores with the verifier's changes the final reward by 0.31 points on average on the 1–5 scale. This error is small relative to the signal GRPO actually uses: the mean positive–negative reward margin of the verifier is 2.447 (Table 2), roughly eight times the measurement error, so an error of this size rarely changes which response in a pair is preferred.