acceptodds
Under review as a conference paper at ICLR 2027

RubriVox: From Rule-Based Reward Modeling to Spoken Dialogue Policy Optimization

Abstract

RL (Reinforcement learning) has become the mainstream post-training paradigm for large language models, but its application in end-to-end speech dialogue models is limited by the difficulty of reward modeling. Unlike text-only assistants, spoken dialogue models must be evaluated along both semantic and paralinguistic dimensions. However, existing audio-language reward models often collapse these factors into a single holistic and opaque score, making failures hard to interpret and providing weak learning signals for downstream optimization. We propose RubriVox, a verifier-based RL framework for spoken dialogue models. At its core, RubriVox trains an audio-language Reasoning Reward Model (RubriVox-Verifier) as a structured reward verifier. RubriVox-Verifier decomposes spoken-response evaluation into semantic and paralinguistic critic rules, routes each rule to the proper evidence source for verification, and aggregates the verified rule-level judgments into a scalar reward. This design yields more interpretable and fine-grained feedback than holistic black-box reward scoring. Experiments show that when used for downstream RL, RubriVox provides a principled reward signal for aligning spoken dialogue models with user intent and vocal appropriateness. Demo could be found in https://rubrivox.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spoken Dialogue, Reward Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DQXeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: RubriVox breaks down the evaluation of spoken responses into critic rules covering both semantics and paralinguistics. Each rule is then directed to the evidence source best suited to verify it, and the verified judgments at the rule level are combined into a single scalar reward. The resulting RubriVox-Verifier is trained using contrastive verifier GRPO; afterwards it is frozen and supplies scalar rewards to the downstream spoken-policy GRPO.

Contributions: This work develops a contrastive spoken-verifier data pipeline that combines speech-suitable prompt filtering, difficulty-aware response selection, and Contrastive rule-tree search to construct discriminative rule-level verifier traces; proposes RubriVox-Verifier and contrastive verifier GRPO, where RubriVox-Verifier produces evidence-routed rule-level reward traces while contrastive verifier GRPO optimizes verifier rollouts with format, calibration, stability, and discrimination rewards; and builds RubriVox, a full verifier-guided RL framework for spoken dialogue models, showing that structured RubriVox-Verifier rewards outperform Gemini factorized and holistic rewards in downstream policy optimization.

Critical strengths
  1. The paper addresses a topic with a very worthwhile focus: reward modeling for spoken dialogue indeed needs to cover both semantic and paralinguistic dimensions at the same time, and existing approaches that use LALMs as judges tend to produce holistic, uninterpretable scores.

  2. The reward design is reasonably structural. Instead of having the model produce a single overall score, it first generates three weighted atomic rules with at least one semantic rule and one paralinguistic rule, then scores each rule separately and aggregates the scores according to their weights. The resulting reward is therefore decomposable and traceable: when a failure occurs, one can pinpoint which rule and which type of evidence are responsible. Relative to a black-box holistic reward, this constitutes a substantive design advance.

  3. Using RubriVox, both the reward model and the downstream policy improve over the base model and over Gemini-based holistic/decoupled rewards. In addition, the human evaluation also shows that the generated rules are highly discriminative, which adds a certain degree of persuasiveness.

Critical weaknesses
  1. The claim on "interpretability or credit assignment" is not validated. The paper repeatedly asserts that the rule-level reward is "decomposable and traceable" and provides "precise credit assignment," yet after Eq. 3, GRPO sees only the scalar RruleR_{rule}; there is no per-rule advantage, no token-level credit, and no rule-level ablation demonstrating that decomposition actually improves credit assignment. "Interpretable" is supported only by a case study (Figure 4) and by rule discriminability; there is no faithfulness test (e.g., rule swap, rule deletion, counterfactual audio, or whether humans can predict the verifier's score from the trace).

  2. The RubriVox-Verifier first sees the response and then generates a rubric for it, rather than scoring all responses with a fixed rubric. The method feeds the full interaction (user speech, 1s silence, assistant response) into the verifier, so rule generation already knows the candidate response, unlike the response-blind SFT teacher. In policy RL, each response to the same question induces its own rules and its own scalar reward, yet GRPO compares these rewards as if they came from a common scale. They do not: one response may score higher because it elicits vaguer, easier rules or weights the paralinguistic dimension it excels at, while another may be penalized for triggering stricter rules. The policy may thus learn to make the verifier hand it easy questions rather than to answer better, which is structural reward hacking. This also contradicts the paper's own requirement that evaluation stay consistent under the same question: verifier RL uses shared-trajectory scoring, but policy RL does not. Consequently, Table 1 gains cannot be cleanly attributed to response quality, since within-group reward differences conflate better responses with looser rules, and re-scoring under fixed rules could flip the ranking.

  3. A further concern is data quality: the target responses here are synthesized with TTS models such as CosyVoice3, and the positive and negative samples most likely come from TTS style conditions or spoken-dialogue model outputs, with the paper not clearly explaining how the negatives were obtained. The verifier may therefore learn "traces of the TTS style condition" rather than the emotion, speaking rate, and tone of real human speech. VStyle and VoiceBench are also mainly read-aloud / single-turn QA rather than natural conversation. For a paper motivated by "spoken dialogue", this is a construct-validity problem: what it measures as "paralinguistic quality" may not equal paralinguistic quality in real interaction. In my view, the authors should disclose an analysis and quality check of the constructed data.

Questions for authors
  1. Section 3.2 generates rules from the full interaction including the candidate response, while Section 3.1 requires evaluation to be consistent across responses to the same question. In policy RL (Eq. 14–15), does each rollout use its own response-conditioned rule set, or is one rule set shared across the GRPO group? If per-response, how do you ensure the rewards are comparable and prevent the policy from inducing lenient rubrics? A query-only vs. per-response ablation, plus rule-set overlap/variance, would materially affect my assessment.

  2. What are the sizes, sources, and annotation protocols of the held-out semantic/paralinguistic test sets? Are the labels human, TTS style conditions, or model-generated? What is the inter-annotator agreement, and how was contamination with training prompts controlled? Is Table 2 accuracy per-response 3-way or pairwise, and what is the chance level?

7 comments
AuthorsOpenReview ↗

Response to Reviewer DQXe(1/3)

Thank you for your detailed and constructive review. We appreciate your recognition that spoken-dialogue reward modeling must cover semantic and paralinguistic quality jointly. We are also grateful that you view our decomposable, traceable rule-level reward as a substantive design advance over black-box holistic rewards. We address your concerns below; revisions in the PDF are marked in blue.

W1. Interpretability and credit assignment.

We agree. Policy GRPO uses one advantage per response (Eq. 15), and RubriVox does not provide per-rule or token-level advantages. What the trace provides is an explicit decomposition of each response's reward. Under a given rubric, a one-point drop on a speaking-rate rule weighted at 30% lowers the reward by exactly 0.30 (Eq. 3), and the trace records the evidence behind that score. Rubric-based RL methods use the same response-level aggregation of checklist or rubric scores while keeping a scalar policy advantage [1–3]. We have replaced "precise credit assignment" with "explicit rule-level contributions to the response reward" in the paper.

Are the rule-level scores faithful to the response? The reward is a deterministic function of the trace (Eq. 3), so the remaining question is whether each rule-level score is correct. We tested this with human experts. RubriVox evaluated 200 spoken responses to 100 questions from the held-out set, with three rules per response, yielding 600 response–rule items. Five experts independently scored every item on the verifier's 1–5 scale. They saw the question, the response audio, the rule, and any tool measurements, but not the verifier's own scores or analyses . We averaged the expert scores for each rule and recombined them using the verifier's weights:

sˉji=15∑h=15sji(h)\bar{s}_{ji}=\frac{1}{5}\sum_{h=1}^{5}s_{ji}^{(h)}, Rjhuman=∑i=13wji100,sˉji\quad R_j^{\text{human}}=\sum_{i=1}^{3}\frac{w_{ji}}{100}\\,\bar{s}_{ji}.

ComparisonItemsMean absolute error ↓
Verifier rule score vs. mean expert score600 rules0.62
Verifier reward vs. expert-based reward200 responses0.31

Under the same rules and weights, replacing the experts' scores with the verifier's changes the final reward by 0.31 points on average on the 1–5 scale. This error is small relative to the signal GRPO actually uses: the mean positive–negative reward margin of the verifier is 2.447 (Table 2), roughly eight times the measurement error, so an error of this size rarely changes which response in a pair is preferred.

AuthorsOpenReview ↗

Response to Reviewer DQXe(2/3)

W2 & Q1. Response-conditioned rubrics: comparability within GRPO groups and risk of lenient rubrics

Direct answer. Yes. In downstream policy GRPO (§3.5), each rollout receives its own rubric from the frozen verifier. We agree that this makes cross-response comparability essential. It is an explicit training objective of the verifier (§§3.3–3.4), and we verified it empirically with the analyses below.

How consistency is trained. In SFT, the teacher generates rules with all candidate responses hidden, and the selected rubric (rules and weights) is shared by the traces of every positive and negative response in the pool; only the scores differ, each from that response's own evidence. The verifier is thus trained to keep the rubric fixed when only the response changes. Verifier RL then adds two terms: rrobustr^{\text{robust}} (Eq. 9) rewards rubrics whose criteria match, in content and type, those generated from the opposite response, and rdiscr^{\text{disc}} (Eq. 10) applies each rubric to both responses — criteria, weights, routes, and instructions fixed — paying a bonus only if the positive response scores strictly higher. A lenient rubric that gives both responses full marks therefore forfeits rdiscr^{\text{disc}}. Removing rrobustr^{\text{robust}} lowers held-out discrimination from 80.8% to 76.4% (semantic) and 78.5% to 75.3% (paralinguistic) (Table 2).

Does rescoring under a shared rubric change the ranking? We took 200 held-out positive–negative pairs and 200 four-response GRPO groups from policy training. An LLM judge (GPT-6-Astra) matched rules by requirement and criterion type, allowing differences in wording and order; for GRPO groups it also required equal strictness, so a more lenient rule counted as a mismatch. Whenever rubrics differed, we rescored all responses in the case under one shared rubric. No preference was reversed:

SettingCasesFull rubric matchRescored (mismatch)Reversals
Held-out pairs20072%560
Policy GRPO groups20078%440

Why we control for shared rubrics rather than query-only rubrics. You suggest a query-only condition. We want to explain why we report a shared-rubric control instead. The verifier is trained throughout (§§3.3–3.4) on the full interaction a=[au;,1,s silence;,ay]a=[a_u;\\,1\\,\text{s silence};\\,a_y], so feeding it the user turn alone is out of distribution: its rule generation, weighting, and routing have never been optimized for that input, and the resulting degradation would measure the input mismatch rather than the effect of rubric sharing. We therefore target your underlying concern directly — that within a GRPO group, responses are compared under different rubrics — while keeping the verifier in distribution.

Do the downstream gains persist when rubrics are shared by construction? We retrained VITA-Audio with a within-group control. For each four-response group, the frozen verifier generated rules and routes from the first response, and we then applied that single rubric to all four responses, scoring each from its own evidence. Within a group, every response is now compared under identical criteria, weights, and routes, so a response can no longer gain by inducing a more lenient rubric. This is the condition under which your concern predicts the gains should disappear.

VITA-AudioVStyle Avg ↑Alpaca ↑OBQA ↑IFEval ↑Adv ↑
Base policy2.553.8374.326.294.2
RubriVox, per-response rubrics2.894.1477.130.698.3
RubriVox, shared rubrics2.834.0978.228.697.3

With shared rubrics, VITA-Audio still improves over the base policy on all five metrics (VStyle 2.55→2.83, OBQA 74.3→78.2, IFEval 26.2→28.6) and stays close to the per-response setting, so the gains do not depend on within-group rubric variation.

Why reward hacking would be visible. We agree a frozen verifier can in principle be gamed, so it is worth stating what that failure would look like. If the policy learned to elicit lenient rubrics rather than to answer better, its verifier reward would rise while externally scored quality stayed flat or fell — every metric in Table 1 comes from VoiceBench and VStyle evaluators that are independent of RubriVox and never see its rubrics. We observe the opposite: the gains appear on those external metrics for all three backbones, and on human-recorded sets (W3). Two structural constraints also limit the room for such a strategy. Each rubric must contain at least one semantic and one paralinguistic rule with weights summing to 100 (§3.2), so a policy can shift weight but cannot drop a dimension it is weak on; and the verifier is frozen, so leniency it does not already exhibit cannot be induced over training.

AuthorsOpenReview ↗

Response to Reviewer DQXe(3/3)

W3. Synthetic speech, label construction, and transfer to human speech

Semantic pairs. Labels come from the response model's own errors: Qwen2.5-Omni samples eight answers per query, a teacher checks their transcripts against the task references, and we keep queries with both correct and incorrect answers, each four-response pool containing at least one positive and one negative (§3.3). Pools mix spoken-dialogue model outputs with speech from CosyVoice3, IndexTTS 2.5, and CosyVoice2 (Appendix B.3), so semantic labels depend on content, not on the speech source. To verify the references themselves, GPT-5 received each original question, its spoken rewrite, and the dataset's answer key or reference response, and flagged ambiguous questions, unsupported answers, and rewrites that changed the facts the answer depends on. A human reviewer examined every flagged item and a random sample of accepted items per task type.

Paralinguistic pairs cover eight attributes (accent, age, emotion, gender, language, pitch, speaking rate, volume). GPT-5 writes a user turn and two replies for a conversational setting in which one attribute matters. For explicit understanding, one reply describes the relevant attribute of the user's voice correctly and the other incorrectly. For explicit generation, both replies answer the request and the same TTS engine renders one with the requested delivery and the other with a deliberately mismatched delivery — so engine-specific artifacts cannot separate a pair; only the targeted delivery differs. For implicit cases, the label reflects whether the reply's tone fits the user's speech and the situation. We also label replies actually produced by spoken-dialogue models with the same criteria.

Quality checks. Scripted clips are discarded if their Whisper-large-v3 WER exceeds 5%; emotion-conditioned clips must be recognized by emotion2vec as the intended emotion with score ≥ 0.5; and Gemini 2.5 Pro checks that the audio and dialogue context support each label. Five experts then independently audited 600 paralinguistic training samples covering all eight attributes. All 600 labels received majority support: 579 (96.5%) unanimously, 10 by four of five, and 11 by three of five.

Transfer to human speech. In Table 1, all three backbones improve on VoiceBench's human-recorded subsets (CommonEval, WildVoice, SD-QA). For example, SD-QA rises by 2.0, 1.9, and 1.8 points for VITA-Audio, KimiAudio, and Step-Audio-2-mini. To probe paralinguistic quality more directly, we also evaluated the policies on the human-recorded EchoMind-Human set [4]. Its open-ended task scores context fit (CtxFit), use of the speaker's vocal cues (SpeechRel), and vocal empathy of the reply (VES) on 5-point scales (before → after RubriVox):

BackboneCtxFitSpeechRelVES
VITA-Audio4.00 → 4.123.00 → 3.082.13 → 2.40
KimiAudio3.43 → 3.502.54 → 2.693.23 → 3.24
Step-Audio-2-mini3.88 → 4.023.21 → 3.403.14 → 3.20

All three policies improve in context fit and in using the speaker's vocal cues, and VITA-Audio gains most in vocal empathy (+0.27).

Q2. Held-out data, annotation, and chance levels

Test-set size, sources, and split. During data construction, we held out 203 semantic pairs and 279 paralinguistic pairs, for a total of 482 positive–negative pairs. The semantic pairs were drawn from the reference-checked response pools described in §3.3, and the paralinguistic pairs were drawn from the spoken tasks described in W3. We split by query so that responses to the same query stayed together. We also checked the held-out prompts against those used for verifier SFT, verifier RL, and downstream policy RL and removed detected duplicates.

Label provenance and agreement. Please ref to W3. We have performed the same quality check method on all data in the test set.

Table 2 metrics. As defined at the start of §4.2, Accuracy is per-response binary accuracy. A prediction is correct when the model’s judgment of whether the response meets the relevant requirement matches its ground-truth label. Uniform random guessing gives 50% expected accuracy. Discrimination is pairwise. A pair is counted as correct when the positive response receives a strictly higher score than the negative response. A tie receives zero credit. A uniformly random tie-free ordering gives 50% expected discrimination.

References

[1] Checklists Are Better Than Reward Models For Aligning Language Models.

[2] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.

[3] Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning.

[4] EchoMind: An Interrelated Multi-Level Benchmark for Evaluating Empathetic Speech Language Models.

Reviewer_EL5bOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the difficulty of designing reliable rewards for spoken dialogue models, which must consider both semantic content and speech quality. It proposes RubriVox, a framework that generates weighted evaluation rules, selects suitable evidence sources, and uses contrastive GRPO to train a reward verifier. Experiments show better reward discrimination and improved downstream performance than Gemini-based reward methods on VoiceBench and VStyle.

Critical strengths
  • The paper studies an important problem in spoken dialogue: how to give useful reward signals for both answer content and voice quality.
  • The paper is well organized.
Critical weaknesses
  • The human studies provide useful evidence for the proposed method. However, further evaluation on unseen data would help show its ability to generalize. More details about the source and quality of the voice-quality labels would also strengthen the evaluation.
  • The results are promising, but direct comparisons with existing speech reward models, such as WavReward, UniSRM, SpeechLLM-as-Judges, and GSRM, are missing. Such comparisons would help better show the advantages of the proposed method.
Questions for authors
  • How are the positive and negative labels for voice quality created and checked?
  • Can the authors compare RubriVox with one or more existing speech reward models, such as WavReward, UniSRM, or GSRM, on the same test set?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EL5b(1/2)

Thank you for your positive assessment. We appreciate your recognition that useful rewards for both answer content and voice quality are an important problem in spoken dialogue, that the paper is well organized, that the human studies provide useful evidence, and that the results are promising. We address your two suggestions below; revisions in the PDF are marked in blue.

W1 & Q1. Generalization to unseen data, and how voice-quality labels are created and checked

Evaluation on unseen data. Beyond the human studies, both the verifier and the policies it trains are evaluated on data unseen during training. Table 2 uses 482 held-out pairs (203 semantic, 279 paralinguistic), split by query and deduplicated against all training prompts. The downstream policies are evaluated only on external benchmarks (VoiceBench, VStyle), including VoiceBench's human-recorded subsets, on which all three backbones improve. We have additionally evaluated all three policies on the human-recorded EchoMind-Human set [1], whose open-ended task scores context fit (CtxFit), use of the speaker's vocal cues (SpeechRel), and vocal empathy of the reply (VES) on 5-point scales (before → after RubriVox):

BackboneCtxFitSpeechRelVES
VITA-Audio4.00 → 4.123.00 → 3.082.13 → 2.40
KimiAudio3.43 → 3.502.54 → 2.693.23 → 3.24
Step-Audio-2-mini3.88 → 4.023.21 → 3.403.14 → 3.20

All three improve in context fit and in incorporating cues from a human speaker's voice, and VITA-Audio gains most in vocal empathy (+0.27).

How paralinguistic (voice-quality) labels are created. The data cover eight attributes: accent, age, emotion, gender, language, pitch, speaking rate, and volume. GPT-5 writes a user turn and two replies for a conversational setting in which one attribute matters. If the user asks about their own voice, one reply describes it correctly and the other incorrectly. If the user requests a delivery style, both replies answer the request and the same TTS engine renders one with the requested delivery and the other with a deliberately mismatched delivery — so engine-specific artifacts cannot separate a pair, only the targeted delivery differs. If no style is requested, the label reflects whether the reply's tone fits the user's speech and the situation. We also label replies actually produced by spoken-dialogue models with the same criteria (Appendix B.4). Semantic labels instead come from reference-based correctness of the response model's own answers (§3.3).

How they are checked. Scripted clips are discarded if their Whisper-large-v3 WER exceeds 5%; emotion-conditioned clips must be recognized by emotion2vec as the intended emotion with score ≥ 0.5; and Gemini 2.5 Pro checks that the audio and dialogue context support each label. Five experts then independently audited 600 paralinguistic training samples covering all eight attributes. All 600 labels received majority support, and 579 (96.5%) were confirmed unanimously. For semantic data, GPT-5 received each original question, its spoken rewrite, and the source dataset's answer key or reference response, and flagged ambiguous questions, unsupported answers, and rewrites that changed the facts the answer depends on; a human reviewer then examined every flagged item and a random sample of accepted items per task type.

References

[1] EchoMind: An Interrelated Multi-Level Benchmark for Evaluating Empathetic Speech Language Models.

AuthorsOpenReview ↗

Response to Reviewer EL5b(2/2)

W2 & Q2. Comparison with existing speech reward models on the same test set

We evaluated UniSRM, and SDiaReward on the same held-out set of 203 semantic and 279 paralinguistic pairs used for RubriVox in Table 2 with official checkpoints and default prompts. Discrimination is the percentage of pairs in which the positive response receives a strictly higher reward than the negative one; ties receive zero credit.

Reward modelSemantic discrimination (%) ↑Paralinguistic discrimination (%) ↑
UniSRM42.440.1
SDiaReward48.850.9
RubriVox-Verifier-Routed-RL80.878.5

RubriVox exceeds the strongest baseline by 32.0 points on semantic discrimination and by 27.6 points on paralinguistic discrimination . Values below 50% mainly reflect ties, which receive zero credit.

Downstream policy RL. Discrimination on pairs does not by itself establish that a reward model is useful for optimization, so we also used SDiaReward, SageLM, and UniSRM as the GRPO reward for the same three policies.

How pairwise judges were used as rewards. SageLM and UniSRM produce pairwise preferences rather than per-response scores, so we converted them by round-robin comparison within each GRPO group. With Gπ=4G_\pi=4 rollouts per question, a group yields (42)=6\binom{4}{2}=6 comparisons, and each response scores 1 for a win, 0 for a loss, and 0.5 for a tie, normalized by its three opponents:

rjπ=13∑k≠j(I[,j≻k,]+0.5,I[,j∼k,])∈[0,1].r^{\pi}_j=\frac{1}{3}\sum_{k\neq j}\Big(\mathbb{I}[\\,j\succ k\\,]+0.5\\,\mathbb{I}[\\,j\sim k\\,]\Big)\in[0,1].

Every response faces the same three opponents, so these scores are directly comparable within the group — which is all GRPO needs, since the advantage is group-normalized (Eq. 15) and therefore depends only on the within-group ordering and spread. Two aspects of this protocol favor the baselines rather than us: ties receive 0.5 instead of the zero credit used in our discrimination metric, and each group costs 6 judge calls against the 4 scoring calls RubriVox requires. Everything else was held fixed across reward sources: their default prompts, our GRPO hyperparameters, and number of training steps.

RubriVox gives the best VStyle Avg, Alpaca, OBQA, and IFEval on every backbone, and the best Adv on two of the three:

BackboneReward sourceVStyle Avg ↑Alpaca ↑OBQA ↑IFEval ↑Adv ↑
VITA-AudioBase2.553.8374.326.294.2
SDiaReward2.063.5870.127.390.0
SageLM2.073.1762.225.487.5
UniSRM2.383.2466.828.186.3
RubriVox2.894.1477.130.698.3
KimiAudioBase2.564.4683.561.1100.0
SDiaReward2.083.8570.155.499.4
SageLM2.013.6465.750.999.8
UniSRM2.393.9368.159.399.2
RubriVox2.734.5286.264.398.5
Step-Audio-2-miniBase2.433.5278.745.786.2
SDiaReward2.143.2261.540.887.5
SageLM2.013.0365.140.289.4
UniSRM2.353.4866.845.989.4
RubriVox2.543.7881.548.292.1

One likely contributor to the weaker results is reward resolution rather than reward direction. When a judge returns frequent ties, the within-group reward variance collapses and the group-normalized advantage is driven by the few decided comparisons, so the update follows whichever distinctions that judge happens to make confidently.

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer EL5b,

Thank you again for your time and constructive comments on our paper. We have posted a detailed response, including new experiments and corresponding revisions to the paper .

We understand that this is a busy period. When convenient, we would be grateful if you could take a look and let us know whether our response addresses your concerns. If any questions remain, we would be glad to discuss them further before the discussion period closes.

Thank you for your consideration.

Best regards,

The Authors

Reviewer_ngbzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RubriVox, a verifier-based RL framework for spoken dialogue models. The core component, RubriVox-Verifier, is a generative audio LLM RM that routes each spoken interaction into a compact set of weighted atomic rules. This work focuses on training an audio LLM as a router: direct eval, tool call, or external multimodal RMs, then evaluates the performance of the speech dialogue LLMs trained with RubriVox-Verifier.

Critical strengths
  1. RM is critical for developing and optimizing Speech LLMs. This paper investigate how rubric-based rewards benefits speech LLMs reinforcement learning.
  2. The authors conduct comprehensive experiments and analyses, evaluates the performance of the speech dialogue LLMs trained with RubriVox RM, addressing most of techical concerns.
Critical weaknesses
  1. The writing is hard to follow in places. e.g. Fig. 2 appears in the intro but lacks an explanation (V*, b, different between q and a), which actually hinders the reader's understanding.
  2. This paper still does not clarify why RMs is needed as a router for the adaptive distribution of scoring tasks. Typically, reachers need to set or adjust rewards based on the shortcomings of existing models, or select reward ratios based on the query’s label, which is reasonable. When the number of tools is limited and there are only three rubtics (K=3), the problem scale can easily be determined by the reacher.
  3. Lack of related works about speech dialogue RMs, e.g. SageLM[1], SpeechJudge[2], SDiaReward[3]. Especially, SageLM generated none black-box, explainable scorer rewards.
  4. Table 1 shows none significant performance improvement after RubriVox RL training. Therefore, a significance test is required.

[1] Ge et al., SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgement, AAAI 2026
[2] Zhang et al., SpeechJudge: Towards Human-Level Judgment for Speech Naturalness, ICLR 2026.
[3] Lu et al., SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness, ACL 2026

Questions for authors

Overall caption of Fig. 3 is needed, or separate it into two figures. Otherwise, readers cannot quickly jump from the image to the corresponding descriptions.
Additionally, I'm not quite sure why all the models except “ours” are grayed out in Fig. 3a, or which model is referred to as the “baseline” in Fig. 3b.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ngbz(1/3)

Thank you for your careful review. We appreciate your recognition that reward modeling is critical for speech LLMs, and that our experiments and analyses of rubric-based rewards for spoken-dialogue RL are comprehensive and address most technical concerns. We respond to the remaining points below; revisions in the PDF are marked in blue.

W1. Notation in Figure 2.

Thank you for pointing this out. qq is the filtered text prompt used to build response pools and search rules. a=[au;,1,s silence;,ay]a=[a_u;\\,1\\,\text{s silence};\\,a_y] is the spoken interaction the verifier receives: user speech aua_u, then response speech aya_y. V∗\mathcal{V}^* is the teacher-resolved target trace used for verifier SFT (Eq. 6). b∈+,−b\in\\{+,-\\} indicates whether a rule trajectory τb,j\tau_{b,j} is generated from the interaction containing the positive or the negative response (§3.4). We have added these definitions to the caption of Figure 2 and point readers to §§3.2–3.4 for details.

W2. Why learn query-specific rules and evidence routing?

We agree that researchers can use a model’s known weaknesses to set reward priorities and can prepare rules, weights, and routes before policy training. The difficulty in open-ended spoken dialogue is deciding what counts as a good answer to each query. Many queries admit several valid answers, and the semantic and delivery requirements vary with the user’s intent. Even with (K=3), a useful rubric must capture those requirements precisely enough to distinguish better answers from worse ones.

Our verifier-training pipeline (§3.3) already performs this offline construction and illustrates the work it requires. For each retained query, we build a pool containing both positive and negative spoken responses. The teacher proposes candidate three-rule rubrics from the query alone. We then apply each candidate rubric to the same response pool and reward it when every positive response scores above every negative one. Extending this pipeline to new queries means collecting additional labeled responses, searching over candidate rubrics, and verifying their discrimination. That preparation cost grows with the number of queries covered.

RubriVox-Verifier is trained on the resulting traces to amortize it: during RL it generates weighted rules, routes, and scores for unseen queries with no labeled pool or search. On held-out, query-disjoint pairs it reaches 80.8% semantic and 78.5% paralinguistic discrimination (Table 2), and the same frozen verifier improves VStyle Avg and OBQA on all three policy backbones (Table 1). These results demonstrate both reuse on unseen data and effectiveness in downstream policy training.

AuthorsOpenReview ↗

Response to Reviewer ngbz(2/3)

W3. Related speech reward models (SageLM, SpeechJudge, SDiaReward)

Thank you for these references. SageLM and SDiaReward were cited in §2 of the submission. We have now added SpeechJudge and expanded the discussion around two differences.

How the explanation relates to the reward. We agree that SageLM is not a black box: it explains its judgments with multi-aspect rationales. The difference is where the explanation sits relative to the number. Its judgments are not computed from explicit, query-specific weighted criteria, and acoustic properties are not routed to measurement tools. In RubriVox the reward is a deterministic weighted sum of rule-level scores (Eq. 3), each obtained through an explicit evidence route, so the trace does not accompany the reward — it determines it.

Task and evaluation. SageLM and SDiaReward evaluate the judge itself; SpeechJudge targets TTS naturalness, where the text to be spoken is given in advance. Our task is RL for open-ended spoken dialogue, where the policy must learn both what to say and how to say it. We therefore used each reward model as the GRPO reward for the same three policies.

How pairwise judges were used as rewards. SageLM and UniSRM produce pairwise preferences rather than per-response scores, so we converted them by round-robin comparison within each GRPO group. With Gπ=4G_\pi=4 rollouts per question, a group yields (42)=6\binom{4}{2}=6 comparisons, and each response scores 1 for a win, 0 for a loss, and 0.5 for a tie, normalized by its three opponents:

rjπ=13∑k≠j(I[,j≻k,]+0.5,I[,j∼k,])∈[0,1].r^{\pi}_j=\frac{1}{3}\sum_{k\neq j}\Big(\mathbb{I}[\\,j\succ k\\,]+0.5\\,\mathbb{I}[\\,j\sim k\\,]\Big)\in[0,1].

Every response faces the same three opponents, so these scores are directly comparable within the group — which is all GRPO needs, since the advantage is group-normalized (Eq. 15) and therefore depends only on the within-group ordering and spread. Two aspects of this protocol favor the baselines rather than us: ties receive 0.5 instead of the zero credit used in our discrimination metric, and each group costs 6 judge calls against the 4 scoring calls RubriVox requires. Everything else was held fixed across reward sources: default prompts, our GRPO hyperparameters, and number of training steps.

RubriVox gives the best VStyle Avg, Alpaca, OBQA, and IFEval on every backbone, and the best Adv on two of the three:

BackboneReward sourceVStyle Avg ↑Alpaca ↑OBQA ↑IFEval ↑Adv ↑
VITA-AudioBase2.553.8374.326.294.2
SDiaReward2.063.5870.127.390.0
SageLM2.073.1762.225.487.5
UniSRM2.383.2466.828.186.3
RubriVox2.894.1477.130.698.3
KimiAudioBase2.564.4683.561.1100.0
SDiaReward2.083.8570.155.499.4
SageLM2.013.6465.750.999.8
UniSRM2.393.9368.159.399.2
RubriVox2.734.5286.264.398.5
Step-Audio-2-miniBase2.433.5278.745.786.2
SDiaReward2.143.2261.540.887.5
SageLM2.013.0365.140.289.4
UniSRM2.353.4866.845.989.4
RubriVox2.543.7881.548.292.1

One likely contributor to the weaker results is reward resolution rather than reward direction. When a judge returns frequent ties, the within-group reward variance collapses and the group-normalized advantage is driven by the few decided comparisons, so the update follows whichever distinctions that judge happens to make confidently.

AuthorsOpenReview ↗

Response to Reviewer ngbz(3/3)

W4. Statistical significance of the downstream gains.

Appendix C of the submission reports the mean ± SD over three independent runs on VITA-Audio; we agree a formal significant test makes this easier to assess. We compared RubriVox with VITA-Base using two-sided Welch's tt-tests (three runs per condition, differing only in seed), with Holm correction keeping the family-wise error rate across the five metrics below 0.05. All five improvements remain significant after correction:

Metric ↑RubriVox, mean ± SDVITA-Base, mean ± SDGainHolm (p)
VStyle Avg2.89 ± 0.032.55 ± 0.02+0.340.00102
Alpaca4.18 ± 0.083.84 ± 0.02+0.340.01376
OBQA77.07 ± 0.3474.29 ± 0.22+2.780.00199
IFEval30.6 ± 0.526.2 ± 0.4+4.400.00149
Adv98.3 ± 0.394.2 ± 0.5+4.100.00199

Experiments demonstrate that the base model shows significant improvement after being trained with RubricVox.

Q1. Figure 3 captions, colors, and baseline.

We have split the former Figure 3 into Figures 3 and 4, each with its own caption. Figure 3 (rule discriminativeness): the gray bars in the former panel (a) only highlighted our model and carried no statistical meaning. All models now share the same colors for semantic and paralinguistic results, and stars mark the sample-weighted overall accuracy. Figure 4 (rule-guided paralinguistic judgment): the former "Baseline" is Qwen3-Omni (Initial), the verifier backbone before RubriVox training, and is now labeled as such.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.