- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed the idea to incorporate stateful controls into speech LLMs. The states control the models when to wait, when to think, and when to answer. Furthermore, the author proposed 6 types of rewards to optimize the performance. The model is evaluated on several spoken reasoning question-answering tasks. With DAPO with 6 rewards the model gets improvement on average accuracy. They also evaluate their method on a real human recording of spoken question reasoning.
Critical strengths
- The novel design of integrating states prediction in the streaming speech LLMs
- The design of applying DAPO to improve the model to learn better state control as well as reasoning.
- Provide analysis on human-recorded real audio of spoken reasoning QA.
Critical weaknesses
-
The improvement on the main results are marginal (Synthetic SRQA, Table3).
The accuracies of all variants of the proposed methods are very close (<5%). With a basic controller, the average accuracy is already 67.6%. With all 6 rewards by DAPO, the accuracy is 70.3%. Comparing with the latency, in table 9 (correct me if this is not the right table to refer to), the latency (numbers of tokens after question finish) is almost the same. The maximum gap between the best and the worse settings is smaller than 2 tokens.
-
The performance differences on Table 4 (real audio) is mixed and the differences are marginal.
In table 4, DAPO is only good at decreasing the latency but the accuracy also falls. Furthermore, the accuracies and latencies differences are marginal. Gap of Acc. is still < 5% and Latency is still < 2 tokens.
Questions for authors
- What is "Base controller"? I cannot find a formal definition in the paper.
- It is weird that the latencies are not shown together with the accuracies in Table 3. Reader have to look into the Appendix for table 9.
- I think it would be better to emphasize Qwen2.5-Omni-7B as a reference (top line for accuracy, baseline for latency) in Table 3. Shih et al. is using Moshi as their backbone, so there isn't too much to compare with them.
- It's better to show the confidence interval in the tables given that the numbers are already very close.
4 comments
Revised PDF uploaded
Dear Reviewer Jt78,
We have uploaded the revised paper; all changes are marked in blue. The items from our response are in place:
- Q1. The Base controller is defined in Section 4.
- Q2, Q3. Table 3 now shows post-end tokens and RTF beside accuracy and uses native Qwen2.5-Omni-7B as the reference; the literature Moshi rows moved to Appendix D.4.
- Q4. Confidence intervals for the main benchmark (Table 3) and for Real Audio Bench (Table 4).
We would be glad to answer any remaining questions. If the response and the revision address your concerns, we would be grateful if you would consider updating your assessment.
I appreciate the authors' effort for the explanation.
They addressed most of my concerns in the question section. For Q4, the author only shows the confidence interval for the accuracies. Could they also provide the confidence intervals for the latencies (Post-end tokens)? Given that the numbers are also very close across the setups.
Regarding the weakness section: From the results of Table 3 and Table 4 (Real Audio), we see that DAPO variants are only outperforming SFT on Table 3 but not Table 4. I wonder what the explanation for this is. Does it mean that DAPO is optimized on the synthetic data and degrades the model's generalization ability to real audio?
Response to Reviewer Jt78's follow-up
We thank the reviewer for the follow-up and are glad that most of the questions are resolved. We answer the two remaining points below.
Confidence intervals for post-end tokens
For the streaming Base, the 95% interval is ±0.13 tokens on the main benchmark and ±0.55 tokens on Real Audio Bench.
- Main benchmark. For Base, the six-task macro-average post-end length is 10.44 tokens, with a 95% task-stratified bootstrap interval of [10.31, 10.58] (10,000 resamples, items resampled within each task).
- Real Audio Bench (186 items). Here the interval is wider: 6.52 for Base with [5.99, 7.09]. Six-reward DAPO has the lowest observed mean in Table 4 (6.33, against 6.52 for Base and 6.64 for SFT), and we do not claim a significant residual difference among the three on this set.
Intervals for the remaining rows, with paired intervals for the differences, will be added in the final version using the same procedure.
Why DAPO outperforms SFT in Table 3 but not in Table 4
Every DAPO variant has higher observed accuracy than Base on human recordings. What does not carry over to this set is DAPO's additional gain over SFT.
| Real Audio Bench (186 items) | Correct | Difference from Base | Difference from SFT |
|---|---|---|---|
| Streaming Base | 119 | – | – |
| SFT | 128 | +4.8 [+1.6, +8.6] | – |
| DAPO, 4 rewards | 122 | +1.6 [−1.6, +5.4] | −3.2 [−7.0, +0.5] |
| DAPO, 5 rewards | 126 | +3.8 [0.0, +8.1] | −1.1 [−4.8, +2.7] |
| DAPO, 6 rewards | 121 | +1.1 [−2.7, +4.8] | −3.8 [−8.1, 0.0] |
Differences are in accuracy points, with 95% paired-bootstrap intervals (10,000 resamples of the 186 items).
- The comparison does not isolate robustness to human speech. SFT and DAPO are both trained on synthesized speech only. Real Audio Bench changes the acoustic domain and the item distribution at the same time, so a difference between the two stages on this set cannot be attributed to human audio alone.
- The differences on 186 items are small. Relative to SFT, the DAPO variants differ by 2 to 7 items. No significant difference is detected for the four- and five-reward variants; for the six-reward variant the interval ends at zero. Relative to Base, the three variants are 3, 7 and 2 items higher.
- One plausible explanation is specialization. DAPO optimizes the reward on its own training distribution, which includes a small share (below 2%) of training-split examples in the benchmark task formats. Its gain over SFT is largest on those formats: 4.2 points in Table 3 (p ≤ 5.2×10⁻⁷). The results show that this gain does not transfer uniformly to a different evaluation distribution. We have not tested this explanation directly.
- This is how the paper states it. Section 5.2 says that SFT has the highest observed accuracy on this set and that six-reward DAPO is not the most accurate configuration there; the primary claim rests on the 8,959-item benchmark.
We hope this resolves the remaining points, and would be grateful if the reviewer would consider updating the assessment.
Response to Reviewer Jt78
We sincerely thank the reviewer for recognizing the stateful control design, the use of DAPO to learn state control and reasoning together, and the analysis on human-recorded audio. We adopt all four presentation suggestions (Q1–Q4) and then address the size of the gains (W1, W2).
Q1. What is the "Base controller"?
The Base controller is the unmodified Qwen2.5-Omni-7B, with no adapter, run under the same streaming protocol as the trained controllers: the same prompts, 0.5 s decision grid, action parser, token caps and scorer. Only the weights differ, which is what makes Base→DAPO a matched comparison. The submission uses the term in its tables without defining it; we will add this definition to Section 4.
Q2, Q3, Q4. Table 3: latency beside accuracy, Qwen as the reference, confidence intervals
We agree with all three. The post-end tokens, the Qwen reference and the Real Audio intervals are in the submission but split between the main text and the appendix (Table 9); the intervals for the main benchmark are new. We will bring them together in Table 3, with Qwen2.5-Omni-7B as the reference row and the Moshi rows moved to the appendix:
W1. Size of the gains on the main benchmark
The gain is small in absolute points and large relative to the gap it addresses: DAPO closes 64% of the gap between the streaming Base and the Qwen reference, and the result is statistically reliable.
W2. Real Audio (Table 4)
Real Audio Bench is a 186-item transfer check on human speech. Every learned controller is above Base on it; SFT is the most accurate, and six-reward DAPO is not the best configuration on this set.
Summary of revisions
We hope these clarifications and revisions address the concerns, and respectfully ask the reviewer to reconsider the assessment.