- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitation that existing streaming‑reasoning large‑language models rely heavily on supervised imitation learning and lack the ability to adaptively allocate computation. The authors propose AdaSR together with the HRPO (Hierarchical Relative Policy Optimization) algorithm. HRPO performs hierarchical credit assignment for the two‑phase streaming‑reasoning structure (streaming phase + deep‑reasoning phase), letting the model autonomously decide when to reason or skip computation for incrementally arriving inputs. Experiments are conducted on Qwen3‑1.7B/4B/8B across mathematical and QA benchmarks. Compared with SFT baselines and vanilla GRPO, AdaSR‑HRPO achieves a better trade‑off among reasoning accuracy, token‑level computation cost and streaming latency, and also yields improved out‑of‑domain generalization.
Critical strengths
-
HRPO resolves the inherent cross‑phase credit‑assignment paradox in streaming reasoning. Conventional GRPO applies identical sequence‑level advantage to all tokens and cannot distinguish the contribution of local streaming thoughts and final deep integration; HRPO decomposes advantages into streaming‑local, deep‑local and global signals assigned to corresponding token ranges, aligning RL update signals with the temporal structure of streaming reasoning and fixing a core technical bottleneck for this task.
-
This work delivers a practical end‑to‑end RL training stack for streaming‑reasoning rollout. The authors adapt veRL and vLLM with custom streaming attention masks and grouped position encoding. Partial‑observation constraints are strictly enforced during both trajectory sampling and log‑probability evaluation, ensuring training‑inference alignment and improving real‑world deployability.
-
Systematic ablations verify the necessity of every core module within AdaSR‑HRPO. The paper adopts component‑wise advantage‑level reward normalization to avoid signal corruption from summed‑reward normalization. Comprehensive ablations over advantage granularity, reward components and hyperparameters provide solid empirical evidence for interpreting performance gains.
Critical weaknesses
-
Simulated text‑only streaming diverges from the paper’s motivating real‑world scenarios. Though the introduction draws heavy motivation from speech and video multimodal streaming, all experiments are built on sentence‑segmented text simulation. There are no multimodal evaluations, so we cannot confirm whether HRPO’s performance gains transfer beyond this synthetic text setting, weakening evidence for real‑world applicability.
-
The baseline suite omits key state‑of‑the‑art RLVR methods. Comparisons are limited to SFT StreamingThinker and vanilla GRPO, without DAPO, Dr‑GRPO or GDPO baselines. Without controlled comparisons, it is hard to isolate performance improvements brought by hierarchical stage‑wise advantage assignment from gains originating from GDPO‑style decoupled multi‑reward normalization, creating ambiguity about the true source of empirical improvements.
-
Critical hyper‑parameters lack cross‑dataset generalization validation. Key hyper‑parameters λ and η are only tuned on GSM‑symbolic and directly reused across all other benchmarks, without reported cross‑dataset hyper‑parameter sweeps. It remains unknown whether these hyper‑parameters generalize across diverse reasoning tasks; heavy per‑dataset re‑tuning would introduce large practical adoption overhead for AdaSR‑HRPO.
Questions for authors
-
What artifacts or distribution mismatches are introduced by simulating streaming inputs through sentence‑level text segmentation? Could you characterize gaps between this text‑simulated setting and real‑world incremental multimodal observations?
-
HRPO‑sentence and HRPO‑token exhibit degraded performance relative to stage‑level HRPO. Is this performance drop caused by reinforcement‑learning optimization instability or broken cross‑segment reasoning dependencies? Can you provide diagnostic metrics such as per‑segment intermediate reasoning accuracy to identify the root cause?
2 comments
Part 2
W3: Critical hyper‑parameters lack cross‑dataset generalization validation.
We fully agree that validating hyperparameter generalization across datasets is crucial. To clarify, we evaluated the key hyperparameters on GSM-Symbolic P1, GSM-Symbolic P2, and MetaMathQA, and selected the configuration with the best overall performance across these datasets. The corresponding hyperparameter ablations are reported in Table 6 of the appendix.
With the selected configuration, our method consistently outperforms both SFT and GRPO across the evaluated datasets, including OOD settings. The cross-dataset OOD results are reported in Table 4 of the main paper and are reproduced below for convenience. These results provide evidence that the selected hyperparameters remain effective beyond a single dataset. We will clarify the hyperparameter-selection procedure and the corresponding cross-dataset validation in the revised manuscript.
| Model | Method | GSM-Infinite Acc. ↑ | GSM-Infinite Total Len. ↓ | LogicNLI Acc. ↑ | LogicNLI Total Len. ↓ |
|---|---|---|---|---|---|
| Qwen3-4B | SFT-based | 0.479 | 273.408 | 0.474 | 564.248 |
| Qwen3-4B | GRPO | 0.509 | 285.38 | 0.445 | 595.263 |
| Qwen3-4B | HRPO | 0.546 | 265.65 | 0.489 | 591.985 |
| Qwen3-8B | SFT-based | 0.519 | 259.738 | 0.514 | 536.036 |
| Qwen3-8B | GRPO | 0.554 | 262.550 | 0.490 | 547.642 |
| Qwen3-8B | HRPO | 0.606 | 233.772 | 0.549 | 520.947 |
Table 4. OOD performance comparison on GSM-Infinite (within-task distribution shift) and LogicNLI (cross-task generalization). Total Len. denotes total generated tokens.
Q2:HRPO‑sentence and HRPO‑token exhibit degraded performance relative to stage‑level HRPO. Identify the root cause of the performance drop.
Thank you for this question. We first clarify the differences among the three settings. HRPO assigns separate local advantages to the streaming and deep-reasoning stages. HRPO-sentence assigns streaming advantages at the sentence level. HRPO-token further localizes format-related credit to the corresponding boundary tokens. The remaining reward components follow the same assignment scheme as HRPO.
Our reward design aims to support adaptive token allocation across reasoning units while discouraging unnecessary computation.
Stage-level HRPO regularizes total streaming length, allowing a fixed budget to be redistributed across reasoning units (Eq. 6). Together with trajectory-level correctness, this permits more reasoning on demanding units and shorter thoughts or skipping elsewhere. The efficiency reward discounts streaming computation to reflect its overlap with input arrival (Eq. 7).
Sentence-level and token-level assignment pressures each thought to be shorter, potentially penalizing useful longer reasoning. We hypothesize that this restricts adaptive allocation and contributes to the accuracy drop. HRPO-token inherits this potential limitation because its non-format rewards retain sentence-level assignment.
To further examine this explanation, we quantify the distribution of token counts across individual streaming reasoning units under all three settings, as summarized below.
| Method | Mean streaming tokens / rollout | Skipped units (%) | Normalized Gini, all units | Normalized Gini, non-skipped units |
|---|---|---|---|---|
| HRPO (stage-level) | 116 | 17% | 0.48 | 0.35 |
| HRPO-sentence | 112 | 12% | 0.38 | 0.28 |
| HRPO-token | 113 | 13% | 0.35 | 0.26 |
The normalized Gini coefficient measures how evenly streaming reasoning tokens are allocated across reasoning units within a rollout. Let denote the number of reasoning units and the number of streaming reasoning tokens in unit . A skipped unit has . The normalized Gini coefficient is defined as
The coefficient ranges from 0 to 1. A value of 0 means that all units receive the same number of tokens. A value of 1 means that all tokens are allocated to a single unit. Higher values indicate a more concentrated allocation.
Part 1
Thank you for raising this point. Streaming reasoning aims to enable models to reason while information is still arriving, allowing them to think as they observe. This involves two closely related questions: how to define local reasoning boundaries and how to optimize the reasoning process under partial observations. Our paper focuses on the latter, using reinforcement learning to optimize streaming reasoning. To simplify the former, we segment the input into complete sentences as the units of incremental observation.
The motivation for this choice is that complete sentences provide semantically coherent units for reasoning. In real-world speech scenarios, incremental ASR could convert incoming speech into text, which could then be segmented according to the completeness of the recognized semantic units. Similarly, for video input, descriptions generated from observations available so far could provide local textual context for streaming reasoning. These offer possible interfaces between our formulation and real-world incremental inputs.
We acknowledge that end-to-end multimodal processing and finer-grained, dynamically determined reasoning boundaries would more closely reflect real-world streaming scenarios. Our text-simulated setting simplifies these aspects and does not capture challenges such as perception errors, uncertain semantic boundaries, information loss through textual representations, and additional processing latency. As an initial step, our work studies RL-based reasoning optimization under simplified input boundaries. Addressing more complex boundary identification and end-to-end multimodal observations is beyond the scope of the present paper and remains a direction for future work. We will clarify this scope and the corresponding limitations in the revised manuscript.
We apologize for the lack of clarity in our baseline description. The GRPO baselines in the main table already use GDPO-style component-wise reward normalization. The original GRPO-style variant, which aggregates multiple rewards before normalization, is included in the ablation study in Appendix Table 6. Under the same hyperparameter settings, this variant exhibits a sharp decline in both accuracy and generation length, consistent with an overly dominant length-related signal.
To address the reviewer’s concern, we additionally conduct controlled comparisons incorporating the key optimization components of DAPO and Dr. GRPO. We apply these components to both GRPO and HRPO while keeping the remaining settings fixed, allowing us to isolate their effects and examine whether the benefits of stage-level advantage assignment persist under different optimization choices. The results are summarized in the table below.
We observe only modest gains from token-mean aggregation. One possible explanation is limited variation in generated reasoning lengths during training, under which sequence-level and token-level averaging assign similar relative weights to the sampled trajectories. We therefore interpret the limited gains as potentially specific to the length distribution in our setting.