acceptodds
Desk rejected at ICLR 2027

Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large reasoning models typically follow a read-then-think paradigm: they observe the complete input, reason over a static context, and then produce an answer. Yet many real-world scenarios involve continuously evolving inputs, such as audio and video streams, where latency-sensitive reasoning can benefit from processing information incrementally as it arrives rather than waiting for the entire input to complete. Recent streaming reasoning methods allow models to think while reading, but they largely rely on supervised imitation of pre-constructed trajectories, which limits their flexibility. In this paper, we propose AdaSR, an adaptive streaming reasoning framework that enables models to reason during input streaming and perform final deliberation once the stream is complete, learning when to think, and how much computation to allocate across different stages. To optimize this hierarchical reasoning process, we introduce Hierarchical Relative Policy Optimization (HRPO), which decomposes policy optimization into streaming reasoning and deep reasoning phases, providing more fine-grained advantage assignment instead of uniformly distributing a single sequence-level advantage over all tokens. HRPO integrates format, accuracy, and adaptive thinking rewards to enforce valid reasoning protocols, preserve final task performance, and encourage latency-aware computation allocation. Experiments show that AdaSR achieves a better balance among reasoning accuracy, computational efficiency, and streaming latency compared with supervised fine-tuning baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Streaming Reasoning,RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Cwv2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limitation that existing streaming‑reasoning large‑language models rely heavily on supervised imitation learning and lack the ability to adaptively allocate computation. The authors propose AdaSR together with the HRPO (Hierarchical Relative Policy Optimization) algorithm. HRPO performs hierarchical credit assignment for the two‑phase streaming‑reasoning structure (streaming phase + deep‑reasoning phase), letting the model autonomously decide when to reason or skip computation for incrementally arriving inputs. Experiments are conducted on Qwen3‑1.7B/4B/8B across mathematical and QA benchmarks. Compared with SFT baselines and vanilla GRPO, AdaSR‑HRPO achieves a better trade‑off among reasoning accuracy, token‑level computation cost and streaming latency, and also yields improved out‑of‑domain generalization.

Critical strengths
  1. HRPO resolves the inherent cross‑phase credit‑assignment paradox in streaming reasoning. Conventional GRPO applies identical sequence‑level advantage to all tokens and cannot distinguish the contribution of local streaming thoughts and final deep integration; HRPO decomposes advantages into streaming‑local, deep‑local and global signals assigned to corresponding token ranges, aligning RL update signals with the temporal structure of streaming reasoning and fixing a core technical bottleneck for this task.

  2. This work delivers a practical end‑to‑end RL training stack for streaming‑reasoning rollout. The authors adapt veRL and vLLM with custom streaming attention masks and grouped position encoding. Partial‑observation constraints are strictly enforced during both trajectory sampling and log‑probability evaluation, ensuring training‑inference alignment and improving real‑world deployability.

  3. Systematic ablations verify the necessity of every core module within AdaSR‑HRPO. The paper adopts component‑wise advantage‑level reward normalization to avoid signal corruption from summed‑reward normalization. Comprehensive ablations over advantage granularity, reward components and hyperparameters provide solid empirical evidence for interpreting performance gains.

Critical weaknesses
  1. Simulated text‑only streaming diverges from the paper’s motivating real‑world scenarios. Though the introduction draws heavy motivation from speech and video multimodal streaming, all experiments are built on sentence‑segmented text simulation. There are no multimodal evaluations, so we cannot confirm whether HRPO’s performance gains transfer beyond this synthetic text setting, weakening evidence for real‑world applicability.

  2. The baseline suite omits key state‑of‑the‑art RLVR methods. Comparisons are limited to SFT StreamingThinker and vanilla GRPO, without DAPO, Dr‑GRPO or GDPO baselines. Without controlled comparisons, it is hard to isolate performance improvements brought by hierarchical stage‑wise advantage assignment from gains originating from GDPO‑style decoupled multi‑reward normalization, creating ambiguity about the true source of empirical improvements.

  3. Critical hyper‑parameters lack cross‑dataset generalization validation. Key hyper‑parameters λ and η are only tuned on GSM‑symbolic and directly reused across all other benchmarks, without reported cross‑dataset hyper‑parameter sweeps. It remains unknown whether these hyper‑parameters generalize across diverse reasoning tasks; heavy per‑dataset re‑tuning would introduce large practical adoption overhead for AdaSR‑HRPO.

Questions for authors
  1. What artifacts or distribution mismatches are introduced by simulating streaming inputs through sentence‑level text segmentation? Could you characterize gaps between this text‑simulated setting and real‑world incremental multimodal observations?

  2. HRPO‑sentence and HRPO‑token exhibit degraded performance relative to stage‑level HRPO. Is this performance drop caused by reinforcement‑learning optimization instability or broken cross‑segment reasoning dependencies? Can you provide diagnostic metrics such as per‑segment intermediate reasoning accuracy to identify the root cause?

2 comments

W1 & Q1: Simulated text‑only streaming diverges from the paper’s motivating real‑world scenarios. Describe gaps between this text‑simulated setting and real‑world incremental multimodal observations.

Thank you for raising this point. Streaming reasoning aims to enable models to reason while information is still arriving, allowing them to think as they observe. This involves two closely related questions: how to define local reasoning boundaries and how to optimize the reasoning process under partial observations. Our paper focuses on the latter, using reinforcement learning to optimize streaming reasoning. To simplify the former, we segment the input into complete sentences as the units of incremental observation.

The motivation for this choice is that complete sentences provide semantically coherent units for reasoning. In real-world speech scenarios, incremental ASR could convert incoming speech into text, which could then be segmented according to the completeness of the recognized semantic units. Similarly, for video input, descriptions generated from observations available so far could provide local textual context for streaming reasoning. These offer possible interfaces between our formulation and real-world incremental inputs.

We acknowledge that end-to-end multimodal processing and finer-grained, dynamically determined reasoning boundaries would more closely reflect real-world streaming scenarios. Our text-simulated setting simplifies these aspects and does not capture challenges such as perception errors, uncertain semantic boundaries, information loss through textual representations, and additional processing latency. As an initial step, our work studies RL-based reasoning optimization under simplified input boundaries. Addressing more complex boundary identification and end-to-end multimodal observations is beyond the scope of the present paper and remains a direction for future work. We will clarify this scope and the corresponding limitations in the revised manuscript.

W2: without DAPO, Dr‑GRPO or GDPO baselines.

We apologize for the lack of clarity in our baseline description. The GRPO baselines in the main table already use GDPO-style component-wise reward normalization. The original GRPO-style variant, which aggregates multiple rewards before normalization, is included in the ablation study in Appendix Table 6. Under the same hyperparameter settings, this variant exhibits a sharp decline in both accuracy and generation length, consistent with an overly dominant length-related signal.

To address the reviewer’s concern, we additionally conduct controlled comparisons incorporating the key optimization components of DAPO and Dr. GRPO. We apply these components to both GRPO and HRPO while keeping the remaining settings fixed, allowing us to isolate their effects and examine whether the benefits of stage-level advantage assignment persist under different optimization choices. The results are summarized in the table below.

We observe only modest gains from token-mean aggregation. One possible explanation is limited variation in generated reasoning lengths during training, under which sequence-level and token-level averaging assign similar relative weights to the sampled trajectories. We therefore interpret the limited gains as potentially specific to the length distribution in our setting.

SettingGSM-Symbolic P1GSM-Symbolic P2MetaMathQA
GRPO0.083 / 740.054 / 740.194 / 90
GRPO (GDPO-style,KL=0)0.867 / 3070.756 / 3840.823 / 318
HRPO (GDPO-style,KL=0)0.871 / 2920.788 / 3700.826 / 302
GRPO (GDPO-style,KL=0.1)0.858 / 3030.741 / 3760.816 / 320
HRPO (GDPO-style,KL=0.1)0.864 / 2900.776 / 3640.820 / 299
GRPO (GDPO-style + Clip-Higher,KL=0)0.872 / 3110.764 / 3930.826 / 321
HRPO (GDPO-style + Clip-Higher,KL=0)0.874 / 2940.794 / 3750.832 / 309
GRPO (GDPO-style + Clip-Higher + Token-mean,KL=0)0.875 / 3040.771 / 3810.825 / 314
HRPO (GDPO-style + Clip-Higher + Token-mean,KL=0)0.873 / 2890.795 / 3670.839 / 307
GRPO (GDPO-style + Fixed-norm,KL=0)0.865 / 2940.760 / 3680.820 / 305
HRPO (GDPO-style + Fixed-norm,KL=0)0.868 / 2810.790 / 3570.829 / 294

W3: Critical hyper‑parameters lack cross‑dataset generalization validation.

We fully agree that validating hyperparameter generalization across datasets is crucial. To clarify, we evaluated the key hyperparameters on GSM-Symbolic P1, GSM-Symbolic P2, and MetaMathQA, and selected the configuration with the best overall performance across these datasets. The corresponding hyperparameter ablations are reported in Table 6 of the appendix.

With the selected configuration, our method consistently outperforms both SFT and GRPO across the evaluated datasets, including OOD settings. The cross-dataset OOD results are reported in Table 4 of the main paper and are reproduced below for convenience. These results provide evidence that the selected hyperparameters remain effective beyond a single dataset. We will clarify the hyperparameter-selection procedure and the corresponding cross-dataset validation in the revised manuscript.

ModelMethodGSM-Infinite Acc. ↑GSM-Infinite Total Len. ↓LogicNLI Acc. ↑LogicNLI Total Len. ↓
Qwen3-4BSFT-based0.479273.4080.474564.248
Qwen3-4BGRPO0.509285.380.445595.263
Qwen3-4BHRPO0.546265.650.489591.985
Qwen3-8BSFT-based0.519259.7380.514536.036
Qwen3-8BGRPO0.554262.5500.490547.642
Qwen3-8BHRPO0.606233.7720.549520.947

Table 4. OOD performance comparison on GSM-Infinite (within-task distribution shift) and LogicNLI (cross-task generalization). Total Len. denotes total generated tokens.

Q2:HRPO‑sentence and HRPO‑token exhibit degraded performance relative to stage‑level HRPO. Identify the root cause of the performance drop.

Thank you for this question. We first clarify the differences among the three settings. HRPO assigns separate local advantages to the streaming and deep-reasoning stages. HRPO-sentence assigns streaming advantages at the sentence level. HRPO-token further localizes format-related credit to the corresponding boundary tokens. The remaining reward components follow the same assignment scheme as HRPO.

Our reward design aims to support adaptive token allocation across reasoning units while discouraging unnecessary computation.

Stage-level HRPO regularizes total streaming length, allowing a fixed budget to be redistributed across reasoning units (Eq. 6). Together with trajectory-level correctness, this permits more reasoning on demanding units and shorter thoughts or skipping elsewhere. The efficiency reward discounts streaming computation to reflect its overlap with input arrival (Eq. 7).

Sentence-level and token-level assignment pressures each thought to be shorter, potentially penalizing useful longer reasoning. We hypothesize that this restricts adaptive allocation and contributes to the accuracy drop. HRPO-token inherits this potential limitation because its non-format rewards retain sentence-level assignment.

To further examine this explanation, we quantify the distribution of token counts across individual streaming reasoning units under all three settings, as summarized below.

MethodMean streaming tokens / rolloutSkipped units (%)Normalized Gini, all unitsNormalized Gini, non-skipped units
HRPO (stage-level)11617%0.480.35
HRPO-sentence11212%0.380.28
HRPO-token11313%0.350.26

The normalized Gini coefficient measures how evenly streaming reasoning tokens are allocated across reasoning units within a rollout. Let KK denote the number of reasoning units and LiL_i the number of streaming reasoning tokens in unit ii. A skipped unit has Li=0L_i = 0. The normalized Gini coefficient is defined as

G=∑1≤i<j≤K∣Li−Lj∣(K−1)∑i=1KLi.G = \frac{\displaystyle\sum_{1 \le i < j \le K} |L_i - L_j|}{\displaystyle(K-1)\sum_{i=1}^{K} L_i}.

The coefficient ranges from 0 to 1. A value of 0 means that all units receive the same number of tokens. A value of 1 means that all tokens are allocated to a single unit. Higher values indicate a more concentrated allocation.

Reviewer_YVFnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper studies an algorithm that thinks as the model is reading new tokens, followed by a final deep thinking. The main contribution is a hierarchical analog of the GRPO algorithms, which authors tout as capable to do credit assignment per different segments of thinking, unlike GRPO.

The tokens are divided to three classes, streaming, deep local and trajectory global. For each section, the algorithm builds separate reward components, thus doing different token advantages for tokens within the same trajectory. The paper demonstrates improved results over vanilla GRPO

Critical strengths

motivation: it is super clear that a streaming approach is useful in setups where tokens arrive sequentially and we would like to amortize cost of reasoning. The only caveat is that tokens can sometimes be removed after they arrive (such as when a user is typing), but for many examples (such as say video of a live even such as autonomous driving) this is a fair assumption and does not constitute a major limitation.

The VLLM integration is non-trivial and useful.

Consistent gains over SFT, and modest but real gains over GRPO.

Critical weaknesses
  • The hierarchical framing is definitely overstating the contributions. The paper does a great and correct job motivating credit assignment. But then, looking at equation 8, the local advantages are basically built from format and length rewards alone. The only signal about whether these local reasoning tokens were useful is due to the final accuracy reward which as far as I can see applies identically to all tokens. What HRPO actually does is merely a per segment shaping of format and length, which is narrower than the hierarchy claim, which has a connotation that the hierarchy is designed to enable more accurate reasoning.

To add to this, the local signal is using small value of lambda (0.05), and is also further divided by the length of the segment, which means that its effect is going to be marginal. Table 6 actually shows that by increasing lambda, the approach will actually perform poorly, which to me means that what's touted as a hierarchical approach is better viewed as some kind of a regularizer.

Also accuracy gains relative to GRPO are less than 0.9 range, which seems to be quite small (for example smaller than reported standard deviations)

As for the notion that the model thinks adaptively and decides when to think or skip, we need to see more analysis and ablations on how frequently does the model decide to think or skip, as well as its impact on final accuracy.

Finally, the tasks are mostly math word problems artificially split into sentences, and the evaluated setting is text only. The introduction motivates audio, video, agents, and sensors, but these setups are not actually tested. In other words, the setup seems to be too contrived.

Questions for authors
  • What does the local streaming advantage encode beyond format validity and length?

  • Is the AdaSR-GRPO baseline GRPO-style (sum rewards then normalize once) or GDPO-style (normalize per component, then add)? Please add a GRPO baseline using GDPO-style normalization and per-phase length rewards, collapsed into a single sequence-level advantage, so the effect of hierarchical token-range assignment can be separated from the effect of reward decoupling. In other words, what source of gain do we have here beyond GDPO style advantage computation? is it just that we apply these advantages to token sets, rather than entire trajectory like GDPO does? can we have that GDPO baseline?

  • Can you provide more information on skipping behavior?

5 comments

I did not get any response on my review so I assume that the submission is being abandoned. I did want to encourage authors to pursue this direction despite my concerns. hierarchical credit assignment is indeed super important.

W1 & Q1:What does the local streaming advantage encode beyond format validity and length?

Thank you for the question. We clarify the role of the local streaming advantage from two perspectives: reward content and credit assignment.

Reward content. We agree that the local streaming advantage captures format validity and token efficiency, without an additional semantic-quality score. Since task success depends on cooperation between the streaming and deep reasoning phases, the global accuracy reward supervises their joint outcome and indirectly guides the usefulness of streaming reasoning.

Credit assignment. Our GRPO baseline assigns the same trajectory-level advantage to both phases. Positive global feedback can therefore reinforce redundant streaming thoughts, while negative feedback arising from deep-phase errors or inefficiency can also affect useful ones. HRPO retains global correctness and efficiency objectives while assigning local format and length feedback to the corresponding phase. A successful but verbose trajectory can thus receive both positive task-level feedback and local pressure to reduce computation.

The hierarchy concerns the scope and joint optimization of local and global feedback, supporting the accuracy–efficiency trade-off without claiming direct semantic credit assignment for intermediate reasoning.

W2. The local signal is using small value of lambda (0.05), and is also further divided by the length of the segment. How much does the local signal contribute in HRPO?

We use a weak adaptive length reward to control the reasoning cost of accuracy gains. Two coefficients are relevant here. In Eq. (8), η\eta controls the weight of the length reward. In Eq. (5), λ\lambda controls the weight of the phase-local objective. Both default to 0.05.

Table 6 illustrates the role of η\eta. On Qwen3-1.7B, with λ\lambda fixed at 0.05, removing the adaptive length reward raises mean accuracy across three datasets from 82.83% to 83.37% (+0.53 percentage points, using unrounded values), while mean generation length rises from 321.62 to 369.35 tokens (+14.84%).

DatasetAccuracy, defaultAccuracy, reward removedChange (pp)Tokens, defaultTokens, reward removedChange in tokens (%)
GSM-symbolic P187.10%88.40%+1.30292.00331.89+13.66%
GSM-symbolic P278.80%80.00%+1.20370.26433.08+16.97%
MetaMathQA82.60%81.70%−0.90302.60343.08+13.38%
Mean82.83%83.37%+0.53321.62369.35+14.84%

Generations become longer on all three datasets, but accuracy improves on only two. This trade-off motivates a small η\eta, which leaves room for additional reasoning while placing a modest cost on extra tokens.

Regarding the reviewer’s concern about λ\lambda that further divided by the length of the segment, Eq. (5) first averages the token-level clipped surrogate within each phase and then applies λ\lambda to the sum of the two phase means. For a phase of length LL, the factor 1/L1/L averages LL token contributions, just as 1/∣oi∣1/|o_i| averages the global surrogate over the full response. Thus, the phase mean enters the objective with weight λ\lambda. The normalization alone does not imply that the phase’s contribution to the update decreases in proportion to its length.

W3: Accuracy gains relative to GRPO are less than 0.9 range, which seems to be quite small (for example smaller than reported standard deviations)

We agree that some individual accuracy gains are modest relative to the reported variation across seeds and should be interpreted cautiously. Across all six multi-seed settings in Table 7, however, HRPO achieves higher mean accuracy than GRPO while generating fewer tokens. We observe the same pattern in all 12 model–benchmark comparisons in Table 1, with accuracy gains exceeding 0.9 percentage points in nine of them.

The advantage is more pronounced at 8B, particularly on out-of-domain tasks. From 4B to 8B, HRPO’s accuracy gain over the same-size GRPO baseline increases on each of the four in-domain benchmarks. The average gain rises from 1.10 to 2.28 percentage points, while the reduction in mean generation length grows from 3.32% to 8.45%. On GSM-Infinite and LogicNLI (Table 4, OOD task), the gains are already 3.7 and 4.4 points at 4B and reach 5.2 and 5.9 points at 8B, respectively. HRPO also generates fewer tokens on both OOD tasks at both scales.

W3: Accuracy gains relative to GRPO are less than 0.9 range, which seems to be quite small (for example smaller than reported standard deviations)

We agree that some individual accuracy gains are modest relative to the reported variation across seeds and should be interpreted cautiously. Across all six multi-seed settings in Table 7, however, HRPO achieves higher mean accuracy than GRPO while generating fewer tokens. We observe the same pattern in all 12 model–benchmark comparisons in Table 1, with accuracy gains exceeding 0.9 percentage points in nine of them.

The advantage is more pronounced at 8B, particularly on out-of-domain tasks. From 4B to 8B, HRPO’s accuracy gain over the same-size GRPO baseline increases on each of the four in-domain benchmarks. The average gain rises from 1.10 to 2.28 percentage points, while the reduction in mean generation length grows from 3.32% to 8.45%. On GSM-Infinite and LogicNLI (Table 4, OOD task), the gains are already 3.7 and 4.4 points at 4B and reach 5.2 and 5.9 points at 8B, respectively. HRPO also generates fewer tokens on both OOD tasks at both scales.

W4 & Q2: Provide more information on skipping behavior?

We first compared the skip rates across different methods during streaming stage.

DatasetMetricSFTGRPOHRPO
GSM-symbolic P1Skip rate5.78%6.26%7.21%
Acc94.6%95.8%97.0%
GSM-symbolic P2Skip rate8.35%8.31%9.63%
Acc85.6%89.9%92.3%
MetaMathQASkip rate12.35%13.94%21.27%
Acc88.5%92.5%95.1%
PubMedQASkip rate2.22%2.46%4.73%
Acc78.5%81.7%84.6%
avgSkip rate7.18%7.74%10.71%
Acc86.8%89.9%92.2%
Reviewer_JibHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaSR extends supervised streaming reasoning with reinforcement learning. Its HRPO objective combines phase-local and trajectory-global clipped objectives, assigning format/length advantages to streaming and deep-reasoning tokens and accuracy/efficiency advantages across the rollout. The implementation preserves streaming visibility in vLLM rollout and actor likelihood computation. Main results on Qwen3 models show higher accuracy and fewer generated tokens than the paper’s GRPO baseline, with additional OOD tests and partial multiseed validation. The main unresolved issues are whether improvements are isolated to hierarchical assignment, how contradictory ablation entries should be interpreted, and how the method compares at matched accuracy and exposed latency.

Critical strengths

S1. Well-motivated setting Separating computation before and after input completion matches a real latency asymmetry and provides a sensible setting for adaptive computation.

S2. Implementation addresses an important correctness requirement Appendix E explains how rollout and policy evaluation use consistent partial-input masks and position semantics. This matters because inconsistent visibility would invalidate the policy ratios.

S3. Multiple scales and useful supplementary checks The study includes three model sizes, GRPO and SFT comparisons, phase-granularity/reward ablations, two OOD tasks, and multiseed results for 1.7B and 4B. The reported token-efficiency improvements are encouraging.

Critical weaknesses

W1. Credit-assignment attribution is not isolated HRPO changes component-wise normalization, phase-dependent loss weighting, and separately clipped local/global objectives together. The manuscript does not clearly specify a flat baseline that keeps the same normalized reward components and comparable loss scales while changing only assignment. Moreover, the local rewards encode format and length rather than semantic usefulness, and every token within a phase still shares one local signal. Appendix B derives the gradient algebra but does not demonstrate that erroneous early reasoning is correctly credited. The stronger claim of resolving temporal credit assignment is therefore not yet established.

W2. Reported settings and results are inconsistent Table 6 has two GRPO rows with visible coefficients Acc=1, Format=2, α=0.5, η=0.05, but P2 accuracy is 0.054 in one and 0.758 in the other. Table 3’s full-reward P1 GRPO accuracy is 0.867, versus 0.850 in Tables 1–2 despite the same displayed lengths. Table 7 usefully supplies multiseed results, but several HRPO–GRPO mean gains are small relative to the reported standard deviations, and the seed count is not specified there. These issues make the strength and reproducibility of the comparison uncertain.

W3. The latency advantage needs a matched operating-point analysis At 150 words/minute, Table 5 reports 1.505 s for HRPO versus 1.403 s for SFT and 1.595 s for GRPO under vLLM. Thus the large read-then-think speedup is largely a shared streaming benefit; HRPO is not faster than SFT in that table. The conclusion appropriately calls these delays comparable, but a superior accuracy–latency frontier needs matched accuracy/budget comparisons and sensitivity to input rate or backlog. The fixed α token proxy alone does not establish that frontier.

Questions for authors

Q1. Can the authors define and evaluate a flat-assignment baseline using the same component-wise normalization, reward information, and calibrated loss weights, and isolate phase assignment from normalization and clipping choices?

Q2. Which configurations or seeds explain the conflicting entries in Tables 1–3 and 6? Please provide corrected tables, exact seed counts, data split sizes, and the location at which each reward coefficient is applied.

Q3. How does the accuracy–post-input-latency frontier compare with StreamingThinker and GRPO when reasoning budgets are varied, and does the ordering persist at faster input rates?

3 comments

W2 & Q2: Reported settings and results are inconsistent Table 6 has two GRPO rows with visible coefficients, but P2 accuracy is 0.054 in one and 0.758 in the other. Clarify reward weighting, seed counts, data splits, and the robustness of the reported gains.

We apologize for the unclear presentation of the ablation settings and the reporting errors. We clarify the discrepancies below.

1. Different reward-normalization procedures in Table 6. The two GRPO rows share the displayed coefficients but differ in the order of reward aggregation and normalization. The row with P2 accuracy of 0.054 uses weighted reward aggregation followed by normalization, corresponding to the original GRPO-style treatment of multiple rewards. Under these settings, optimization becomes overly influenced by the length-related signal, resulting in excessively short generations and a substantial accuracy drop. The row with P2 accuracy of 0.758 instead normalizes each reward component separately before weighted aggregation, following GDPO-style normalization. Thus, identical numerical coefficients do not imply equivalent optimization signals. We will explicitly distinguish these settings in the table.

2. Correction to Tables 1–2. The P1 GRPO accuracy of 0.850 was reported in error. We have corrected it to 0.867, consistent with the corresponding full-reward result in Table 3. We thank the reviewer for identifying this discrepancy.

3. Clarification of the evaluation protocol in Table 7. We used a fixed seed of 42 and performed three inference runs for each experimental setting. These are repeated-inference evaluations, rather than experiments with independent training seeds. We have clarified the protocol and will correct the “multiseed” terminology accordingly. We acknowledge that these repetitions do not establish robustness across training seeds, particularly for the smaller reported gains.

We have updated the manuscript PDF with the corrected results and clarified experimental descriptions.

W3 & Q3: The latency advantage needs a matched operating-point analysis At 150 words/minute, Table 5 reports 1.505 s for HRPO versus 1.403 s for SFT and 1.595 s for GRPO under vLLM. Thus the large read-then-think speedup is largely a shared streaming benefit; HRPO is not faster than SFT in that table. The conclusion appropriately calls these delays comparable, but a superior accuracy–latency frontier needs matched accuracy/budget comparisons and sensitivity to input rate or backlog. The fixed α token proxy alone does not establish that frontier.

We agree with the reviewer’s assessment of the source of the speedup. As illustrated in Appendix F, Figure 4, the substantial acceleration over read-then-think primarily comes from overlapping CoT decoding with input arrival. This is a shared benefit of streaming execution. Table 5 does not show HRPO being faster than SFT: at 150 words/minute, their post-input latencies under vLLM are 1.505 s and 1.403 s, respectively, compared with 1.595 s for GRPO.

Our goal in applying RL is to improve reasoning accuracy while largely preserving the responsiveness of streaming execution. Specifically, we investigate whether reasoning under progressively available inputs can approach or exceed the reported read-then-think baseline. In Table 1, our GRPO- and HRPO-based streaming methods outperform streaming SFT in all 12 model–dataset settings and exceed the reported offline baseline in 11 settings. These findings support the value of RL for reasoning under partial observations.

The small length-reward weight controls the additional computation used to obtain these accuracy gains. Final-answer correctness alone cannot distinguish between two correct trajectories with substantially different reasoning costs. We therefore introduce length and efficiency signals with a small weight, η=0.05\eta=0.05, in Eq. 8. Table 6 illustrates this trade-off on Qwen3-1.7B with λ=0.05\lambda=0.05:

SettingMean accuracyMean generation length
Default adaptive length reward, η=0.05\eta=0.0582.83%321.62 tokens
Without adaptive length reward83.37%369.35 tokens
Change after removal+0.53 percentage points*+47.73 tokens (+14.84%)

Removing the adaptive length reward increases generation length on all three datasets, while accuracy improves on only two. The small value of η\eta therefore preserves room for useful reasoning while moderately constraining its computational cost.

The different token costs reflect the stages’ different effects on user waiting time. Streaming computation can overlap with incoming input, whereas deep reasoning occurs after the input ends and directly delays the answer. Accordingly, Eq. 7 uses αLs+Ld\alpha L_s+L_d to assign streaming tokens a lower but nonzero cost. This fixes the relative cost of the two stages, while the policy determines the actual token allocation.

We agree that this proxy alone does not establish a superior accuracy–latency frontier, particularly when faster input creates a backlog. Our fixed-α\alpha, fixed-rate evaluation provides an initial controlled study of RL for streaming reasoning. Establishing the frontier requires varying reasoning budgets for all methods and measuring accuracy and post-input latency at multiple input rates. We will clarify that our current results support improved accuracy with comparable streaming latency under the reported protocol.

W1 & Q1: Can HRPO’s gains be attributed to phase-specific credit assignment after controlling for reward normalization, loss scaling, and clipping?

We apologize for the unclear presentation of the ablation settings. Both HRPO and GRPO in the main experiments use GDPO-style component-wise normalization followed by weighted aggregation, with the same reward-component coefficients, as described in the Methods section. Thus, component-wise normalization and reward weighting are shared between the two methods.

The distinction lies in their credit-assignment objectives. GRPO applies a global advantage across the trajectory, whereas HRPO assigns local signals to the corresponding reasoning phases while retaining a trajectory-level global signal. The separately clipped local and global objectives are part of this phase-structured formulation. Our comparison therefore evaluates HRPO’s phase-structured objective, including its assignment and clipping design. We will clarify these shared settings and methodological differences in the revised manuscript.

We additionally compare Clip-Higher, token-mean loss aggregation, and KL regularization under matched GRPO and HRPO settings to examine the effects of these optimization choices. Please see our response to Reviewer Cwv2, Part 1, for the detailed configurations and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.