acceptodds
Desk rejected at ICLR 2027

Real-Time Streaming Reasoning with Temporal Self-Distillation Policy Optimization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Real-time streaming reasoning enables language models to reason incrementally as inputs arrive under progressively revealed observations. While this paradigm reduces response latency, effectively optimizing the intermediate reasoning process remains challenging, as high-quality segment-level supervision is difficult to obtain. Outcome-based reinforcement learning provides a natural alternative because final task success can often be verified directly. However, a streaming trajectory consists of multiple reasoning segments generated under different partial observations, whereas the outcome reward only evaluates the trajectory as a whole. To bridge this gap, we propose Temporal Self-Distilled Policy Optimization (TSDPO), which combines outcome-based policy optimization with on-policy self-distillation. TSDPO leverages training-only privileged information to retrospectively evaluate sampled reasoning segments under the same policy, producing segment-level support signals that are calibrated across streaming positions and used to redistribute trajectory-level advantages over time. This provides fine-grained supervision for streaming reasoning without requiring external teachers or annotated reasoning trajectories. Experiments on Qwen3-1.7B and Qwen3-4B show that TSDPO improves answer accuracy while producing more compact reasoning trajectories, and preserves the low-latency characteristics of real-time streaming inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Streaming Reasoning, RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WUreOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TSDPO, a method that tackles what the authors call “temporal credit mismatch” in streaming reasoning for LLMs. The basic issue is that GRPO gives every token in a trajectory the same advantage score, but in a streaming setup the model is generating reasoning segments under different partial views of the input, so treating them all the same doesn’t make sense.

TSDPO keeps the same trajectory-level advantage from GRPO but redistributes it across segments based on an answer-support score. The idea is to score each segment twice with the frozen old policy: once under normal streaming visibility, once with the verified answer appended; then use the likelihood gap as a proxy for how much that segment “supports” the answer.

They evaluate on Qwen3-1.7B and Qwen3-4B across GSM-Symbolic P1/P2, MetaMathQA, and PubMedQA. TSDPO beats GRPO on accuracy in all 8 settings (gains of 0.3–3.0%) and also reduces total generation length. Ablations show segment-level credit works better than both token-level (which collapses) and stage-level alternatives.

Critical strengths

The temporal credit mismatch is a real problem and the paper does a good job making the case for why it matters. The solution is appealing because it doesn’t need external teachers, process annotations, or extra reward models, it just uses the structure that’s already there (the verified answer and the frozen policy). The fact that it reduces to GRPO when λ=0 means you’re not throwing away anything, just adding a knob.

Beating GRPO on both accuracy and length in all 8 settings is a clean result, even if individual gains are modest. But what I found more valuable than the main results was the granularity ablation showing that token-level collapses, stage-level helps but not enough, and segment-level hits the sweet spot.

Critical weaknesses

Everything runs on Qwen3 at two sizes. We have no idea if this works on Llama, Mistral, or anything else. The benchmarks lean heavily on math. And the most obvious baseline to compare against (Self-distilled RLVR) is missing entirely, with the justification that numerical results weren’t available. That's the most relevant prior work and it’s not in the paper. The one OOD test (GSM-Infinite) shows an accuracy drop, which makes me question the generality of the approach.

The whole method depends on the likelihood gap being a good proxy for segment importance, but there’s no experiment that actually tests this. You could run a simple intervention, like drop or scramble the highest-scored segments and see if accuracy tanks, and that would go a long way. Without it, we’re taking on faith that the proxy is measuring what we think it’s measuring. The token-level collapse also suggests there’s something fragile about the scoring at finer granularities.

Questions for authors
  1. Can you validate the answer-support scores causally? For instance, if you ablate the top-weighted segments from successful trajectories, does accuracy drop proportionally? If the scores are actually tracking causal importance, the paper becomes much stronger.

  2. How sensitive is performance to λ, τ, and c? You report fixed values but no sensitivity analysis. If TSDPO only works in a narrow hyperparameter regime, that’s important to know.

  3. What’s the actual training cost overhead vs. GRPO? The extra answer-conditioned forward passes aren’t free. If TSDPO takes twice the training compute for 1% of accuracy, that has practical implications.

1 comment

Following reviewer rDyz's observation and the lack of response from the authors, I am lowering my score.

Reviewer_F9EKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In streaming reasoning, TSDPO uses the frozen pre-update model to compare the log-likelihood of the same reasoning segment with and without the correct answer, obtaining an answer-support score that determines its training weight.

Critical strengths

1.The correct answer is used only to score existing segments, not for online generation, allowing the model to learn using only the input available at each step without an external teacher or additional step-level annotations. 2.Across both model sizes and all four main benchmarks, TSDPO achieves higher accuracy and shorter outputs than group relative policy optimization (GRPO), with the three-seed aggregate results for the 4B model showing the same trends in mean accuracy and total generation length.

Critical weaknesses

1.Equation 2 averages losses over all tokens, whereas Equation 5 first averages within each segment and then across segments, but Table 1 does not specify which averaging scheme the GRPO baseline uses, potentially conflating the effects of averaging and weighting. 2.On GSM-Infinite, TSDPO produces outputs approximately 22% shorter than GRPO but reduces accuracy from 50.60% to 49.17%, showing that shorter reasoning may come at the cost of accuracy.

Questions for authors

1.Could the authors specify the GRPO baseline's loss-averaging scheme and compare the unweighted version (λ = 0) with the full method under matched averaging and KL regularization settings? 2.What accuracies do TSDPO and GRPO achieve on GSM-Infinite when both are constrained to the same generation-token budget?

Reviewer_rDyzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates RL for streaming (“think-while-reading”) reasoning, arguing that GRPO’s uniform trajectory-level advantage may lead to temporal credit mismatch across streaming segments. To address this, it introduces TSDPO, which re-scores each sampled segment under online and answer-conditioned histories and uses the difference in mean token log-likelihoods as a proxy for answer support. The scores are calibrated position-wise and converted into weights that redistribute the GRPO advantage while preserving the mean segment-level advantage within each trajectory. The authors report higher accuracy and shorter total generation length than GRPO in all eight model–benchmark combinations covering Qwen3-1.7B/4B, GSM-Symbolic P1/P2, MetaMathQA, and PubMedQA.

Critical strengths
  1. Clear motivation with a concise redistribution mechanism. In streaming inference, different segments are generated based on distinct partial observations; thus, a uniform, trajectory-level advantage estimate may fail to capture the varying contributions of these segments. To address this, the paper proposes a redistribution mechanism (Equations 9 and 21) that preserves the average segment advantage while adjusting the credit assigned to each segment via temporal weighting. When λ=0\lambda=0, this approach reverts to a uniform distribution of segment advantages—a design that is both intuitive and clearly linked to GRPO.

  2. Prudent discussion with well-defined boundaries for conclusions. The paper explicitly acknowledges that scores serve as proxy metrics rather than verified estimates of causal contribution. It also addresses the limitations of out-of-distribution (OOD) evaluation on a single dataset and distinguishes between post-input-completion latency estimates and end-to-end latency. These clarifications help readers accurately interpret the meaning of the metrics and the scope of the conclusions supported by the experimental results.

Critical weaknesses
  1. There are serious concerns regarding the origin of the experimental results and their correspondence to the described methodology.

All 32 reported values for TSDPO in Table 1 of this paper are identical to those for AdaSR-HRPO from the paper AdaSR (arXiv:2606.14694v1). Similarly, all 32 reported values for the GRPO baseline are also identical, yet the paper fails to clarify the relationship between the two works. There are discrepancies in the algorithms, reward formulations, and training configurations described in the two papers—for instance, training batch sizes (128 vs. 16), group sizes (8 vs. 12), and KL penalty settings (0.01 vs. False). Furthermore, the GRPO baseline results in AdaSR—which match those presented here—incorporate format and length rewards, whereas Equation (4) in this paper utilizes an accuracy-based reward.

The authors must clarify whether existing experimental results or checkpoints were reused, and provide the implementation code, complete configuration details, and original experiment logs. This issue also casts doubt on the validity of the results presented in the main tables, as well as the ablation studies. Until a satisfactory explanation is provided, I cannot confirm that these results reliably validate the proposed method.

  1. There is a lack of transparency and reporting of uncertainty regarding the supplementary experiments.

In Table 6, the "privileged self-distillation" baseline underperforms GRPO by 0.011–0.012 across all four benchmarks. To ensure the reliability of these comparisons, the authors should provide training curves and results from individual experimental runs. Regarding the latency figures in Tables 5 and 10—which are estimated based on sequence length and throughput—the authors must specify the statistical definition and calculation method used for the error bars (derived from three evaluation runs) and provide the raw measurement data. Additionally, experiments using multiple random seeds were conducted only for the Qwen3-4B model; the reported improvements for Qwen3-1.7B lack a corresponding assessment of uncertainty.

Questions for authors
  1. Why do different methods and training configurations yield identical results, item for item?

Although this paper differs from AdaSR (arXiv:2606.14694v1) in terms of optimization objectives, reward mechanisms, and training hyperparameters, all 32 values reported for TSDPO in Table 1 are identical to those for AdaSR-HRPO; the same applies to the 32 values reported for the GRPO baseline. Please explain this consistency and clarify which specific algorithm, reward setup, and training configuration were actually used to produce these results. In particular, please specify: Do the GRPO and TSDPO implementations in this paper rely solely on accuracy-based rewards? Furthermore, have any experimental results, evaluation outputs, or post-training checkpoints from AdaSR been reused? If these results were obtained through independent training using the methods proposed in this paper, please provide the corresponding code version, complete configuration files, and raw execution logs to establish a verifiable link between the reported figures and your proposed approach.

  1. How are the “±” values for latency in Table 10 calculated, and what is the source of this variation?

The paper states that latency is estimated based on generation length and throughput, with evaluations conducted three times. Please clarify whether the “±” notation represents the standard deviation, standard error, or another statistical metric. Additionally, specify which factors varied across the three evaluation runs—for instance, were the outputs regenerated, and were generation length and throughput measured separately for each run? Please provide the generation length, throughput, and calculated latency for each of the three evaluations, along with the aggregation method used. This will allow for the reproduction of the mean values and error margins in Table 10, as well as the speedup ratios presented in Table 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment by Authors

We apologize for the incorrect results reported in the submitted manuscript. During AI-assisted reformatting of the experimental tables, results from AdaSR that were present in the editing context were inadvertently substituted for the original results of our proposed method. This explains the exact duplication identified by the reviewer. We failed to detect this substitution before submission and take full responsibility for the resulting reporting error and our inadequate verification of the AI-assisted edits. We will carefully review and revise the manuscript, checking the reported results against the original experimental records.

Regarding the method itself, we adopt the same datasets and evaluation protocol as AdaSR to establish a common basis for comparison. OPSD’s contribution lies in using full-context supervision during training to improve reasoning under partial, incrementally arriving inputs through self-distillation. This addresses a central challenge of streaming reasoning: learning to make useful progress before the complete input becomes available. We believe this research question remains meaningful, and we recognize that the effectiveness of our approach must be demonstrated through corrected, verifiable experimental evidence. We will revise the manuscript accordingly to clarify the methodological contribution and substantiate the empirical claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.