- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TSDPO, a method that tackles what the authors call “temporal credit mismatch” in streaming reasoning for LLMs. The basic issue is that GRPO gives every token in a trajectory the same advantage score, but in a streaming setup the model is generating reasoning segments under different partial views of the input, so treating them all the same doesn’t make sense.
TSDPO keeps the same trajectory-level advantage from GRPO but redistributes it across segments based on an answer-support score. The idea is to score each segment twice with the frozen old policy: once under normal streaming visibility, once with the verified answer appended; then use the likelihood gap as a proxy for how much that segment “supports” the answer.
They evaluate on Qwen3-1.7B and Qwen3-4B across GSM-Symbolic P1/P2, MetaMathQA, and PubMedQA. TSDPO beats GRPO on accuracy in all 8 settings (gains of 0.3–3.0%) and also reduces total generation length. Ablations show segment-level credit works better than both token-level (which collapses) and stage-level alternatives.
Critical strengths
The temporal credit mismatch is a real problem and the paper does a good job making the case for why it matters. The solution is appealing because it doesn’t need external teachers, process annotations, or extra reward models, it just uses the structure that’s already there (the verified answer and the frozen policy). The fact that it reduces to GRPO when λ=0 means you’re not throwing away anything, just adding a knob.
Beating GRPO on both accuracy and length in all 8 settings is a clean result, even if individual gains are modest. But what I found more valuable than the main results was the granularity ablation showing that token-level collapses, stage-level helps but not enough, and segment-level hits the sweet spot.
Critical weaknesses
Everything runs on Qwen3 at two sizes. We have no idea if this works on Llama, Mistral, or anything else. The benchmarks lean heavily on math. And the most obvious baseline to compare against (Self-distilled RLVR) is missing entirely, with the justification that numerical results weren’t available. That's the most relevant prior work and it’s not in the paper. The one OOD test (GSM-Infinite) shows an accuracy drop, which makes me question the generality of the approach.
The whole method depends on the likelihood gap being a good proxy for segment importance, but there’s no experiment that actually tests this. You could run a simple intervention, like drop or scramble the highest-scored segments and see if accuracy tanks, and that would go a long way. Without it, we’re taking on faith that the proxy is measuring what we think it’s measuring. The token-level collapse also suggests there’s something fragile about the scoring at finer granularities.
Questions for authors
-
Can you validate the answer-support scores causally? For instance, if you ablate the top-weighted segments from successful trajectories, does accuracy drop proportionally? If the scores are actually tracking causal importance, the paper becomes much stronger.
-
How sensitive is performance to λ, τ, and c? You report fixed values but no sensitivity analysis. If TSDPO only works in a narrow hyperparameter regime, that’s important to know.
-
What’s the actual training cost overhead vs. GRPO? The extra answer-conditioned forward passes aren’t free. If TSDPO takes twice the training compute for 1% of accuracy, that has practical implications.
Following reviewer rDyz's observation and the lack of response from the authors, I am lowering my score.