- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SemCredit, an offline method for assigning finer-grained credit from final-answer rewards. It splits multiple rollouts for the same problem into paragraphs, clusters similar segments into semantic states, and organizes their state sequences into a prefix trie. Differences in downstream success rates between a node and its parent are used to reweight trajectory-level advantages. Policy updates focus on high-entropy tokens within an early decision horizon, while other positions receive KL regularization. Experiments cover four Qwen models on six mathematical benchmarks and a separately trained model on three medical QA benchmarks. SemCredit achieves the highest average mathematical performance among the controlled offline baselines.
Critical strengths
1.The method provides a concrete way to reuse outcome-labeled rollouts. Constructing local training signals from completed trajectories, without step-level labels or additional continuation sampling, is a useful direction. The connection between semantic segments and sparse token updates is clearly specified.
2.The controlled results are encouraging across model scales. Table 1 reports improvements in the six-benchmark average for all four models under a shared offline LoRA setup. Table 3 also shows that adding the semantic-trie weighting improves over the same early-token selection across all four models. The separately trained medical experiments provide additional evidence that the approach can be applied beyond mathematical tasks.
Critical weaknesses
- The methodological advance over closely related work is not sufficiently established. High-entropy sparse updates and deriving local credit from multiple trajectories are already well explored. TEMPO estimates prefix values from descendant outcomes; SALT derives step advantages through a trajectory graph; and T-STAR merges functionally similar steps into a tree and propagates outcome rewards. The paper cites these methods, which I appreciate. However, its controlled experiments mainly compare against offline GRPO, DAPO with Forking Tokens, and ReasonMaxxer. They do not establish an advantage over closely related structured-credit methods. SemCredit’s distinguishing choices appear to be the semantic prefix representation, the sign-preserving multiplier, and the early decision horizon. A comparison isolating semantic versus exact-prefix grouping under the same offline data and objective would be particularly informative. Without stronger evidence for these choices, the contribution currently appears incremental.
- The ablations do not yet establish why the weighting helps or how robust the results are. Equations (4)–(5) use nonnegative weights, so each segment retains the sign of its trajectory-level advantage. The method changes update strength: a correct segment in a failed trajectory still receives negative policy updates when its tokens are selected. The weights also combine local outcome statistics with node-frequency shrinkage. Table 3 shows that this combined weighting helps, but does not separate useful semantic credit from frequency-based weighting or changes in effective gradient scale. Frequency-only weighting or shuffled local weights with matched scale would help clarify the mechanism. The semantic grouping also needs more validation. Embedding similarity may merge mathematically different steps or separate equivalent ones, while local success estimates can be noisy for nodes with few observations. Finally, training uses at most 50 mathematical problems per model, but variability across training runs is not reported. Eight evaluation samples per problem do not establish stability across training seeds or selected problem sets. These uncertainties matter for both the proposed explanation and the smaller reported gains.
Questions for authors
1.Closest prior work: What is the most important methodological distinction from TEMPO, SALT, and T-STAR? Do you have an existing comparison that isolates this distinction under the controlled offline setting?
2.Source of improvement: Do you have ablations separating the local outcome multiplier from node-frequency shrinkage and overall weight scale? What evidence attributes the improvement specifically to semantic outcome information?