- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether native expert-routing signals in Mixture-of-Experts models can serve as step-level process supervision. The authors construct a confidence score from three statistics: the overall routing probability mass assigned to selected experts, the routing mass of the initial tokens in each reasoning step, and the variation of routing signals inside one step. The authors proposed 2 methods: RoCo-Rethink identifies the reasoning step with the lowest routing confidence and regenerates the remaining trajectory. RoCo-PRM uses a strong MoE teacher to produce confidence labels and distills them into a standalone process reward model, allowing it to guide both MoE and dense target models. These methods are evaluated using three MoE models, two dense target models, and six benchmark.
Critical strengths
- The proposed method proposed a novel process supervision method, which leverages the architectural internal signals into reasoning supervision.
- The writting is clear and easy to follow.
- The experiments are thorough.
Critical weaknesses
- The authors do not discuss the potential computational and storage costs introduced by the proposed method.
- The head-routing feature uses only the first three tokens of a step, which might be only connection words and contain limited information.
- The proposed method may be sensitive to step segmentation. Including more ablation of reasoning traces might be helpful.
- The reported Spearman correlations between routing statistics and external verifier scores are generally low, with many results around 0.05–0.25.
Questions for authors
Please refer to the Weaknesses.
3 comments
Response to Reviewer PYoM — Part 2: Segmentation sensitivity and routing informativeness (W3–W4)
W3. How sensitive is RoCo to step segmentation?
Sentence-level refinement changed few selected revision starts and retained the same aggregate revision success.
-
A refinement that changes step boundaries.
On 100 initially incorrect OLMoE–GSM8K trajectories, we preserved the original newline boundaries and split steps further at sentence boundaries.
- 56 of 100 trajectories gained additional boundaries.
- The refinement added 79 boundaries.
- The total step count increased from 706 to 785.
-
Comparing actual revision starts.
We compared selected starts by their absolute token offsets in the original generation. This identifies where regeneration actually begins even when step indices change under the new segmentation.
Among the 56 trajectories whose segmentation changed, the selected start moved in 3 cases (5.4%).
-
Revision success.
With eight revisions evaluated per selected start, both segmentations yielded 47 successful revisions out of 800 (5.88%).
The selected revision start was therefore retained in most affected trajectories, and aggregate revision success was unchanged under this sentence-level refinement.
W4. How informative are routing statistics given their modest correlations with external verifiers?
Routing features jointly add predictive information beyond step characteristics and logit entropy.
-
What the additional analysis tests.
We agree that Table 3 shows modest rank correlations between individual routing statistics and verifier scores. We tested whether the three features jointly contribute information after controlling for step token length, relative position, formatting indicators, and logit entropy.
We repeated grouped cross-validation across the same 18 model–task–verifier settings as Table 4. The control and augmented predictors used identical samples and folds.
-
Controlled prediction results.
Candidate steps Control AUROC Control + routing AUROC Settings improved All steps 0.6079 0.6317 14/18 Non-answer steps 0.5939 0.6185 15/18 For the non-answer analysis, we excluded explicit answer steps while retaining the verifier labels computed on the original trajectories.
Adding routing features improved macro-average AUROC by 0.0238 with all steps and 0.0246 after excluding answer steps. This supports their incremental predictive value given the specified controls.
-
Direct revision-point evaluation.
We also evaluated 100 initially incorrect OLMoE–GSM8K trajectories, restricting all selectors to the same non-answer candidates and using eight cached revisions per candidate.
Selector Estimated revision success Longest step 6.75% Last non-answer step 2.50% Uniform random non-answer step 5.00% RoCo 9.13% RoCo’s advantage persisted against length-based and fixed-position selection rules. This complements the verifier-based analysis by testing how often revision from the selected step corrects the final answer.
Gentle Reminder to Revisit Our Rebuttal and Evaluation
Dear Reviewer PYoM,
Thank you again for your positive assessment and support of our work. We sincerely appreciate your recognition of the method’s novelty, clear presentation, and thorough experiments.
Your questions helped us strengthen the evaluation with measurements of computational overhead, ablations of the head-routing feature, step-segmentation analysis, and additional controls for routing informativeness.
We would be grateful if you could revisit our responses and consider whether the additional evidence warrants an updated assessment. We hope these results address your concerns and would welcome any remaining questions.
Thank you very much for your time and continued support.
Best regards,
The Authors
Response to Reviewer PYoM — Part 1: Computational overhead and head-routing features (W1–W2)
We appreciate your questions about the practical cost of routing-based revision and the information contained in its head-token feature. We measured scoring overhead separately from generation and tested alternative head-token windows.
W1. Computational and storage overhead
Routing replay and step scoring account for about 2.1% of RoCo-Rethink’s measured incremental revision latency.
Latency measurements.
We evaluated all 1,319 GSM8K test problems with OLMoE. All methods started from shared initial trajectories, and the measured latency includes scoring and revision.
RoCo’s routing replay and scoring took 0.210 s per problem, or approximately 2.1% of its total revision latency. Relative to Random Rethink, the total latency difference was −0.16 s per problem (95% CI [−0.83, 0.53]).
These measurements include the scoring cost introduced by RoCo. Total latency also depends on the suffix regenerated and the number of rounds executed.
Storage requirements.
During replay, we aggregate router statistics into step-level scores and retain those scores for revision-point selection. The procedure requires no persistent storage of full router probabilities.
W2. Do the first three tokens provide a useful routing signal?
The three-token head feature improved revision success in a controlled ablation.
Ablation protocol.
We evaluated 100 initially incorrect OLMoE trajectories from GSM8K. All variants used the same candidate steps and eight cached revisions per candidate. We varied the number of head tokens and tested skipping explicit step numbers and bullet markers.
Results.
Adding the original three-token feature increased success by 1.375 percentage points (paired 95% CI [0.50, 2.50]). The three-token window had the highest observed success rate among the tested lengths.
Interpretation.
The head feature is computed from routing behavior at the beginning of a step. Its usefulness can therefore be tested even when the initial words contain little semantic content. Removing the feature reduced revision success, while skipping explicit numbering and bullet markers retained the same aggregate success rate.
These results support the empirical usefulness of the three-token routing feature. The prefix-skipping experiment specifically tests explicit formatting markers.