- The PatternRM versus length-penalty comparison is hard to interpret as a clean comparison of reward types.
Table 4 is not a simple win for PatternRM. The length-penalty baseline gets lower aggregate Trigger, while PatternRM looks better on downstream performance and accuracy retention. The authors' explanation, that length penalty mainly works by compressing responses, is plausible and supported by the decomposition. But the two interventions are not matched. Appendix G reports much larger average reward adjustments for length penalty than for PatternRM, and the methods also differ in topic coverage and invocation frequency. This means the experiment changes the feedback signal, the strength of the penalty, and the scope of application all at once. I would be more convinced by a comparison where the average reward adjustment or the response-length reduction is matched.
- The reward design feels under-justified.
PatternRM uses fixed weights: 0.05 for CoT leakage and repetition, 0.02 for contradiction and performative reasoning, with a total cap of 0.1. The overall idea is reasonable: correctness remains primary, and PatternRM only nudges the model toward cleaner correct answers. Still, the particular numbers are not explained. Were they chosen by pilot experiments, label reliability, expected severity, or simply by hand? This matters because the paper later suggests that contradiction and performative reasoning may need more careful calibration. Even a small ablation over the category weights, cap, or invocation probability would make this part of the method feel much less ad hoc.
- The analysis still leaves response length, task difficulty, and behavior quality tangled together.
The paper does show that longer responses tend to have higher Trigger, and the within-prompt length penalty is a reasonable way to avoid some crude cross-question length comparisons. But the method still does not model question difficulty, required reasoning depth, or necessary answer length. The length-bin decomposition helps, but it is not causal; the appendix explicitly notes that the same bin can contain different prompts and difficulty levels across variants. This leaves an important possibility open: length penalty may look strong on aggregate because it removes verbose failure opportunities, while also hurting harder examples that genuinely need longer reasoning or more structured explanation.
- The method-level evidence is much narrower than the benchmark-level evidence.
The benchmark survey is broad: it covers several model families and a wide Qwen3.5 scale range. The RL validation, however, is only on Qwen3-VL-4B and Qwen3-VL-8B, and appears to use one run per configuration. That is a narrow basis for conclusions about PatternRM as a general training method. I am also not sure how far the binary response-level labels can go: they are useful for measuring incidence, but they do not tell us whether a failure affects the final answer, a key reasoning step, or a minor explanatory sentence.
Response to Reviewer y6RQ
Response to Q1: Extending PatternEval and PatternRM to the thinking process
Thank you for the positive assessment and for this suggestion, which we tested directly. Our RL policies are Instruct models without a separate thinking channel. We therefore used the 12 thinking models already evaluated on PatternEval: seven open models from the Qwen3.5, Qwen3.6 and Qwen3-VL families and five API models, with 2,253 to 2,387 questions each. We applied the training PatternRM unchanged, with the same prompt, to the thinking trace as if it were the response, and separately to the final answer. CoT leakage is expected inside a trace, so we analyse only repetition, contradiction and performative reasoning.
Table I. PatternRM hit rate (%), mean of 12 thinking models.
Table II. Final answers by whether the thinking trace has the pattern, pooled over 12 models (%). Every with/without difference is significant (95% bootstrap interval excludes 0), with the same sign in at least 9 of the 12 models.
Repetition and contradiction are three to five times more frequent in thinking traces than in final answers. A contradiction in the trace goes with a final accuracy of 29.6%, against 66.0% without one. Patterns also propagate: after a hit in the trace, the same pattern appears in 16 to 42% of final answers, against at most 4.4% otherwise. Smaller models are more affected: 17.3% of Qwen3.5-4B traces contain a contradiction, against about 1% for the strongest API models. So PatternRM already gives a discriminative signal on thinking traces without re-prompting.
These results are correlational, since harder questions also cause more self-correction. PatternRM was not calibrated on traces, and we have not checked its precision there by hand. RL on the thinking channel would need to separate exploratory self-correction from contradiction and to weight the two channels separately; we leave it to future work. Details are in Appendix C.3 (Table 9).