acceptodds
Under review as a conference paper at ICLR 2027

Beyond Correctness: Benchmarking Response Behaviors and Improving Non-Thinking Outputs in Hybrid-Thinking MLLMs

Abstract

Hybrid-thinking multimodal large language models (MLLMs) support thinking and non-thinking modes, but answer correctness alone does not capture the quality of their final responses. We introduce PatternEval, a failure-enriched diagnostic benchmark of 2,415 multimodal prompts across nine task categories. It evaluates correctness separately from four response-pattern failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Across 17 evaluated model configurations, non-thinking response-pattern failure rates exceed those in thinking mode by 2.86–48.64 percentage points, motivating direct evaluation of the response delivered through each interface. We further develop PatternRM, a reward model that supplies pattern-specific penalties to augment verifier-based group relative policy optimization (GRPO). Incorporating PatternRM into verifier-based reinforcement learning reduces non-thinking response-pattern failures while largely preserving accuracy on PatternEval. Comparison with an explicit length penalty reveals a trade-off: length regularization achieves lower aggregate failure rates, whereas GRPO with PatternRM achieves higher downstream scores while reducing failures relative to the verifier-based baseline. Together, our findings highlight the value of response-pattern metrics for evaluating reinforcement learning beyond task accuracy and demonstrate that integrating pattern-aware reward modeling with verifier-based training can improve both response quality and downstream task performance.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Hybrid Thinking, Response-Pattern Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_y6RQOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PatternEval to evaluate four types of response issues beyond accuracy—Chain-of-Thought (CoT) spillage, repetition, contradiction, and "disguised reasoning"—and retrains PatternRM to provide behavioural penalties for GRPO.

Critical strengths
  1. This paper provided PatternEval, a useful benchmark to evaluate LLMs' response-pattern failures. This is highly valuable for subsequent research. Its Meaningful evaluation goes beyond correctness.
  2. Broad diagnostic coverage. PatternEval contains 2,415 prompts across nine categories, evaluates 17 model configurations, and validates candidate judges against 2,500 human-labelled responses.
  3. Promising training results. With PatternRM, RLVR not only improve response quality but also downstream task performance.
Critical weaknesses

None

Questions for authors
  1. Can PatternEval & PatternRM extend to the thinking process? Like Response Repetition, Logical Contradiction & Performative Reasoning, these issues remain critical in the reasoning content.
1 comment
AuthorsOpenReview ↗

Response to Reviewer y6RQ

Response to Q1: Extending PatternEval and PatternRM to the thinking process

Thank you for the positive assessment and for this suggestion, which we tested directly. Our RL policies are Instruct models without a separate thinking channel. We therefore used the 12 thinking models already evaluated on PatternEval: seven open models from the Qwen3.5, Qwen3.6 and Qwen3-VL families and five API models, with 2,253 to 2,387 questions each. We applied the training PatternRM unchanged, with the same prompt, to the thinking trace as if it were the response, and separately to the final answer. CoT leakage is expected inside a trace, so we analyse only repetition, contradiction and performative reasoning.

Table I. PatternRM hit rate (%), mean of 12 thinking models.

RepetitionContradictionPerformative
Thinking trace6.56.09.0
Final answer1.22.07.8

Table II. Final answers by whether the thinking trace has the pattern, pooled over 12 models (%). Every with/without difference is significant (95% bootstrap interval excludes 0), with the same sign in at least 9 of the 12 models.

Pattern in the traceFinal accuracy: with / withoutSame pattern in the final answer: with / without
Repetition46.6 / 65.016.3 / 0.2
Contradiction29.6 / 66.020.1 / 0.9
Performative32.8 / 66.842.2 / 4.4

Repetition and contradiction are three to five times more frequent in thinking traces than in final answers. A contradiction in the trace goes with a final accuracy of 29.6%, against 66.0% without one. Patterns also propagate: after a hit in the trace, the same pattern appears in 16 to 42% of final answers, against at most 4.4% otherwise. Smaller models are more affected: 17.3% of Qwen3.5-4B traces contain a contradiction, against about 1% for the strongest API models. So PatternRM already gives a discriminative signal on thinking traces without re-prompting.

These results are correlational, since harder questions also cause more self-correction. PatternRM was not calibrated on traces, and we have not checked its precision there by hand. RL on the thinking channel would need to separate exploratory self-correction from contradiction and to weight the two channels separately; we leave it to future work. Details are in Appendix C.3 (Table 9).

Reviewer_SkA1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PatternEval, a benchmark for evaluating response behaviors in hybrid-thinking MLLMs beyond answer correctness. It covers four failure patterns: chain-of-thought leakage, repetition, contradiction, and performative reasoning. The authors evaluate 17 model configurations under thinking and non-thinking modes and further propose PatternRM, a reward model integrated into GRPO to reduce undesirable response patterns. Experiments show improvements in response quality and downstream task performance.

Critical strengths
  1. The paper addresses a meaningful problem by evaluating response quality beyond answer correctness, particularly for hybrid-thinking models. PatternEval covers multiple multimodal tasks and provides useful insights into different response failures across 17 model configurations.
  2. The experimental results are promising, showing that incorporating PatternRM into GRPO reduces response-pattern failures and improves downstream performance. The analyses of response length and length-penalty training further help explain the observed improvements.
Critical weaknesses
  1. The methodological novelty is relatively limited, as the four response-pattern categories largely reflect known generation issues. Moreover, the benchmark relies heavily on LLM-based judgments, whose reliability remains questionable for contradiction and performative reasoning.
  2. The experimental evidence raises concerns about the effectiveness and generalization of PatternRM. Potential data overlap between training and evaluation is not fully ruled out. Additionally, after controlling for response length, contradiction and performative reasoning become worse at both model scales, suggesting that the proposed reward does not consistently address all targeted failure categories.
Questions for authors
  1. Could the authors further clarify the novelty of PatternRM compared with directly using existing reward models or LLM-based feedback to penalize undesirable response behaviors?
  2. How reliable are the automatic evaluations for contradiction and performative reasoning, given their relatively low F1 scores? Could the authors provide additional human validation of the improvements after RL?
  3. Could the authors verify item-level separation between the training data and the evaluation benchmarks, or provide results on a fully independent test set?
  4. Could the authors explain why contradiction and performative reasoning become worse after length standardization, and whether different reward weights could improve these behaviors?
4 comments
AuthorsOpenReview ↗

[Part 1/4] Response to W1 & Q1

Response to W1 & Q1: Novelty compared with existing reward models and LLM feedback

Thank you for this question. We agree that each of the four failures is known on its own. Our contribution is to make them measurable and trainable for hybrid-thinking MLLMs. PatternEval measures them separately from correctness. PatternRM penalizes the named failure inside a correct response, so correctness always dominates. Two results show that this localization, not the use of LLM feedback as such, is what helps.

First, we trained two controls on Qwen3-VL-8B. They use the same judge and, in every batch, give each group exactly the same rewards as PatternRM, but move the penalty to another correct response of the same group (Appendix G.2). Shorter responses have fewer chances to fail, so we compare Trigger only on the questions where the two responses differ in length by at most 25% (length-matched Trigger).

Table I. Qwen3-VL-8B, step 450 (%). PatternEval Acc and Trigger are on all PatternEval questions; downstream Acc is the 11-task mean ("Overall" in Table 5). Each method is compared with GRPO on its own length-matched questions, so GRPO's value in that column differs by row. GRPO and PatternRM use Table 4; the swap controls are scored on the 2,387 questions with a reference answer.

The penalty goes toPatternEval AccTriggerLength-matched Trigger: GRPO → method (questions)Downstream Acc
none (GRPO)50.4446.38–61.24
the flagged response (PatternRM)50.5730.8925.69 → 20.86 (1,117)66.00
a random correct response (swap-random)49.0643.6538.95 → 37.77 (1,366)59.79
the longest correct response (swap-longest)49.7526.9918.41 → 19.23 (364)65.07

With reward statistics matched batch by batch, only penalizing the flagged response gives significantly cleaner outputs at matched length (the 95% paired-bootstrap interval of the difference excludes 0), and it gives the largest downstream gain.

Second, the same backbone used zero-shot detects these failures less well. We compare PatternRM with its own backbone on the 2,500 human-labelled responses.

Table II. Macro-F1 and macro precision (%) on 2,500 human-labelled responses (Table 3).

JudgeF1, image + textF1, textPrecision, text
Qwen3.5-27B, zero-shot, no thinking62.863.555.7
Qwen3.5-27B, zero-shot, thinking70.569.073.4
PatternRM, no thinking–71.375.9

Without thinking, the zero-shot backbone trails PatternRM by about 8 points in F1 and 20 points in precision. With thinking it comes close, but it then generates a reasoning trace in each of about 614 judge calls per training step. PatternRM also returns per-category decisions, which can be weighted separately and applied to thinking traces (Appendix C.3). Feedback that names and locates the failure trains better than a holistic score, both for text with RL [1] and for MLLMs, where segment-level corrections on 1.4k samples outperform holistic preferences on 10k [2].

Together, these two results indicate that PatternRM is a better reward than its zero-shot backbone at the same judge cost. A false positive puts the penalty on a response without that failure, which is what swap-random does on purpose, and swap-random gives no cleaner outputs at matched length and the lowest downstream accuracy in Table I. Without thinking, the zero-shot backbone's flags are false 44% of the time on average over the four classes, against 24% for PatternRM, and 63% and 70% for contradiction and performative reasoning (40% and 42% for PatternRM). A larger share of its penalties would therefore act like swap-random. This matches a known weakness of zero-shot LLM judges: they reward empty reasoning openers such as "Let's solve this problem step by step", an RL policy learns to exploit this, and fine-tuning the judge on such negatives nearly eliminates these false positives [3].

Frontier zero-shot judges may give a better reward. GPT-5.5 and Seed-2.0-Pro reach 77.2 and 73.4 macro-F1 on text (Table 1), mainly through higher recall, although their precision (71.1 and 70.6) stays below PatternRM's 75.9. They are costly as rewards, however: an 8B run makes about 614 judge calls per step, about 276,000 API calls over 450 steps. Seed-2.0-Pro is also our evaluation judge, so training on it would remove the separation between reward and evaluation. Details are in Appendix G.2.

[1] Wu et al. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. NeurIPS 2023.

[2] Yu et al. RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. CVPR 2024.

[3] Zhao et al. One Token to Fool LLM-as-a-Judge. NeurIPS 2025 MATH-AI Workshop.

AuthorsOpenReview ↗

[Part 2/4] Response to W1 & Q2

Response to W1 & Q2: Reliability of contradiction and performative labels after RL

Thank you for raising this. Pattern labels after RL come from the evaluation judge, Seed-2.0-Pro, which belongs to a different model family from the training reward model. Its F1 on the 2,500 human-labelled responses is 91.2, 87.3, 62.5 and 60.0 for CoT leakage, repetition, contradiction and performative reasoning (Table 3).

To check the improvements after RL independently of the evaluation judge, we drew 250 question pairs from 8B GRPO and PatternRM and blinded the order within each pair. Two annotators with computer-science backgrounds, who divided the pairs between them, labeled them by hand, and GPT-5.6-Sol re-judged them under the same rubric. We had selected 120 of these pairs because the evaluation judge flagged contradiction or performative reasoning. They are biased toward that judge, so we report the other 130 pairs (80 length-matched, 50 random).

Table III. Pattern rates on the 130 8B pairs, GRPO → PatternRM (%), and pairwise agreement of the three label sources on all 250 pairs (500 responses). Bold marks a significant change.

CoTRepConPRTrigger
Human labels26.92 → 20.0030.77 → 12.316.15 → 8.465.38 → 6.9242.31 → 32.31
Seed-2.0-Pro (evaluation judge)23.08 → 16.9223.08 → 10.774.62 → 6.156.92 → 6.9231.54 → 25.38
GPT-5.6-Sol (external judge)29.23 → 25.3844.62 → 22.314.62 → 6.926.92 → 10.0052.31 → 40.77
κ with human labels, Seed-2.0-Pro0.820.680.470.290.58
κ with human labels, GPT-5.6-Sol0.870.620.430.520.68
κ, Seed-2.0-Pro vs. GPT-5.6-Sol0.830.590.440.320.58

Under human labels, leakage, repetition and Trigger fall significantly, by more than under the evaluation judge, and contradiction and performative reasoning change little (+2.31 and +1.54 points, not significant). GPT-5.6-Sol gives the same picture: repetition and Trigger fall significantly, and contradiction and performative reasoning change little (+2.31 and +3.08, not significant). All three give similar rates for these two classes. On the 120 selected pairs, the evaluation judge reports a significant contradiction increase of 21.67 points, while human labels give 3.33 and GPT-5.6-Sol 5.83 (neither significant).

Every pair of label sources agrees least on contradiction and performative reasoning, in line with the evaluation judge's F1. Most of the evaluation judge's disagreement on these two classes lies in responses with leakage or repetition: annotators generally attribute such problems to those two classes and do not label the other two separately, whereas the judge often adds them. On responses that neither flags for leakage or repetition, its κ with human labels is 0.59 for contradiction and 0.50 for performative reasoning. We therefore report these two classes per class and not only through Trigger. A severity re-judge of all 4,101 flagged responses with two judges also confirms the reductions (Appendix G.4).

Details are in Appendix D.3 (Table 12).

AuthorsOpenReview ↗

[Part 3/4] Response to W2 & Q3

Response to W2 & Q3: Item-level train–test separation

Thank you for this request. Of the 44,200 RL training prompts, 40,084 (31,382 images) remain after filtering, and we de-duplicated all of them against every evaluation set. Images were matched with a 256-bit difference hash (Hamming distance ≤ 10), and text with character 3–5-gram TF-IDF (cosine ≥ 0.9). A same-image check on number sets or answers also catches cross-lingual paraphrases, such as Chinese GeoQA originals in a training source and their English versions in MathVerse.

Table IV. Overlap between RL training items and evaluation sets.

Evaluation setItemsSame questionSame image, any question
PatternEval2,41500
MathVista (testmini)1,00052 (5.2%)90
MathVerse (testmini)3,94050 (1.3%)468
ChartQA2,5002330
AI2D / DynaMath3,088 / 5,010035 / 22
Other six sets11,64200

PatternEval has no overlap with the training items, so it is a fully independent test set. No evaluation item matches a training item in both image and text. Every same-question case is a translated or re-worded version with the same image.

We then removed every downstream item that shares an image with training, the most conservative criterion.

Table V. Downstream accuracy scored as in Table 5, GRPO → PatternRM (%), on all items and after removing every item that shares an image with training. Only MathVista, MathVerse, ChartQA, AI2D and DynaMath contain such items. At 8B the per-item outputs behind Table 5 were not kept, so the change from the removal is measured on a re-evaluation of the nearest saved checkpoints.

All itemsDe-duplicated
4B, 11-task mean56.01 → 65.0255.90 → 64.77
8B, 11-task mean61.24 → 66.0061.14 → 65.79
30B-A3B, 11-task mean51.38 → 56.8951.21 → 56.53
4B, MathVerse52.69 → 61.8852.10 → 59.79
4B, MathVista72.20 → 78.4072.75 → 78.79

The 11-task gains barely change, and on MathVerse, the set with the most shared images, the gain stays large. All runs use the same training data, so any overlap affects GRPO and PatternRM alike. The same-question check can miss MathVerse's vision-only variants and letter-only answers; the image-level count bounds these cases. PatternRM's supervision corpus is also separate: it comes from rollouts collected during RL training of a different in-house model and was de-duplicated against PatternEval and the 2,500-response judge set (Appendix E.1). Details are in Appendix G (Dataset separation, Table 14).

AuthorsOpenReview ↗

[Part 4/4] Response to W2 & Q4

Thank you for pointing this out. The size of the increase depends on the estimator. The length-bin standardization in the submission (Table 18) reweights PatternRM's few remaining long responses to GRPO's long tail. The same-question matched-length comparison also holds the question fixed.

Table VI. Length-matched rates, GRPO → PatternRM (%). Bold marks a significant increase; all three Trigger reductions are significant.

4B (1,034 questions)8B (1,117 questions)30B-A3B (1,137 questions)
Contradiction7.74 → 9.285.01 → 7.166.24 → 7.21
Performative11.22 → 10.836.27 → 7.8812.66 → 13.90
Trigger29.79 → 26.1125.69 → 20.8641.95 → 37.82

The increase is significant only at 8B, and we do not explain it away. Under the length-bin standardization of Table 18, the length penalty's increases at 8B are larger than PatternRM's, so part of the effect comes with any reward that shortens responses.

The 8B increase lies mostly where PatternRM does not act:

  • Of the +2.15 contradiction increase, +2.06 (significant) is in responses with wrong answers, and PatternRM penalizes only correct responses.
  • Of the performative increase, +1.43 (significant) co-occurs with leakage or repetition and only +0.18 stands alone. The human annotation convention counts only stand-alone hits, and under it the evaluation judge is more precise.
  • On the 130 blinded pairs, human labels and the external judge show increases of similar size that are not significant at that sample size (Part 2). When the external judge only re-checks the evaluation judge's own flags, it confirms a significant +2.15 for contradiction, so we cannot rule out a small real increase at 8B.

These two classes are harder both to learn and to detect, so a new weight does not fix them.

Harder to learn. PatternRM penalizes only correct responses, and these two failures occur mostly in wrong ones. At 8B, 79% of GRPO's contradiction hits and 85% of its performative hits on PatternEval are in wrong answers; among correct responses their rates are 4.3% and 5.3%, against 18.4% for both leakage and repetition. The few penalties they receive are also noisy: about 40% of PatternRM's flags for these classes are false, and Part 1 shows that misplaced penalties (swap-random) do not make outputs cleaner. Leakage and repetition show in the surface form of a response, whereas a contradiction spans two distant claims and performative reasoning depends on whether a step makes progress. By the training reward model's own labels, PatternRM roughly halves leakage and repetition during training (25.0% → 13.0% and 18.2% → 8.9% from the first 50 steps to steps 300–460) but barely moves contradiction and performative reasoning (6.3% → 6.8% and 11.0% → 8.9%). With reversed weights (0.05 for these two classes), these rates do fall (4.8% → 3.4% and 12.7% → 10.8%), yet the matched-length increases under the evaluation judge grow to +3.61 and +2.41, both significant. Lowering this reward's flags thus does not lower what an independent judge finds, as RL policies exploit the errors of the reward they optimize [3].

Harder to detect. Three measurements rank the classes identically: evaluation-judge F1 (91.2, 87.3, 62.5, 60.0), PatternRM F1 (93.3, 83.9, 55.1, 52.8) and the evaluation judge's agreement with human labels (κ 0.82, 0.68, 0.47, 0.29). Two strong judges also agree far less with each other on these two classes (κ 0.44 and 0.32, against 0.83 and 0.59; Part 2). This is not specific to our judges: GPT-4 detects self-contradictory reasoning with 52.2% F1 and humans with 66.7% [1], and the best critic finds errors in long reasoning traces with only 40.8% F1 [2].

So these two classes need a specialized bad pattern detector, for example one trained with step-level labels, more than a new weight. Details are in Appendix D.3 (Tables 11 and 12) and F.1. Appendix I.6 shows a real contradiction in a PatternRM response, and Appendix I.7 a borderline label.

We thank the reviewer again for the constructive comments. We hope the additional results address your concerns, and we would be grateful if you would reconsider your score.

[1] Liu et al. Self-Contradictory Reasoning Evaluation and Detection. Findings of EMNLP 2024.

[2] He et al. Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? ACL 2025.

[3] Zhao et al. One Token to Fool LLM-as-a-Judge. NeurIPS 2025 MATH-AI Workshop.

Reviewer_FbKVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that answer correctness is not enough for evaluating hybrid-thinking MLLMs, since the final response can still contain leaked reasoning, repetition, contradictions, or unsupported reasoning-like text. To study this, the authors introduce PatternEval and use it to measure correctness separately from four response-pattern failures. They also train PatternRM, a reward model used as an auxiliary penalty in verifier-based GRPO.

I like the problem framing. The distinction between a correct answer and a well-behaved final response is real, and the benchmark analysis gives useful evidence that these failures are not rare. My main concern is with the training claim. The comparison against length penalty is informative, but not clean enough to establish that pattern-aware rewards are intrinsically better than length-based regularization. Several design choices, especially reward strength, category weights, model scope, and task difficulty, need tighter controls.

Critical strengths
  1. The paper focuses on a genuinely important failure mode. A final answer can be correct but still unusable or low-quality because it repeats itself, contradicts itself, or exposes reasoning that should not be shown. This is especially relevant for hybrid-thinking systems, where the user-visible answer and the reasoning channel should be evaluated separately.

  2. PatternEval is a useful diagnostic setup. Separating correctness from four concrete response-pattern labels makes the evaluation more informative than a single accuracy number. The cross-model survey also helps show that the issue is not just an artifact of one model family.

  3. I appreciate that the paper does not stop at measurement. The PatternRM experiment tries to turn the diagnostic labels into a training signal, and the result that verifier-only GRPO can increase response-pattern failures is an interesting warning about optimizing only for final-answer accuracy.

  4. The additional analyses are helpful. The meta-judge evaluation, response-length plots, and length-bin decomposition make the empirical story more transparent than simply reporting aggregate Trigger and accuracy.

Critical weaknesses
  1. The PatternRM versus length-penalty comparison is hard to interpret as a clean comparison of reward types.

Table 4 is not a simple win for PatternRM. The length-penalty baseline gets lower aggregate Trigger, while PatternRM looks better on downstream performance and accuracy retention. The authors' explanation, that length penalty mainly works by compressing responses, is plausible and supported by the decomposition. But the two interventions are not matched. Appendix G reports much larger average reward adjustments for length penalty than for PatternRM, and the methods also differ in topic coverage and invocation frequency. This means the experiment changes the feedback signal, the strength of the penalty, and the scope of application all at once. I would be more convinced by a comparison where the average reward adjustment or the response-length reduction is matched.

  1. The reward design feels under-justified.

PatternRM uses fixed weights: 0.05 for CoT leakage and repetition, 0.02 for contradiction and performative reasoning, with a total cap of 0.1. The overall idea is reasonable: correctness remains primary, and PatternRM only nudges the model toward cleaner correct answers. Still, the particular numbers are not explained. Were they chosen by pilot experiments, label reliability, expected severity, or simply by hand? This matters because the paper later suggests that contradiction and performative reasoning may need more careful calibration. Even a small ablation over the category weights, cap, or invocation probability would make this part of the method feel much less ad hoc.

  1. The analysis still leaves response length, task difficulty, and behavior quality tangled together.

The paper does show that longer responses tend to have higher Trigger, and the within-prompt length penalty is a reasonable way to avoid some crude cross-question length comparisons. But the method still does not model question difficulty, required reasoning depth, or necessary answer length. The length-bin decomposition helps, but it is not causal; the appendix explicitly notes that the same bin can contain different prompts and difficulty levels across variants. This leaves an important possibility open: length penalty may look strong on aggregate because it removes verbose failure opportunities, while also hurting harder examples that genuinely need longer reasoning or more structured explanation.

  1. The method-level evidence is much narrower than the benchmark-level evidence.

The benchmark survey is broad: it covers several model families and a wide Qwen3.5 scale range. The RL validation, however, is only on Qwen3-VL-4B and Qwen3-VL-8B, and appears to use one run per configuration. That is a narrow basis for conclusions about PatternRM as a general training method. I am also not sure how far the binary response-level labels can go: they are useful for measuring incidence, but they do not tell us whether a failure affects the final answer, a key reasoning step, or a minor explanatory sentence.

Questions for authors
  1. Can the authors compare PatternRM and length penalty under matched strength, for example by controlling average reward adjustment, response-length reduction, or Trigger reduction before comparing downstream scores?

  2. How sensitive are the RL results to PatternRM's category weights, cap value, and invocation probability? Why are CoT leakage and repetition weighted more heavily than contradiction and performative reasoning?

  3. Can the authors stratify the results by task family, response length, and some difficulty proxy such as verifier confidence or required answer length? This would clarify whether length penalty mainly helps easy verbose cases while hurting harder ones.

  4. Can the authors add robustness evidence beyond the current Qwen3-VL-4B/8B single-run setting, either with multiple seeds or with a small additional experiment on another scale or architecture?

10 comments
AuthorsOpenReview ↗

[Part1/4] Response to W1 & Q1

Response to W1 & Q1: Comparison at matched strength

Thank you for this suggestion. We agree that the submitted comparison changes the feedback signal, the penalty strength and the scope at once. Matching the mean reward adjustment would not fix this. In a GRPO group whose rollouts are all correct, the length penalty's normalized advantage is (qi−qˉ)/σq(q_i-\bar q)/\sigma_q and does not depend on its coefficient (Appendix G, Eq. 4). A smaller coefficient lowers the mean adjustment but not the update. We therefore match the reward distribution itself.

We trained two controls on Qwen3-VL-8B. They use the same judge, invocation probability, weights and cap as PatternRM. In each group, the penalties are moved, unchanged in number and value, from the flagged correct responses to other correct ones: to random ones (swap-random) or to the longest ones (swap-longest). So in every batch, each group gets exactly the same rewards and advantages as PatternRM would give it; only the penalized response differs. We also trained a weak length penalty with c = 0.2, whose maximum penalty equals PatternRM's cap of 0.1. All runs share initialization, data and GRPO settings. Scope also differs less than stated: the length penalty covers all three training sources, including document questions.

Shorter responses have fewer chances to fail, so a lower Trigger can come from length alone. The length-matched Trigger therefore compares two runs only on the PatternEval questions where their responses differ in length by at most 25% of the longer one.

Table I. Qwen3-VL-8B, step 450 (%). The first four result columns are on PatternEval; downstream Acc is the 11-task mean ("Overall" in Table 5). Each method is compared with GRPO on its own length-matched questions, so GRPO's value in that column differs by row. GRPO, PatternRM and the submitted length penalty use Tables 4 and 15; the new runs are scored on the 2,387 questions with a reference answer, which moves rates by under 0.5 points.

MethodPatternEval AccTriggerMean length (chars)Length-matched Trigger: GRPO → method (questions)Downstream Acc
GRPO50.4446.383,708–61.24
+ PatternRM50.5730.891,36025.69 → 20.86 (1,117)66.00
swap-random49.0643.653,67438.95 → 37.77 (1,366)59.79
swap-longest49.7526.9973418.41 → 19.23 (364)65.07
Length Penalty, c = 0.250.2140.783,16536.23 → 36.00 (1,314)61.94
Length Penalty, c = 0.6 (submitted)48.6427.9181424.07 → 22.49 (378)63.53

With reward statistics matched batch by batch, penalizing a random correct response lowers downstream accuracy from 61.24 to 59.79, while penalizing the flagged response raises it to 66.00. swap-longest has the same penalty strength but always penalizes the longest correct response, so it acts as a length penalty: its responses are the shortest and its Trigger the lowest, yet at matched length it is not cleaner than GRPO (18.41 → 19.23). It also shortens answers that need long reasoning (median 751 vs. 1,636 characters for PatternRM; Part 3). Among these runs, only PatternRM is significantly cleaner at matched length (25.69 → 20.86; 95% paired-bootstrap interval excludes 0). The weak length penalty changes little. So the gain comes from where the penalty lands, not from its strength or from shorter outputs.

The same test explains the submitted result, where the length penalty has the lowest Trigger (Table 4). Table 18 reweights each run's PatternEval responses to GRPO's distribution over eight character-length bins. After this reweighting, the length penalty's Trigger is 56.43 at 4B and 51.06 at 8B, above GRPO's 51.43 and 46.38, so its whole reduction comes from shorter responses. A length bin can hold different questions for different runs, so Table II also fixes the question.

Table II. Submitted runs (4B step 510, 8B step 450) on PatternEval (%). Trigger is from Table 4, and all four reductions over all questions are significant. Bold marks a significant length-matched reduction.

ScaleMethodTrigger: GRPO → methodLength-matched Trigger: GRPO → method (questions)
4B+ PatternRM51.43 → 39.5029.79 → 26.11 (1,034)
4B+ Length Penalty51.43 → 32.4644.40 → 45.21 (491)
8B+ PatternRM46.38 → 30.8925.69 → 20.86 (1,117)
8B+ Length Penalty46.38 → 27.9124.07 → 22.49 (378)

With the question fixed, the result is the same: the length penalty is not significantly cleaner than GRPO at either scale, and PatternRM is significantly cleaner at both. At 8B the length penalty also lowers PatternEval accuracy significantly (50.44 → 48.64), while PatternRM does not (50.44 → 50.57). Details are in Appendix G.1–G.2 (Table 19, Figure 7).

AuthorsOpenReview ↗

[Part 2/4] Response to W2 & Q2

Response to W2 & Q2: Category weights, cap and invocation probability

Thank you for asking. Each value has a reason (Appendix F.1):

  • The cap of 0.1 is a constraint. Correct responses get rewards in [0.9, 1] and wrong ones get 0, so correctness always dominates and the pattern term only reorders correct responses.
  • Invocation probability 0.6 judges at least one rollout in 99.93% of groups, with about 614 judge calls per step against 1,024 at p = 1.0.
  • The weight ordering follows label reliability. PatternRM's F1 is 93.3 and 83.9 for CoT leakage and repetition, against 55.1 and 52.8 for contradiction and performative reasoning (Table 3).

The magnitudes were set in preliminary RL on a different in-house model, before the Qwen3-VL runs, and were not re-tuned. There, larger pattern penalties lowered Trigger quickly but cost accuracy, while small ones kept accuracy (Part 4, Table VII). We therefore cap the pattern term at a tenth of the correctness reward.

We also ran two ablations on Qwen3-VL-8B.

Table III. Qwen3-VL-8B, step 450. PatternEval rates (%) as in Table 4, whose GRPO and original rows are reused; the two new runs are scored on the 2,387 questions with a reference answer, which moves rates by under 0.5 points. Downstream Acc is the 11-task mean of Table 5.

VariantWeights CoT/Rep/Con/PRpAccTriggerCoTRepConPRDownstream Acc
GRPO––50.4446.3833.8337.1810.6417.3961.24
Original0.05/0.05/0.02/0.020.650.5730.8917.3914.5310.9713.2966.00
Reversed weights0.02/0.02/0.05/0.050.650.4435.2722.4121.6212.5714.2965.58
Invocation p = 1.00.05/0.05/0.02/0.021.049.8533.0120.1120.7410.5612.5766.36

Accuracy barely moves: PatternEval accuracy stays within 0.8 points and downstream accuracy within 0.5 points of the original setting. The original setting has the lowest Trigger, and p = 1.0 does not improve on it although it needs 1.7× the judge calls. Reversing the weights weakens the reduction of leakage and repetition and does not lower contradiction or performative reasoning further than the original weights (12.57 and 14.29, against 10.97 and 13.29), although the training reward model's own hit rates for these two classes do fall (4.8% → 3.4% and 12.7% → 10.8% from the first 50 steps to steps 300–460). For these two classes the bottleneck is detector reliability, not the weight (Appendix D.3).

We did not add cap or equal-weight runs, because the reward structure limits what they can change (Eq. 2):

  • GRPO normalizes rewards within each group. In a group whose rollouts are all correct, the rewards are 1 − zP, so scaling all weights and the cap by the same factor leaves every advantage unchanged, as for the length penalty (Appendix G, Eq. 4). The overall scale acts only in groups that also contain wrong responses, and any cap below 1 keeps every correct response above every wrong one.
  • The weights sum to 0.14, so the cap of 0.1 binds only when a response is flagged for leakage, repetition and a third class. A cap of 0.2 would never bind, and a cap of 0.05 would give a response flagged only for leakage the same penalty as one flagged for all four classes.
  • What remains is the ratio between classes. The reversed run tests it (2.5 : 1 against 1 : 2.5), and equal weights lie between the two.

Details are in Appendix F.1 and Tables 11 and 19.

AuthorsOpenReview ↗

[Part 3/4] Response to W3 & Q3

Response to W3 & Q3: Response length, difficulty and task family

Thank you for this question. We break the results down along proxies that do not depend on the compared models (Appendix H.1):

  • Task family: the three domains of Table 5.
  • PatternEval difficulty: the pass rate of 40 frozen external models, in four bands.
  • Reasoning demand: the median length of those models' correct answers, in thirds.

Our verifier is a binary LLM judge without calibrated confidence, so the external pass rate takes the place of verifier confidence.

Table IV. Downstream accuracy by task family (%), domain means as in Table 5.

ScaleMethodMath (5 tasks)Logic (2 tasks)Document (4 tasks)Overall
4BGRPO49.8332.8475.3156.01
+ PatternRM59.9043.3482.2865.02
+ Length Penalty56.2740.6463.1655.94
8BGRPO53.5136.8883.1061.24
+ PatternRM60.2741.6385.3466.00
+ Length Penalty56.6139.6684.1063.53
swap-random51.6436.8681.4459.79
swap-longest60.2140.8183.2665.07
30B-A3BGRPO51.5533.4860.1251.38
+ PatternRM61.4641.1759.0356.89

The length penalty gains on math and logic at both scales but loses on documents at 4B (63.16 vs. 75.31). PatternRM is highest in every family at 4B and 8B; at 30B-A3B it gains on math and logic and is slightly lower on documents. Splitting downstream items by whether the shared Instruct initialization solves them, PatternRM is significantly above GRPO in every stratum at 4B and 8B (Table 22).

Table V. PatternEval, Qwen3-VL-8B, step 450. Accuracy (%) by external pass rate, with the number of questions, and median response length (characters) on the third of questions whose correct external answers are longest.

Hard (477)Mid-hard (411)Mid-easy (432)Easy (1,064)Median length, long-demand third
GRPO6.5020.6840.7485.625,952
+ PatternRM6.9219.2243.5285.151,636
+ Length Penalty5.2418.7338.1983.93711
swap-longest6.9218.9840.9784.49751

No variant changes accuracy significantly in any band, and neither does PatternRM at 30B-A3B. PatternRM keeps longer answers where the question needs them: in the long-demand third its median response is more than twice as long as the length penalty's. In the medium-demand third, the length penalty lowers accuracy significantly (51.04 → 47.41), while PatternRM does not (51.04 → 50.00). At matched length, PatternRM's Trigger falls significantly in the easy and mid-easy bands (16.99 → 12.14 on 618 questions; 32.00 → 23.50 on 200), but not in the hardest band (47.22 → 45.83 on 144).

So the length penalty's aggregate advantage does not come with losses concentrated on hard questions. PatternRM loses no accuracy on hard questions, but we do not claim that it is cleaner there. Details are in Appendix H.1 (Table 22, Figure 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.