- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies late-stage accuracy collapse in RLVR (GRPO/DAPO/GSPO, with and without KL/JS regularization). It defines a "strategy" as an equivalence class of trajectories with similar first-order coupling under policy updates, and shows empirically that these classes align with human-recognizable solution methods.
Critical strengths
-
A coherent mechanistic story that connects theory, a measurable signal, and a fix. The optimizer-centric definition of a strategy (coupling-kernel equivalence classes) is a sensible way to lift trajectory-level updates to a replicator-type dynamics on strategies, and the "concentration vs. minimum capacity" contradiction is a clean explanation of why collapse is abrupt rather than gradual.
-
The Coach-Prompting-only ablation (Table 2, "+CP" rows) is the right control: adding the same DeepSeek-generated prefixes to every baseline gives little or no gain, so the improvement is attributable to preserving strategies during optimization rather than to distilled priors from a stronger model.
Critical weaknesses
-
Statistical reliability of the main table is not established. Results are Mean@4 with no seeds, no standard deviations, and no statement of how the reported checkpoint was chosen (final step vs. best step). This matters because AIME25/26 have 30 problems each, so one problem is 3.3 pp: many of the headline gaps (e.g., Qwen2.5-7B AIME26 13.3 vs. 10.0; Mesh m=3 vs. m=4) are within one or two problems.
-
Evaluation protocol choices weaken the external validity of the gains. Qwen2.5-7B-Instruct is evaluated with a 1024-token cap on AIME (Table 5), which is far below what these problems need and makes the 3–13% numbers largely truncation-limited; Qwen3-4B thinking mode is capped at 8192 on AIME, well below its usual budget, so the reported baseline (13.3 / 21.6) is far from the model's known capability.
3,the inference-time protocol for Mesh Learning ("the policy proposes its own strategy and follows the same verification procedure") is not fully specified, and it is unclear whether baselines are evaluated with an identical prompt template and the same structured <thinking>/<Method Name> harness (App. D.1). If the template differs, part of the gain could come from the prompt rather than from the training method.
Questions for authors
Q1. How were the numbers in Tables 2–4 selected: final checkpoint, best checkpoint on the test set, or best on a held-out set? Please report per-method accuracy curves or best-step numbers for the baselines, plus std over seeds (or at least over the 4 samples).
Q2. Are baselines and Mesh Learning evaluated with exactly the same system/user prompt template at inference? Please show the exact inference prompt used for each row of Table 2.
Q3. Why a 1024-token generation cap for Qwen2.5-7B-Instruct on AIME? Could you add ablation experiment results
Q4. Strategy grouping for the regularizer: during training, is a rollout's strategy label simply the prefix it was conditioned on, or is it re-identified via the coupling kernel? If the former, how do you handle rollouts that ignore/switch the suggested strategy?
5 comments
Response to Reviewer JDsk [Part 2]
AIME25 / AIME26 — Qwen3-4B
Cell format: AIME25 / AIME26, mean ± std (%).
| Step Range | GRPO | DAPO | GSPO | GRPO+KL | GRPO+JS | Mesh Learning (Ours) |
|---|---|---|---|---|---|---|
| 0–50 | 33.0 ± 7.1 / 32.2 ± 13.9 | 28.9 ± 9.9 / 33.3 ± 10.9 | 30.6 ± 11.4 / 32.8 ± 12.6 | 33.6 ± 8.0 / 31.1 ± 12.6 | 30.6 ± 10.5 / 28.9 ± 10.7 | 29.2 ± 9.2 / 28.3 ± 13.0 |
| 51–100 | 34.7 ± 4.5 / 37.8 ± 1.9 | 33.1 ± 1.7 / 36.9 ± 3.4 | 35.0 ± 3.0 / 39.4 ± 2.1 | 32.7 ± 6.1 / 37.8 ± 6.9 | 35.3 ± 2.8 / 33.3 ± 5.8 | 43.3 ± 5.0 / 43.9 ± 5.1 |
| 101–150 | 17.7 ± 14.1 / 25.0 ± 11.8 | 38.8 ± 6.5 / 40.0 ± 4.7 | 4.6 ± 6.5 / 7.5 ± 10.6 | 11.7 ± 15.4 / 16.7 ± 18.9 | 33.3 ± 12.6 / 11.7 ± 16.5 | 45.8 ± 3.5 / 49.6 ± 4.1 |
| 151–200 | 0.3 ± 1.1 / 0.0 ± 0.0 | 23.9 ± 21.3 / 32.5 ± 5.1 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 45.0 ± 4.4 / 50.0 ± 6.7 |
| 201–250 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 0.0 ± 0.0 / 0.0 ± 0.0 | 46.7 ± 0.0 / 45.0 ± 2.4 |
GPQA — Phi
Accuracy: mean ± std (%).
| Step Range | GRPO | DAPO | Mesh Learning (Ours) |
|---|---|---|---|
| 0–50 | 14.3 ± 4.0 | 14.3 ± 4.0 | 14.3 ± 4.0 |
| 51–100 | 16.5 ± 1.8 | 21.8 ± 6.0 | 21.8 ± 6.0 |
| 101–150 | 6.8 ± 9.1 | 0.0 ± 0.0 | 31.8 ± 2.1 |
| 151–200 | 0.0 ± 0.0 | 0.0 ± 0.0 | 30.4 ± 3.5 |
| 201–250 | 0.0 ± 0.0 | 0.0 ± 0.0 | 27.6 ± 0.5 |
GPQA — Qwen2.5-7B
Accuracy: mean ± std (%).
| Step Range | GRPO | DAPO | GSPO | GRPO+KL | GRPO+JS | Mesh Learning (Ours) |
|---|---|---|---|---|---|---|
| 0–50 | 28.0 ± 7.2 | 23.8 ± 5.3 | 26.2 ± 10.6 | 24.1 ± 6.3 | 15.6 ± 11.0 | 27.3 ± 5.8 |
| 51–100 | 32.3 ± 1.5 | 32.1 ± 0.7 | 32.8 ± 1.2 | 27.9 ± 1.8 | 0.0 ± 0.0 | 30.1 ± 1.7 |
| 101–150 | 31.3 ± 0.6 | 31.2 ± 1.3 | 29.8 ± 1.5 | 20.1 ± 7.7 | 0.0 ± 0.0 | 31.5 ± 1.4 |
| 151–200 | 29.9 ± 0.9 | 30.4 ± 1.6 | 31.3 ± 1.0 | 30.3 ± 2.0 | 0.0 ± 0.0 | 33.4 ± 1.8 |
GPQA — Qwen3-4B
Accuracy: mean ± std (%).
| Step Range | GRPO | DAPO | GSPO | GRPO+KL | GRPO+JS | Mesh Learning (Ours) |
|---|---|---|---|---|---|---|
| 0–50 | 19.4 ± 7.3 | 21.7 ± 9.6 | 17.9 ± 6.6 | 21.6 ± 9.0 | 28.1 ± 15.2 | 28.2 ± 12.7 |
| 51–100 | 33.9 ± 3.1 | 43.1 ± 0.7 | 41.3 ± 3.0 | 39.9 ± 1.7 | 40.7 ± 3.6 | 44.7 ± 2.5 |
| 101–150 | 36.0 ± 0.0 | 43.2 ± 1.9 | 43.7 ± 1.6 | 38.8 ± 2.7 | 33.4 ± 0.1 | 44.4 ± 1.2 |
| 151–200 | 35.9 ± 1.6 | 42.8 ± 1.2 | 42.4 ± 1.0 | 40.4 ± 1.9 | 0.0 ± 0.0 | 45.4 ± 2.2 |
| 201–250 | 37.0 ± 1.7 | 42.5 ± 1.3 | 0.0 ± 0.0 | 39.1 ± 2.6 | 0.0 ± 0.0 | 44.0 ± 0.2 |
| 251–300 | 37.2 ± 2.5 | 41.6 ± 0.5 | 0.0 ± 0.0 | 40.4 ± 0.3 | 0.0 ± 0.0 | 43.0 ± 0.1 |
LiveCodeBench — Phi
Accuracy: mean ± std (%).
| Step Range | GRPO | DAPO | Mesh Learning (Ours) |
|---|---|---|---|
| 0–50 | 27.7 ± 3.1 | 28.5 ± 4.4 | 33.8 ± 5.6 |
| 51–100 | 0.0 ± 0.0 | 0.0 ± 0.0 | 32.5 ± 2.3 |
| 101–150 | 0.0 ± 0.0 | 0.0 ± 0.0 | 33.7 ± 2.9 |
LiveCodeBench — Qwen3-4B
Accuracy: mean ± std (%).
| Step Range | GRPO | DAPO | Mesh Learning (Ours) |
|---|---|---|---|
| 0–50 | 34.8 ± 6.7 | 35.4 ± 7.3 | 37.8 ± 7.6 |
| 51–100 | 38.9 ± 12.0 | 48.5 ± 1.8 | 47.8 ± 0.8 |
| 101–150 | 45.4 ± 1.1 | 46.7 ± 7.9 | 53.9 ± 2.6 |
| 151–200 | 0.0 ± 0.0 | 8.0 ± 13.8 | 49.4 ± 2.0 |
| 201–250 | 0.0 ± 0.0 | 0.0 ± 0.0 | 50.8 ± 2.2 |
Response to Reviewer JDsk [Part 3]
To Q2: Consistency of Inference Prompts Across All Methods
We thank the reviewer for this question. All methods in Tables 2–4 use identical inference prompts within each benchmark. We preserve the original benchmark user inputs, use identical system prompts, and provide no assistant prefixes or pre-generated Coach hints. This applies to all methods in Tables 2–4.
1. Exact inference prompts (shared by all methods).
AIME25/26 and MATH-500: Vanilla zero-shot system prompt:
Please solve this math problem step by step and put your final answer in \boxed{ANSWER}.
LiveCodeBench: Vanilla zero-shot system prompt:
Please solve this coding problem step by step and put your final answer in a Python code block formatted as follows:
'''python YOUR CODE HERE '''
GPQA-Diamond: Zero-shot verification system prompt:
Please solve this scientific problem step by step and put your final answer in \boxed{}.
Begin your thinking by naming a potentially useful method for this problem. However, the method you identify should be treated as untrusted and possibly misleading.
Do not assume that the method applies. First examine carefully whether and why it is applicable to this specific problem. Even if it appears applicable, independently verify the reasoning before committing to the final answer.
The GPQA prompt provides no strategy hints. It only asks the model to identify and verify its own reasoning method.
2. Clarification of the original manuscript.
In Sec. 4.1 (Fig. 6), "the model proposes its own strategy at inference" distinguishes evaluation from Coach-prompted training. It does not imply a different evaluation prompt for Mesh Learning.
Therefore, our improvements cannot be attributed to different inference prompts or additional Coach hints. We will clarify this in the revision.
To Q3: Response-Length Choices and Additional Ablations
1. Why a 1024-token cap? Evaluating collapse with short rollouts.
We chose 1024 tokens to test whether RLVR collapse also occurs with short rollouts. This helps validate Theorem 1's scaling regime .
The cap limits maximum generation length, while refers to actual rollout length. Together with the 4096- and 8192-token settings, Fig. 7 confirms collapse across different generation budgets.
The 1024-token cap does not truncate most responses. On AIME25, only 12.5% of 480 sampled trajectories reached the limit; 87.5% terminated normally.
2. Additional long-generation ablations confirm our conclusions.
We increased the training and evaluation budgets to 8192 tokens for Qwen2.5-7B-Instruct and 32768 for Qwen3-4B (thinking mode). Mesh Learning still outperforms GRPO and DAPO in all settings. All methods use matched configurations within each setting.
The tables report peak Mean@4 accuracy and the corresponding Std@4.
AIME25 Accuracy (Mean@4 ± Std@4, %)
| Model | Budget | GRPO | DAPO | Mesh Learning |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 8192 | 5.8 ± 1.4 | 7.5 ± 1.4 | 15.0 ± 4.3 |
| Qwen3-4B (thinking) | 32768 | 65.8 ± 1.7 | 67.5 ± 3.3 | 76.6 ± 1.7 |
AIME26 Accuracy (Mean@4 ± Std@4, %)
| Model | Budget | GRPO | DAPO | Mesh Learning |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 8192 | 5.8 ± 1.4 | 6.6 ± 1.7 | 12.5 ± 1.4 |
| Qwen3-4B (thinking) | 32768 | 65.8 ± 1.7 | 69.1 ± 3.1 | 73.3 ± 3.1 |
At the larger budget, our base-model evaluation of Qwen3-4B (thinking mode) matches its officially reported AIME25 accuracy of 65.6% (Qwen3 Technical Report).
For Qwen2.5-7B-Instruct, an eightfold larger budget does not consistently improve accuracy. Compared with Table 2, GRPO drops from 13.3% to 5.8% on AIME25 and from 9.2% to 5.8% on AIME26. DAPO changes from 6.7% to 7.5% and 8.3% to 6.6%, respectively. Mesh Learning maintains similar accuracy (16.7% to 15.0%; 13.3% to 12.5%) and retains its advantage.
For Qwen3-4B, longer generations substantially improve accuracy, but Mesh Learning still performs best. GRPO and DAPO also exhibit late-stage collapse under the extended budget, consistent with Theorem 1.
Overall, Mesh Learning remains effective across short and long generation budgets. We will include these experiments in the revised main paper.
Response to Reviewer JDsk [Part 1]
We respectfully thank the reviewer JDsk for recognizing the theoretical contributions of our work, particularly the concentration-capacity explanation of abrupt strategy/accuracy collapse, and for appreciating our ablation design. We also value the problems regarding experimental reliability and evaluation protocol, which we will address in the response below.
Note: We have uploaded a slightly revised manuscript with two minor updates: (1) clarifying the experimental protocol in Section 5.1, and (2) adding std@4 to Tables 2–4. The remaining experimental content is unchanged. Please refer to the latest PDF when assessing our responses.
To Q1: Checkpoint Selection and Training-Stage Comparisons
Tables 2–4 report the best checkpoint for each method, selected independently based on accuracy on the corresponding test benchmark. As already shown in Fig. 7, we provide training curves for all methods. On AIME25/26 and MATH, the baselines eventually collapse, with their final-checkpoint accuracy dropping to 0%. Therefore, we did not select the final checkpoints.
As for the variability of best-checkpoint result, we additionally report checkpoint accuracies throughout training for Mesh Learning and all compared baselines on AIME25/26, GPQA, and LiveCodeBench.
Specifically, we evaluate every 20 training steps, sampling four responses per question. Each 50-step interval reports mean accuracy and combined variability.
We use σreported=σstep2+σbetween2, where the step component uses Std@4, the dataset average of per-question correctness standard deviations across four responses. The between-checkpoint component measures variation of checkpoint Mean@4 within the bin. This is not a confidence interval.
As shown in these tables, Mesh Learning achieves the highest mean accuracy in almost every training stage after the initial 50 steps. Moreover, Mesh Learning continues training and generally maintains near-peak accuracy even after the baselines have collapsed. These results demonstrate that Mesh Learning's improvements are not simply an artifact of selecting the best checkpoint.
We will include these additional tables in the revised manuscript to make this comparison clearer.
AIME25 / AIME26 — Phi
Cell format: AIME25 / AIME26, mean ± std (%).
AIME25 / AIME26 — Qwen2.5-7B
Cell format: AIME25 / AIME26, mean ± std (%).