Given the two citation issues above, I did get it to extract all the references and do a quick check online to see if any others had been debunked/similar. I copy the report below.
To be honest, it didn't do a great job - eg, I gave it #1 as an example, including the Notion doc, and it didn't even find #2 on its own. So, the report shouldn't be thought of as comprehensive.
I did get agents to check it after it was written, for accuracy, but naturally individual claims should be verified manually. The two points raised in §Weaknesses are still the primary issues.
Findings requiring qualification or clarification
Priority: baseline adequacy in two prominently cited RL studies. The issue is whether their comparisons establish the claimed size and interpretation of the RL benefit against a well-elicited starting model. Critics need not demonstrate that RL worsens performance. Different evaluation protocols prevent causal subtraction of scores; they do not dispose of a challenge to the adequacy of the reported baseline.
Chandak, Goel and Prabhu’s Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims raises this objection for both Spurious Rewards and RL with One Training Example.
1. Spurious Rewards: the headline gain needs a stronger-baseline comparison
Confirmed prompt sensitivity and baseline-adequacy concern. The primer’s +21.4-point Qwen2.5-Math-7B result (§4.1, Figure 2a; §7) faithfully reproduces Spurious Rewards, v2, Figure 1, p 2: 49.4% before training and 70.8% after random-reward training. Yet the same source’s Appendix I, Tables 4–5, pp 25–26 reports 68.8% without training under another prompt.
The two untrained prompt scores differ by 19.4 points, numerically close to the 21.4-point within-protocol RL increase. These are distinct comparisons: the prompt difference does not decompose the RL gain, and 70.8−68.8 is not a causal estimate of RL’s effect. The reported 49.4→70.8 increase remains a within-protocol result, but does not quantify the benefit over a well-elicited starting model. Faithful transcription alone does not address that baseline-adequacy objection. The source acknowledges prompt sensitivity; the primer should attach this concrete qualification to its central example, rather than rely on its general evaluation discussion (§6.3).
Notion’s overview compares the different, non-Math Qwen2.5-7B model. Its numbers should not be transferred to this example: the Math-model concern above is independently supported by the primary paper itself.
2. RL with One Training Example: the stronger starting-model score is genuine
Confirmed stronger starting-model score under a different evaluation. The original paper, v1, §3.3/Table 4 reports DeepSeek-R1-Distill-Qwen-1.5B improving from 71.0% to 78.0% MATH500. The official model card separately reports 83.9% for the released starting model under its evaluation protocol, before the one-example training. Notion’s overview gives 71.9 rather than the source’s 71.0; that discrepancy does not invalidate the stronger-baseline objection.
The v1 comparison uses an 8,192-token cap and greedy MATH500 evaluation (§3.1): its reported seven-point increase is within those conditions. The model card specifies a 32,768-token cap and, for benchmarks requiring sampling, temperature 0.6, top-p 0.95 and 64 responses per query to estimate pass@1. Its 83.9 is not a matched counterfactual for v1. More informatively, the revised paper, v3, Table 4/Appendix B.5 compares 82.9% → 83.9% under its own matched 32k sampled protocol: a much smaller, one-point gain under those conditions. This supports the concern about the claimed gain’s dependence on evaluation, without erasing the within-v1 result. The trained 83.9 does not refute the independently reported starting-model 83.9.
Relevance to the primer: §6.1/Figure 5, Tables B.1–B.2 row 6.1a and B.6 principally use Qwen2.5-Math-1.5B, 36.0% → 73.6%, not that DeepSeek subexperiment. The DeepSeek comparison warrants checking baseline adequacy in one-example studies, but does not itself overturn the plotted Qwen result. The primer should make that model/protocol scope clear when using the study to motivate data efficiency. A corresponding stronger untrained Qwen baseline was not established in this audit.
3. The s1 uncertainty discussion omits published paired item-bootstrap evidence
Confirmed inferential omission, not an arithmetic error. Appendix B.1, p 29 says the selected-versus-random AIME24 difference cannot be separated from question-sampling noise without per-problem results. Its approximate independent-binomial standard errors recompute correctly. However, the published s1 paper, §5.1/Table 2, p 20280 already reports a 95% paired item-bootstrap interval of [−26.7, −3.3] percentage points for random 1K minus selected s1K, excluding zero.
The primer should report this item-level paired result alongside its independent-binomial approximation. Conversely, s1’s 59K-versus-selected intervals include zero; this does not establish equivalence. These item-based analyses do not quantify training-run or evaluation-implementation variability. s1 discusses the latter—evaluation nondeterminism—in Appendix C, p 20289, not training-seed robustness.
4. Figure 5’s normalisation is not uniquely specified
Unresolved reporting detail, not a demonstrated impossible value. The purple one-prompt OPD annotation is reproducible: the reported scores 42.4 → 51.7, against 53.7 for full data, retain 82.3% of the improvement (Hou et al., v2, Tables 1–2).
For the green RLVR curve, Figure 5 does not identify the metric, full-data reference and checkpoint convention underlying “96%”. Different legitimate choices give different fractions. For example, best-MATH500 scores 36.0 → 74.0 against 75.4 yield 96.4%, whereas best-six-task-average scores give other values (Wang et al., v3, Table 3, p 7; Tables 8–9, p 21). Providing the plotted values and normalisation would resolve this without new experiments.
Verified checks and evaluation limits
- Denominators: single-run AIME scores in Figure 3a/Table B.2 are attainable on 30 questions. TTRL’s 12.9% and 40.2% are also attainable because they average 16 samples; requiring multiples of 1/30 would be incorrect.
- Process errors: Figure 2b’s rates reproduce as 7/200, 94/500, 161/500 and 259/500. These are selected correct-final-answer cohorts, not error prevalence in arbitrary training corpora (ProcessBench, §3.3/Table 2).
- Redrawn data: Figure 2c matches Skywork’s 16-rollout distributions (§6.2); Figure 2d matches ReSyn’s fixed-16K-instance ablation (§5.2/Table 5); Figure 3c reproduces R-Zero’s pseudo-label accuracy against its assumed GPT-4o oracle: 79%, 69% and 63% (§4.4/Table 5). This is not independently established mathematical correctness.
- Derivations and budgets: the no-signal probabilities in Table C.4, advantages in C.1 and pass@1/pass@64 counterexample in B.7 recompute correctly. B.4’s noise identity is valid under its stated class-conditional assumptions, not a general result for clipped GRPO. B.6–B.7 expose rollout, selection and annotation burdens; quantities vary by study and some costs are not stated. One prompt does not mean one training observation.
- Leakage interpretation: this primer trains no models or releases new training data. Its cited test-time adaptation and benchmark-selected checkpoints raise evaluation-exposure and selection concerns, not demonstrated hidden answer-label leakage in a new experiment. No new regression/R-squared estimate or empirical run-level error bars are presented; B.1 does calculate approximate analytical binomial standard errors.
What the criticism screen changes
Of the seven articles criticised in Notion, only Spurious Rewards and RL with One Training Example are cited in the primer; both support substantive claims. Their baseline concerns should be distinguished from the peripheral historical corrections below.
There are genuine historical source defects, but their citation roles matter. FinQA’s released retriever code had a label-leaking table-formatting bug corrected in 2022 (official update notes); UltraFeedback corrected overall-score labels (official repository). Both appear here as background entries in A.3/Table A.1, not measurements supporting the central seven lessons. Hu et al., v5, has two distinct Table 3 issues: the PDF, p 8 and HTML disagree on correlation signs (eg Qwen2.5-3B/GRACE DS: +0.781 versus −0.781); in the PDF, the five displayed Qwen2.5-3B/RSR values average −0.6918, versus the reported −0.603. Neither issue directly refutes the separate nine-teacher Figure 1 observation used by the primer (§5.2, §7, Table B.6). These specific defects do not justify calling all cited evidence collectively “debunked”.
For positioning, Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning, Findings ACL 2025, §7/Figures 4–6 already examines teacher–student and task dependence in CoT distillation but is absent from the bibliography. It already documents the narrow observation that stronger teachers need not transfer best. The primer’s broader cross-route synthesis and reporting framework remain distinct; this comparison alone does not settle its overall novelty or value. This is older peer-reviewed work; arXiv-only work and work peer-reviewed on or after 17 July 2026 were not treated as mandatory comparisons.
Unchecked: underlying prediction logs, immutable dataset/checkpoint provenance and the companion search/figure records were not supplied. Numerical feasibility and source agreement do not independently reproduce model runs, seed variability or every source’s contamination controls. Detailed Notion section bodies were inaccessible; its overview/table and the relevant primary-source numbers, not every allegation, were checked. Uncorroborated allegations and inaccessible evidence remain unresolved, not confirmed defects.
Response to Reviewer sLr7
Thank you for the review. We are glad you found the organized synthesis of scattered evidence, and the learner and route as shared variables, valuable. Your two weaknesses pointed at things we could usefully fix: the positioning against Luo et al. and Yu et al. is now explicit in §2 and Table A.2, several figures are redrawn, and Table B.5 now separates a wrong verdict from a mishandled reward. Section 7 now also reports two of the three comparisons you asked about.
W1. Positioning.
We agree, and thank you for pointing to these two surveys. In brief: Luo et al. organize the literature by data operation, from selection to synthesis, and Yu et al. by RL technique under data scarcity; like other recent surveys, such as Chen et al. (2025a) on the capabilities of long chains of thought, they map which methods exist. The primer complements them with a question they leave open, one a practitioner meets whenever two results disagree: which of them holds for a given model and training method? We read each finding together with the learner it was measured on, the route by which its data reached that learner (SFT, RL or on-policy distillation) and what its comparison held fixed. Read this way, six of the seven conflicts we collect differ mainly in learner or route (Appendix B.2), and our new experiments test this reading directly: in every family, the best teacher for the smallest student is the worst for the largest. §2 now states this boundary in more detail, and Table A.2 sets these surveys side by side by the question each starts from.
W2. Figures.
Thank you for the suggestion. Figures 1–4 are redrawn at their printed size: no label in them is now smaller than 6 pt, where most were before, down to 3.9 pt in Figure 2. The new Figure 6 follows the same standard.
W2. Tables.
Thank you; Table B.5 now separates them. Summed step rewards and one score per response form their own group, Reward aggregation and granularity, and the caption states that this group falls outside the derivation above the table, which concerns errors in the checker's verdicts.
Q1. Which comparison first?
The teacher-by-student crossing, and we have now run it. Its prediction is the sharpest, since s1 and OpenThoughts point in different directions about which teacher is better. Across three teachers and ten students from three model families, the ranking reverses between the smallest and the largest student of each family. We also ran the comparison of random against systematic checker errors. Please see Q2 in our main response for both results.
We hope these changes address your concerns, and we would be glad to answer any further questions.