acceptodds
Under review as a conference paper at ICLR 2027

Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-Evolution

Abstract

Recent breakthroughs in large language models (LLMs) on reasoning tasks rely heavily on massive, high-quality datasets—typically human-annotated and thus difficult to scale. While data synthesis or distillation offers a promising alternative, existing methods struggle with inconsistent data quality and an inability to dynamically adapt to the evolving capabilities of the model, leading to suboptimal training signals. To address these limitations, we introduce Socratic-Zero, a fully autonomous framework that generates high-quality training data from minimal seed examples through the co-evolution of three agents: the Teacher, the Solver, and the Generator. The Solver continuously refines its reasoning by learning from preference feedback on both successful and failed trajectories; the Teacher adaptively crafts increasingly challenging questions based on the Solver's weaknesses; and the Generator distills the Teacher's question-design strategy to enable scalable, high-fidelity curriculum generation. This closed-loop system produces a self-improving curriculum without requiring large pre-existing task collections or human annotations. Remarkably, starting from only 100 seed questions, our Socratic-Solver-8B achieves an average gain of +20.2 percentage points over prior data synthesis methods across seven mathematical reasoning benchmarks (AMC23, AIME24-25, Olympiad, MATH-500, Minerva, and GSM8K). Beyond mathematical reasoning, Socratic-Zero further generalizes to a diverse set of domains and modalities, including science reasoning, deep research, software engineering, multimodal chart understanding, and multimodal coding. Furthermore, when using synthetic data from our Socratic-Generator-32B to train student models, they outperform those trained on data from state-of-the-art commercial LLMs on these benchmarks, including Qwen3-235B-A22B, DeepSeek-V3.1-671B, GPT-5, Gemini-2.5-Pro, Grok-4, and Claude-4.1-Opus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qMVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Socratic-Zero, a framework for bootstrapping mathematical reasoning from seed data through three agents; A Solver (Qwen3-8B-Base) is trained with DPO on its own trajectories. A frozen Teacher (Qwen3-235B-A22B) verifies answers and rewrites failed problems into new targeted ones. A Generator (Qwen3-32B) distills the Teacher’s problem rewriting through SFT, with each example weighted by how close the Solver’s success rate on the new problem is to 0.5. The main experiment Starts from a 1,500-example SFT model and 100 seed problems on Math domain, and the proposed method report a +20.2-point average gain on seven math benchmarks, outperforming prior self-play methods. The paper also reports transfer gains on science QA, deep research, SWE-bench, and multimodal chart and code tasks.

Critical strengths
  1. Timely and relevant problem. Self-improvement and synthetic curriculum generation are active directions (e.g., Absolute Zero, R-Zero, SPICE), so a framework that reduces dependence on annotated data is timely and important topic.
  2. A practical idea for amortizing an expensive curriculum designer. Distilling a large Teacher’s failure-driven problem refinement into a smaller, trainable Generator is a sensible engineering idea. Weighted supervised finetuning on the distillation data by how close each problem is to the Solver’s capability boundary is a simple, interpretable way to target useful difficulty.
  3. Broad experimental coverage. The paper examines many components on ablation, including verifier choice, Teacher scale, seed source, Teacher solution injection, DPO vs. GRPO, and reward shape. This breadth would make the work informative to practitioners.
Critical weaknesses
  1. The empirical evidence is not reliable enough to support the main claims. For example, the reported base accuracies of Qwen3-8B-Base (48.8 on MATH-500, 63.4 on GSM8K) are far below commonly reported values for this model; both benchmarks are near saturation for recent models. This inflates the reported gains and suggests a non-standard evaluation setup. Despite this ambiguity on the evaluation setup, author used reported number for other baselines and provide "approximate estimates" for methods whose per-benchmark breakdown is not explicitly reported, without clear explanation on which baseline numbers were actually reproduced under the same base model, SFT initialization and evaluation protocol, and which were estimated. 1.2. Insufficient statistical support despite small differences. Some table shows small differences, e.g. in 0.4 gain over the second best line on table 4, the differences are reported without variance. 1.3 I might misunderstood, but In Table5, the generator, which is trained to imitate the Teacher, ends up outperforming the teacher. This is counter-intuitive result of distillation but the paper didn't analyze it.

  2. Overstated framing and limited novelty relative to prior self-play work. For example, the title highlighted the method as "data-free", yet the method requires a cold-start SFT on 1,500 high-quality MATH problem-solution pairs, 100 carefully curated seeds. Also, the introduction criticizes reliance on curated data and on “human-curated knowledge boundaries”, but knowledge boundary moves from human annotation to the 235B-scale Teacher and the method does not show self-improvement beyond it. Although the method is presented as "paradigm-shifting", co-evolutionary task generation and solving is already established (Absolute Zero, R-Zero, SPICE, SPIRAL). The remaining components are standard: preference optimization on self-generated trajectories with teacher-provided positives, which closely resembles RL with teacher-guided or context distillation, and SFT distillation of a teacher’s outputs with a smooth difficulty weight. The work reads as a variant of existing self-play with added frontier-teacher distillation.

  3. The true self-play setting underperforms. Although the paper positions itself as self-play method, it relies on large-scale Teacher model. And when the Teacher matches the Solver in scale (Qwen3-8B, Table 7a), the gain is +10.7, below Absolute Zero (+16.7) and R-Zero (+18.8). As those baselines operate without a much larger external model, this result raises a concern about whether the method fits for true self-play frame.

  4. Table 2 is uninterpretable, as it has no baselines. It is also unclear whether these are math-trained models evaluated out of domain or separate models trained with domain-specific Socratic-Zero loops; the base models and tasks differ (e.g., Qwen2.5-7B-Instruct for deep research, a VL model for charts). Gains such as +36 on NQ from a math curriculum would need substantial explanation.

[minor]

  • Figure 1(b) uses GPT- and Gemini-style logos for the Teacher and Generator, while the paper uses open-weight Qwen models. This is misleading, because whether a component can be fine-tuned is central to the method.
  • Figure 2 occupies a lot of space but conveys little concrete information. The geometric shapes and bookshelf icons do not relate to the math setting. A concrete example of a failure, its refinement and its utility weight would be more informative.
  • Calling a fixed, non-optimized Teacher an “agent” in a “co-evolution” framework is a stretch. Only the Solver and Generator evolve.
Questions for authors
  • Which rows in Table 1 were reproduced by author under the matched same base model, SFT initialization, and which were estimated?
  • Why are the base-model accuracies on MATH-500 and GSM8K so much lower than published figures for Qwen3-8B-Base? What prompt, answer extraction and generation length were used, and do the gains persist under a standard evaluation?
  • Since the core difference from other self-play is the usage of Teacher model, was a standard distillation baseline evaluated, e.g., SFT on Teacher-generated problems and solutions from the same 235B model at matched data volume?
  • Regarding the verification, is V(q, y) an external checker given to the Teacher, or the Teacher’s own judgment? For Teacher-generated problems, how is the correctness of new problems and answers ensured?
  • Section 4.2.1 says human annotators judged validity. If humans were involved, please describe, the number of problems, the protocol and inter-annotator agreement. Also, as the domain is mainly about complicated math problem, please describe how the author make sure their expertise on math domain.
  • In table2, was each model trained on math and evaluated out of domain, or trained with a domain-specific loop? What are the seeds, Teacher and verifier for non-math domains, and how do appropriate baselines (e.g., the same amount of Teacher distillation) perform?
  • Can authors report variance across seeds for all main tables, and include a compute-matched comparison that accounts for Teacher inference cost?
  • The claim that Absolute Zero and R-Zero “lack effective quality control” should be made precise. Both include filtering mechanisms, such as executor-based validation and difficulty and repetition filtering, so please specify which aspect is missing and how Socratic-Zero addresses it.
9 comments
AuthorsOpenReview ↗

Response to Reviewer qMVd (Part 1 of 3)

Thank you for the careful review. We address the concerns below; key manuscript revisions are marked in blue.

Q1. About Baselines and Evaluation Settings. All trained baselines in Table 1 except SPICE were reproduced across all seven mathematical benchmarks: Static Augmentation, Distillation (memath_math), DPO (Polaris), DPO (MATH-full), LLM2LLM, Absolute Zero, SPIRAL, and R-Zero. They share our Qwen3-8B-Base checkpoint, 1,500-pair MATH Level-5 SFT initialization, verifier, and zero-shot Mean@32 evaluation; the Base Model and Initial SFT rows were also evaluated locally. SPICE's training-document collection was unavailable, so its scores are estimates based on published results and locally evaluated reference checkpoints. Its approximately 82K training instances are estimated from its reported configuration; other baseline data scales follow their original papers.

The base-score discrepancy comes from evaluation settings. We use the Appendix B.1 zero-shot prompt with a chat template, reasoning steps and a boxed answer; temperature 0.7, top-p 0.9, and at most 8,192 new tokens. Mean@32 averages correctness over 32 independent responses per problem and then over problems. MathRule extracts and normalizes answers against benchmark references, with model checks for ambiguous cases. The official Qwen3 report uses four-shot chain-of-thought prompting. With that setup and greedy decoding, our GSM8K score is 89.74%, matching the official 89.84% within 0.10 points. Our reported 35.9% → 56.1% gain uses the same zero-shot Mean@32 protocol throughout.

The requested direct-distillation baseline was already evaluated; we apologize that it was not presented prominently enough. The first three conditions below share the 235B Teacher, 8B Solver, 100 seeds, SFT initialization and approximately 7.9K generated examples. Direct distillation uses Teacher problem–solution pairs generated in advance; adaptive SFT uses verified successful Solver trajectories; the full method uses the adaptive curriculum with DPO.

Training settingSolver objectiveGenerated training examplesAverage accuracy (%)
Direct Teacher distillationSFT~7.9K39.1
Adaptive-curriculum SFT variantSFT~7.9K48.9
Socratic-ZeroDPO~7.9K56.1
LLM2LLM (original Table 1)SFT~16K40.9

The full method exceeds direct distillation by 17.0 points at matched data volume. We clarified sources and evaluation in Section 4.1 (p. 5, lines 252–263), Table 1 (p. 6, lines 270–276), and Appendix H (p. 20, lines 1032–1044), and brought the distillation comparison into Table 8 (p. 9, lines 463–478; protocol: Appendix Q, p. 26, lines 1387–1393).

Q2. About Verification and Human Calibration. MathRule is the primary checker in V(q, y): it extracts and normalizes the Solver's final answer against the curriculum reference, which is implicit in this notation. During bootstrap, the Teacher supplies supplementary semantic judgments when rule-based extraction is insufficient or ambiguous. The 100 seed problem–solution pairs provide initial references. For a refined problem, the Teacher constructs its reference from the parent problem and solution, the failed Solver trajectory, and the refinement context. We sample three independent Teacher solutions and retain pairs with majority-consistent final answers. If all eight Solver attempts fail, the Teacher re-solves and reviews the problem for mathematical errors, ambiguity and missing information; failed pairs are excluded from Solver and Generator training. Rule/model disagreements also trigger checks. During continuation, the Generator takes over reference construction and model-side verification, with MathRule still primary.

We apologize for the unclear human-evaluation description. Human review calibrated the judge's prompt: ten evaluators with mathematics-competition experience blindly reviewed 1,000 model-accepted problems in total and cross-checked model-flagged cases. Each problem received three judgments, resolved by majority vote, checking mathematical validity, clarity and answer definiteness against references. After prompt calibration, model and human judgments showed good agreement; the calibrated model judge performed the full Table 3 evaluation. Section 4.2.1 now states this correctly (p. 7, line 364). The verification and filtering procedures are clarified in Section 3.1 (p. 4, lines 167–175) and Appendix I (pp. 20–21, lines 1072–1119).

AuthorsOpenReview ↗

Response to Reviewer qMVd (Part 2 of 3)

Q3. About Domain-Specific Training and Baselines. Table 2 reports separate domain-specific Teacher–Student training. We apologize that this scope was unclear. Qwen3-235B-A22B teaches the three text domains; Qwen3-VL-235B-A22B teaches the two multimodal domains. Students use DPO with rule-based verification, including execution feedback for software engineering. Generator training and continuation are evaluated in the mathematical experiments. The four Instruct-based settings start directly from seeds, without the 1,500-pair SFT.

DomainStudent modelSeed initializationGenerated training examples
Science reasoningQwen3-8B-Base100 SciBench problems5K
Deep researchQwen2.5-7B-Instruct500 DeepDive tasks30K
Software engineeringQwen3-4B-Instruct-2507128 SWE-smith tasks, one per repository36K
Multimodal chart reasoningQwen2.5-VL-7B-Instruct100 ChartQAPro examples5K
Multimodal codingQwen2.5-VL-7B-Instruct430 examples from DesignBench’s screenshot-to-code subset10K

Seeds initialize the curriculum; generated examples are the training data. In SWE, we compare both data source and training objective using Qwen3-4B-Instruct-2507 and the same SWE-bench Verified harness. Direct SWE-smith training uses its full ~50K task pool, which also supplies our 128 seeds, one per repository. Synthesized-data SFT uses the same 36K examples as our method. Both controls match our Solver training budget; R-Zero, SPIRAL and Absolute Zero are reproduced at 36K examples.

MethodTraining task poolSWE-bench Verified (%)
Base—21.6
R-Zero36K21.3
SPIRAL36K24.3
Absolute Zero36K24.9
Direct training on SWE-smith~50K25.3
SFT on Socratic-Zero data36K27.4
Socratic-Zero36K30.0

Our gains over SWE-smith and synthesized-data SFT are 4.7 and 2.6 points. We clarified settings in Table 2 (p. 6, lines 291–295), added the SWE comparison to Table 9 (p. 10, lines 486–497), and documented models, seeds, data and verification in Appendix S (p. 27, Table 20; control protocols: lines 1449–1456).

Q4. About Repeated Runs, Compute, and the All-8B System. We report three independent runs for the main Solver (56.1 ± 0.2%) and each 47K continuation branch (Teacher: 62.8 ± 0.3%; Generator: 67.5 ± 0.2%), with mean ± standard deviation across runs. Continuation starts from a shared checkpoint. The separate 10K-problem data-utility evaluation uses five downstream Student training runs: 39.61 ± 0.17% for Socratic-Generator-32B versus 38.79 ± 0.23% for Claude-4.1-Opus. These results appear in Table 1 (p. 6, line 276), Tables 5 and 7 (pp. 8–9), and below Table 4 (p. 7).

Table 13 estimates total compute in equivalent A100 GPU hours, including Teacher inference. All our configurations use an 8B Solver; Teacher-only rows report bootstrap, while Generator rows include Generator training and guided continuation. The breakdown is Solver training + Teacher inference (+ Generator training).

MethodEstimated total cost (A100h)Average accuracy (%)
Absolute Zero~38452.6
SPIRAL~67247.0
R-Zero~57654.7
SPICE>100053.8 (estimate)
Socratic-Zero (8B Teacher)~98 = 96 + 247.6
Socratic-Zero (235B Teacher)~112 = 96 + 1656.1
Socratic-Zero (8B Teacher, 8B Generator)~382 = 320 + 2 + 6059.3
Socratic-Zero (235B Teacher, 32B Generator)~492 = 332 + 16 + 14467.5

The complete all-8B system achieves 59.3%, exceeding Absolute Zero by 6.7 points at comparable estimated compute (382 vs. 384 A100h) and R-Zero by 4.6 points at lower compute (382 vs. 576 A100h). The 47.6% row isolates 8B-Teacher bootstrap; its compute is about one-quarter of Absolute Zero's and one-sixth of R-Zero's. We apologize for the typo in Table 7a: the correct bootstrap score and gain are 47.6% and +11.7 points over 35.9%. We corrected that entry (p. 9, line 441) and highlighted the complete-system comparison in the main text (p. 10, lines 514–521), with cost accounting in Appendix A.4 (p. 14, lines 736–741; Table 13). SPICE follows the estimated-score protocol in Q1.

AuthorsOpenReview ↗

Response to Reviewer qMVd (Part 3 of 3)

Q5. About Quality-Control Mechanisms. Thank you for the suggestion to make the comparison more precise. Absolute Zero uses execution-based validation, including program-integrity and determinism checks; R-Zero uses Solver answer consistency, difficulty filtering, repetition penalties and format checks. Our contribution connects the content of a failed Solver trajectory to targeted mathematical refinement and reference validation: the Teacher uses the parent problem, reference and failure analysis to construct a revised problem–solution pair, which passes consistency filtering, MathRule checking and failure-triggered review. The Generator learns this refinement policy through difficulty-weighted training. We made this distinction explicit in the Introduction (p. 1, lines 43–50) and Related Work (p. 2, lines 94–99); Q2 and Appendix I (pp. 20–21) describe the validation procedure.

Q6. About the Generator's Curriculum Advantage. The Generator learns through reward-weighted distillation: Teacher refinements provide training examples, while the Solver's success rate supplies their utility reward. WSFT prioritizes problems near 50% success, specializing the Generator in curricula that challenge the current Solver. This follows the reward-guided intuition of offline reinforcement learning and is implemented through weighted SFT. The fixed Teacher provides refinement examples; the trainable Generator learns which refinements are most useful for the Solver, explaining its advantage in curriculum utility.

Both continuation branches start from the same Qwen3-8B Solver checkpoint at 56.1% and use DPO at matched cumulative data scales. Qwen3-235B-A22B supplies the Teacher branch; Socratic-Generator-32B supplies the Generator branch.

Cumulative examplesTeacher-guided Solver (%)Generator-guided Solver (%)Difference (pp)
~8K, shared starting point56.156.1—
~16K59.259.5+0.3
~28K62.364.1+1.8
~47K62.8 ± 0.367.5 ± 0.2+4.7

The 47K statistics cover three independent runs per branch. Table 4 separately evaluates generated-data utility using the same DeepSeek-R1-Distill-Llama-8B Student: 36.04% for our Generator versus 35.20% for the Teacher. The weighted objective is defined in Section 3.3 (p. 5, lines 221–246), trainability in Appendix N.1 (p. 24, lines 1246–1251), and continuation roles and statistics in Tables 5 and 7 (pp. 8–9).

Q7. About Terminology and Presentation. “Data-free” describes automatic curriculum expansion from a small seed set, without additional externally curated problem–solution training data. The 1,500-example SFT primarily aligns instruction following for the Base Solver; it is optional when the starting model already has that ability. Table 2's deep research, SWE, chart reasoning and multimodal coding experiments start directly from Instruct models and small seed curricula. This follows the zero-style self-play convention: related methods also specify prompts and initialization, such as R-Zero's Solver/Challenger templates (Appendix A.2) and Absolute Zero's prompted buffer initialization (Section 3.3.1).

Within this loop, “co-evolution” refers to the Solver and its learned curriculum policy: the Solver and Generator are optimized, while the Teacher remains fixed during bootstrap. “Autonomous” describes the automatic generation–verification–training cycle driven by Solver feedback. Our contribution is learning and transferring a failure-conditioned refinement policy, supported by the direct-distillation, all-8B and Generator–Teacher comparisons above. We clarified these definitions and initialization settings in the Introduction (p. 2, lines 69–75), Section 3.1 (p. 3, lines 151–161), Appendix O (p. 25, lines 1335–1338), and Appendix S (p. 27, lines 1426–1430).

For the figures, the icons were schematic illustrations of roles. We will label Figure 1 with the actual Qwen3-235B-A22B Teacher, Qwen3-8B-Base Solver and Qwen3-32B Generator, clearly distinguishing fixed and trainable components. Figure 2 will include a concrete mathematical example linking a failed Solver attempt, targeted problem refinement and the success rate used for its utility weight. These changes make the same learning loop explicit in both terminology and illustration.

Reviewer_CJznOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Socratic-Zero, an automatic curriculum learning framework built around a Solver, a Teacher, and a Generator. The Solver learns from preferred and rejected reasoning trajectories through DPO. The fixed Teacher generates new problems based on the Solver’s failures and checks the answers. The Generator learns the Teacher’s problem-generation strategy through weighted supervised fine-tuning, with more weight given to problems near the Solver’s current ability level. The method achieves strong results with a relatively small amount of generated data.

Critical strengths
  1. The main idea is interesting. Instead of generating a fixed training set, the method uses the Solver’s current failures to update the curriculum during training.

  2. The difficulty control is reasonable. The method gives the highest weight to problems with a Solver success rate near 50%, which matches the idea of training near the model’s current ability boundary.

  3. The evaluation covers several parts of the framework. The paper studies Solver performance, problem validity, Teacher scale, reward design, data scaling, and whether the Generator can replace the Teacher.

  4. The Generator has practical value. Distilling the problem-generation strategy from a large Teacher into a 32B Generator may reduce the long-term cost of curriculum generation.

Critical weaknesses
  1. The “data-free” and “fully autonomous” claims are too strong. The method still uses 1,500 MATH Level-5 examples for cold-start SFT, 100 selected seed problems, and a Qwen3-235B Teacher. A more accurate description would be automatic evolution after a small-data cold start.

  2. The three-agent co-evolution claim is somewhat overstated. The Teacher remains fixed, and the main-stage curriculum is mostly generated by the Teacher. The Generator only replaces the Teacher in later continuation experiments. The main process is closer to joint Solver and Generator training under a fixed Teacher.

  3. The reward ablation shows only small differences. The Gaussian reward improves the average result by only about 0.2–0.4 points over the other weighting functions. This provides limited evidence that a 50% success rate is the best curriculum difficulty.

  4. The cross-domain experiments lack important details. The science, deep research, software engineering, and multimodal experiments use different models, but the paper does not clearly explain whether they follow the same three-agent pipeline, how much data is generated, or how correctness is checked.

Questions for authors
  1. Under the same Teacher, generated-data size, and compute budget, does the adaptive curriculum clearly outperform one-time static data generation?
  2. How do the authors define “co-evolution” when the Teacher remains fixed and the Generator does not directly generate the main-stage curriculum?
  3. Is the target success rate of 0.5 consistently effective across different Solver sizes and task domains?
  4. Do the cross-domain experiments use the complete Solver–Teacher–Generator loop? Please provide the data size, training process, and verification method for each domain.
  5. The Generator-guided setting clearly outperforms Teacher-guided training at around 47K examples. Is this result based on one run or multiple independent runs? Can the authors report the mean and standard deviation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CJzn (Part 1 of 2)

Thank you for the careful review. Key clarifications in the revised manuscript are marked in blue.

Q1. About One-Time Static Generation

We have evaluated the one-time static-generation baseline with the same Qwen3-235B-A22B Teacher, Qwen3-8B-Base Solver, 100 seed problems, and approximately 7.9K generated training examples. All configurations start from the same checkpoint after cold-start SFT on 1,500 MATH Level-5 problem–solution pairs and use the same seven-benchmark Mean@32 evaluation.

Training settingCurriculum constructionSolver objectiveAvg. (%)
One-time static generation / direct Teacher distillationGenerate the training set in advanceSFT39.1
Adaptive-curriculum SFT variantUpdate problems using Solver feedbackSFT48.9
Socratic-ZeroUpdate problems using Solver feedbackDPO56.1

In the static baseline, the Teacher generates problem–solution pairs before Solver training. In the adaptive SFT variant, the curriculum evolves with Solver feedback, and SFT uses verified successful Solver trajectories. Socratic-Zero uses the adaptive curriculum and trains the Solver on verified preference pairs with DPO.

The full method exceeds one-time static generation by 17.0 percentage points at the same Teacher and generated-data budget; the adaptive SFT result also supports the value of feedback-guided data generation. Pipeline compute is reported separately in Table 13 as estimated A100 GPU hours, including Teacher inference.

We now present this comparison in Table 8 (p. 9, lines 463–478) and specify the training protocol in Appendix Q (p. 26, lines 1387–1393). The key evidence and clarifications are marked in blue.


Q2. About the Definition of Co-Evolution

Co-evolution refers to the joint development of the Solver and its Solver-dependent curriculum policy. The Solver and Generator are the trainable components, while the Teacher provides guidance with fixed parameters during bootstrap. The two phases have distinct roles:

  1. Teacher-guided bootstrap. The Teacher generates targeted refinements from the Solver's current failures. The Solver learns through DPO, while the Generator learns the refinement policy through weighted SFT, using the current Solver's success rates to weight training examples. As the Solver improves, its failures, the generated curriculum, and the Generator's training signals change accordingly. The Generator therefore learns throughout the main phase.

  2. Generator-guided continuation. The trained Generator takes over curriculum generation and produces new problems conditioned on Solver feedback. These problems support further Solver training, directly connecting the learned curriculum policy to subsequent Solver improvement.

We have explicitly distinguished these phases and the trainable components in Section 3.1 (p. 3, lines 151–155 and the following phase description) and the Figure 2 caption, with the key clarifications marked in blue.


Q3. About Difficulty Targets Across Model Scales and Domains

We use the same target μ=0.5 and Gaussian width σ=0.2 across the reported mathematical Solver scales. Difficulty is measured by each problem's success rate under the current Solver, so the problems receiving high weight adapt to the model's capabilities. The shared setting gives the following results (Table 19):

SolverReference checkpointReference accuracy (%)Stage-3 accuracy (%)Gain (points)
Qwen3-8B-BaseCold-start SFT35.956.1+20.2
GLM4-9B-ChatZero-shot35.252.3+17.1
Qwen3-14B-BaseZero-shot43.060.3+17.3

Scores are averages across the seven mathematical benchmarks; each gain uses the reference checkpoint named in its row.

The separate Generator-weighting ablation in Table 6 holds σ=0.2 fixed and varies the target μ:

Target μ0.30.40.50.60.7
Average accuracy (%)35.3235.3735.7235.5035.43

The midpoint achieves the highest average among the tested centers, with smooth sensitivity to neighboring targets. It favors problems that yield both successful and failed Solver attempts, providing useful contrasts for preference learning.

For other task domains, Table 2 evaluates adaptive curriculum generation and DPO training in the Teacher–Student setting with rule-based verification. The Generator-weighting results above concern the mathematical experiments; the domain-specific training settings are detailed in Q4 and Appendix S.

We have clarified the shared setting in the method (p. 5, lines 234–236) and added the Solver-scale results and midpoint rationale in Appendix R (Table 19, p. 27). The key clarifications are marked in blue.

AuthorsOpenReview ↗

Response to Reviewer CJzn (Part 2 of 2)

Q4. About Domain-Specific Training Settings

The five domain extensions in Table 2 evaluate the Teacher–Student adaptive-curriculum component of Socratic-Zero. Generator training and Generator-guided continuation are evaluated in the mathematical experiments. For the domain extensions, the Teacher is Qwen3-235B-A22B for science reasoning, deep research, and software engineering, and Qwen3-VL-235B-A22B for the two multimodal domains.

DomainStudent checkpointCurriculum seedsGenerated training examplesVerification
Science reasoningQwen3-8B-Base100 SciBench problems5KRule-based
Deep researchQwen2.5-7B-Instruct500 DeepDive tasks30KRule-based
Software engineeringQwen3-4B-Instruct-2507128 SWE-smith tasks, one per repository, with the corresponding repositories36KExecution-based
Multimodal chart reasoningQwen2.5-VL-7B-Instruct100 ChartQAPro examples5KRule-based
Multimodal codingQwen2.5-VL-7B-Instruct430 DesignBench screenshot-to-code tasks10KRule-based

Training process. The Instruct checkpoints used in deep research, software engineering, and both multimodal domains begin directly from their seed curricula, without the preliminary 1,500-example SFT. Each Student is trained within its target domain using domain-specific seeds and prompts. The Teacher generates or refines problems using Student feedback; verified Student trajectories provide preference pairs for DPO updates. Correctness feedback is rule-based in all five domains, with execution feedback used for software engineering. Seed counts describe curriculum initialization, while the training-data column counts generated examples. Table 2 reports each Student's performance after this domain-specific training.

We have clarified the experimental scope and data volumes in the Table 2 caption (p. 6, lines 291–295) and main text, and added the domain-specific settings in Appendix S (p. 27, lines 1426–1433; Table 20). The key clarifications are marked in blue.


Q5. About Repeated Runs for the Continuation Comparison

Both continuation settings were evaluated over three independent runs each. They start from the same Qwen3-8B-Base Solver checkpoint, which achieves 56.1% at approximately 8K cumulative training examples, and continue to approximately 47K cumulative examples using the same DPO objective.

Continuation setting / curriculum sourceCumulative training examplesIndependent runsMean ± standard deviation (%)
Teacher-guided: Qwen3-235B-A22B-Instruct-2507~47K362.8 ± 0.3
Generator-guided: Socratic-Generator-32B~47K367.5 ± 0.2

For each run, the metric is the average accuracy across the seven mathematical benchmarks. We then report its mean and standard deviation across the three independent runs. Generator-guided continuation achieves a 4.7-percentage-point improvement in mean accuracy over Teacher-guided continuation.

We have added the run counts and statistics to the continuation-results discussion and the captions of Table 5 (p. 8, lines 378–383) and Table 7 (p. 9, lines 432–436). The key additions are marked in blue.


Q6. About Framing and Initialization

The 1,500-example SFT primarily supports instruction following for the Base Solver used in our main experiment. When the starting checkpoint already has sufficient instruction-following ability, this stage can be skipped and training begins directly from a small seed curriculum. The Instruct-based experiments in Table 2 follow this route: deep research, software engineering, multimodal chart reasoning, and multimodal coding.

“Data-free” describes the generation of the expanding training curriculum from a small seed set, without requiring additional externally curated problem–solution data. We follow the self-generated-curriculum paradigm of zero-style self-play. Related methods also specify task prompts and templates: R-Zero, Appendix A.2 provides Solver/Challenger prompts, and Absolute Zero, Section 3.3.1 specifies prompted buffer initialization. Our seed initialization and model-specific instruction alignment are explicitly reported.

“Autonomous” refers to the generation–verification–training loop, which expands the curriculum and updates the models using Solver feedback.

We retain this terminology and clarify the optional instruction-alignment stage in the introduction (p. 2, lines 69–75) and Appendix O (p. 25, lines 1335–1338). Appendix S (p. 27, lines 1426–1430) specifies the Instruct-based settings that start directly from seeds. The key clarifications are marked in blue.

Thank you for the detailed response. The additional experiments and clarifications address my concerns to some extent. I will maintain my current score.

AuthorsOpenReview ↗

Appreciation for Your Constructive Feedback and Positive Assessment

We sincerely thank you for the time and effort you have dedicated to reviewing our paper and for your insightful and constructive feedback, which has been invaluable in improving the quality of our work. We will carefully incorporate the additional experimental results and clarifications arising from our discussion into the final version of the paper. We also greatly appreciate your positive assessment of our work.

Reviewer_CafiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a three-agent loop consisting of a Solver, Teacher, and Generator. The Solver learns from teacher-labeled correct and incorrect trajectories through DPO. A frozen Qwen3-235B Teacher rewrites failed problems, while a Qwen3-32B Generator distills this strategy through weighted SFT, emphasizing problems with a solver success rate near 0.5. The Solver is first trained on 1,500 MATH Level-5 examples, then evolves from 100 seed problems to produce about 7.9K training examples. The resulting Qwen3-8B Solver reportedly gains 20.2 points across seven math benchmarks and outperforms Absolute Zero, R-Zero, and SPICE. The Generator’s data also shows higher downstream utility than data generated by GPT-5 and Claude-4.1-Opus.

Critical strengths
  1. The method distills the teacher’s problem-generation strategy into a smaller generator and prioritizes problems with a solver success rate near 0.5. The design is intuitive and supported by the ablations in Table 6.
  2. Table 7a studies different teacher sizes, while Appendix A.3 tests several verifier configurations, providing useful analysis.
Critical weaknesses
  1. Some baseline results in Table 1 are estimated, and the main method compares a 235B teacher against 8B-scale self-play methods. In the more size-comparable setting, the 8B-teacher variant reaches only 47.6, below R-Zero and Absolute Zero.
  2. The generator is defined to produce new problems only, while DPO still requires verification and reference solutions. Table 5 does not explain where these signals come from, so it is unclear whether the generator truly replaces the teacher.
Questions for authors
  1. Which baseline results in Table 1 are reproduced and which are estimated? Do these methods use the same SFT initialization and verifier?

  2. During generator-guided training, who provides verification and reference solutions? If the 235B teacher is still required, please revise the claim that it is replaced and explain why the same 56.1 result has different gains in Tables 7a and 11.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cafi

Thank you for the careful review. Key clarifications in the revised manuscript are marked in blue.

Q1. About Baseline Provenance and Shared Settings

All trained baselines in Table 1 except SPICE were fully reproduced on all seven mathematical benchmarks. They use the same Qwen3-8B-Base checkpoint, cold-start SFT on 1,500 MATH Level-5 problem–solution pairs, verifier, and zero-shot Mean@32 evaluation protocol as Socratic-Zero.

SPICE was not reproduced end to end because the training-document collection used in its original experiments was not publicly available. Its scores are approximate estimates based on the published results and locally evaluated reference checkpoints. Its training-data scale (approximately 82K instances) is estimated from the configuration reported in the original paper. The data scales of the other baselines likewise follow their original papers.

We have clarified these sources and shared settings in Section 4.1 (p. 5, lines 252–263), the Table 1 caption (p. 6, lines 270–276), and Appendix H (p. 20, lines 1032–1038). The key clarifications are marked in blue.


Q2. About Generator Replacement and Gain Calculation

During Generator-guided continuation, Socratic-Generator-32B takes over problem generation, reference-solution construction, and model-side verification, with MathRule providing the primary correctness checks. The Solver forms DPO preference pairs from verified responses; when all sampled attempts fail, the Generator-provided reference serves as the positive example. The 235B Teacher is used in the preceding bootstrap phase, and continuation proceeds with the Solver, Generator, and MathRule. We have clarified these roles in the method (p. 4, lines 172–175) and Table 5 caption (p. 8, lines 378–383), with the key revisions marked in blue.

Both tables use the same 35.9 SFT baseline, so the correct gain is 56.1 − 35.9 = 20.2 percentage points. We apologize for the gain-calculation error in the original Table 11 (now Table 13). We have corrected +13.5 to +20.2 and explicitly stated the reference baseline in its caption (p. 14, lines 745–749). The underlying 56.1 result is unchanged.


Q3. About Comparisons at the 8B Scale

The 47.6% result in Table 7a corresponds to the 8B-Teacher bootstrap configuration. The complete all-8B system reaches 59.3%. We summarize the corresponding data volumes, estimated compute, and average accuracy from Tables 1, 5, and 13 below; all configurations use an 8B Solver.

Method / configurationGenerated training examplesEstimated compute (A100 GPU hours)Average accuracy (%)
Absolute Zero~40K38452.6
SPIRAL~51K67247.0
R-Zero~24K57654.7
SPICE~82K>1,00053.8*
Ours: 8B Teacher, bootstrap~4.8K (single round)9847.6
Ours: 235B Teacher, main result~7.9K11256.1
Ours: 8B Teacher + 8B Generator, complete system~16K (single round)38259.3
Ours: 235B Teacher + 32B Generator, complete system~47K49267.5

Here, "bootstrap/main result" reports Teacher-guided Solver training before Generator-guided continuation. "Complete system" includes bootstrap with the listed Teacher followed by continued Solver training guided by the listed Generator. The model sizes identify these respective roles; the Solver remains 8B throughout.

Data volumes are approximate and cumulative unless marked "single round." Our compute estimates include Teacher inference and, where applicable, Generator training. *SPICE accuracy is an approximate estimate, as explained in Q1.

At the 8B scale, the bootstrap configuration uses approximately one-quarter of Absolute Zero's estimated compute and one-sixth of R-Zero's. The complete all-8B system improves accuracy over Absolute Zero by 6.7 points at comparable estimated compute (382 vs. 384 A100 GPU hours), and over R-Zero by 4.6 points with lower estimated compute (382 vs. 576).

In data efficiency, the main configuration achieves higher accuracy than Absolute Zero and R-Zero with 67–80% fewer generated training examples (7.9K vs. 40K and 24K). Generator-guided continuation further raises accuracy to 67.5% at 47K cumulative examples, demonstrating sustained gains as the curriculum expands.

The revised main text highlights the complete all-8B comparison (p. 10, lines 514–521), with the key clarifications marked in blue.

Dear Reviewer Cafi,

Thank you again for the time and effort you dedicated to reviewing our paper, as well as your constructive feedback. Following your suggestions, we have provided comprehensive additional results and follow-up discussions in our response, and we would love to engage with you further.

As the author-reviewer discussion period is coming to a close soon, we wanted to gently check in and see if our responses and new additions have addressed your concerns. If any questions remain, we would be more than happy to clarify further. We truly appreciate your time and look forward to hearing your thoughts!

Best regards, The Authors

Thank you for the additional results and detailed clarifications. Your responses have addressed my concerns about the baseline comparisons, the Generator’s role during continued training, and the consistency of the reported gains. I therefore maintain my positive rating of weak acceptance.

Dear Reviewer Cafi,

Thank you for your constructive feedback and the time and effort you devoted to reviewing our submission. We appreciate your positive assessment of our work and are glad that our additional results and clarifications addressed your concerns about the baseline comparisons, the Generator’s role during continued training, and the consistency of the reported gains. We will incorporate the points raised during our discussion into the revised manuscript. Thank you again for your thoughtful review.

Best regards,
The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We thank all three reviewers for their thoughtful feedback and for recognizing the value of adaptive curriculum generation, interpretable difficulty weighting, and our broad experimental analysis.

We have revised the manuscript to clarify the experimental settings, make the key comparisons more prominent, and incorporate additional evidence. Key substantive revisions are marked in blue. The table below summarizes how we address the main shared concerns; full protocols, results, and page/line references appear in our individual replies.

ReviewersConcernResponse
Cafi, qMVdBaseline provenance and evaluation consistencyWe have clarified baseline sources and the shared evaluation protocol. Section 4.1, Table 1 and Appendix H now distinguish reproduced results from SPICE estimates, specify common initialization and verification, and explain published-score differences with a GSM8K alignment check.
CJzn, qMVdAdaptive generation versus direct Teacher distillationWe have brought the matched-data distillation comparison into main-text Table 8. Using the same Teacher and generated-data volume, the comparison separates static distillation, adaptive-curriculum SFT and the full method, making the roles of curriculum construction and training objective explicit.
Cafi, qMVdSame-scale performance and resource efficiencyWe have distinguished bootstrap from the complete all-8B system and clarified compute accounting. The main text highlights the complete system’s advantage at comparable estimated compute; Table 13 includes Teacher inference and Generator training and specifies how data volumes are counted.
Cafi, CJzn, qMVdGenerator replacement, curriculum advantage and repeatabilityWe have clarified the Generator’s takeover and added repeated-run statistics. The method explains its generation, reference-construction and verification roles, with MathRule as the primary checker. Tables 5 and 7 report the continuation comparison over three independent runs per branch, supporting the learned Generator’s curriculum advantage.
CJzn, qMVdCross-domain settings and baseline comparisonsWe have documented all five domain settings and added SWE controls. Table 2 and Appendix S specify the Teacher–Student scope, models, seeds, data volumes and verification. Main-text Table 9 compares the full method with SFT on the same synthesized data and direct training on the larger SWE-smith task pool.
CJzn, qMVdMeaning of data-free, autonomous and co-evolutionWe have clarified the terminology and when instruction-alignment SFT is needed. The Introduction, Section 3.1 and Appendix O distinguish the fixed Teacher from the trainable Solver and Generator. The four Instruct-based settings start directly from seeds, illustrating that the 1,500-example SFT is optional.
qMVdVerification, human assessment and quality controlWe have made the verification pipeline and human-calibration role explicit. Section 4.2.1 now states that a human-calibrated model judge supplies Table 3’s validity metric. Appendix I details reference construction, consistency filtering and failure-triggered checks, while Related Work precisely distinguishes these from prior filtering mechanisms.

We have also corrected the identified typographical and gain-calculation errors and aligned the corresponding tables and text. We appreciate the reviewers' suggestions, which have helped make the evidence and experimental scope clearer throughout the paper.

Best regards, The Authors

AuthorsOpenReview ↗

General Response to the Follow-up Discussion

Dear Area Chair and Reviewers,

Thank you for the questions raised during the discussion. Reviewer qMVd’s follow-up prompted three additional experiments on questions shared across the reviews. The revised manuscript reports the results in Appendix T, pp. 29–31, with the additions highlighted in blue.

1. Training directly from Qwen3.5-9B-Base (Table 21, p. 29). All trained Solvers started from the same base checkpoint without cold-start SFT and used approximately 20K training examples over five iterations. We reproduced Absolute Zero, R-Zero, and SPICE with 4K examples per iteration. Both Socratic-Zero configurations used three Teacher-guided iterations followed by two guided by a 9B Generator. The Teacher was 9B in the all-9B configuration and 27B in the second configuration.

MethodGSM8K 0-shotGSM8K 4-shot CoTMATH-500AIME 2025Avg.
Qwen3.5-9B-Base88.6391.7468.2020.2167.20
Absolute Zero91.5192.4273.8031.1572.22
R-Zero91.3692.3474.2029.7971.92
SPICE91.8193.1075.4033.4473.44
Socratic-Zero, all-9B93.2593.8679.0040.2176.58
Socratic-Zero, 27B Teacher95.1597.1983.2049.0681.15

Scores are percentages. Both GSM8K settings and zero-shot MATH-500 use one greedy answer per problem. AIME 2025 uses zero-shot Avg@32 at temperature 0.7. The final column averages the four settings, including both GSM8K protocols. The all-9B system exceeded the strongest reproduced baseline, SPICE, by 3.14 points on average and 6.77 points on AIME 2025.

For Socratic-Zero, the Teacher consulted Nemotron-CC-Math to generate the first 1,000 seed problems. Subsequent iterations generated their curricula without further document access. Using 1,000 problems drawn directly from the documents as seeds gave comparable results. We also corrected the original Qwen3-8B comparison by attributing the Absolute Zero scores to the SPICE paper and removing the earlier SPICE estimates.

2. Which curriculum decisions help the Solver? (Table 22, p. 30). Two paired ablations started from the same Stage-3 Qwen3-8B Solver checkpoint (56.1%) and continued to approximately 16K cumulative examples. Weighting Generator training by problem difficulty raised the Solver’s average across seven math benchmarks from 57.8% with uniform weighting to 59.5%. In a separate comparison, both Teachers generated new problems at each iteration. Giving the Teacher access to the Solver’s failed attempts raised the average from 57.3% to 59.2%. Appendix T.2 also reports variability in the 10K-problem data-quality evaluation. Across five independent downstream training runs per condition, Socratic-Generator-32B scored 39.61 ± 0.17 and Claude-4.1-Opus scored 38.79 ± 0.23 (mean ± standard deviation).

3. Distillation and self-play beyond mathematics (Tables 23–24, p. 31). Direct distillation trains the Solver on Teacher solutions to existing problems. Socratic-Zero also generates new problems in response to the Solver’s errors. On MMLU-Pro, Socratic-Zero reached 65.8 with 5K examples, compared with 56.3 for distillation on 75.4K ScienceQA-derived examples and 53.4 for R-Zero on approximately 24K. The same ordering held on GPQA and SuperGPQA. In deep research, Socratic-Zero used 30K examples and exceeded distillation on 50K SearchQA tasks by 10 points on NQ, 4 on TriviaQA, and 5 on HotpotQA. The existing software-engineering comparison holds the generated problems fixed. On the same 36K examples, SFT scored 27.4 on SWE-bench Verified and the full method scored 30.0 (Table 9, p. 10).

Best regards,
The Authors

AuthorsOpenReview ↗

General Response at the Close of the Discussion

Dear Area Chair and Reviewers,

Thank you for considering the revised manuscript and the additional experiments. We especially appreciate the support of Reviewers Cafi and CJzn, both of whom maintained their positive ratings (weak acceptance) after reading our responses. Cafi confirmed that we had addressed all three concerns identified in their follow-up. CJzn stated that the additional experiments and clarifications had partially addressed their concerns. We also thank qMVd for reviewing the new comparisons and ablations.

Question raisedReviewer feedbackWhat we clarified or tested
Were the baselines comparable, and where did the scores come from?Cafi confirmed that the baseline-comparison concern was addressed. qMVd welcomed the new 9B comparison and retained reservations about the original table.We corrected the attribution of the original Absolute Zero result and removed the SPICE estimates. In Table 21, we directly reproduced Absolute Zero, R-Zero, and SPICE. All three baselines and both Socratic-Zero configurations used the same Qwen3.5-9B-Base Solver, five iterations, approximately 20K examples per configuration, and the same evaluation protocol within each column.
When does the Generator take over, and what does it do?Cafi confirmed that the Generator’s role during continued training was clear.The revision identifies the handoff from the Teacher. The trained Generator then constructs the next round of questions and reference answers, and MathRule serves as the primary checker of Solver answers.
Are the reported gains consistent across the text and tables?Cafi confirmed that the reporting-consistency concern was resolved.We distinguished newly generated examples from cumulative data and identified the starting checkpoint and endpoint for each continuation comparison.
What does adaptation add? Does the 47K result hold across runs? Which stages transfer across domains?CJzn maintained their positive assessment and reported that the additions partially addressed their concerns overall.Table 8 compares static and adaptive curricula with the same Teacher and data volume. From a shared checkpoint, three independent runs per branch at 47K examples yielded 62.8 ± 0.3% for Teacher-guided and 67.5 ± 0.2% for Generator-guided continuation. We also documented the Teacher–Student protocol, data volume, and verification method for each of the five non-math domains.
Do gains persist with a same-size Teacher and standard prompting?qMVd welcomed the 9B experiment and raised the weighting of GSM8K in the four-setting average.With a 9B Solver, Teacher, and Generator, four-shot GSM8K accuracy rose from 91.74% to 93.86%. On MATH-500 and AIME 2025 separately, all-9B Socratic-Zero scored 79.00% and 40.21%, versus 75.40% and 33.44% for the strongest reproduced baseline on each.
What do failure conditioning and difficulty weighting contribute? How does the method compare beyond math?qMVd acknowledged the uniform-SFT ablation and new domain baselines, while retaining concerns about training-seed variability.From the same 56.1% checkpoint to the same 16K endpoint, giving the Teacher access to failed attempts raised accuracy from 57.3% to 59.2%. Difficulty-weighted Generator training reached 59.5% versus 57.8% with uniform SFT. We added distillation controls in science and deep research and self-play baselines in science and software engineering.

Remaining request: independent training seeds. We consider it beyond the reasonable scope of a rebuttal to retrain every newly requested configuration under multiple seeds. In Table 21 alone, each extra seed requires five rounds of curriculum generation and training for each of five configurations. We completed the requested direct reproductions and controlled ablations. There was insufficient time to repeat every configuration in Tables 21–24 before the discussion deadline.

Our original seven-benchmark math evaluation used Avg@32, averaging 32 sampled answers per problem at a fixed checkpoint. The new 9B comparison likewise used Avg@32 on AIME 2025 and greedy decoding on GSM8K and MATH-500. Across independent training runs, the main 8B result was 56.1 ± 0.2% over three runs, and the 47K continuation comparison above also used three runs per branch. The separate 10K data-quality experiment used five downstream training runs per condition. Given the repeated-run evidence, we expect additional seeds to preserve our main conclusion that the learned Generator improves continued training.

Thank you again for your time and thoughtful feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.