acceptodds
Under review as a conference paper at ICLR 2027

-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Labeled QA Data

Abstract

Deep search agents remain difficult to train at scale due to limited labeled data and weak credit assignment in long-horizon search. Self-play offers a scalable way to reduce data dependence, but conventional approaches optimize models only with sparse outcome rewards, limiting learning efficiency. We find that the task-generation process in self-play can itself serve as a scalable source of privileged supervision for self-distillation. Specifically, self-play naturally produces a Question Construction Path (QCP), which captures the reverse solution process and provides rich intermediate signals for self-distillation without curated privileged information. Leveraging this insight, we propose Privileged Information Self-Play (-Play), a self-evolution framework that couples self-play with self-distillation by directly exploiting privileged information generated during task construction. In -Play, self-play generates training tasks together with QCPs for self-distillation in a low-cost and scalable manner, while self-distillation converts QCPs into token-level supervision to accelerate self-play evolution. This closed-loop interaction enables more efficient model improvement without relying on labeled QA data or curated privileged information. Across multiple benchmarks and model scales, data-free -Play surpasses fully supervised search agents and improves evolutionary efficiency by 1.5–3.1 over conventional self-play.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-play, self-distillation, agentic search

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_tcP9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes π-Play, which augments search-agent self-play with privileged self-distillation. An examiner generates question–answer pairs and question construction paths (QCPs); an EMA teacher conditions on the QCP to provide token-level reverse-KL guidance along student rollouts, while the student also receives outcome-based GRPO updates. Experiments cover three Qwen3 configurations and seven QA benchmarks, with comparisons against supervised and self-play search agents. The paper includes QCP/answer-removal ablations, three-seed results for π-Play, and training-step and per-step timing analyses.

Critical strengths
  1. A useful source of privileged supervision within an existing self-play loop. The main contribution is the reuse of task-construction traces as teacher context. This gives intermediate artifacts a concrete training role without requiring externally labeled QA pairs or a stronger teacher. It is a meaningful combination of examiner–student self-play and privileged self-distillation. The contribution should be understood primarily as the QCP-based supervision source and its integration, since HRPO and feedback-conditioned self-distillation are established components (Yue et al., 2026, https://arxiv.org/abs/2601.07055; Hübotter et al., 2026, https://arxiv.org/abs/2601.20802).

  2. Ablations provide evidence beyond the final aggregate score. Tables 3–4 distinguish QCP guidance from answer-only context and from removing distillation. On Qwen3-4B-Instruct, the full-QCP total is 277.2, versus 265.5 without distillation, 264.1 with the answer alone, and 273.7 after removing answers and answer-bearing snippets. Appendix C also reports three-seed results and a 4.3% mean per-step overhead in the measured 4B-Instruct setting. These controls support a practical benefit from the additional context, although the precise causal role of path structure remains less certain.

Critical weaknesses
  1. QCP correctness and indispensable multi-hop structure are not sufficiently validated. The method treats construction paths as reliable privileged supervision, but the student matching the examiner's answer is not an independent correctness check. The provided examples also expose a gap between intended and realized structure: Figure 7 is labeled hop = 2 but contains two search calls, whereas Appendix E.2 requires n−1 calls; its final question associates the deployment of Lunokhod 2 with the Luna 19 spacecraft even though the displayed supporting passage identifies Luna 21. Figure 8 similarly contains three searches for a nominal three-hop task and asks a question that does not obviously require all preceding links. These examples do not establish that the full dataset is unreliable, but they weaken the claims of verified construction paths and controlled hop complexity. Answer removal is helpful, yet it does not isolate structured paths from the same evidence supplied as an unordered document set.
  2. The precision and scope of the comparative gains need clarification. The final total-score margins over Dr.Zero in Table 1 are 4.5, 10.9, and 5.4, equivalent to approximately 0.64, 1.56, and 0.77 percentage points in the seven-benchmark macro average. Table 10 gives uncertainty for π-Play but single values for the baselines, so it does not establish paired significance. The Qwen3-4B per-benchmark values also differ substantially between Tables 1 and 10 despite almost identical totals, making the checkpoint/seed relationship important. The 1.5–3.1× claim measures steps to a baseline target, whereas timing is reported only for 4B-Instruct; broader wall-clock or GPU-hour savings are therefore not yet established.
Questions for authors
  1. Validation of generated tasks and paths. How are answer correctness, factual relations, actual hop counts, and non-skippable dependencies verified? Please reconcile the Figure 7/8 examples with Appendix E.2, and report an independently checked sample with error and shortcut rates. A matched-context comparison using the same retrieved evidence with path order/structure removed would materially clarify what QCP contributes.
  2. Statistical comparison and checkpoint selection. Could the authors provide paired multi-seed results for Dr.Zero and π-Play, define the ± statistic in Table 10, and explain whether Table 1 and Table 10 use the same training endpoint and checkpoint-selection rule? Please state what data were used to select λ and K, and how evaluation questions were kept separate from generated training tasks.
  3. End-to-end efficiency. For each model size, what are the cumulative wall-clock time or GPU-hours, generated tokens, and search calls required to reach the same evaluation target? Please include examiner training and student-data generation, and clarify whether the step-efficiency target or best checkpoint was selected using test results.
6 comments
AuthorsOpenReview ↗

Authors Response (Part 1/2)

Response


Q1.1 Validation of generated QA pairs and QCPs

Regarding Figures 7 and 8, we acknowledge that the actual question construction paths may contain more search calls than prescribed by Appendix E.2. These examples illustrate the examiner's actual construction process rather than verified minimal solution paths.

We randomly sampled 50 generated QA/QCP pairs for human and GPT-6 evaluation, obtaining average error and shortcut rates of 7% and 39%, respectively. Some errors may stem from outdated or inaccurate information in the underlying Wikipedia corpus. These issues highlight limitations of the current knowledge corpus and task-generation implementation, which could be addressed through improved generation and verification techniques within the self-play framework. Moreover, questions admitting shorter reasoning paths still serve as valid training data if their answers are correct and unambiguous.

GPT-6HumanAverage
Error Rates8%6%7%
Shortcut Rates44%34%39%

Importantly, our core contribution is to demonstrate that self-play generated QCPs can serve as a scalable source of privileged information for self-distillation. We acknowledge that more advanced generation and verification techniques could further improve QCP quality and potentially enhance π-Play's performance.

Q1.2 Effectiveness and Structural Contribution of QCPs

(a) Effect of QCP conditioning on teacher accuracy.

To assess the effectiveness of QCPs, we evaluated the teacher with and without QCPs on 200 randomly generated questions, masking answer-related information in the QCPs to prevent answer leakage. On Qwen3-8B, QCP conditioning more than doubles teacher accuracy at the initial checkpoint (23.0%→48.5%), demonstrating the value of QCP as privileged information.

ModelTeacher w/o QCPTeacher w/ QCPΔ
Qwen3-8B23.048.5+25.5

(b) Ablation of QCP content and order.

We further compared QCP with retrieved evidence only and shuffled QCP. π-Play w/ QCP achieved the highest score, outperforming retrieved evidence only and Shuffled QCP. These results suggest that path structure and ordering contribute beyond retrieved evidence. Meanwhile, Shuffled QCPs retain some performance gains, indicating that the teacher model exhibits a degree of robustness to disruptions in path ordering.

NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleTotal
π-Play w/o Distillation41.262.343.637.729.013.338.4265.5
π-Play w/ Retrieved Evidence41.463.242.737.332.513.937.2268.2
π-Play w/ shuffled path41.362.243.438.332.013.241.6272.0
π-Play w/ QCP41.863.244.038.532.313.444.0277.2

Q2. Statistical comparison and checkpoint selection.

  • Multi-seed results for Dr.Zero. We added three-seed results for Dr.Zero on Qwen3-4B-Instruct. Dr.Zero obtains an average score of 267.5, compared with 276.6 for π-Play, confirming a consistent performance advantage.
NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleTotal
Dr.Zero42.1±0.762.8±1.144.3±0.438.4±0.329.8±1.313.8±0.736.3±0.9267.5±2.0
π-Play41.6±0.662.8±0.443.8±0.138.4±0.232.5±0.313.9±0.343.7±1.2276.6±1.7
  • Checkpoint-selection protocol. For checkpoint selection, we use the final checkpoint after three training iterations (150 student training steps) for all methods. We also identified an inconsistency in Table 10, where the Qwen3-4B results were reported at Iteration 2 (100 steps). We will correct this to report the final checkpoint at Iteration 3, ensuring a consistent evaluation protocol.
π-Play iter3NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleTotal
Qwen3-4B40.4±1.460.9±1.344.2±0.932.5±0.631.4±0.610.0±1.231.2±2.9250.7±1.5
  • Data used to select λ and K. For the distillation hyperparameter K and λ, we selected its value based on the total performance across the seven benchmarks.

Q3. End-to-end efficiency.

In Figure 4, we use Dr.Zero's performance after three iterations as the reference threshold. Using linear interpolation between evaluation checkpoints, π-Play achieves 3.1×, 2.4×, and 1.5× improvements in step efficiency on Qwen3-4B, Qwen3-4B-Instruct, and Qwen3-8B, respectively.

We further measure the end-to-end wall-clock time required to reach the same performance on 8×H20 GPUs, including examiner training and student-data generation. π-Play achieves 2.5×, 2.1×, and 1.5× wall-clock efficiency, respectively.

ModelDr.Zero Timeπ-Play TimeWall-clock Speedup
Qwen3-4B24.4h9.9h2.5×
Qwen3-4B-Instruct35.1h17.1h2.1×
Qwen3-8B47.9h33.0h1.5×
AuthorsOpenReview ↗

Authors Response (Part 2/2)

Summary


Thank you for the valuable review and constructive feedback. We appreciate your recognition of the meaningful framework design, QCPs as a useful source of privileged supervision, and extensive ablation studies.

We are also encouraged by the shared affirmations from Reviewers kPEv, mz8M, and Hem3 regarding:

  • Motivation and Novelty: (kPEv: A well-motivated use of information already available during self-play), (mz8M: The framework is clearly motivated), (Hem3: A well-motivated way to obtain dense supervision).

  • Framework Design: (kPEv: A concrete integration of privileged distillation into self-play), (mz8M: The framework is clearly motivated and reasonably straightforward to implement).

  • Practical Applicability: (mz8M: The approach is practically appealing)

  • Empirical Evidence and Analysis: (kPEv: Ablations support the value of trajectory information... useful reporting on practical cost and repeatability), (mz8M: The empirical evaluation includes several relevant analyses), (Hem3: The QCP ablations, including answer-removal controls, provide useful evidence).

We hope that our responses have fully addressed your concerns, and we welcome any further suggestions and discussions. If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.

Thanks for the response. The human audit of 50 generated questions found a 6% error rate and a 34% shortcut rate. The response also clarifies that Figures 7 and 8 show the paths taken during question construction. The new evidence-only and shuffled-path comparisons address my question about what QCPs add beyond the retrieved documents.

The three-seed comparison on 4B-Instruct shows a 1.30-point gain in macro-average EM over Dr.Zero. The response explains the checkpoint discrepancy in Table 10 and reports roughly 1.5–2.5× end-to-end speedups on eight H20 GPUs at matched score targets. This addresses my main question about training time.

Remaining questions. Were λ and K selected using the same examples as Table 1, and were the generated training questions checked for overlap with the evaluation questions? Please also define the ± statistic and clarify whether wall-clock time to target was interpolated between evaluation checkpoints.

Reviewer_Hem3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes π-Play, which combines self-play task generation with privileged self-distillation. An examiner generates QA pairs and question construction paths, an EMA teacher uses these paths to provide token-level guidance, and the student learns from both distillation and outcome rewards. Experiments use Qwen3-4B and 8B without external labeled QA training data.

Critical strengths
  1. Reusing question construction paths as privileged information is a well-motivated way to obtain dense supervision from self-play.

  2. The QCP ablations, including answer-removal controls, provide useful evidence that construction paths contribute beyond simply exposing the answer.

Critical weaknesses
  1. Quality control for self-generated QA pairs is underspecified. Search grounding alone does not guarantee correct, unambiguous answers, and no independent quality audit or benchmark-overlap analysis is reported.

  2. The EMA teacher uses τ=0.05 without an update-strategy ablation. A frozen-teacher control and sensitivity to τ would help establish whether teacher co-evolution is necessary.

Questions for authors

See the weaknesses above.

3 comments
AuthorsOpenReview ↗

Authors Response

Response


W1. Quality of Self-Generated QA Pairs and Benchmark Overlap

A1.1. Quality of self-generated QA pairs

To assess the quality of self-generated QA pairs, we randomly sampled 50 pairs for human and GPT-6 evaluation, obtaining an average question–answer consistency rate of 93%, suggesting that most generated QA pairs are reliable. We further observed that some inconsistencies stem from outdated or inaccurate information in the underlying Wikipedia corpus. Using a more up-to-date and reliable knowledge corpus could further improve the accuracy of generated QA pairs and potentially enhance π-Play's performance.

GPT-6HumanAverage
Answer Consistency92%94%93%

A1.2. Benchmark-overlap analysis

we compared all questions generated throughout the training of Qwen3-4B-Instruct against the seven benchmarks to assess potential train–test overlap. Specifically, we computed TF-IDF cosine similarity and flagged question pairs with similarity above 0.75 as potential overlaps. As shown below, the overlap rates are consistently close to zero across all seven benchmarks, suggesting that π-Play's gains primarily reflect improved search and reasoning capabilities rather than memorization of similar questions.

BenchmarkNQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleOverall
Potential Overlaps0.11%0.10%2.31%0.26%0.27%0.17%0.00%0.46%

W2. Ablation Study of EMA Updates

A2. To evaluate the effectiveness of teacher co-evolution, we conducted an additional ablation comparing a frozen teacher (τ=0) with EMA teacher updating (τ=0.05) under Qwen3-4B-Instruct. As shown below, EMA improves the total score from 270.0 to 277.2, demonstrating the benefit of dynamically updating the teacher during training.

τ=0τ=0.05
π-Play270.0277.2

Summary


Thank you for the valuable review and constructive feedback. We appreciate your recognition of our well-motivated core insight, the thorough QCP ablation studies, and the clarity of our writing.

We are also encouraged by the shared affirmations from Reviewers R1 (kPEv), R2 (mz8M), and R4 (tcP9) regarding:

  • Motivation and Novelty: (kPEv: A well-motivated use of information already available during self-play), (mz8M: The framework is clearly motivated), (tcP9: A useful source of privileged supervision).

  • Framework Design: (kPEv: A concrete integration of privileged distillation into self-play), (mz8M: The framework is clearly motivated and reasonably straightforward to implement), (tcP9: A meaningful combination of self-play and privileged self-distillation).

  • Practical Applicability: (mz8M: The approach is practically appealing)

  • Empirical Evidence and Analysis: (kPEv: Ablations support the value of trajectory information... useful reporting on practical cost and repeatability), (mz8M: The empirical evaluation includes several relevant analyses), (tcP9: Ablations provide evidence beyond the final aggregate score).

We hope that our responses have fully addressed your concerns, and we welcome any further suggestions and discussions. If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear reviewer,

Thank you again for your careful consideration of our response. As the rebuttal period is nearing its end, we would sincerely appreciate any further comments or questions you may have. We would be very happy to discuss them in more detail.

Best regards,

Submission 19375 Authors

AuthorsOpenReview ↗

Looking forward to further discussions!

Dear Reviewer Hem3,

Thank you for taking the time to review our manuscript and for providing valuable feedback. We have carefully addressed your concerns through additional clarifications, validation, and detailed analyses.

Below, we summarize the main improvements made in response to your feedback:

  1. Quality audit of generated QA pairs. We randomly sampled 50 QA pairs generated during self-play and audited them alongside their QCPs. Answer consistency was 94% in the human assessment and 92% in the GPT-6 assessment, for an average of 93%. This supports the use of the generated QA pairs as training data for π-Play.

  2. Benchmark-overlap analysis. We assessed potential overlap between all questions generated during Qwen3-4B-Instruct training and the evaluation questions from all seven benchmarks. The potential overlap rates were close to zero across all seven benchmarks. This suggests that π-Play’s gains do not come from memorizing similar questions.

  3. Effectiveness of EMA Updates. EMA allows the teacher to track the student’s progress without requiring a separate teacher optimization step. In our ablation, π-Play achieved a higher total score with an EMA-updated teacher than with a frozen teacher 277\.2 vs\. 270\.0. The QCP conditioned teacher’s accuracy also rose from 48.5% at step 0 to 52.0% at step 50 and 57.5% at step 100. Together, these results support EMA as a practical, low-overhead way for the teacher to improve alongside the student.

As the rebuttal period is approaching its end, we would greatly appreciate it if you could consider our responses and revised manuscript. We sincerely welcome further discussion and would be happy to provide additional clarifications.

If our responses have addressed your concerns, we would be very grateful if you could kindly reconsider your evaluation and rating of our work.

Thank you again for your time and consideration.

Best regards,

Submission 19375 Authors

Reviewer_mz8MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes π-Play, a self-evolution framework for search agents that combines self-play with privileged-information self-distillation. An examiner generates question–answer pairs together with their Question Construction Paths (QCPs). A same-scale teacher, updated through an exponential moving average of the student’s parameters, conditions on these QCPs to provide token-level supervision along student-generated trajectories. The student is trained using a combination of outcome-based GRPO and a distillation objective. Experiments across three Qwen model configurations and seven QA benchmarks show improvements over supervised RL and self-play baselines.

Critical strengths
  1. The paper identifies a useful source of supervision: the intermediate retrieval and reasoning artifacts already produced during task construction. Reusing these artifacts as privileged information is a sensible way to connect task generation with student optimization without requiring externally labeled QA data.

  2. The framework is clearly motivated and reasonably straightforward to implement. The student does not require access to QCPs at inference time, which makes the approach practically appealing.

  3. The empirical evaluation includes several relevant analyses. In particular, the authors investigate alternative privileged information, partial QCPs, answer removal, multiple random seeds, and training costs. These experiments provide initial evidence that QCP-conditioned distillation benefits training.

Critical weaknesses
  1. Limited methodological novelty beyond the source of privileged information. The student objective closely follows existing on-policy context/self-distillation methods: student-generated trajectories are regularized toward a privileged-information-conditioned teacher. The main distinction appears to be using QCPs as privileged context, rather than introducing a new distillation mechanism.
  2. The specific value of path structure is not established. The QCP ablations change both information content and structure. Comparing against the same retrieved evidence without construction reasoning, or with shuffled steps, would clarify whether the construction path offers benefits beyond additional factual context.
  3. The teacher update does not directly optimize the stated teacher objective. Equation (2) motivates a reward-maximizing, student-regularized teacher, but the implementation replaces this optimization with an EMA of student parameters. EMA alone does not ensure that the teacher learns to exploit QCPs or provides reliable guidance on student-generated prefixes.
Questions for authors
  1. Does the difficulty reward favor unreliable questions? The examiner receives the highest difficulty reward when only one student attempt succeeds. However, low success rates may also arise from ambiguity, answer-format differences, or incorrect labels. How does the method distinguish useful difficulty from poor question quality?
  2. Does a construction path necessarily provide useful guidance for solving the question? Question construction may involve redundant searches, while the resulting question may admit a shorter solution. Could conditioning the teacher on QCPs bias the student toward inefficient strategies?
4 comments
AuthorsOpenReview ↗

Authors Response (Part 1/2)

Response


W1. Significance of π-Play for self-distillation

We would like to clarify that π-Play's contribution lies in a framework-level innovation rather than a new distillation mechanism or optimization objective. Its primary contribution is to effectively integrate two complementary self-evolution paradigms: self-play and self-distillation.

  • Conventional self-play uses examiner-generated questions but typically discards the corresponding construction trajectories, optimizing the student primarily through sparse outcome rewards. It therefore constitutes a self-evolution paradigm with limited intermediate supervision.

  • Self-distillation, in contrast, can provide dense supervision, but often relies on externally supplied QA data and privileged information, such as expert demonstrations, environmental feedback, or prior knowledge [1-3]. This reliance may make fully autonomous scaling challenging.

In π-Play, self-play generates training tasks together with their QCPs, providing both training data and privileged information for self-distillation in a low-cost and scalable manner, while self-distillation converts QCPs into token-level supervision to accelerate evolution process of self-play. Our key contribution lies in this integration, which enables fully autonomous model improvement without external labeled data.

W2. Does QCP outperform retrieved evidence and shuffled paths?

We compared QCP with retrieved evidence only and shuffled QCP. π-Play w/ QCP achieved the highest score, outperforming retrieved evidence only and Shuffled QCP. These results suggest that path structure and ordering contribute beyond retrieved evidence. Meanwhile, Shuffled QCPs retain some performance gains, indicating that the teacher model exhibits a degree of robustness to disruptions in path ordering.

NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleTotal
π-Play w/o Distillation41.262.343.637.729.013.338.4265.5
π-Play w/ Retrieved Evidence41.463.242.737.332.513.937.2268.2
π-Play w/ shuffled path41.362.243.438.332.013.241.6272.0
π-Play w/ QCP41.863.244.038.532.313.444.0277.2

W3. Teacher Objective and EMA Ablation

We would like to clarify that Eq. (2) serves as a conceptual motivation for teacher co-evolution rather than an objective directly optimized in our implementation. We adopt EMA as a computationally efficient and practical teacher update strategy, allowing the teacher to gradually track student improvements without additional teacher optimization overhead. Furthermore, our additional ablation shows that EMA improves the total score, supporting EMA as a reasonable and effective engineering choice.

τ=0τ=0.05
π-Play270.0277.2

Q1. The rationale behind the difficulty reward

Consistent with prior work on self-play [4–6], our difficulty reward favors challenging yet solvable questions. Specifically, π-Play adopts Dr.Zero’s reward formulation [6] to favor questions with partial student success. Partial student success is a proxy for difficulty relative to the student’s current capabilities. We agree this reward alone cannot distinguish genuinely difficult questions from ambiguous ones or those with incorrect reference answers. Incorporating independent checks for ambiguity and answer correctness into the reward could help address this limitation.

Moreover, to assess question quality separately, we randomly sampled 50 generated QA pairs and asked a human evaluator and GPT-6 to assess question–answer consistency. Their consistency rates were 94% and 92%, respectively, averaging 93% on the audited sample. This suggests that most of the generated QA pairs are accurate.

GPT-6HumanAverage
Consistency92%94%93%
AuthorsOpenReview ↗

Authors Response (Part 2/2)

Q2. Effectiveness of QCP Guidance

We agree that question construction paths may contain redundant searches and do not necessarily represent optimal reverse solution trajectories. However, π-Play does not directly train the student to imitate these paths. Instead, the teacher leverages QCPs as privileged information to provide token-level guidance on student-generated rollouts, rather than directly transferring potentially redundant search actions

(a) Effectiveness of QCP Conditioning.

To assess the effectiveness of QCPs, we evaluated the teacher with and without QCPs on 200 randomly generated questions, masking answer-related information in the QCPs to prevent answer leakage. On Qwen3-8B, QCP conditioning more than doubles teacher accuracy at the initial checkpoint (23.0%→48.5%), demonstrating the value of QCP as privileged information.

ModelTeacher w/o QCPTeacher w/ QCPΔ
Qwen3-8B23.048.5+25.5

(b) Search Behavior under QCP Guidance.

As shown in Table 6, π-Play achieves higher average accuracy while reducing both search count and query redundancy, suggesting that it learns more efficient search strategies.

MethodAccuracy ↑Search Count ↓Query Redundancy ↓
SQLM*35.72.30.43
Dr.Zero38.02.40.48
π-Play39.61.90.37

Moreover, to examine whether fewer queries come at the expense of evidence support, we randomly sampled 100 questions from each MuSiQue hop group (2-, 3-, and 4-hop) and evaluated both methods on the same questions. We measured evidence-supported accuracy as the percentage of sampled questions answered correctly for which DeepSeek-V4-Flash judged the passages retrieved before answering to be jointly sufficient to derive the answer. π-Play issued fewer queries on average and showed lower query redundancy than Dr.Zero in all three groups. Its evidence-supported accuracy matched Dr.Zero’s on 2- and 4-hop questions and was one percentage point higher on 3-hop questions. These results suggest that π-Play searched more efficiently without sacrificing evidence-supported accuracy on the sampled questions.

MethodAccuracy & Evidence ↑Search Count ↓Query Redundancy↓
Difficulty: 2-hop
Dr.Zero12.0%2.360.47
π-Play12.0%2.030.35
Difficulty: 3-hop
Dr.Zero2.0%2.840.43
π-Play3.0%2.590.41
Difficulty: 4-hop
Dr.Zero2.0%3.520.49
π-Play2.0%3.000.46

Summary


Thank you for the valuable review and constructive feedback. We appreciate your recognition of our sensible and well-motivated framework design, the value of QCPs as a useful source of privileged supervision, its practical applicability, and the extensive supporting empirical analyses.

We are also encouraged by the shared affirmations from Reviewers kPEv, Hem3, and tcP9 regarding:

  • Motivation and Novelty: (kPEv: A well-motivated use of information already available during self-play), (Hem3: A well-motivated way to obtain dense supervision), (tcP9:9: A useful source of privileged supervision).

  • Framework Design: (kPEv A concrete integration of privileged distillation into self-play), (tcP9: A meaningful combination of self-play and privileged self-distillation).

  • Empirical Evidence and Analysis: (kPEv Ablations support the value of trajectory information... useful reporting on practical cost and repeatability), (Hem3: The QCP ablations, including answer-removal controls, provide useful evidence), (tcP9: Ablations provide evidence beyond the final aggregate score).

  • Clarity and Presentation: (kPEv: Clear and easy to follow), (Hem3: Clear and easy to follow).

We hope that our responses have fully addressed your concerns, and we welcome any further suggestions and discussions. If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.

Reference


[1] "Self-Distillation Enables Continual Learning." ICML (2026)

[2] "OpenClaw-RL: Train Any Agent Simply by Talking." ArXiv (2603.10165)

[3] "On-Policy Context Distillation for Language Models." ArXiv (2602.12275)

[4] "Self-Questioning Language Models." ICLR Workshop (2026).

[5] "Absolute Zero: Reinforced Self-play Reasoning with Zero Data." NeurIPS (2025).

[6] "Dr. Zero: Self-Evolving Search Agents without Training Data." COLM (2026).

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear reviewer,

Thank you again for your careful consideration of our response. As the rebuttal period is nearing its end, we would sincerely appreciate any further comments or questions you may have. We would be very happy to discuss them in more detail.

Best regards,

Submission 19375 Authors

AuthorsOpenReview ↗

Looking forward to further discussions!

Dear Reviewer mz8M,

Thank you for taking the time to review our manuscript and for providing valuable feedback. We have carefully addressed your concerns through additional clarifications, validation, and detailed analyses.

Below, we summarize the main improvements made in response to your feedback:

  1. Value of path structure in QCP. We compared full QCPs, retrieved evidence alone, and shuffled QCPs as sources of privileged information. The full QCPs achieved the highest overall score, suggesting that preserving the path structure and order adds value to student supervision beyond the retrieved evidence alone.

  2. Effectiveness of EMA Updates. EMA allows the teacher to track the student’s progress without requiring a separate teacher optimization step. In our ablation, π-Play achieved a higher total score with an EMA-updated teacher than with a frozen teacher 277\.2 vs\. 270\.0. The QCP conditioned teacher’s accuracy also rose from 48.5% at step 0 to 52.0% at step 50 and 57.5% at step 100. Together, these results support EMA as a practical, low-overhead way for the teacher to improve alongside the student.

  3. Effectiveness of QCP Guidance. In our ablation on 200 generated questions, conditioning the teacher on answer-masked QCPs improved its accuracy at every evaluated checkpoint, with the largest gain at step 0 (23.0%→48.5%). On MuSiQue, π-Play made fewer searches and issued less redundant queries at every hop level, while increasing the proportion of answers that were both correct and supported by retrieved evidence. This pattern suggests that QCP guidance helps the student search more efficiently. These results provide empirical support for our core claim that the task-generation process ie˙,˙QCPi\.e\., QCP in self-play can itself serve as a scalable source of privileged supervision for self-distillation.

As the rebuttal period is approaching its end, we would greatly appreciate it if you could consider our responses and revised manuscript. We sincerely welcome further discussion and would be happy to provide additional clarifications.

If our responses have addressed your concerns, we would be very grateful if you could kindly reconsider your evaluation and rating of our work.

Thank you again for your time and consideration.

Best regards,

Submission 19375 Authors

Reviewer_kPEvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes π-Play, a self-play framework for training search agents that augments an examiner–student setup with a privileged teacher. The examiner generates questions through multi-turn search, and its question-construction path is reused as additional context for the teacher. The student combines outcome-based reinforcement learning with token-level distillation from this teacher, whose parameters track the student through an exponential moving average. Experiments on three Qwen3 configurations and seven question-answering benchmarks evaluate final performance, training efficiency, and the usefulness of different forms of privileged information. The main idea is to turn an existing by-product of self-play into a richer training signal without requiring annotated reasoning trajectories separately.

Critical strengths
  1. A well-motivated use of information already available during self-play. The question-construction path contains evidence and intermediate search decisions that an outcome reward alone does not expose. Reusing this trajectory for teacher conditioning is a practical way to obtain denser supervision, while keeping the student's inference-time inputs unchanged. The contribution is a concrete integration of privileged distillation into search-agent self-play.

  2. The ablations support the value of trajectory information beyond answer exposure. Table 3 compares full paths with answer-only, partial-path, and no-distillation variants. The full-path configuration scores 277.2 in aggregate, versus 265.5 without distillation and 264.1 with the answer alone. Table 4 further removes answers and answer-bearing snippets while retaining much of the benefit. These controls make the case for using the construction process substantially stronger than a comparison against Dr.Zero alone.

  3. Useful reporting on practical cost and repeatability. Table 13 measures an average per-step overhead of 4.3%, suggesting that the additional teacher pass is manageable in the tested setup. Table 10 also reports three-seed results for π-Play. These are valuable additions for a method whose main appeal includes more efficient training, although the comparative uncertainty remains incomplete.

Critical weaknesses
  1. Robust superiority over the closest baseline is not yet established. The aggregate improvements over Dr.Zero in Table 1 translate to approximately 0.6–1.6 average exact-match points across the seven benchmarks. Table 10 reports variability for π-Play, but not corresponding repeated runs of Dr.Zero, so it is difficult to assess the stability of these margins. In addition, the data used to select checkpoints and the model-specific distillation settings are not clearly identified. This is a question about comparative reliability and selection protocol, not evidence that test-set tuning occurred.

  2. The efficiency claim needs a more precise target and cost definition. Figure 4 provides evidence of earlier attainment of a reference score, but the reported 1.5–3.1× factors concern training steps rather than measured end-to-end time to that score. The per-step timing experiment is helpful, yet does not by itself account for the full self-play cycle, including question generation. Moreover, Dr.Zero is evaluated for three iterations; this establishes a finite-budget reference, not necessarily its converged performance. The paper should distinguish faster attainment of that reference from faster convergence at equal total compute.

  3. The reliability of the privileged signal and the credit-assignment interpretation are only partially validated. The QCP ablations show that construction paths are useful, but do not directly establish their correctness, necessary reasoning depth, or the teacher's advantage over the student. For example, the nominal three-hop task in Figure 8 appears answerable from a single retrieved passage. Likewise, fewer searches and the reported query-redundancy measure in Section 5.4 do not by themselves distinguish better credit assignment from answering with less evidence. These issues matter because reliable, informative construction paths are central to the method's explanation, rather than merely an implementation detail.

Questions for authors
  1. Can you provide matched repeated-seed results for Dr.Zero on at least one representative model, and clarify how the runs in Tables 1 and 10 were selected? Which held-out data determined the checkpoints and the distillation hyperparameters?

  2. How are the threshold-crossing steps in Figure 4 computed, and what are the corresponding end-to-end times, including question generation and teacher inference? Please distinguish the three-iteration Dr.Zero reference from an empirically established convergence point.

  3. On an audited sample, what fraction of examiner-generated questions has a correct answer and genuinely requires its assigned number of hops? Does conditioning on the QCP measurably improve the teacher's answer accuracy over the unconditioned student?

  4. How is query redundancy defined in Section 5.4? A breakdown by question difficulty, including correctness and evidence support, would help interpret whether the reduced number of searches reflects more effective reasoning.

4 comments
AuthorsOpenReview ↗

Authors Response (Part 1/2)

Response


Q1. Multi-seed analysis of Dr.Zero and checkpoint selection

A1.

  • Multi-seed analysis: We added multi-seed results for Dr. Zero on Qwen3-4B-Instruct, reported as mean ± standard deviation across three random seeds. Dr. Zero obtains an average score of 267.5, compared with 276.6 for π-Play, confirming a consistent performance advantage. ||NQ|TriviaQA|PopQA|HotpotQA|2WikiMQA|MuSiQue|Bamboogle|Total| |---|---|---|---|---|---|---|---|---| |Dr.Zero|42.1±0.7|62.8±1.1|44.3±0.4|38.4±0.3 |29.8±1.3|13.8±0.7|36.3±0.9|267.5±2.0| |π-Play|41.6±0.6|62.8±0.4|43.8±0.1|38.4±0.2|32.5±0.3|13.9±0.3|43.7±1.2|276.6±1.7|
  • Checkpoint selection: For evaluation, we use the final checkpoint after three training iterations (150 steps) for all methods.
  • Distillation hyperparameter selection: We selected the distillation hyperparameters K and λ on Qwen3-4B-Instruct based on the total score across all seven benchmarks, computed using the same evaluation examples as in Table 1. We reused the selected K unchanged for Qwen3-4B and Qwen3-8B. For λ, we used the results on Qwen3-4B-Instruct to guide the settings for the other two models, without running separate sweeps on either model.

Q2. End-to-end efficiency & Dr.Zero's convergence performance

A2. In Figure 4, we use Dr.Zero's final score after three iterations as the reference threshold, corresponding to 245.2, 266.3, and 274.9 for Qwen3-4B, Qwen3-4B-Instruct, and Qwen3-8B, respectively. Using linear interpolation between adjacent evaluation checkpoints, we estimate the steps required for π-Play to reach these thresholds, yielding 3.1×, 2.4×, and 1.5× improvements in step efficiency, respectively.

A2.1. End-to-end efficiency

As shown in the table below, we compare the end-to-end wall-clock time required to reach the same performance on 8×H20 GPUs, including task generation, examiner training and teacher inference. π-Play achieves 2.5×, 2.1×, and 1.5× wall-clock efficiency, respectively.

ModelDr.Zero Timeπ-Play TimeWall-clock Speedup
Qwen3-4B24.4h9.9h2.5×
Qwen3-4B-Instruct35.1h17.1h2.1×
Qwen3-8B47.9h33.0h1.5×

A2.2. Dr.Zero's convergence performance

To validate the three-iteration reference, we extend Dr.Zero training to 5 iterations. As shown below, additional iterations yield no consistent performance improvement, supporting the use of the third iteration as a convergence reference. This observation is also consistent with Section 4.2 of Dr.Zero [1], which reports that further training beyond the third iteration yields marginal or no improvements.

π-PlayDr.Zero iter3Dr.Zero iter4Dr.Zero iter5
Qwen3-4B249.7245.2242.2239.3
Qwen3-4B-Instruct277.2266.3266.8266.4
Qwen3-8B280.3274.9274.7275.2

Q3. QCP quality and its benefits for teacher guidance

A3.1. QCP Quality Audit

We randomly sampled 50 generated QA/QCP pairs for human and GPT-6 evaluation, obtaining average answer consistency and intended-hop consistency rates of 93% and 61%. Some inconsistencies may stem from outdated or inaccurate information in the underlying Wikipedia corpus. Although some questions admit shorter reasoning paths than specified, they may still serve as valid training data if their answers are correct and unambiguous.

GPT-6HumanAverage
Answer Consistency92%94%93%
Hop Consistency56%66%61%

We emphasize that our core contribution lies in effectively integrating self-play and self-distillation, and demonstrating that self-play-generated QCPs can serve as a scalable source of privileged information for self-distillation. More advanced generation and verification techniques within the self-play framework could further improve QCP quality and potentially enhance π-Play's performance.

A3.2. Effectiveness of QCP Guidance.

To evaluate the benefits of QCPs, we evaluated the teacher with and without QCPs on 200 randomly generated questions, masking answer-related information in the QCPs to prevent answer leakage. QCP conditioning consistently improves accuracy across training steps, with gains of 25.5, 7.0, and 4.0 percentage points at Steps 0, 50, and 100, respectively. These results suggest that QCPs are particularly beneficial in the early stages of training, when the teacher capabilities are relatively limited.

Teacher checkpointStep 0Step 50Step 100
Teacher w/o QCP23.045.053.5
Teacher w QCP48.552.057.5
Δ+25.5+7+4
AuthorsOpenReview ↗

Authors Response (Part 2/2)

Q4. Definition of query redundancy and comparison across question difficulty levels

A4.1. Definition of query redundancy.

We measure query redundancy by averaging pairwise TF-IDF cosine similarities [2][3] within each trajectory and then averaging the resulting scores across trajectories. Let NN be the number of trajectories, nin_i the number of queries in trajectory ii, and vijv_{ij} the TF-IDF vector of its jthj_{th} query. The redundancy score for trajectory ii is given by

\begin{cases} \displaystyle \frac{2}{n_i(n_i-1)} \sum_{1 \le j < k \le n_i} \cos\\!\left(v_{ij}, v_{ik}\right), & n_i \ge 2, \\\\ 0, & n_i \le 1. \end{cases}$$ The overall score is $\frac{1}{N}\sum_{i=1}^{N} r_i$. Lower scores indicate less lexical overlap under this metric. **A4.2. Search behavior by question difficulty.** MuSiQue categorizes questions in the test set as 2-hop, 3-hop, or 4-hop according to reasoning complexity. We randomly sampled 100 MuSiQue questions from each hop group (2-, 3-, and 4-hop) and evaluated both methods on the same questions. For each group, we report the mean number of search queries and the percentage of questions answered both correctly and with sufficient retrieved evidence. We used DeepSeek-V4-Flash to judge whether the passages retrieved before answering jointly provided enough information to derive the model’s answer. |Method|Accuracy & Evidence ↑|Search Count ↓|Query Redundancy↓| |---|---|---|---| |**Difficulty: 2-hop**|||| |Dr.Zero|12.0%|2.36|0.47| |π-Play|12.0%|2.03|0.35| |**Difficulty: 3-hop**|||| |Dr.Zero|2.0%|2.84|0.43| |π-Play|3.0%|2.59|0.41| |**Difficulty: 4-hop**|||| |Dr.Zero|2.0%|3.52|0.49| |π-Play|2.0%|3.00|0.46| π-Play issues fewer search queries and has lower query redundancy in all three groups. Its evidence-supported accuracy matches Dr.Zero’s on 2- and 4-hop questions and is one percentage point higher on 3-hop questions. This result shows that **π-Play matched or exceeded Dr.Zero’s evidence-supported accuracy while issuing fewer queries on average in each of the three hop groups**, suggesting more efficient search behavior. ## `Summary` --- Thank you for the valuable review and constructive feedback. We appreciate your recognition of our **well-motivated core contribution**, **the value of QCPs as privileged supervision**, and **the clarity of our writing**. We are also encouraged by the shared affirmations from Reviewers mz8M, Hem3, and tcP9 regarding: - **Motivation and Novelty**: (**mz8M:** The framework is clearly motivated), (**Hem3:** A well-motivated way to obtain dense supervision), (**tcP9:** A useful source of privileged supervision). - **Framework Design**: (**mz8M:** The framework is clearly motivated and reasonably straightforward to implement), (**tcP9:** A meaningful combination of self-play and privileged self-distillation). - **Practical Applicability**: (**mz8M:** The approach is practically appealing) - **Empirical Evidence and Analysis**: (**mz8M:** The empirical evaluation includes several relevant analyses), (**Hem3:** The QCP ablations, including answer-removal controls, provide useful evidence), (**tcP9:** Ablations provide evidence beyond the final aggregate score). - **Clarity and Presentation**: (**Hem3:** Clear and easy to follow). We hope that our responses have fully addressed your concerns, and we welcome any further suggestions and discussions. **If our responses have resolved your concerns, we would sincerely appreciate your consideration for raising the score.** ## `Reference` [1] "Dr. Zero: Self-Evolving Search Agents without Training Data." COLM (2026) [2] "DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL." ArXiv (2509.10446) [3] "Recommending Queries by Extracting Thematic Experiences from Complex Search Tasks." Entropy (2018)
AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear reviewer,

Thank you again for your careful consideration of our response. As the rebuttal period is nearing its end, we would sincerely appreciate any further comments or questions you may have. We would be very happy to discuss them in more detail.

Best regards,

Submission 19375 Authors

AuthorsOpenReview ↗

Looking forward to further discussions!

Dear Reviewer kPEv,

Thank you for taking the time to review our manuscript and for providing valuable feedback. We have carefully addressed your concerns through additional clarifications, validation, and detailed analyses.

Below, we summarize the main improvements made in response to your feedback:

  1. Multi-seed Analysis of Dr. Zero. We added three-seed results for Dr. Zero on Qwen3-4B-Instruct. π-Play achieved a higher total score than Dr. Zero 276\.6 ± 1\.7 vs\. 267\.5 ± 2\.0, mean ± standard deviation, supporting a consistent performance advantage.

  2. End-to-end Efficiency. In addition to our analysis of step efficiency, we added an analysis of end-to-end wall-clock efficiency. π-Play achieves wall-clock speedups of 2.5×, 2.1×, and 1.5× on Qwen3-4B, Qwen3-4B-Instruct, and Qwen3-8B, respectively, further demonstrating its efficiency gains in both training steps and wall-clock time.

  3. Effectiveness of QCP Guidance. Our audit of 50 randomly sampled generated QA pairs and their QCPs found average answer and intended-hop consistency rates of 93% and 61%, respectively. Questions that require fewer hops than intended can still be useful for training if their answers are correct and unambiguous. On 200 generated questions, answer-masked QCPs improved teacher accuracy at every evaluated checkpoint, with the largest gain at Step 0 230˙23\.0%→48\.5%. These results provide empirical support for our core claim that the task-generation process i.e.,QCPi.e., QCP in self-play can itself serve as a scalable source of privileged supervision for self-distillation.

  4. Query Redundancy by Difficulty Level. On MuSiQue, π-Play reduced search count and query redundancy at every hop level while increasing the proportion of answers that were both correct and supported by retrieved evidence. This is consistent with our core claim that QCPs generated during self-play can serve as effective privileged supervision for improving the student’s search behavior.

As the rebuttal period is approaching its end, we would greatly appreciate it if you could consider our responses. We sincerely welcome further discussion and would be happy to provide additional clarifications.

If our responses have addressed your concerns, we would be very grateful if you could kindly reconsider your evaluation and rating of our work.

Thank you again for your time and consideration.

Best regards,

Submission 19375 Authors

Comments on the paper

1 comment

First, we thank Reviewers kPEv, mz8M, Hem3, and tcP9 for their constructive feedback and for recognizing the following strengths of our work.

  • Motivation and Novelty: ∗∗kPEv∗∗:Awell−motivateduseofinformationalreadyavailableduringself−play**kPEv**: A well-motivated use of information already available during self-play, ∗∗mz8M∗∗:Theframeworkisclearlymotivated**mz8M**: The framework is clearly motivated, ∗∗Hem3∗∗:Awell−motivatedwaytoobtaindensesupervision**Hem3**: A well-motivated way to obtain dense supervision, ∗∗tcP9∗∗:Ausefulsourceofprivilegedsupervision**tcP9**: A useful source of privileged supervision.

  • Framework Design: ∗∗kPEv∗∗:Aconcreteintegrationofprivilegeddistillationintoself−play**kPEv**: A concrete integration of privileged distillation into self-play, ∗∗mz8M∗∗:Theframeworkisclearlymotivatedandreasonablystraightforwardtoimplement**mz8M**: The framework is clearly motivated and reasonably straightforward to implement, ∗∗tcP9∗∗:Ameaningfulcombinationofself−playandprivilegedself−distillation**tcP9**: A meaningful combination of self-play and privileged self-distillation.

  • Practical Applicability: ∗∗mz8M∗∗:Theapproachispracticallyappealing**mz8M**: The approach is practically appealing

  • Empirical Evidence and Analysis: **kPEv**: Ablations support the value of trajectory information\.\.\. useful reporting on practical cost and repeatability, ∗∗mz8M∗∗:Theempiricalevaluationincludesseveralrelevantanalyses**mz8M**: The empirical evaluation includes several relevant analyses, ∗∗Hem3∗∗:TheQCPablations,includinganswer−removalcontrols,provideusefulevidence**Hem3**: The QCP ablations, including answer-removal controls, provide useful evidence, ∗∗tcP9∗∗:Ablationsprovideevidencebeyondthefinalaggregatescore**tcP9**: Ablations provide evidence beyond the final aggregate score.

  • Clarity and Presentation: ∗∗kPEv∗∗:Clearandeasytofollow**kPEv**: Clear and easy to follow, ∗∗Hem3∗∗:Clearandeasytofollow**Hem3**: Clear and easy to follow

Our core claim is that the task-generation process ie˙,˙QCPi\.e\., QCP in self-play can itself provide a scalable source of privileged supervision for self-distillation. The original and additional experiments support this claim from multiple angles.

  1. Main performance. π-Play outperforms the self-play baselines across all three models (Table 1), and this advantage persists in the multi-seed comparison (Table 10).

  2. Ablations of QCP components. The QCP ablations in Table 3 and Table 4 show that QCPs remain useful even after explicit answers and answer-bearing snippets are removed.

  3. Ablations of QCP structure. The QCP variants outperformed both the retrieved-evidence only and shuffled-QCP variants in overall score (Response W2 to Reviewer mz8M, Response Q1.2 to Reviewer tcP9), suggesting that preserving the original path structure and order adds value to student supervision.

  4. Direct benefit to the teacher. At the initial Qwen3-8B checkpoint, conditioning on answer-masked QCPs raised teacher accuracy from 23.0% to 48.5%, demonstrating their value as privileged context for the teacher (Response Q3 to Reviewer kPEv, Response Q2 to Reviewer mz8M).

At the framework level, we effectively effectively integrates two complementary self-evolution paradigms by reusing QCP as privileged information.

  • Self-play typically trains the student on self-generated questions using sparse outcome rewards, with limited intermediate supervision.

  • Self-distillation provides dense supervision but often relies on externally supplied QA data or privileged information, making autonomous scaling more difficult.

In π-Play, self-play generates training tasks together with their QCPs, providing both training data and privileged information for self-distillation in a scalable manner, while self-distillation converts QCPs into token-level supervision to accelerate evolution of self-play. This integration enables fully autonomous and more efficient model improvement without externally labeled QA data. Our efficiency analysis further shows that π-Play improves efficiency in terms of both training steps and wall-clock time compared with the self-play baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.