- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Knowledge-to-Prompt (K2P), which transfers knowledge from a large teacher to a smaller, frozen student through optimized natural-language prompts, without ground-truth training labels. K2P synthesizes reusable instructions from teacher solutions, refines them using paired teacher and student responses, and guides search and selection with answer agreement. The experimental results on reasoning tasks show improvements compared to other baselines.
Critical strengths
- The setting is practical and low-resource. Improving a frozen student without fine-tuning or ground-truth labels is valuable.
- The authors compare K2P against PLD, APE, GEPA, SPO, and label-free variants. The baselines are comprehensive and cover sufficient diversity of related works.
- The analysis and theoretical explanations are detailed. The paper examines teacher solution contributions, feedback refinement, archive selection, and sensitivity to different sample sizes.
Critical weaknesses
- The label-free problem is barely stressed. The paper assumes that teacher errors can reward shared mistakes, but teacher accuracy is 94–98% on every task (saturated), so the teacher is nearly a gold labeler. It would be interesting to test weaker teachers, harder tasks, or noisier data.
- The theory is loosely connected to the experiments. For example, the key conditions (κ, ε_t, γ, ω) are never estimated. The binary and independent-flip noise model is unrealistic, since teacher errors correlate with question difficulty and student errors.
Questions for authors
The paper's presentation is dense and too long. It puts many key results into the 57-page Appendix (token usage, bootstrap tables), which makes the actual takeaway hard to extract.
4 comments
Response to Reviewer sZN2 (2/4) — W1 continued — Phi teacher
W1. Testing the label-free setting with unreliable teachers and corrupted answers (continued)
Full construction with Phi-4-mini-instruct. Phi supplies solutions, references, synthesized prompts, and refinement responses throughout construction. Both methods share references and follow the original student evaluation protocol. GEPA-T receives the paired K2P construction cost as its budget. On QuaRTz, the shared teacher-side extractor was fixed before construction. The table reports mean ± sample SD on separate test sets.
| Task | Student | Construction seeds | Zero-shot | K2P | GEPA-T |
|---|---|---|---|---|---|
| Colored Objects | Qwen 0.8B | 3 | 49.00 | 56.63 ± 9.33 | 56.03 ± 12.70 |
| Colored Objects | Ministral 3B | 3 | 69.10 | 92.43 ± 2.61 | 70.23 ± 0.59 |
| Tracking | Qwen 0.8B | 3 | 21.18 | 66.99 ± 3.08 | 72.75 ± 6.01 |
| Tracking | Ministral 3B | 3 | 78.04 | 89.93 ± 4.32 | 82.03 ± 3.22 |
| QuaRTz | Qwen 0.8B | 3 | 82.53 | 78.15 ± 2.56 | 81.12 ± 0.77 |
| QuaRTz | Ministral 3B | 3 | 85.20 | 87.33 ± 2.53 | 89.20 ± 0.19 |
Colored Objects. K2P's three-seed mean gains over zero-shot with Phi are 7.63 pp for Qwen and 23.33 pp for Ministral.
Tracking. K2P exceeds zero-shot in all six runs. Its mean is 5.75 pp below GEPA-T for Qwen and 7.91 pp above GEPA-T for Ministral. Phi produces usable answers on 66/160 search/selection questions, so this setting includes both answer errors and formatting failures.
The Tracking comparison uses the same initialization protocol for all three seeds, synthesizing until eight prompts pass the original admission checks. All attempts, including rejected outputs, are included in K2P's construction cost and GEPA-T's matched budget. This protocol was fixed after the initial run stopped at 32 synthesis attempts with two admitted prompts and before further generation. The three seeds required 611, 285, and 406 attempts.
Interpreting QuaRTz. K2P's three-seed mean exceeds zero-shot for Ministral and falls below it for Qwen. GEPA-T has higher means for both students. Retrospective tests of all 70 candidates across six frozen archives show that each Qwen archive's highest test score is below zero-shot. Each Ministral archive contains prompts that outperform zero-shot, and the selected prompts score 0–4.34 pp below their archives' highest test scores. Reselection using gold or 9B references on the same 80-question selection pool yields mixed changes in test accuracy.
We have added these results to Sections 6.2 and 7.2. Appendix B.10 reports initialization details, per-seed candidate scores, and reselection diagnostics. Appendix B.11 presents the full-construction corruption curve (Figure 7) and the fixed-candidate analysis.
Our response to W2 further relates real teacher errors to candidate rankings.
Response to Reviewer sZN2 (3/4) — W2 — Theory and empirical diagnostics
W2. Empirical checks of theoretical conditions and correlated teacher errors
Thank you for highlighting these issues. We agree that the theoretical quantities need a clearer empirical connection and that correlations between teacher errors, question difficulty, and student errors deserve explicit treatment. We measured expected proposal gains at 96 recorded search states from the 24 four-round construction runs. We also measured agreement–accuracy mismatch across 60 combinations of archive, data role, and reference teacher. These diagnostics connect the theoretical quantities to observed construction behavior.
Empirical diagnostics of search progress and agreement
Larger sets a stronger progress target and can require a larger shortfall allowance . Likewise, larger strengthens the agreement–accuracy condition and can require a larger residual . We therefore evaluate several reference values of and and measure the corresponding shortfalls and residuals on the recorded states and candidate archives.
| Quantity | Meaning | Empirical evaluation |
|---|---|---|
| Required fraction of the remaining agreement gap closed in expectation | Reference values 0.25, 0.5, and 1 | |
| Allowance for expected-progress shortfall | Estimated conditional shortfall from eight proposal gains per state | |
| Scale relating accuracy and agreement deficits | Reference values 0.25, 0.5, 0.75, and 1 | |
| Residual mismatch in that relation | Smallest residual covering each observed archive at fixed |
Conditional expected search progress. At each of the 96 recorded states, we drew eight fresh proposals from the same recorded parent, holding the feedback, teacher decoding, student evaluation seeds, parser, admission rules, and 80-question search pool fixed. All draws enter the means, with zero accepted gain for invalid or non-improving proposals.
- Mean accepted gain is 1.41 pp, with conditional Monte Carlo SE 0.12 pp and 95% Hoeffding interval [0.43, 2.39] pp.
- 190/768 proposals improve agreement (24.74%). The SD across the three construction-seed-specific mean gains is 0.54 pp.
At each state, we first average the eight gains, then compute the positive shortfall against . The two reference gaps are the retrospective gap from the parent to its run's final archive best and , an upper bound on the gap to any admissible prompt on the fixed search pool. Table entries average the statewise shortfalls.
| Reference | Estimated mean shortfall using archive gap, pp | Estimated mean shortfall using gap upper bound, pp |
|---|---|---|
| 0.25 | 0.40 | 2.45 |
| 0.50 | 1.40 | 6.22 |
| 1.00 | 3.95 | 13.85 |
At , the target is to close half the remaining gap in expectation. The estimated mean shortfall is 1.40 pp against the archive gap and 6.22 pp against the gap upper bound. These measurements describe the recorded parent/feedback states with student evaluation seeds fixed. The theoretical bound uses the conditional gap over the full admissible prompt space. Averaging the resulting true conditional shortfalls over construction histories gives a sufficient . The full-space and population guarantees use the assumptions stated in Section 5.
In the 24 construction trajectories that supplied these states, 17 have positive total accepted gain and 27/96 slots improve agreement. Strict acceptance keeps the parent's search score nondecreasing.
Agreement and accuracy. We evaluated 60 combinations of archive, data role, and reference teacher. Gold accuracy uses canonical exact matching on the same 80-question construction pools. For each reference , we use archive optima in Equation (7) and find the smallest required . At , mean residuals over the three Colored Objects selection archives are as follows, on a 0–1 scale.
| Student | Minimum archive residual with original teacher | Minimum archive residual with Phi |
|---|---|---|
| Qwen 0.8B | 0.0125 | 0.3083 |
| Ministral 3B | 0.0250 | 0.1083 |
Phi's larger residuals show a weaker agreement–accuracy relationship within these fixed candidates, linking the teacher-quality experiment to Equation (7). On QuaRTz/Qwen, the original-teacher residual averages 0.0458, and selection attains the archive's best gold score in all three pools. Thus positive residuals can coexist with successful selection.
Response to Reviewer sZN2 (4/4) — W2 continued and Q1 — Correlated errors and appendix
W2. Empirical checks of theoretical conditions and correlated teacher errors (continued)
Correlated teacher errors
The binary example gives an idealized sufficient condition for agreement to preserve accuracy rankings under classical independent classification noise. With valid, aligned scoring, a constant error probability , and teacher flips independent of the student's answer conditional on input and truth, it yields and . Angluin and Laird, 1988
The general candidate-comparison bound retains the joint event that the teacher is wrong and two candidate answers differ, allowing errors to correlate with question difficulty and student outputs. Appendix C.5, Equation (C6), also handles unusable references and scoring-rule differences.
A fixed-pool bound also permits correlated errors. Under aligned scoring, agreement and correctness coincide whenever the teacher reference is correct. If is the fraction of incorrect or unusable teacher references, the fixed-pool accuracy and agreement scores therefore differ by at most for every prompt. Taking suprema gives the sufficient choice , on that pool. On the fixed 80-question selection pools, the residual bounds are 0.025/0.025/0/0.150 on Tracking/Colored/TabMWP/QuaRTz with the original teacher, and 0.800 on Colored with Phi. The Phi bound is loose for percentage-point accuracy comparisons, motivating the more informative archive residuals and ranking measurements. A population guarantee uses the population error rate.
Actual ranking changes. On the same frozen Colored Objects archives constructed with Qwen3.5-9B, we count strict reversals between teacher-agreement and gold-accuracy rankings using each teacher's selection references.
| Student | Reversals with original teacher | Reversals with Phi | Mean gold selection loss with original teacher | Mean gold selection loss with Phi |
|---|---|---|---|---|
| Qwen 0.8B | 0/198 | 29/198 | 0.00 pp | 1.67 pp |
| Ministral 3B | 5/198 | 58/198 | 0.00 pp | 3.33 pp |
Counts cover 66 pairs in each of three archives, sharing candidates and questions. Selection loss is the archive-best minus selected gold accuracy on the 80-question selection pool. Teacher-reference accuracies are 98.75% and 60.00%, respectively.
Sections 7.1–7.2 now summarize proposal gains, progress shortfalls, and selection reliability. Appendix B.12 details the diagnostics, and Appendix C.14 connects them to the theory.
Q1. Appendix organization, token costs, and bootstrap takeaways
Thank you very much for pointing this out. We have shortened the appendix exposition, clarified its section hierarchy, and added an appendix-wide roadmap with linked sections and page numbers. The main text also summarizes construction costs and the bootstrap findings.
- The opening guide explains the organization into reproduction details and numerical records in Appendix A, empirical analyses in Appendix B, proofs and supporting analyses in Appendix C, and extended related work in Appendix D. New subheadings separate data, construction rules, and templates in A.2, baseline protocols and costs in A.3, and the two sample-size studies in B.4.
- Repeated explanations in A–C have been consolidated, including admission and selection rules, bootstrap procedures, and theoretical interpretation. Cross-references lead to the full protocols, while C.14 concisely maps the empirical diagnostics to the theoretical mechanisms.
Section 6.1 specifies the model weights and construction-cost accounting, with complete costs in Table A2 (Appendix A.3). Section 6.2 reports actual shared-budget costs and links to Table B15. It also summarizes the paired-bootstrap findings and explains that the pointwise 95% intervals quantify test uncertainty for frozen prompts, alongside the construction-seed standard deviations in Table 1. Full intervals appear in Table A4 (Appendix A.4).
Response to Reviewer sZN2 (1/4) — W1 — Controlled answer corruption
Thank you very much for reviewing our work and for your valuable comments on teacher reliability, the connection between theory and experiments, and the paper presentation. We have carefully addressed your comments, with detailed point-by-point responses provided below. We have also uploaded a revised manuscript with all changes highlighted in blue.
W1. Testing the label-free setting with unreliable teachers and corrupted answers
Thank you very much for raising this point. We agree that the original experiments provide limited coverage of unreliable teachers. We tested full construction with Phi-4-mini-instruct and controlled final-answer corruption across the source, search, and selection pools. Phi-4-Mini technical report
We use Phi to test construction with less reliable teacher references. Its reference accuracy on the 160 search/selection questions is 26.88% for Tracking, 63.75% for Colored Objects, and 87.50% for QuaRTz, with unusable answers counted as incorrect. On the same Colored Objects questions, Qwen3-8B and Ministral-3-8B achieve 95.63% and 78.13%, respectively. Model descriptions appear in the Qwen3 report and Ministral 3 report.
With Phi throughout construction, K2P's three-seed mean exceeds zero-shot in five of six task/student settings, while its comparison with GEPA-T varies across settings.
Controlled answer corruption throughout construction. On QuaRTz, we flip final A/B answers in all three 80-question pools (source, search, selection), updating the full response's final answer and parsed reference while retaining the original reasoning. Qwen3.5-9B performs synthesis and refinement. Each corrupted condition reruns synthesis, search, feedback, refinement, and selection before testing on all 784 questions. The 0% condition reuses the original runs.
Values are mean ± sample SD over three construction seeds, each with its own corruption mask at each rate. Masks are shared across students and nested across rates. Reference accuracies are averaged across masks and measured with gold answers after construction.
For Ministral, the 20% and 30% conditions select the same final prompt for each seed and reuse the same cached test outputs.
Ministral exceeds zero-shot in every run, including a 5.06 pp mean gain at 50% flipping. Qwen's means are below zero-shot at every rate and fall by 6.04 pp from 0% to 50%. Test accuracy varies non-monotonically across rates.