- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper distinguishes between generation and verification as two ways a pretrained teacher can expose its knowledge. When a teacher has learned a factual relation in one direction (A -> B), it may fail to generate the relation in the reverse direction (B -> A) while still being able to recognize the correct relation when evaluated in the direction it knows. Based on this observation, the paper introduces directional label distillation, where the teacher does not generate the reverse answer directly. Instead, it scores a set of candidate answers using the known direction, aggregates scores from multiple teacher channels, and selects the highest-scoring candidate as a pseudo-label for training the student.
The paper defines a known-direction scoring function in which the candidate changes while the continuation being scored remains in the direction where the teacher has stronger knowledge.
It also compares against reverse-direction scoring and addresses the effect of candidate name priors using PMI-like prior correction. The experiments further show that student accuracy largely follows pseudo-label quality: students reproduce the selected labels at very high rates, so improvements in label accuracy translate into improvements in student accuracy.
The method is evaluated on Wikidata parent–child relations. Known-direction scoring consistently produces more accurate labels than reverse-direction alternatives. Students trained with known-direction labels achieve approximately ~15 percentage points higher open-ended accuracy than students trained with prior-corrected reverse labels.
Critical strengths
- Distinction between teacher generation and verification: The paper identifies an important phenomenon: a pretrained model may fail to generate a fact in one direction while still being able to recognize the correct fact when candidates are scored in a direction where its knowledge is better expressed.
- Directional label distillation is a simple and effective formulation: The paper formalizes the above insight through a known-direction score, selects the candidate with the highest score, and uses the selected answer as a pseudo-label for student distillation. This turns the conceptual observation into a simple, practical distillation procedure, and the experiments show that the resulting labels consistently lead to substantially better student accuracy than reverse-direction alternatives.
Critical weaknesses
- Computational cost: The candidate-scoring procedure may be substantially more computationally expensive than conventional distillation, since multiple teachers must score potentially many candidates. Although the paper acknowledges this limitation, it does not quantify the computational overhead or demonstrate that the proposed approach remains practical as the candidate inventory or number of teacher channels grows.
- Potential pretraining exposure: The paper makes a commendable effort to prevent adaptation-time exposure by removing forward edges involving the relevant children before training, so the student does not see those facts in the forward training data. However, this does not establish that the student has never seen these facts during pretraining. The evaluation relies on relatively well-known people and parent-child relationships from Wikipedia, which could easily have appeared in the pretraining corpus of Qwen3 or other models. The warm-start controls show that reverse-label training is important, but do not establish that the observed gains represent acquisition of previously unknown knowledge rather than improved elicitation or memorization of knowledge already present in the student. This is particularly relevant given the paper's broader claim about knowledge transfer.
- Hard-label SFT vs. distillation: The student is ultimately trained only on the selected final answer, making the training procedure resemble pseudo-labeling/SFT more than conventional knowledge distillation with a dense teacher signal. While the teacher's verification scores are used to construct higher-quality labels, the paper does not investigate whether retaining the teacher's score distribution or other soft supervision provides additional benefits. Clarifying the distinction between directional label distillation and pseudo-label-based SFT, and comparing against soft-distillation alternatives, would strengthen the methodological contribution.
- Generality beyond the studied factual setting: It remains unclear whether directional verification continues to work for harder reasoning tasks and more general knowledge relations. The experiments focus on parent-child relations with a fixed candidate inventory, while the paper's central motivation is broader: that a model may be able to verify knowledge in a direction where it cannot generate it.
Questions for authors
Please address the concerns raised above (Critical Weaknesses Section).
16 comments
Response to Reviewer KPfP (2/9)
W1, inventory size and channel count. Scoring a candidate list costs the same at any inventory size, the full-inventory scan grows only linearly and is shared by every label source, and label accuracy stops rising at eight channels.
- Inventory size. Label selection scores only the candidate list, so its cost does not depend on the inventory, and the 64-name lists of the unscreened cohort and of the three new relations need no teacher scan. Building the acquired pools, where three teachers scan all 8,218 names, takes about 41 A100-seconds for each query and scoring direction at the same rate (three teachers times 8,218 names at 1.6 to 1.7 ms, new Appendix L) and grows linearly with inventory size and teacher count (Appendix E, page 21, page 22 in the revision). This scan is shared by every label source that picks from these pools, including all reverse comparators and the list-mapped generation of Table 20, so every label source starts from the same scan. A full scan of this kind, run for the 512 unscreened parents with no inserted answer, gives Table 16, where the known-direction pipeline labels 66.21% correctly against 39.06% for the summed-DC pipeline. For inventories far larger than 8,218 names a cheaper proposal stage can run in front of the scorer (Section 7 of the revision), and the label-selection cost of Table R3 stays the same because it depends only on the list.
- Number of channels. A channel is one teacher and template pair (Section 2), so each added channel is one more scoring pass, about 107 A100-seconds for 1,000 queries of 64 candidates for each 7 to 8B teacher we timed. On the 1,500 acquired pools label accuracy is 90.20, 91.53, 91.07 and 90.80% with 6, 8, 10 and 12 channels (Table 28), so it stops rising at eight. The eight-channel labels of the paper's default-exposure controls, four teachers with two templates each and so twice the four-teacher scoring passes of Table R3, already exceed each single-teacher pipeline, 91.53% against 81.13% to 83.67% (Table 26b).
- The cost is paid once and offline. Labeling 1,000 queries with all four teachers takes 0.12 A100 GPU-hours of scoring, or 0.25 with the one-time loading of the four teachers, less than half of one MDM-0.6B training run, which took 0.53 A100 GPU-hours including model loading and evaluation (mean of eight runs on A100-SXM4 cards). Every student reuses the labels, and the teachers are discarded at inference (Section 2.3), so the label source does not change the student's inference cost. Two students that differ only in their label source both answer 1,000 queries in 15.34 seconds at batch size 64 on one RTX A6000 card, scaled from their 512 trained reverse queries (new Appendix L).
Response to Reviewer KPfP (3/9)
W2. On invented facts that neither the student nor the untouched teacher holds, students trained on known-direction labels answer 96.8% of the reverse queries, against 1.9% with summed-DC labels and 0.0% without reverse labels, so they learn the facts stated in their labels and do not recall them from pretraining.
"The warm-start controls show that reverse-label training is important, but do not establish that the observed gains represent acquisition of previously unknown knowledge rather than improved elicitation or memorization of knowledge already present in the student."
This question is central to understanding how language models acquire knowledge, because distillation helps only if the student comes to hold knowledge it lacked, and the reversal curse is the case in which the student cannot infer that knowledge on its own. The evidence below shows that the students learn the facts stated in their labels.
We ran the direct test of acquisition, with facts the student does not hold before training (Table R6, new Appendix M and Table 37 of the revision, to which Section 3 of the revision now points). We invented 1,000 parent and child pairs with pseudoword surnames, each used once and absent from our 10,505-pair corpus. The untouched Qwen3-8B completes neither direction for any pair (0 of 1,000 each) and picks the true child at chance (1.2% against 1/64 = 1.6%). After LoRA training only on child-first sentences, the teacher recalls 96.2% of parents yet names the true child for 0 of 1,000 parents. MDM-0.6B students then follow the paper's recipe, with forward training on the real corpus only, so the students see the invented facts only in the reverse labels.
Table R6. Invented facts, 1,000 trained reverse queries with 64-name lists. Teacher label accuracy and student open and matched accuracy (%, mean ± sample SD over three seeds).
| Label source | Label accuracy | Student open accuracy | Student matched accuracy |
|---|---|---|---|
| Known direction | 100.0 | 96.8 ± 0.9 | 99.4 ± 0.3 |
| Summed DC | 1.9 | 1.9 ± 0.0 | 1.9 ± 0.0 |
Reverse labels from mean and summed scores are 2.2% and 1.8% correct, and students trained without reverse labels answer 0.0% of the queries. The student therefore acquires facts that neither it nor the untouched teacher holds, and it learns them from the known-direction labels.
Three results already in the submitted paper show the same on real facts.
- Students trained on wrong labels almost never produce the true child. The reverse-mean students see 645 wrong labels among the 1,390 exposure-free queries, and under inventory matching they name the true child for at most 5 of them in any run and for none under open scoring (Table 12, columns F, Appendix D, page 19, page 20 in the revision). Students reproduce wrong labels as faithfully as right ones. On the 666 exposure-free queries where the two label sources disagree, reverse-mean labels are 5.26% correct and known-direction labels 82.73% correct, and after inventory matching students return their selected label for 96.60% and 97.25% of these queries (Table 11, Appendix D, page 19, page 20 in the revision). None of the known-direction, reverse and summed-DC students turns a wrong label into a whole-answer match (Appendix D, same page). If training elicited knowledge already in the student, wrong labels would be corrected far more often.
- Students trained without reverse labels, MDM-4B and AR-8B models larger than the MDM-0.6B students of Table 1, stay below 3% open accuracy on all 1,500 trained queries of the screened cohort, including with the any-order masked objective MLM-U (Table 3b, page 8), so their pretraining, even together with every true forward pair, does not make them generate these reverse answers. Their inventory-matched accuracy in the same table, 28.16% to 44.38%, stays below the 69.80% that matching on the surname alone reaches with the original inventory, so it reflects surname cues that the matching step can use. Enlarging the matching inventory with parent-only names cuts the warm-start MDM-4B students from 39.56% to 10.69%, while students trained on reverse labels change by less than one point (Section 5.2, Table 27b, page 28, page 29 in the revision).
- Student answers follow the labels over the whole range of label quality, and matched accuracy stays within about one point of label accuracy from random labels, 14.13% correct and 13.91% matched, to gold labels, 100% correct and 98.91% matched (Table 3a, page 8). The same holds in the new student settings of Section 4.5. After inventory matching every student arm stays within 0.26 points of its label accuracy on the three new relations (new Appendix J), and within 0.46 points on three-seed means with teachers of up to 70B parameters (new Appendix K) and on two-hop queries (new Appendix O).
Response to Reviewer KPfP (1/9)
We thank the reviewer for the careful reading and for recognizing the important phenomenon the paper identifies and its simple and effective formulation. The revision answers each of the four weaknesses with a new measurement, and text revised in response to the reviews is set in blue. All four teachers label 1,000 queries in about seven minutes on one A100 card, at the same cost as reverse scoring (Appendix L). On invented facts that neither the student nor the untouched teacher holds, students trained on known-direction labels answer 96.8% of the reverse queries against 1.9% with summed-DC labels (Appendix M). Students trained on the selected label lead students trained on the teachers' score distribution in every paired seed at four temperatures (Appendix N). The advantage holds on three further relations (Appendix J) and on two-hop queries, where students trained on known-direction labels lead in all 24 seed-paired comparisons (Appendix O). New Section 4.5 on page 8 reports the results beyond parent and child facts in the main text, and each point below gives its place in the revised PDF.
W1. All four teachers label 1,000 queries of 64 candidates in 427 A100-seconds, about seven minutes on one card and less than a quarter of one MDM-0.6B training run, or less than half with teacher loading included, at the same cost as reverse scoring. With OLMo-2-7B as the single teacher, each correct label costs less than one seventh as much as with generation from the same teacher, and label accuracy peaks at eight channels, so the channel count need not grow.
We timed the label sources of the paper on one A100 80GB card, on the 512 uniform lists of the unscreened cohort with 64 candidates each, and scaled the times to 1,000 queries (Table R3, new Appendix L and Table 36 of the revision). Label accuracy is on the 512 uniform lists (Tables 13, 18 and 20).
Table R3. A100-seconds to label 1,000 queries of 64 candidates, scaled from the 512 uniform lists, and label accuracy (%) on the 512 uniform lists. Summed DC adds one context-only score for each distinct name, paid once.