Questions
I group my questions into two parts. Part A covers issues the authors should address for me to maintain my score; most concern clarity and presentation. Part B covers what I see as the central limitations of the work; convincing responses to these would lead me to raise my score.
A. Would not lower the score
A1. The illustrative example in §3.1 is disconnected from the surrounding argument.
The max(xs) / min(xs) / max(0,max(xs)) example (ll. 185–188) is placed between the linearization argument and the definition of δ. The main text returns to it only in the final sentence of P3. Can you move the example to where it is used, as motivation for P3, with a clearer discussion?
A2. What does the decomposition into transfer mass and adverse share add beyond net transfer?
§3.2 introduces aQ and cQ as two coordinates of a source, and the discussion after Eq. (9) presents a trade-off between them. However, Proposition 1 depends on them only through their product κQ, and so as does the favorable region.
P1 and P3 are likewise stated in terms of κQ (Eqs. 63, 65).
Please explain what aQ and cQ capture that κQ does not. If they carry separate meaning, I suggest organizing §3 around them: the transition from §3.2 to §3.3 could state which values of aQ and cQ are desirable and derive P1–P3 as interventions that move them. Otherwise, I suggest simplifying the exposition to κQ.
A3. Limitations of the linearized feature model.
The model takes ϕ to be the score gradient at θ0 and holds these features fixed, a tangent (NTK-style) linearization of the network. The appendix acknowledges this assumption (final paragraphs of §F.2.2 and §F.3). The main text, however, contains no discussion of limitations. For one, P2 relies on the decomposition ϕ=ϕdom(y)+ϕpair(x,y) and on its "order-invariant part." Neither is a property of score-gradient features. Could you add a limitations discussion to the main text covering these points, and state which conclusions are expected to hold outside the linearized regime?
A4. Which sources does each figure include?
The pool of sources changes across figures without being stated. Fig. 1a ("every smaller Base source available") and the "best observed smaller Base source" in Fig. 2a/b include only Qwen2.5 sources, while Fig. 1b uses Llama-1B. Please state the source pool in each figure caption, and explain why the "best" panels are restricted to Qwen2.5. Also explain why a cross-family source was chosen for the headline mixture in Fig. 1b; the in-family Qwen2.5-3B mixture in Fig. 2c could serve this role.
A5. Terminology for models and reject sources.
The naming is inconsistent and in places contradicts standard usage:
- "Base" refers to instruction-tuned checkpoints. Qwen2.5-0.5B-Instruct is a "Subscale Base" source (Tab. 9), and Llama-3.2-1B-Instruct is a "Llama Base" source (§C.1).
- "Base" also denotes the student's original model in the Fig. 2 legend and Tab. 4, while Tab. 3 calls the same row "Vanilla."
- Tab. 12 labels a smaller 7B source "vanilla 7B," although "vanilla" otherwise refers to the model at the student's scale.
- One concept appears under at least five names: smaller Base, subscale, strict-subscale, strictly smaller Base, and Subscale Base.
Please adopt one naming scheme for student checkpoints and reject sources, avoid calling instruction-tuned models "Base", and add a short glossary table.
A6. The "Reassigned" and "Aligned" controls.
As far as I can tell, "Aligned" is simply the standard smaller-source construction: its values in Tab. 7 match the endpoints in Tab. 9. "Reassigned" is presented as Aligned with the prompt correspondence removed. Can you rename the arms (e.g., "prompt-matched" vs. "prompt-shuffled") and list every difference between them?
A7. Repulsion, attraction, and coupling.
Repulsion and attraction are defined in §3.3 only through the sign of γ in Eq. (12), within a single clause. Their operational meaning, favoring lower vs. higher reference likelihood, appears only in §4.4. "Reference coupling" is central to the abstract's design principle, yet it is never formally defined in §3. Please define repulsion, attraction, and coupling formally where they are introduced, and use one sign convention across Fig. 4a, Tab. 11, and §D.4.
B. Would raise the score
B1. Model family of the reject sources.
Whether the smaller sources come from the student's model family is central to interpreting the main result. The paper does not state it where the sources are defined (§4.1).
- Main scaling matrix uses only Qwen2.5-Instruct sources. This can be inferred from Tab. 9, but Tab. 3 lists only sizes.
- Llama sources appear only in App. C.1, for the 14B/32B/72B students, with a single pointer in §4.2.
- The experiment in Fig. 1b uses the cross-family Llama-3.2-1B-Instruct before the reader has been told that Llama sources are used at all.
The cross-family evidence is also thinner than the main-text pointer suggests:
- There is no 7B student.
- Fig. 5 shows only MATH-500 for the math line.
- On AIME, Llama-1B falls below both Self constructions at 32B (29.2 vs. 30.8/30.4) and at 72B (34.2 vs. 35.4/37.5) (Tab. 5).
Please state the family of every reject source in §4.1, and summarize the cross-family results in the main text. Then either qualify any claim that the result is independent of family, or support it, for example with cross-family sources for the 7B student and AIME results alongside Fig. 5.
B2. Positioning of the theory.
The paper presents its theory as supporting the main claims, but §3.3 positions it as a source of hypotheses to test. Each of P1–P3 ends with "motivates testing," and the introduction states that the analysis "motivates three interventions." Under this framing, the theory predicts secondary phenomena rather than supporting the central claim. The introduction asks why the cheaper frozen source also provides stronger negative supervision. The theory never evaluates κQ for Self versus a smaller source, and model scale does not appear in the analysis.
I would appreciate one of the following:
- (a) State precisely which claims the theory supports and which it does not, and revise the abstract, contributions, and conclusion accordingly. A careful discussion would suffice here.
- (b) Strengthen the link between the theory and the main result, for example by deriving what the model implies for Self versus smaller-source rejects and testing that implication.
B3. The favorable region (Eq. 9) is never developed.
The favorable region is presented as a contribution in the abstract, the introduction, and the conclusion. It is also the formal object behind the claim that a reject distribution Q outperforms Q0. Yet the paper never develops it, as after its definition, Eq. (9) is revisited only once, at the end of §F.3, to note that the region is nonempty. The authors also directly claim aQ and cQ are "not directly observable" (§3.3), and EH is never estimated. No reject source is ever shown to lie inside or outside the region.
B4. Reference-likelihood reselection (P3) is the most practically useful result, but it is under-developed.
Within a fixed candidate bank, selecting rejects with lower likelihood under the SeqKD reference improves performance for every source (Tab. 11). This suggests a cheap, general recipe for constructing rejects, yet the paper treats it as one of three tests. I would like it discussed more explicitly and with more emphasis.