- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an explanation of subliminal learning through an analysis of learning dynamics. Explicitly expanding out a gradient training step, they identify a cross-task kernel which captures the influence of any ghost-output on the student's task prediction. They analyze this kernel and use it to understand different phenomena in subliminal learning, and verify their theory on a toy MNIST model and a larger LLM.
Critical strengths
The exposition and derivation steps are clear, with the theory fully contained in the main body. The problem is well motivated, and the mechanism seems like a reasonable explanation.
Critical weaknesses
While the approach seems like it could work, as it stands now, the theoretical derivations are missing crucial justifications and make informal assumptions which are not obviously true, and are insufficient to defend the strong claims that build on these. For example, taking only the first-order approximation of the gradient update step in Equation 11 is loosely justified by stating the student and teacher will be close in parameter space because they share an initialization and are trained in the overparametrized regime, but the teacher may have undergone substantial training already, so this gap may not be small. This needs a formally stated assumption, and should be verified in the experiments. Relatedly, the analysis for the extension to multistep from the single step case is not clear. The benefit of probing with random tokens also warrants stronger theoretical justification and empirical verification.
The work is in the right direction, but it feels disingenuous to say that this paper gives rigorous mathematical guarantees of subliminal learning at its current state.
Questions for authors
-
Can you guarantee PSD alignment still holds if you include the higher-order remainder terms in equation 11 (making necessary assumptions on their size)? I'm assuming yes, but you should make those assumptions clear.
-
For the multistep case, it is written that Equation 21 implies long term transfer because "Over many iterations, the zero- mean perturbation and interference terms average out". Why must either term necessarily be zero-mean?
-
Can you ensure that all these approximations do not accumulate or interact in a bad way? While each assumption is not completely unreasonable, the errors together can compound—it is not clear whether this is the case here following the informal analysis.
-
For the claim that Gaussian noise is always preferred: why does having full-rank input covariance necessarily imply large overlap of the Jacobians for the task and synthetic sample (nonzero/large kernel)? For example, two individual samples (one from MNIST, one from Gaussian) can activate completely disjoint sets of neurons. It seems to me there is no fundamental guarantee that the Jacobians will overlap just by sampling from Gaussian.