- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a knowledge distillation method for transferring molecular teachers into a 2D graph student by normalizing the teacher gradients while filtering the directions conflicting the task. The idea is supported by many theoretical insights that help to understand the paper better.
Critical strengths
The main strength is that the paper is well-written and the methodology is clear and there are theoretical insights that can help to understand it better.
Critical weaknesses
- As I mentioned before, the theoretical part is one main strength of the paper but at the same time, having too many equations (322 equations!!) brings confusion. Looking at Theorem 2.1, do the assumptions hold in experiments? Can the authors do empirical verifications?
- Also looking at Theorem 2.2 compared with the appendix (Lemma B.37,38,46), the theorem claims to reduce update variance while those Lemmas bound EMA-related perturbations. Could the authors clarify how these results establish the claimed variance ordering?
- Looking at the student models, they are all the same type but have different pretraining. Could the authors add other student architectures?
Questions for authors
Please see the Critical Weaknesses.
9 comments
Table R1b. Actual parameter-space gradient alignment and update magnitude ().
| Dataset | Method | Alignment | Relative Norm | Descent () | Non-conflict (%) |
|---|---|---|---|---|---|
| BACE | Uniform KD | -0.018 | 0.900 | 0.982 | 50.0 |
| FORTE | +0.160 | 0.389 | 1.160 | 86.7 | |
| MUV | Uniform KD | -0.117 | 0.534 | 0.883 | 15.0 |
| FORTE | +0.012 | 0.025 | 1.012 | 65.7 | |
| SIDER | Uniform KD | -0.001 | 0.659 | 0.999 | 45.0 |
| FORTE | +0.014 | 0.177 | 1.014 | 66.3 | |
| ToxCast | Uniform KD | -0.026 | 0.517 | 0.974 | 35.0 |
| FORTE | +0.010 | 0.111 | 1.010 | 84.0 |
Alignment: Normalized inner product between the supervised and auxiliary parameter gradients. Positive values indicate favorable first-order alignment with the main task. Relative Norm: Auxiliary-gradient magnitude relative to the supervised gradient. Descent (): Normalized inner product between the supervised gradient and the combined update direction. Values above 1 indicate a positive first-order contribution from auxiliary supervision. Non-conflict (%): Percentage of audited batches for which the auxiliary and supervised parameter gradients have a nonnegative inner product.
Result. Regarding the first factor in Eq. (14), Uniform KD has negative median Alignment and Descent values below 1.0 on all four datasets, whereas FORTE maintains positive median Alignment and Descent values above 1.0 throughout. Regarding the second factor in Eq. (14), FORTE produces smaller relative auxiliary-gradient norms while improving parameter-space non-conflict rates by 21.3–50.7 percentage points.
(3) Supervised Main-Task Optimization: Do these gradient improvements correspond to fewer training epochs?
We further evaluate whether the observed gradient properties correspond to faster optimization of the supervised main-task loss itself, rather than merely improving final predictive performance. We conduct 72 paired experiments across three additional GNN backbones (GCN, GAT, and GraphSAGE), eight molecular datasets, and three random seeds. For each pair, the reference target is defined as the mean Student-only supervised training loss over epochs 96–100. We record the first epoch at which each method's five-epoch moving-average supervised loss reaches that target.
Table R2. Training epochs required to reach the paired Student-only supervised-loss target.
| Backbone | Pairs | Student-only epochs | FORTE epochs | Paired epoch ratio |
|---|---|---|---|---|
| GCN | 24 | 98.50 [94.75, 100] | 7.00 [6, 8] | 14.29 [12.41, 16.00] |
| GAT | 24 | 100 [97, 100] | 7.00 [6, 9] | 14.07 [11.11, 16.42] |
| GraphSAGE | 24 | 100 [99.75, 100] | 7.00 [6, 7.50] | 14.29 [12.74, 16.67] |
| Overall | 72 | 100 [96.75, 100] | 7.00 [6, 8.25] | 14.29 [11.24, 16.50] |
Column definitions: Pairs: Number of matched dataset–seed experiments for each backbone. Student-only epochs: Epochs required by the supervised-only model to reach its reference loss. FORTE epochs: Epochs required by FORTE to reach the same reference loss in each pair. Paired epoch ratio: Student-only epochs divided by FORTE epochs, computed within each matched pair before aggregation. This measures training-epoch efficiency, not wall-clock speedup.
Result. Across all 72 paired experiments, FORTE reaches the Student-only reference supervised loss in substantially fewer epochs. The overall median paired epoch ratio is 14.29, with similar improvements observed across GCN, GAT, and GraphSAGE. Together, Table R1a examines the restricted geometric distortion, Table R1b evaluates the parameter-space alignment and norm factors in Eq. (14), and Table R2 shows that FORTE reaches a common supervised-loss target in substantially fewer training epochs.
W2: Clarifying the Scope of Theorem 2.2 and Its Connection to Lemmas B.37, B.38, and Corollary B.46
We thank the reviewer for this exceptionally sharp and careful reading of Appendix B.6. We clarify the scope of the informal statement in Theorem 2.2: in our analysis (Eq. 233 of Appendix B.6), the mini-batch update deviation decomposes into ordinary graph-sampling error and current-batch scale-estimation perturbation . Because the routing map is nonlinear with respect to the scale estimate, Appendix B.6 formalizes the variance reduction of this scale-induced update perturbation through its conditional second-moment upper bound (variance proxy), denoted by and in Theorem 2.2. Specifically, Lemma B.37 establishes the exact variance attenuation at the scale-estimation level, Lemma B.38 propagates this attenuation to the second-moment bound of GNN parameter updates, and Corollary B.46 quantifies its contribution to the stochastic optimization bound.
(1) Why Does EMA Reduce Scale-Estimation Variance? (Lemma B.37)
Lemma B.37 (Eq. 235) proves that the deviation of the current-batch EMA scale from its full-data oracle counterpart is:
Conditioned on the previous EMA history , the historical state is fixed, and only the current mini-batch estimate introduces randomness. Therefore:
In contrast, instantaneous estimation () directly uses the current mini-batch (), yielding conditional variance . Since , we have . Thus, EMA reduces the conditional variance contributed by current mini-batch scale noise by the exact factor , with strict inequality whenever , at least one previous observation exists (), and the scale noise has nonzero variance.
(2) How Does This Variance Reduction Propagate to GNN Parameter Updates? (Lemma B.38)
Lemma B.38 proves that perturbations in teacher-scale estimates propagate to GNN parameter updates through a Lipschitz-continuous mapping:
Combining this Lipschitz bound with Lemma B.37, the conditional second moment (variance proxy) of the scale-induced parameter-update perturbation satisfies:
where is the conditional variance of the current mini-batch scale estimate. For instantaneous scale estimation, the corresponding bound contains the same terms without the squared attenuation factors . This directly establishes in Theorem 2.2 (with strict inequality when scale noise and routing sensitivity are nonzero).
(3) Connection to the Stochastic Optimization Error Bound (Corollary B.46)
Corollary B.46 (Eq. 309–310) further shows how this variance proxy enters the overall stochastic convergence bound: the contribution of teacher-scale estimation noise is attenuated to
whereas instantaneous scale estimation lacks the attenuation coefficient. Together, Lemmas B.37, B.38, and Corollary B.46 form the formal basis for the informal summary in Theorem 2.2.
We deeply appreciate the reviewer's insightful question on connecting our theoretical assumptions to empirical observations. We conducted additional experiments to examine the key conditions underlying Theorem 2.1 and their relevance to actual main-task optimization. Specifically, we investigate three questions: (1) Does the GNN Jacobian preserve the restricted task–auxiliary gradient geometry? (2) Does FORTE improve the descent and update-norm factors in Theorem 2.1? (3) Do these improvements correspond to faster optimization of the supervised downstream loss? We report the measurements and results for (1)–(3) below.
(1) Restricted Scaled Isometry (Definition B.2): Does the GNN Jacobian preserve the task–auxiliary gradient geometry?
Theorem 2.1 assumes a smooth supervised objective and restricted scaled isometry of the GNN Jacobian on the subspace spanned by the supervised and auxiliary gradients. The latter condition connects gradient relationships in the student representation space to actual GNN parameter updates.
Experimental verification. We measure the restricted Jacobian distortion δB at the validation-selected checkpoint using the actual GNN Jacobian for each method.
Table R1a. Restricted Jacobian distortion for Uniform KD and FORTE.
Column definitions. Each value is the median restricted Jacobian distortion over the audited validation-checkpoint batches for the corresponding method. Smaller δB indicates better preservation of gradient geometry, up to a common scaling factor.
Result. All reported median distortions are below 0.5. FORTE exhibits lower median distortion than Uniform KD on all four datasets. These measurements characterize the geometry of the audited task–auxiliary subspaces; they do not establish that restricted isometry holds uniformly along the entire optimization trajectory.
(2) Gradient Alignment and Relative-Norm Conditions (Theorem 2.1): Does FORTE provide a more favorable main-task update?
Theorem 2.1 (Eq. 14) compares the convergence bounds of Normalized Uniform KD and FORTE through the direction and magnitude of their parameter-space updates:
sf←n=cfcn=(1−λγnμf)2(ρf1+λκn)2,λγn<1,
where the first factor compares the main-task descent coefficient (μf vs. 1−λγn), and the second factor compares the relative update norm (ρf vs. 1+λκn). When sf←n>1, FORTE has a smaller sufficient iteration bound under the stated assumptions.
Experimental verification. To evaluate both factors in Eq. (14), we use the actual supervised and auxiliary parameter gradients (Gθ,B and Aθ,B) on the shared GNN parameters to directly measure parameter-space alignment, relative auxiliary-gradient norm, the observed first-order descent factor cBobs (where cBobs=1+Alignment under the standard weight λ=1 in Appendix C.1), and the non-conflict rate (⟨Gθ,B,Aθ,B⟩≥0):
Alignment=∥Gθ,B∥22⟨Gθ,B,Aθ,B⟩,Relative Norm=∥Gθ,B∥2∥Aθ,B∥2,cBobs=∥Gθ,B∥22⟨Gθ,B,Gθ,B+λAθ,B⟩.