acceptodds
Under review as a conference paper at ICLR 2027

Learn Only What Helps : Task-Prioritized Distillation from Heterogeneous Teachers for 2D Molecular Property Prediction

Abstract

Accurate molecular property prediction (MPP) can benefit from heterogeneous molecular representations that encode complementary structural and chemical information. Yet obtaining such representations often requires additional molecular views or computations that are costly or impractical for large-scale screening. Knowledge distillation (KD) provides a natural way to transfer such supervision to a 2D graph student during training while retaining efficient 2D-only inference. However, existing multi-teacher KD methods largely focus on how to combine supervision from multiple teachers, but overlook whether a given teacher signal helps the downstream objective at a given student update. We argue that this is particularly significant for heterogeneous molecular teachers: differences in gradient magnitude can cause some teachers to dominate the shared update, while directional disagreement can make even a properly scaled teacher interfere with task descent. To address this challenge, we propose First-Order Routing of Teacher Expertise (FORTE), which reframes heterogeneous distillation as a task-prioritized routing problem rather than a teacher aggregation problem. Instead of assigning supervision to all available teachers, FORTE uses the downstream task gradient to determine both whether a teacher update is admissible and how strongly it should influence the student. Compatible teachers are routed through and a null route abstains when none qualifies, yielding an auxiliary update that is both aligned with and bounded by the task gradient.Theoretically, we show that, under the stated assumptions, FORTE improves the convergence bounds for the supervised downstream objective over conventional teacher aggregation, yielding a smaller sufficient iteration budget. Experimental results over a wide range of benchmarks show consistent improvements in both classification and regression, demonstrating the effectiveness and generality of our proposed method.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
molecular property prediction, knowledge distillation, heterogeneous teachers, gradient routing, graph neural networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NB1uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a knowledge distillation method for transferring molecular teachers into a 2D graph student by normalizing the teacher gradients while filtering the directions conflicting the task. The idea is supported by many theoretical insights that help to understand the paper better.

Critical strengths

The main strength is that the paper is well-written and the methodology is clear and there are theoretical insights that can help to understand it better.

Critical weaknesses
  • As I mentioned before, the theoretical part is one main strength of the paper but at the same time, having too many equations (322 equations!!) brings confusion. Looking at Theorem 2.1, do the assumptions hold in experiments? Can the authors do empirical verifications?
  • Also looking at Theorem 2.2 compared with the appendix (Lemma B.37,38,46), the theorem claims to reduce update variance while those Lemmas bound EMA-related perturbations. Could the authors clarify how these results establish the claimed variance ordering?
  • Looking at the student models, they are all the same type but have different pretraining. Could the authors add other student architectures?
Questions for authors

Please see the Critical Weaknesses.

9 comments

W1: Experimental Validation and Clarity of Theorem 2.1

We deeply appreciate the reviewer's insightful question on connecting our theoretical assumptions to empirical observations. We conducted additional experiments to examine the key conditions underlying Theorem 2.1 and their relevance to actual main-task optimization. Specifically, we investigate three questions: (1) Does the GNN Jacobian preserve the restricted task–auxiliary gradient geometry? (2) Does FORTE improve the descent and update-norm factors in Theorem 2.1? (3) Do these improvements correspond to faster optimization of the supervised downstream loss? We report the measurements and results for (1)–(3) below.

(1) Restricted Scaled Isometry (Definition B.2): Does the GNN Jacobian preserve the task–auxiliary gradient geometry?

Theorem 2.1 assumes a smooth supervised objective and restricted scaled isometry of the GNN Jacobian on the subspace spanned by the supervised and auxiliary gradients. The latter condition connects gradient relationships in the student representation space to actual GNN parameter updates.

Experimental verification. We measure the restricted Jacobian distortion δB\delta_B at the validation-selected checkpoint using the actual GNN Jacobian for each method.

Table R1a. Restricted Jacobian distortion for Uniform KD and FORTE.

DatasetUniform KD: Median δB\delta_BFORTE: Median δB\delta_B
BACE0.4780.367
MUV0.3300.324
SIDER0.4570.368
ToxCast0.2450.218

Column definitions. Each value is the median restricted Jacobian distortion over the audited validation-checkpoint batches for the corresponding method. Smaller δB\delta_B indicates better preservation of gradient geometry, up to a common scaling factor.

Result. All reported median distortions are below 0.5. FORTE exhibits lower median distortion than Uniform KD on all four datasets. These measurements characterize the geometry of the audited task–auxiliary subspaces; they do not establish that restricted isometry holds uniformly along the entire optimization trajectory.

(2) Gradient Alignment and Relative-Norm Conditions (Theorem 2.1): Does FORTE provide a more favorable main-task update?

Theorem 2.1 (Eq. 14) compares the convergence bounds of Normalized Uniform KD and FORTE through the direction and magnitude of their parameter-space updates:

sf←n=cncf=(μf1−λγn)2(1+λκnρf)2,λγn<1,s_{f\leftarrow n} = \frac{c_n}{c_f} = \left(\frac{\mu_f}{1-\lambda\gamma_n}\right)^2\left(\frac{1+\lambda\kappa_n}{\rho_f}\right)^2, \qquad \lambda\gamma_n < 1,

where the first factor compares the main-task descent coefficient (μf\mu_f vs. 1−λγn1-\lambda\gamma_n), and the second factor compares the relative update norm (ρf\rho_f vs. 1+λκn1+\lambda\kappa_n). When sf←n>1s_{f\leftarrow n}>1, FORTE has a smaller sufficient iteration bound under the stated assumptions.

Experimental verification. To evaluate both factors in Eq. (14), we use the actual supervised and auxiliary parameter gradients (Gθ,BG_{\theta,B} and Aθ,BA_{\theta,B}) on the shared GNN parameters to directly measure parameter-space alignment, relative auxiliary-gradient norm, the observed first-order descent factor cBobsc_B^{\mathrm{obs}} (where cBobs=1+Alignmentc_B^{\mathrm{obs}} = 1 + \mathrm{Alignment} under the standard weight λ=1\lambda=1 in Appendix C.1), and the non-conflict rate (⟨Gθ,B,Aθ,B⟩≥0\langle G_{\theta,B}, A_{\theta,B}\rangle \ge 0):

Alignment=⟨Gθ,B,Aθ,B⟩∥Gθ,B∥22,Relative Norm=∥Aθ,B∥2∥Gθ,B∥2,cBobs=⟨Gθ,B,Gθ,B+λAθ,B⟩∥Gθ,B∥22.\mathrm{Alignment} = \frac{\langle G_{\theta,B}, A_{\theta,B}\rangle}{\Vert G_{\theta,B}\Vert_2^2}, \qquad \mathrm{Relative\ Norm} = \frac{\Vert A_{\theta,B}\Vert_2}{\Vert G_{\theta,B}\Vert_2}, \qquad c_B^{\mathrm{obs}} = \frac{\langle G_{\theta,B},G_{\theta,B}+\lambda A_{\theta,B}\rangle}{\Vert G_{\theta,B}\Vert_2^2}.

Table R1b. Actual parameter-space gradient alignment and update magnitude (λ=1\lambda=1).

DatasetMethodAlignmentRelative NormDescent (cBobsc_B^{\mathrm{obs}})Non-conflict (%)
BACEUniform KD-0.0180.9000.98250.0
FORTE+0.1600.3891.16086.7
MUVUniform KD-0.1170.5340.88315.0
FORTE+0.0120.0251.01265.7
SIDERUniform KD-0.0010.6590.99945.0
FORTE+0.0140.1771.01466.3
ToxCastUniform KD-0.0260.5170.97435.0
FORTE+0.0100.1111.01084.0

Alignment: Normalized inner product between the supervised and auxiliary parameter gradients. Positive values indicate favorable first-order alignment with the main task. Relative Norm: Auxiliary-gradient magnitude relative to the supervised gradient. Descent (cBobsc_B^{\mathrm{obs}}): Normalized inner product between the supervised gradient and the combined update direction. Values above 1 indicate a positive first-order contribution from auxiliary supervision. Non-conflict (%): Percentage of audited batches for which the auxiliary and supervised parameter gradients have a nonnegative inner product.

Result. Regarding the first factor in Eq. (14), Uniform KD has negative median Alignment and Descent values below 1.0 on all four datasets, whereas FORTE maintains positive median Alignment and Descent values above 1.0 throughout. Regarding the second factor in Eq. (14), FORTE produces smaller relative auxiliary-gradient norms while improving parameter-space non-conflict rates by 21.3–50.7 percentage points.

(3) Supervised Main-Task Optimization: Do these gradient improvements correspond to fewer training epochs?

We further evaluate whether the observed gradient properties correspond to faster optimization of the supervised main-task loss itself, rather than merely improving final predictive performance. We conduct 72 paired experiments across three additional GNN backbones (GCN, GAT, and GraphSAGE), eight molecular datasets, and three random seeds. For each pair, the reference target is defined as the mean Student-only supervised training loss over epochs 96–100. We record the first epoch at which each method's five-epoch moving-average supervised loss reaches that target.

Table R2. Training epochs required to reach the paired Student-only supervised-loss target.

BackbonePairsStudent-only epochsFORTE epochsPaired epoch ratio
GCN2498.50 [94.75, 100]7.00 [6, 8]14.29 [12.41, 16.00]
GAT24100 [97, 100]7.00 [6, 9]14.07 [11.11, 16.42]
GraphSAGE24100 [99.75, 100]7.00 [6, 7.50]14.29 [12.74, 16.67]
Overall72100 [96.75, 100]7.00 [6, 8.25]14.29 [11.24, 16.50]

Column definitions: Pairs: Number of matched dataset–seed experiments for each backbone. Student-only epochs: Epochs required by the supervised-only model to reach its reference loss. FORTE epochs: Epochs required by FORTE to reach the same reference loss in each pair. Paired epoch ratio: Student-only epochs divided by FORTE epochs, computed within each matched pair before aggregation. This measures training-epoch efficiency, not wall-clock speedup.

Result. Across all 72 paired experiments, FORTE reaches the Student-only reference supervised loss in substantially fewer epochs. The overall median paired epoch ratio is 14.29, with similar improvements observed across GCN, GAT, and GraphSAGE. Together, Table R1a examines the restricted geometric distortion, Table R1b evaluates the parameter-space alignment and norm factors in Eq. (14), and Table R2 shows that FORTE reaches a common supervised-loss target in substantially fewer training epochs.

W2: Clarifying the Scope of Theorem 2.2 and Its Connection to Lemmas B.37, B.38, and Corollary B.46

We thank the reviewer for this exceptionally sharp and careful reading of Appendix B.6. We clarify the scope of the informal statement in Theorem 2.2: in our analysis (Eq. 233 of Appendix B.6), the mini-batch update deviation decomposes into ordinary graph-sampling error (d~t−dt)(\tilde{d} _ t - d _ t) and current-batch scale-estimation perturbation (d^t−d~t)(\hat{d} _ t - \tilde{d} _ t). Because the routing map is nonlinear with respect to the scale estimate, Appendix B.6 formalizes the variance reduction of this scale-induced update perturbation through its conditional second-moment upper bound (variance proxy), denoted by σEMA,t2\sigma _ {\mathrm{EMA},t}^2 and σinst,t2\sigma _ {\mathrm{inst},t}^2 in Theorem 2.2. Specifically, Lemma B.37 establishes the exact variance attenuation at the scale-estimation level, Lemma B.38 propagates this attenuation to the second-moment bound of GNN parameter updates, and Corollary B.46 quantifies its contribution to the stochastic optimization bound.

(1) Why Does EMA Reduce Scale-Estimation Variance? (Lemma B.37)

Lemma B.37 (Eq. 235) proves that the deviation of the current-batch EMA scale ν^k,t\hat{\nu} _ {k,t} from its full-data oracle counterpart νk,t∘\nu _ {k,t}^\circ is:

ν^k,t−νk,t∘=αk,tEMA(yk,tB−yk,tN),αk,tEMA=1−ρ1−ρnk,t−+1.\hat{\nu} _ {k,t} - \nu _ {k,t}^\circ = \alpha _ {k,t}^{\mathrm{EMA}}\left(y _ {k,t}^B - y _ {k,t}^N\right), \qquad \alpha _ {k,t}^{\mathrm{EMA}} = \frac{1-\rho}{1-\rho^{n _ {k,t}^- + 1}}.

Conditioned on the previous EMA history Ft−1\mathcal{F} _ {t-1}, the historical state is fixed, and only the current mini-batch estimate yk,tBy _ {k,t}^B introduces randomness. Therefore:

Var(ν^k,t∣Ft−1)=(αk,tEMA)2Var(yk,tB∣Ft−1).\mathrm{Var}\left(\hat{\nu} _ {k,t} \mid \mathcal{F} _ {t-1}\right) = \left(\alpha _ {k,t}^{\mathrm{EMA}}\right)^2 \mathrm{Var}\left(y _ {k,t}^B \mid \mathcal{F} _ {t-1}\right).

In contrast, instantaneous estimation (ρ=0\rho=0) directly uses the current mini-batch (αk,tinst=1\alpha _ {k,t}^{\mathrm{inst}} = 1), yielding conditional variance Var(yk,tB∣Ft−1)\mathrm{Var}(y _ {k,t}^B \mid \mathcal{F} _ {t-1}). Since 0<αk,tEMA≤10 < \alpha _ {k,t}^{\mathrm{EMA}} \le 1, we have (αk,tEMA)2≤1(\alpha _ {k,t}^{\mathrm{EMA}})^2 \le 1. Thus, EMA reduces the conditional variance contributed by current mini-batch scale noise by the exact factor (αk,tEMA)2(\alpha _ {k,t}^{\mathrm{EMA}})^2, with strict inequality whenever 0<ρ<10 < \rho < 1, at least one previous observation exists (nk,t−≥1n _ {k,t}^- \ge 1), and the scale noise has nonzero variance.

(2) How Does This Variance Reduction Propagate to GNN Parameter Updates? (Lemma B.38)

Lemma B.38 proves that perturbations in teacher-scale estimates propagate to GNN parameter updates through a Lipschitz-continuous mapping:

∥d^t(ν)−d~t(ν′)∥2≤Lˉθ,t∥ν−ν′∥2.\Vert \hat{d} _ t(\nu) - \tilde{d} _ t(\nu') \Vert _ 2 \le \bar{L} _ {\theta,t}\Vert \nu - \nu' \Vert _ 2.

Combining this Lipschitz bound with Lemma B.37, the conditional second moment (variance proxy) of the scale-induced parameter-update perturbation ΔdtEMA=d^t(ν^t)−d~t(νt∘)\Delta d _ t^{\mathrm{EMA}} = \hat{d} _ t(\hat{\nu} _ t) - \tilde{d} _ t(\nu _ t^\circ) satisfies:

E[∥ΔdtEMA∥22∣Ft−1]≤Lˉθ,t2∑k=1K(αk,tEMA)2σk,t2⏟σEMA,t2≤Lˉθ,t2∑k=1Kσk,t2⏟σinst,t2,\mathbb{E}\left[\Vert \Delta d _ t^{\mathrm{EMA}} \Vert _ 2^2 \mid \mathcal{F} _ {t-1}\right] \le \underbrace{\bar{L} _ {\theta,t}^2 \sum _ {k=1}^K \left(\alpha _ {k,t}^{\mathrm{EMA}}\right)^2 \sigma _ {k,t}^2} _ {\sigma _ {\mathrm{EMA},t}^2} \le \underbrace{\bar{L} _ {\theta,t}^2 \sum _ {k=1}^K \sigma _ {k,t}^2} _ {\sigma _ {\mathrm{inst},t}^2},

where σk,t2=Var(yk,tB∣Ft−1)\sigma _ {k,t}^2 = \mathrm{Var}(y _ {k,t}^B \mid \mathcal{F} _ {t-1}) is the conditional variance of the current mini-batch scale estimate. For instantaneous scale estimation, the corresponding bound contains the same terms without the squared attenuation factors (αk,tEMA)2(\alpha _ {k,t}^{\mathrm{EMA}})^2. This directly establishes σEMA,t2≤σinst,t2\sigma _ {\mathrm{EMA},t}^2 \le \sigma _ {\mathrm{inst},t}^2 in Theorem 2.2 (with strict inequality when scale noise and routing sensitivity Lˉθ,t\bar{L} _ {\theta,t} are nonzero).

(3) Connection to the Stochastic Optimization Error Bound (Corollary B.46)

Corollary B.46 (Eq. 309–310) further shows how this variance proxy enters the overall stochastic convergence bound: the contribution of teacher-scale estimation noise is attenuated to

O~(∑k=1K(αk,tEMA)2nk,t),\tilde{\mathcal{O}}\left(\sum _ {k=1}^K \frac{(\alpha _ {k,t}^{\mathrm{EMA}})^2}{n _ {k,t}}\right),

whereas instantaneous scale estimation lacks the (αk,tEMA)2(\alpha _ {k,t}^{\mathrm{EMA}})^2 attenuation coefficient. Together, Lemmas B.37, B.38, and Corollary B.46 form the formal basis for the informal summary in Theorem 2.2.

Reviewer_TZBcOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FORTE, a task-prioritized framework for distilling heterogeneous molecular teachers into a student. By using only two-dimensional graphs at inference, FORTE addresses a practically relevant problem setting for large-scale molecular property prediction. Across well-known benchmark datasets, FORTE improves molecular property prediction accuracy for extensive classification and regression tasks.

Critical strengths
  1. The framework exploits complementary molecular representations during training while preserving efficient inference from two-dimensional graphs alone. This mechanism offers a practical approach to improving predictive fidelity without requiring additional modalities during screening.

  2. Experiments across extensive benchmark datasets and three pretrained students demonstrate broad improvements in classification and regression tasks, with scaffold-based evaluation, repeated random seeds, and component ablations providing empirical support for the method.

Critical weaknesses
  1. The central principle closely resembles established gradient-similarity approaches to auxiliary learning. Thus, the technical novelty of the paper is limited to an incremental idea on existing methods. Comparisons with equivalently normalized alternatives are needed to isolate the contribution of FORTE’s routing beyond existing task-prioritized optimization strategies.

  2. Although theoretical claims have been concretely investigated, the convergence analyses depend on restrictive geometric and optimization assumptions whose validity during molecular training remains unverified on real-world molecular datasets. This limits their ability to explain observed improvements or establish practically relevant convergence acceleration.

Questions for authors

See the Weaknesses section.

7 comments

W1: Novelty Beyond Gradient-Similarity Auxiliary Learning

Thank you for your insightful comments! FORTE establishes a stronger theoretical guarantee for the optimization of the pure supervised downstream loss, rather than the combined supervised–distillation objective. Importantly, our result shows that FORTE can actively accelerate main-task optimization under explicit conditions, instead of merely preserving convergence when auxiliary supervision is introduced.

1. Our main theoretical contribution: faster supervised-task convergence Theorem B.16 establishes convergence of the pure supervised downstream loss (excluding the distillation loss), even when supervised learning and knowledge distillation are optimized jointly. Under the stated smoothness and PL conditions, together with a positive task-alignment margin and bounded auxiliary norm–gain coupling, Theorem B.16 establishes: ΔTFORTE≤(1−2μηHF)TΔ0,ΔTsup≤(1−2μηh0)TΔ0,\Delta_T^{\mathrm{FORTE}} \leq (1-2\mu\eta H_F)^T\Delta_0, \qquad \Delta_T^{\mathrm{sup}} \leq (1-2\mu\eta h_0)^T\Delta_0, where HF=h0+λϑηm>h0H_F = h_0 + \lambda\vartheta_\eta m > h_0. Thus, FORTE achieves a strictly smaller theoretical contraction factor and a smaller sufficient iteration budget for the supervised task loss than supervised-only gradient descent at the same admissible step size.

Theorem 2.1 additionally compares FORTE with Raw Uniform KD and Normalized Uniform KD. Raw Uniform KD retains a nonvanishing stationarity floor of order O(λ2braw2)\mathcal{O}(\lambda^2 b_{\mathrm{raw}}^2), while Normalized Uniform KD incurs a conflict penalty (1−λγn)−2(1-\lambda\gamma_n)^{-2}. FORTE eliminates both issues through task-relative calibration, conflict filtering, and abstention.

2. Direct empirical evidence for faster main-task optimization and normalized baseline comparisons To isolate the contribution of FORTE's routing beyond gradient normalization and cosine-similarity gating, we implemented Normalized Uniform KD and Normalized Du-GCS (applying the exact same EMA teacher-gradient magnitude calibration from Eq. 7 to Uniform KD and Du et al., 2018). First, we compare the training epochs required for each method's 5-epoch moving-average supervised loss to reach the Student-only final supervised loss target (mean over epochs 96–100).

Table R1. Epochs required to reach the same supervised main-task loss target (GIN backbone across eight datasets).

MethodMedian epochs ↓\downarrowPaired ratio vs. Student-only ↑\uparrow
Student-only981.00$\times$
Normalized Uniform KD801.14$\times$
Normalized Du-GCS911.08$\times$
FORTE (Ours)714.00$\times$

FORTE reaches the target supervised training loss in a median of 7 epochs (a 14.00×14.00\times speedup), compared with 80 epochs (1.14×1.14\times) for Normalized Uniform KD and 91 epochs (1.08×1.08\times) for Normalized Du-GCS.

Second, we compare downstream test ROC-AUC across all eight classification benchmarks under the same normalized setting:

Table R2. ROC-AUC (%) across eight molecular classification datasets (mean ±\pm SD, three seeds).

DatasetGradNormPCGradNorm. Uniform KDNorm. Du-GCSFORTE (Ours)
BBBP65.6 ±\pm 2.766.7 ±\pm 2.464.43 ±\pm 2.8664.18 ±\pm 2.6673.00 ±\pm 5.58
BACE79.1 ±\pm 1.480.4 ±\pm 0.980.26 ±\pm 0.9279.42 ±\pm 0.8781.60 ±\pm 1.17
ClinTox64.3 ±\pm 2.361.3 ±\pm 7.067.21 ±\pm 5.7664.38 ±\pm 5.8886.10 ±\pm 3.28
Tox2176.3 ±\pm 0.576.6 ±\pm 0.675.78 ±\pm 0.6575.99 ±\pm 1.1279.40 ±\pm 0.47
SIDER60.0 ±\pm 1.060.4 ±\pm 0.159.97 ±\pm 0.6660.20 ±\pm 0.9866.10 ±\pm 0.41
ToxCast63.0 ±\pm 0.563.1 ±\pm 0.664.80 ±\pm 0.4464.47 ±\pm 0.1867.60 ±\pm 0.24
HIV76.2 ±\pm 0.877.1 ±\pm 0.577.51 ±\pm 0.4476.02 ±\pm 2.1279.30 ±\pm 3.06
MUV74.6 ±\pm 1.075.2 ±\pm 0.678.52 ±\pm 1.1575.45 ±\pm 0.7278.70 ±\pm 0.95
Average69.970.171.0670.0176.48

FORTE achieves the highest ROC-AUC on every dataset, outperforming Normalized Uniform KD and Normalized Du-GCS by +5.42 and +6.47 percentage points on average, respectively.

W2: Practical Relevance of the Geometric Assumptions

Thank you for your valuable feedback! We conducted additional experiments on four molecular datasets to evaluate the practical relevance of FORTE's geometric assumptions. Specifically, we examine: (1) whether the GNN Jacobian preserves the geometry of supervised and routed auxiliary gradients (Definition B.2), (2) whether FORTE's latent-space routing leads to improved gradient compatibility and first-order task descent in the actual GNN parameter space (Lemma B.3 and Corollary B.4), and (3) how these measurements relate to the GNN architectural conditions in Appendix B.5.

1. Restricted Scaled Isometry Assumption (Definition B.2): Does the GNN Jacobian preserve the geometry of task and auxiliary gradients? FORTE selects teachers using gradients in the student representation space, whereas optimization updates GNN parameters. Definition B.2 connects these spaces through the following assumption: αB(1−δB)∥v∥22≤∥JB⊤v∥22≤αB(1+δB)∥v∥22,v∈span⁡{gB,aB},\alpha_B(1-\delta_B)\|v\|_2^2 \leq \|J_B^\top v\|_2^2 \leq \alpha_B(1+\delta_B)\|v\|_2^2, \quad v\in\operatorname{span}\{g_B,a_B\}, where αB>0\alpha_B>0 and 0≤δB<10\leq\delta_B<1. Here, gBg_B and aBa_B are the batch-level supervised and routed auxiliary gradients in the representation space, and JBJ_B is the GNN Jacobian. The assumption requires the Jacobian to preserve their geometric relationship up to a common scaling factor, without requiring isometry over the entire representation space.

Experimental verification. We directly compute the restricted Jacobian Gram matrix HB=UB⊤JBJB⊤UB,δB=λmax⁡(HB)−λmin⁡(HB)λmax⁡(HB)+λmin⁡(HB),H_B=U_B^\top J_B J_B^\top U_B, \qquad \delta_B= \frac{\lambda_{\max}(H_B)-\lambda_{\min}(H_B)}{\lambda_{\max}(H_B)+\lambda_{\min}(H_B)}, where UBU_B is an orthonormal basis of the task–auxiliary gradient subspace. We evaluate the complete trainable GNN encoder (1,948,200 parameters), rather than only its projection layer. Each dataset contains 300 audited batches from three model checkpoints, five sampling seeds per checkpoint, and 20 batches per sampling seed. Jacobians are evaluated in evaluation mode with EMA frozen.

Table R3. Direct verification of Definition B.2: restricted Jacobian geometry.

DatasetBatchesRank-2Min. eigenvalueMedian δB\delta_BP90 δB\delta_B
BACE3003000.001000.3090.469
MUV3003000.034400.4780.654
SIDER3003000.094300.3210.558
ToxCast3003000.239000.3380.595
Total1,2001,200———

Batches: Number of evaluated mini-batches; Rank-2: Number of batches in which the task and auxiliary gradients span two independent directions; Min. eigenvalue: Smallest eigenvalue of HBH_B across the audited batches; a positive value means the Jacobian does not collapse the routing subspace; Median δB\delta_B: Typical relative geometric distortion; smaller values indicate better preservation; P90 δB\delta_B: The distortion threshold below which 90% of evaluated batches fall; it characterizes the higher-distortion portion of the distribution.

Result. All 1,200 batches have two-dimensional routing subspaces and strictly positive restricted eigenvalues, satisfying the nondegeneracy requirement of Definition B.2. More importantly, the median distortions range from 0.309 to 0.478, and all P90 values are below 0.7. Thus, the measured geometry is not exactly isometric, but its distortion is quantitatively bounded in the evaluated batches. These results directly characterize the practical applicability of the assumption instead of relying only on δB<1\delta_B<1.

2. Verification of Lemma B.3 and Corollary B.4: Parameter-space gradient compatibility and task descent

Building on the restricted Jacobian geometry verified in Table R3, we further examine whether FORTE's latent-space routing produces favorable gradient alignment and task-descent directions in the actual GNN parameter space. Lemma B.3 establishes the connection between the two spaces:

Gθ,B⊤Aθ,B=αBgB⊤aB+αBgB⊤EBaB,G_{\theta,B}^{\top}A_{\theta,B} = \alpha_B g_B^{\top}a_B + \alpha_B g_B^{\top}E_B a_B,

where the distortion term is bounded by

∣gB⊤EBaB∣≤δB∥gB∥2∥aB∥2.|g_B^{\top}E_B a_B| \leq \delta_B \Vert g_B \Vert_2 \Vert a_B \Vert_2.

This is a worst-case spectral bound based on the operator norm of EBE_B. The actual parameter-space alignment also depends on the directions of the routed gradients relative to the distortion operator.

Experimental verification. Using the full GNN Jacobian on the audited mini-batches, we compute the supervised and auxiliary parameter gradients (Gθ,BG_{\theta,B} and Aθ,BA_{\theta,B}) to directly measure their parameter-space alignment, the auxiliary-gradient relative norm, and the observed first-order task-descent coefficient cBobsc_B^{\mathrm{obs}}:

Alignment=⟨Gθ,B,Aθ,B⟩∥Gθ,B∥22,Relative Norm=∥Aθ,B∥2∥Gθ,B∥2,cBobs=⟨Gθ,B,Gθ,B+λAθ,B⟩∥Gθ,B∥22=1+λ⋅Alignment.\mathrm{Alignment} = \frac{\langle G_{\theta,B}, A_{\theta,B} \rangle}{\Vert G_{\theta,B} \Vert_2^2}, \qquad \mathrm{Relative\ Norm} = \frac{\Vert A_{\theta,B} \Vert_2}{\Vert G_{\theta,B} \Vert_2}, \qquad c_B^{\mathrm{obs}} = \frac{\langle G_{\theta,B}, G_{\theta,B}+\lambda A_{\theta,B} \rangle}{\Vert G_{\theta,B} \Vert_2^2} = 1 + \lambda \cdot \mathrm{Alignment}.

Under the standard distillation weight λ=1\lambda=1 (Appendix C.1), the observed descent coefficient satisfies cBobs=1+Alignmentc_B^{\mathrm{obs}} = 1 + \mathrm{Alignment}. Table R4 compares the parameter-space alignment, relative norm, and first-order task-descent coefficient across the four datasets.

Table R4. Direct parameter-space evaluation of gradient compatibility and first-order task-descent gain (λ=1\lambda=1).

DatasetMethodAlignmentRelative NormDescent (cBobsc_B^{\mathrm{obs}})
BACEUniform KD-0.0180.9000.982
FORTE+0.1600.3891.160
MUVUniform KD-0.1170.5340.883
FORTE+0.0120.0251.012
SIDERUniform KD-0.0010.6590.999
FORTE+0.0140.1771.014
ToxCastUniform KD-0.0260.5170.974
FORTE+0.0100.1111.010

Result. FORTE achieves positive parameter-space alignment and first-order descent coefficients above 1.0 on all four datasets, whereas Uniform KD exhibits negative alignment and descent coefficients below 1.0 throughout. Notably, on datasets with frequent teacher conflicts or missing modalities such as MUV (where 56.0% of molecules have zero eligible teachers; Appendix C.9), molecule-level abstention (ri=0r_i=0) reduces the batch routing factor RBR_B (Eq. 19), which shrinks the relative norm to 0.025 and keeps the alignment small (+0.012) while preventing the severe negative transfer suffered by Uniform KD (-0.117 alignment, 0.883 descent). Combined with the full-trajectory acceleration in Table R1 (where FORTE reaches the target supervised loss 14.00×14.00\times faster than Student-only and outperforms Normalized Uniform KD and Normalized Du-GCS), this behavior directly aligns with Corollary B.18 (Appendix B.4.1), which relaxes the uniform mt≥m>0m_t \ge m > 0 condition of Theorem B.16 to allow abstention/near-zero steps (mt≥0m_t \ge 0) and guarantees task-loss acceleration whenever positive alignment is accumulated across active batches (∑tmt>0\sum_t m_t > 0).

Reviewer_AXSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose FORTE, a teacher selection algorithm to determine whether updates obtains from a teacher is useful in a multi-teacher knowledge distillation setting. Multiple teacher models with different representations are used to obtain latent vector representations. The student model representations are projected onto different teacher latent representations and losses and gradients are calculated based on a differentiable similarity measure. Gradients are then normalized and filtered and teachers are selected to provide gradients used to update the student weights. The student also learns directly from supervised objective and avoids negative transfer when no teachers are selected. The authors perform strong ablations on the whether the aforementioned selection methods make a difference also the effects of multi-teacher distillation on different tasks.

Critical strengths
  • The paper is well written, with clear structure, motivation, and description of methodology. The authors take effort to describe why and how they come up with their solutions, including both plain language and mathematical reasoning, and is a valuable contribution in itself in this problem space. Identifying the magnitude and directional conflicts as key conflicts is a key insight that can be useful beyond the current problem space.
Critical weaknesses
  • While useful, the representational bottleneck will be an issue for any property prediction that is reliant on geometry. In other words, if the property is dependent on the conformation of the molecule, then the the distilled 2D model can, at best, learn the mean of such conformational properties. It is important to note such case. The 2D molecular graph collapses all possible conformations into a single graph and acts as an conformation-invariant bottleneck. This is particularly key as authors point out that task selection as a key problem. As the tasks selected all tend to not depend on such 3D information, and may bias the efficacy of the distillation.
Questions for authors
  • How are the 3D views obtained for the 3D teacher models used to train the 3 student backbones? Line 2716 mentions that the SMILES is used to construct a 3-D conformation. Are multiple conformers used to allow for fine-temperature conditions that these molecules typically exist in?

  • Why is consensus a good choice for conflict identification? One could imagine one model having gradients that are strictly different from all others (due to different representation and latent space), while being the correct choice. And the consensus models are incorrect. Is that scenario mitigated in the framework?

5 comments

W1: Conformational Dependence and the Role of 3D Information in Molecular Distillation

We deeply appreciate the reviewer's sharp physical and chemical insight regarding the representational relationship between 2D graphs and 3D molecular conformations—this touches upon a fundamental question in molecular representation learning. Rather than attempting to recover conformer-specific energies at inference, FORTE leverages 3D geometric and electronic teachers as training-time stereoelectronic priors to improve molecular-level (graph-level) property prediction, while selectively filtering out noisy or task-irrelevant structural signals.

1. Macroscopic Ensemble Nature of Molecular Property Benchmarks In standard molecular property prediction benchmarks (e.g., MoleculeNet; Wu et al., 2018), compounds are indexed at the 2D molecular graph (SMILES) level, and existing databases neither record nor separate individual conformer-specific labels. Physically, in solution and biological environments at finite temperature, flexible molecules exist as a dynamic thermodynamic mixture (Boltzmann ensemble) of interconvertible conformations rather than a single static geometry. Accordingly, experimentally measured endpoints—such as aqueous solubility (ESOL), lipophilicity (Lipo), blood-brain barrier permeability (BBBP), and assay toxicity/activity (Tox21, ClinTox, SIDER, BACE, HIV)—reflect macroscopic ensemble properties determined by the molecule's overall chemical structure, rather than isolated single-conformer states.

2. How 3D Teachers Improve 2D Representations Following established 3D molecular representation and cross-modal distillation frameworks (e.g., GraphMVP, Liu et al., 2022; Uni-Mol, Zhou et al., 2023; IEM, Xiang et al., 2024; EDG, Xiang et al., 2025; D&D, Cho et al., 2025), 3D structural teachers provide privileged stereoelectronic inductive biases during training. Although a 2D molecular graph implicitly determines a molecule's conformational energy landscape, standard 2D message-passing networks struggle to explicitly infer 3D steric bulk, ring puckering, intramolecular hydrogen bonding, 3D pharmacophore distances, and spatial electron-density distributions from 2D connectivity alone. Distilling from 3D geometric (IEM) and electron-density-informed (EDG) teachers injects these stereoelectronic priors into the 2D encoder, enabling better generalization on molecular-level tasks even though the deployed 2D student remains conformation-invariant at inference.

Crucially, prior 3D-to-2D methods transfer 3D features unconditionally for every molecule, which can cause negative transfer when a generated conformer is less representative for a flexible molecule or when a task depends primarily on 2D functional groups. FORTE directly addresses this challenge through task-prioritized routing: it retains 3D teacher supervision when the teacher's gradient aligns with the downstream assay objective, and suppresses or abstains (via the Null Route) when conformational noise causes directional conflict.

Table R1. Teacher-subset ablation: Average ROC-AUC (%) of the GIN student across eight classification datasets (8 benchmarks ×\times 3 seeds = 24 runs under the protocol of Table 1).

Experimental MethodAverage ROC-AUC (%)
Student-only (2D graph)70.6
FORTE + SMILES teacher73.4
FORTE + IEM + EDG (3D structure teachers only)74.2
FORTE + IEM + SMILES74.8
FORTE + EDG + SMILES75.2
FORTE + IEM + EDG + SMILES (Full)76.5

As shown in Table R1, distilling solely from the two structure-related teachers (FORTE + IEM + EDG) improves the 2D student from 70.6% to 74.2% (+3.6%), surpassing the sequence-only teacher (FORTE + SMILES, 73.4%). Combining all three teachers further reaches 76.5% (+5.9% over Student-only) while preserving 2D-only inference. We also clarify that our scope is specifically molecular-level property prediction over 2D chemical graphs, rather than conformer-specific potential energy surface modeling

Q1: Conformer Generation and Finite-Temperature Effects

We thank the reviewer for raising this important question. Following the standard protocols of IEM (Xiang et al., 2024) and EDG (Xiang et al., 2025) (Appendix C.5), we use RDKit to generate a single canonical low-energy 3D conformer from each molecule's SMILES string and use PyMOL to render molecular images from four predefined viewing angles, which represent multi-view projections of the same 3D conformer rather than four independently sampled conformers. For IEM, a frozen ResNet-18 encoder averages features across the four views (Eq. 321); for EDG, multi-view structural features are aggregated and passed through the frozen EDPredictor pre-trained with DFT electron-density supervision (Eq. 322), with both teachers used strictly during training and requiring no conformer generation or DFT computation at inference. Consistent with mainstream large-scale 3D molecular frameworks (e.g., GraphMVP, Liu et al., 2022; Uni-Mol, Zhou et al., 2023; D&D, Cho et al., 2025), we use a single canonical conformer instead of computationally prohibitive finite-temperature multi-conformer sampling, while relying on FORTE's gradient-compatibility filter (Eq. 8–9) to automatically down-weight or reject any teacher signal when this single-conformer approximation introduces task-conflicting gradients.


Q2: Why Use Consensus to Identify Conflicts?

We thank the reviewer for posing this scenario—it cuts directly to the core design philosophy of FORTE and inspired us to conduct the dedicated diagnostic experiment below. FORTE does not rely on teacher consensus. Below, we clarify our independent teacher evaluation mechanism and provide direct empirical verification under a minority-teacher scenario.

1. Task compatibility rather than teacher consensus FORTE evaluates each teacher independently against the supervised downstream task gradient, rather than measuring agreement among teachers. Specifically, Eq. (8) filters each teacher based solely on its cosine similarity with the main-task gradient (sk,i=gi⊤uk,i∥gi∥∥uk,i∥>γs_{k,i} = \frac{g_i^\top u_{k,i}}{\|g_i\|\|u_{k,i}\|} > \gamma). Teachers positively aligned with the main task are retained, conflicting teachers are masked out (bk,i=0b_{k,i}=0), and if no teacher is task-compatible, the Null Route allows the student to update using only the supervised objective.

Because FORTE is a task-prioritized framework, our theoretical and algorithmic objective is the convergence of the pure supervised downstream loss itself. Theorem B.16 proves that joint optimization under FORTE contracts the supervised task loss faster than supervised-only gradient descent at the same admissible step size, and Theorem 2.1 shows tighter convergence bounds than uniform teacher aggregation.

2. Direct experimental verification of the minority task-compatible teacher scenario To directly test the reviewer's scenario—where one teacher is correct (task-compatible) while the other two teachers agree with each other but conflict with the main task—we isolated all diagnostic observations satisfying: cos⁡(gi,uk+,i)>0,cos⁡(gi,uj,i)<0,cos⁡(gi,uℓ,i)<0,cos⁡(uj,i,uℓ,i)>0.5.\cos(g_i, u_{k^+,i}) > 0, \qquad \cos(g_i, u_{j,i}) < 0, \qquad \cos(g_i, u_{\ell,i}) < 0, \qquad \cos(u_{j,i}, u_{\ell,i}) > 0.5. Among 5,760 diagnostic observations, we identified 409 instances (corresponding to 342 distinct molecules after per-dataset deduplication) matching this exact minority-teacher condition.

Table R2. Positive task-gradient alignment rate (%) when only a single minority teacher is task-compatible.

DatasetUniform KDCalibrated Uniform KDFORTE (Ours)
SIDER7.4 ±\pm 3.212.7 ±\pm 3.5100.0
BACE7.1 ±\pm 11.12.8 ±\pm 3.6100.0
MUV15.2 ±\pm 5.223.8 ±\pm 4.6100.0

As shown in Table R2, when two conflicting teachers strongly agree with each other, Uniform KD and Calibrated Uniform KD are dominated by the conflicting majority, resulting in positive task alignment in only 2.8%–23.8% of cases. In contrast, FORTE achieves 100.0% positive task alignment across all three datasets because it masks out the two task-conflicting teachers (bj,i=bℓ,i=0b_{j,i}=b_{\ell,i}=0) and routes supervision exclusively from the single task-compatible teacher k+k^+.

We once again thank the reviewer for these insightful questions, which have significantly strengthened our manuscript.

Dear Reviewer AXSd,

We hope this message finds you well!

We just wanted to gently follow up and see if you’ve had a chance to look over our response. We’d love to hear your thoughts and are happy to clarify anything further!

Thank you so much for your time and consideration!

Best regards, The Authors

Comments on the paper

1 comment
Area_Chair_NN3rOpenReview ↗

Please engage in the discussion

Dear Reviewers,

As we enter the final stage of the author-reviewer discussion, I would like to encourage everyone to actively engage with the authors’ responses and any additional evidence they have provided.

Please take this opportunity to reconsider your initial assessments with an open mind, particularly where the authors have offered substantive clarifications, detailed arguments, or additional experimental results. The goal of the discussion is not to defend initial positions, but to reach the fairest and most informed evaluation of the submission.

I encourage you to engage directly with the authors on any remaining concerns, clearly distinguishing between issues that have been satisfactorily addressed and those that remain unresolved. Where appropriate, please update your reviews and scores accordingly, or explain why your original assessment still stands.

Your active and constructive participation in these final exchanges is essential to reaching a fair, evidence-based decision, and so far only Reviewer QKYr has begun to engage in the discussion.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.