- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to construct specialist teachers and integrate their capabilities into one chemical language model. It estimates task affinities from SFT gradients on disjoint training-data probes and uses a constrained mixed-integer program to form three partially overlapping task groups. Three Qwen3-8B specialists and an additional all-task Qwen3-8B generalist supervise a Qwen3-1.7B student on its own generated prefixes. The distillation advantage mixes generalist and routed-specialist signals, with the specialist weight increasing from 0 to 0.5. Evaluation covers 20 ChemCoTBench subtasks. Teacher comparisons, student baselines, component ablations, probe-resampling audits, and training diagnostics provide evidence of useful specialization and improved student performance, with task-dependent trade-offs.
Critical strengths
-
The paper addresses a useful distinction between developing specialist capability and successfully integrating it. Importantly, ChemOPD and specialist-only MOPD share the specialist checkpoints, routes, student initialization, prompt stream, and actor-update budget. This makes their comparison informative about the supervision rule rather than merely stronger specialists.
-
The teacher-construction stage and evaluation protocol are unusually well specified. The paper separates training memberships from distillation routes, reports teacher-side as well as student-side outcomes, and provides a probe-resampling audit that recovers the same grouping in ten draws.
Critical weaknesses
-
Limited methodological novelty. The framework adds affinity-guided grouping and scheduled generalist-specialist mixing, but these are largely general-purpose extensions of multi-teacher OPD. The paper does not sufficiently establish what is fundamentally new about the combination of these existing components.
-
Limited mechanistic understanding and generality. The results demonstrate useful performance gains, but provide limited explanation of why generalist-specialist integration improves capability transfer, or under what conditions it should be preferred over specialist-only distillation. The experiments are also restricted to Qwen3-8B teachers and a Qwen3-1.7B student, leaving generalization across different teacher-student capacity ratios unclear.
-
Experimental robustness and ablation limitations. The warm-up ablation changes both the timing and average strength of specialist supervision (approximately 0.337 versus a constant 0.5), making it difficult to isolate the benefit of scheduling. An average-weight-matched constant baseline would help disentangle these effects. Additionally, a comparison with task-reward RL on suitable verifiable subtasks would help clarify the practical advantages over alternative post-training approaches.
Questions for authors
To what extent do the conclusions generalize beyond the Qwen3-8B to Qwen3-1.7B setup to other teacher-student sizes in chemistry? Please clarify which conclusions are experimentally supported and which remain hypotheses about broader multi-teacher distillation.
4 comments
Table R1. Capacity comparison with a 0.6B student and 8B teachers.
The SFT student is the all-task initializer; the teacher column gives the routed 8B MIP-specialist results from Table 1. Bold marks ChemOPD results better than MOPD. Lower MAE is better; all other metrics are higher-is-better.
| Task | Metric | SFT student (0.6B) | Routed teachers (8B) | MOPD (0.6B) | ChemOPD (0.6B) |
|---|---|---|---|---|---|
| Addition | P@1 | 0.25 | 0.50 | 0.10 | 0.25 |
| Deletion | P@1 | 0.40 | 0.70 | 0.50 | 0.50 |
| Substitution | P@1 | 0.02 | 0.62 | 0.17 | 0.22 |
| Murcko scaffold | Similarity | 0.43 | 0.77 | 0.44 | 0.47 |
| Equivalence | Accuracy | 0.56 | 0.66 | 0.56 | 0.51 |
| Functional-group count | MAE | 0.12 | 0.08 | 0.09 | 0.08 |
| Ring count | MAE | 0.27 | 0.10 | 0.25 | 0.25 |
| Ring-system scaffold | Similarity | 0.62 | 0.82 | 0.40 | 0.68 |
| QED | SR | 0.57 | 0.81 | 0.48 | 0.65 |
| QED | Delta | 0.121 | 0.188 | 0.098 | 0.158 |
| LogP | SR | 0.36 | 0.78 | 0.31 | 0.47 |
| LogP | Delta | 0.102 | 0.701 | 0.180 | 0.295 |
| Solubility | SR | 0.55 | 0.76 | 0.42 | 0.57 |
| Solubility | Delta | 0.450 | 0.786 | 0.412 | 0.609 |
| DRD2 | SR | 0.36 | 0.59 | 0.22 | 0.37 |
| DRD2 | Delta | -0.007 | 0.136 | 0.015 | 0.009 |
| JNK3 | SR | 0.20 | 0.35 | 0.05 | 0.19 |
| JNK3 | Delta | -0.029 | -0.042 | -0.003 | -0.022 |
| GSK3B | SR | 0.30 | 0.58 | 0.11 | 0.22 |
| GSK3B | Delta | 0.000 | 0.066 | 0.005 | -0.014 |
| Byproduct prediction | FTS | 0.018 | 0.044 | 0.023 | 0.028 |
| Byproduct prediction | Top-1 | 0.00 | 0.00 | 0.00 | 0.00 |
| Major-product prediction | FTS | 0.152 | 0.469 | 0.175 | 0.181 |
| Major-product prediction | Top-1 | 0.01 | 0.26 | 0.00 | 0.00 |
| Mechanism selection | Accuracy | 0.17 | 0.54 | 0.19 | 0.26 |
| NEPP | FTS | 0.268 | 0.640 | 0.256 | 0.431 |
| NEPP | Top-1 | 0.01 | 0.35 | 0.00 | 0.04 |
| Condition prediction | FTS | 0.008 | 0.007 | 0.000 | 0.015 |
| Condition prediction | Top-1 | 0.02 | 0.03 | 0.00 | 0.01 |
| Retrosynthesis | FTS | 0.264 | 0.482 | 0.279 | 0.282 |
| Retrosynthesis | Top-1 | 0.00 | 0.03 | 0.00 | 0.00 |
Weakness 3 — Scheduling with a matched average weight
We added the requested control: the specialist weight stays at α = 36.75/109 ≈ 0.337156 throughout training, with anchor weight 1 − α. This matches the original schedule's average specialist weight. Teachers, routes, initialization, prompt stream, optimizer settings, and the 109-update budget are unchanged.
ChemOPD improves 23 of 31 metrics over this constant-weight baseline, with four ties and four losses. Substitution P@1 rises from 0.38 to 0.70, solubility gain from 0.468 to 0.718, and retrosynthesis FTS from 0.316 to 0.443 (Table 11; Table R2 below). The constant-0.5 control matches the final weight, while this new control matches the average weight. Together, they support a benefit from when specialist guidance is introduced, beyond simply lowering its average weight.
Comparison with task-reward RL. Thank you for this suggestion. Although some tasks have direct verifiers, their different output formats, lengths, and evaluation criteria require task-specific rewards and balancing. JNK3 and GSK3B also use pretrained random-forest oracles; using these predictors as rewards requires additional assessment of predictive uncertainty. This study focuses on capability consolidation through distillation. We plan a controlled task-reward RL comparison on verifiable subtasks in future work.
Table R2. Specialist-weight schedule ablation.
Constant 0.5 matches the final specialist weight, and constant 0.337 matches the average specialist weight. Both retain the generalist anchor. All variants use the original 1.7B student and 8B teachers. Bold marks ChemOPD improvements over constant 0.337. Lower MAE is better; all other metrics are higher-is-better.
| Task | Metric | Constant α=0.5 | Constant α≈0.337 | ChemOPD schedule |
|---|---|---|---|---|
| Addition | P@1 | 0.45 | 0.50 | 0.35 |
| Deletion | P@1 | 0.85 | 0.40 | 0.75 |
| Substitution | P@1 | 0.60 | 0.38 | 0.70 |
| Murcko scaffold | Similarity | 0.50 | 0.57 | 0.58 |
| Equivalence | Accuracy | 0.57 | 0.52 | 0.62 |
| Functional-group count | MAE | 0.09 | 0.10 | 0.10 |
| Ring count | MAE | 0.20 | 0.15 | 0.15 |
| Ring-system scaffold | Similarity | 0.73 | 0.72 | 0.83 |
| QED | SR | 0.84 | 0.82 | 0.86 |
| QED | Delta | 0.191 | 0.186 | 0.167 |
| LogP | SR | 0.63 | 0.58 | 0.68 |
| LogP | Delta | 0.500 | 0.537 | 0.601 |
| Solubility | SR | 0.61 | 0.60 | 0.71 |
| Solubility | Delta | 0.427 | 0.468 | 0.718 |
| DRD2 | SR | 0.45 | 0.50 | 0.55 |
| DRD2 | Delta | 0.055 | 0.062 | 0.098 |
| JNK3 | SR | 0.32 | 0.23 | 0.29 |
| JNK3 | Delta | -0.011 | -0.032 | -0.020 |
| GSK3B | SR | 0.44 | 0.48 | 0.46 |
| GSK3B | Delta | 0.016 | 0.022 | 0.029 |
| Byproduct prediction | FTS | 0.034 | 0.048 | 0.051 |
| Byproduct prediction | Top-1 | 0.00 | 0.00 | 0.00 |
| Major-product prediction | FTS | 0.396 | 0.482 | 0.465 |
| Major-product prediction | Top-1 | 0.15 | 0.14 | 0.18 |
| Mechanism selection | Accuracy | 0.40 | 0.39 | 0.39 |
| NEPP | FTS | 0.505 | 0.490 | 0.521 |
| NEPP | Top-1 | 0.11 | 0.08 | 0.13 |
| Condition prediction | FTS | 0.000 | 0.000 | 0.023 |
| Condition prediction | Top-1 | 0.00 | 0.00 | 0.03 |
| Retrosynthesis | FTS | 0.303 | 0.316 | 0.443 |
| Retrosynthesis | Top-1 | 0.00 | 0.01 | 0.03 |
Thank you again for your valuable feedback. We would be glad to provide further clarification on any remaining questions about our submission.
Dear Reviewer Ugcp, as the rebuttal period is nearing its end, do you feel that our current rebuttal experiments and explanations have adequately addressed your concerns about our work? If so, would you consider updating your rating accordingly?
Thank you for your careful review and suggestions on clarifying the methodological contribution, understanding capability transfer, and strengthening the capacity comparison and scheduling ablation.
Weakness 1 — Methodological contribution
We would like to clarify the methodological contribution of ChemOPD. Our novelty claim is not that gradient-based affinity estimation, constrained grouping, or linear interpolation is individually new. Rather, we address a broader and underexplored design problem in multi-teacher OPD for heterogeneous domains: the teacher collection itself must be constructed to reflect cross-task capability relationships before its knowledge can be effectively consolidated into a student.
This is particularly important for chemical reasoning, where benchmark task categories do not necessarily define independent training domains. Different tasks can share underlying chemical knowledge and benefit from complementary training signals. Treating the teacher collection as a fixed set of specialists trained on predefined, disjoint task categories may therefore overlook useful cross-task interactions. ChemOPD makes teacher construction an explicit part of the OPD framework. It uses estimated task affinities to determine specialist training memberships, permits selected tasks to contribute to multiple specialists, and separately maintains unambiguous prompt-level distillation routes.
The student integration stage addresses the complementary problem: even when specialists acquire useful capabilities, specialist-only OPD does not necessarily transfer these gains effectively. The generalist provides shared cross-task guidance, while the scheduled specialist contribution introduces task-specific corrections during on-policy learning. Thus, the two stages address different but connected requirements: constructing complementary sources of supervision and enabling a compact student to benefit from them. The controlled comparisons and the disjoint-grouping ablation provide empirical evidence for the respective contributions of these design choices.
We therefore position ChemOPD as a framework-level methodological contribution to multi-teacher OPD, rather than claiming a fundamentally new affinity estimator, MIP solver, or algebraically distinct interpolation operator.
Weakness 2 and Question — Transfer to a smaller student
We added the requested Qwen3-8B → Qwen3-0.6B comparison (Table R1 below). This increases the nominal teacher–student capacity ratio from approximately 4.7:1 to 13.3:1. We retain the existing teachers and task routes. At each student size, ChemOPD and MOPD share the SFT initializer, prompt stream, optimizer settings, and 109-update budget.
The overall advantage persists, while individual task outcomes change. ChemOPD improves addition P@1 at 0.6B (0.25 vs. 0.10), but MOPD leads at 1.7B (0.50 vs. 0.35). DRD2 gain shows the reverse pattern: ChemOPD leads at 1.7B but trails at 0.6B. Despite these changes, ChemOPD improves 22 of 31 metrics at both sizes, matching or improving 27/31 metrics at 0.6B and 26/31 at 1.7B (Table 1).
The logs show a similar change in training behavior at both sizes. Using the Appendix I diagnostics, ChemOPD reduces the extreme-candidate rate by 16.0% at 0.6B and 29.4% at 1.7B. Extreme-candidate rates and gradient norms before clipping are lower in every 10-step bin. The proportion of analyzed updates with gradient norm above 1.0 falls from 98.1% under MOPD to 83.5% under ChemOPD at 0.6B; at 1.7B it falls from 47.6% to 36.9%.
These results support the same central claim at both tested capacity ratios: generalist-anchored supervision moderates specialist-derived training signals and helps a compact student integrate heterogeneous capabilities, even though the gains differ by task and student size.