- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper evaluates how 10 general-purpose LLMs balance constraint satisfaction and structural exploration across 15 molecular-generation tasks. It combines validity, success, novelty, uniqueness, and diversity into convergent (CC) and divergent creativity (DC), examining scale, constraints, ICL, temperature, and prompting. It reports negative CC–DC associations, model-family differences, ICL copying, and failed transfer of language-creativity preference training from a shared SFT parent.
Critical strengths
- Multiple models/tasks, within-task comparisons, fixed effects, and alternative metrics provide more insight than a single-metric leaderboard. The authors acknowledge Qwen3’s simultaneous CC/DC gains and inconsistent transfer of some interventions.
- ICL copying diagnostics have practical value: exact copies constitute 30.5%–65.5% of valid outputs despite improved activity-task success and near-ceiling reference-library novelty.
Critical weaknesses
- I could not reproduce the reported FC results from the definitions and aggregation procedure described in the paper. FC = E/G counts successful, novel, unique outputs among all generations. Averaging the five LogP targets first and weighting 11 families equally gives LLaMA3-8B FC = 0.344618: 3.622 sums ten non-LogP tasks; 0.1688 averages five LogP tasks. Fifteen-task weighting also disagrees beyond three-decimal rounding error. FC ≤ SR should hold under identical weights, yet the reported FC = 0.556 exceeds recalculated SR = 0.4344. Page 26 discloses no alternative counts, denominator, or task subset; Figure 9’s run-level view and Table 8’s legacy-corpus note concern other analyses. The scaling-related FC decline survives, but its magnitude and traceability need correction. Component means/SDs cannot recover mean nonlinear CC/DC scores.
- Threshold and valid-sample-size dependence remain unresolved. QED ≥ 0.6, SA ≤ 4, LogP error ≤ 1, and BBB/HIA/DRD2/JNK3/GSK3β scores ≥ 0.5 need protocol or validation support. Thresholding is reasonable, but metric references and high AUC do not establish cutoff-specific positive-prediction reliability. The authors acknowledge conventional cutoffs; their QED scan yields rank correlation as low as 0.067 and shifts of nine positions. Sensitivity of the central association and key model differences remains unclear, as does “absolute success rates are unaffected.” Expected Uniqueness = U/V can decrease with valid-sample count V even under an unchanged conditional molecular distribution, while CC also depends on validity. The fixed-size check covers only Diversity, leaving possible measurement-induced negative association in Uniqueness, DC, and the core regressions untested.
Questions for authors
- Reconcile all four main-text FC values with Tables 19–22, specifying data version, tasks, denominator, and aggregation. Explain FC > SR and whether corrections affect scaling comparisons or other conclusions.
- Identify prior protocols or independent validation data supporting the cutoffs above; provide threshold-specific predictor performance/calibration, especially positive-prediction reliability, and specify how single-task criteria combine in multi-constraint tasks. Using saved continuous scores, extend the QED analysis to associations, key differences, and uncertainty intervals; clarify the statement about unchanged absolute success rates.
- Recompute Uniqueness, Diversity, DC, within-task correlations, and family/model fixed-effects analyses at a common valid-sample count, e.g., n = 40 given the reported minimum of 43. Persistence would strengthen the central claim.
- Please clarify the analysis units and aggregation procedures underlying the regression results. In addition, please clarify how the “six tasks” referenced in the CRE–SFT analysis relate to the sample sizes