- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ChemoGrad is a training-free, per-instance test-time procedure for a frozen chemistry LLM (LatentChem's Qwen3-8B checkpoint, CoT mode). Last-layer hidden states of the first ρ=50% of generated tokens are treated as free variables and updated by a REINFORCE step (running-mean baseline, Adam, lr 0.05). The reward is an external oracle: an RDKit/TDC property or functional-group rule, or, for reaction/caption, similarity to the ground truth. States are decoded through lm_head, the suffix is regenerated, and a candidate is accepted only if the reward improves. Add-ons are hand-written task hints appended to the prompt and a learning-rate-doubling "deepening". On ChemCoTBench molecule optimization, mean Δ rises from 0.60 (published LatentChem) to 0.90 (+51%) with 94–100% success. Gains on editing, reaction prediction and captioning are also reported.
Critical strengths
- A simple, cheap, practical idea (oracle-in-the-loop search on a frozen model). Verifier types are labelled (avail vs gold), and limitations are candidly disclosed: init-to-final pairs favour the final answer, the add-ons cannot be separated, and the BoN comparison holds only at a fixed pool.
- The results hint that single-pass outputs of a four-stage RL-trained model leave headroom when oracle feedback is available at inference. That is useful if shown under matched budgets.
Critical weaknesses
- The headline comparison is neither budget- nor information-matched, and the mechanism is not isolated.
- The baseline is a single published pass under another paper's protocol. ChemoGrad queries the benchmark's scoring function for up to K=30 iterations (candidates per step unspecified) and up to three deepening rounds. It possibly also uses a best-of-8 pool (App. C) and hand-written hints (e.g. "add halogen to an aromatic ring" for LogP) that the baseline never sees. Oracle calls, tokens and wall-clock are not reported.
- Table 3 bundles "enlarged budget, task-prior init, deepened update" into one step (logP 1.04→1.76; solubility 0.90→2.19), and the text concedes no attribution is possible. The base loop alone (1.04/0.90) is below the quoted baseline on solubility (1.17). App. C's paired-difference CIs against the same-protocol greedy CoT baseline imply (if on the Table-1 statistic) a baseline of ≈1.3–1.6 (logP) and ≈1.3–1.7 (solubility). That would put the base loop below it on both.
- For solubility, Fig. 2 init (0.92), Table 3 base loop (0.90) and the quoted baseline (1.17) disagree. A monotone accept rule cannot end below its own start.
- The only budget-aware test (BoN-8 vs loop) uses a different model (vanilla Qwen3-8B-Instruct), one task (logP), and gives the loop more oracle calls.
- Sec. 2.2 itself calls the update a "diversification signal" with improvement enforced by accept/reject, i.e. stochastic proposal plus oracle rejection. Hidden states never re-enter the network; they only supply logits for the first U tokens. Fig. 1's hill-climbing picture is not what Eqs. 4–5 implement (∇z r is absent).
- Needed, on one checkpoint with identical hints and equal oracle calls: BoN-N, noise-only hidden-state perturbation with the same accept rule, token-space prefix resampling, and a standard oracle-guided optimizer.
- Significance claims are misattributed. Sec. 4.1 says tests confirm gains over LatentChem, but the App. C tests are against a local greedy baseline (no per-sample LatentChem data exist). Several win-rate/sign tests are init-vs-final pairs that the authors call "guaranteed to favor the final answer".
- The "transfer" results and success rates rest on reference leakage and metric optimisation.
- Reaction/caption rewards are Tanimoto to the ground-truth product and METEOR to the reference caption, which are test labels. They still appear in the abstract, in Table 1 (unflagged METEOR columns) and in Sec. 4.1, despite the "upper bound" caveat.
- The reaction "0.00→0.85" starts from a local baseline of exactly 0.00 (published LatentChem: 0.52). That suggests a parsing/format failure.
- 94–100% success on classifier oracles is expected for any search that accepts only score increases. There is no held-out oracle and no validity, novelty, similarity or SA analysis, so oracle exploitation is not excluded.
- Fig. 3 rows 1–2 appear to replace large parts of the source (e.g. the azepane tail by a bromoaniline), contradicting "small and local" edits. The Sec. 4.3 case descriptions (benzothiazole urea, N-methylpiperidine) do not match the drawn structures (phenothiazine amide, seven-membered ring).
- Additional: the delta to LatentSeek/MILR (hidden-state policy gradient already exists) is not articulated beyond oracle, hints and lr doubling. "+51%" is the ratio of mean Δ (0.305/0.598), not a "plain average of per-task relative gains" (≈91%). Fig. 2 labels the method "CLO" and uses unlabelled per-panel scales. Table 9's "−sig*" convention contradicts the text for gsk3β. The TDC author list is wrong, and code identifiers appear in prose (RESTART_TARGET).
Questions for authors
-
Report oracle calls, tokens and wall-clock per molecule per configuration. Is the best-of-8 pool part of the Table 1 runs?
-
On the same checkpoint and pipeline, report: the baseline with the same hint, BoN-N at matched calls, noise-only perturbation with the same accept rule, and a classical optimizer at the same budget.
-
What is new relative to LatentSeek?
-
Explain the solubility inconsistency and why the base loop falls below the greedy baseline.
-
Why is the local reaction baseline exactly 0.00? Report format-failure rates.
-
(suggestion) Move gold-verifier results out of the abstract and Table 1, evaluate on held-out oracles, and report validity/novelty/SA.
Response to Reviewer h5nJ: New Controlled Experiments and Method Clarifications
Thank you for the detailed critique. We now have a fixed-pool, three-seed diagnostic addressing the proposal mechanism, plus structural-quality measurements and the legacy audit.
1. Mechanism, controls, and budget
All arms use frozen Qwen3-8B-Base / LatentChem stage-4 resources, identical hints and an identical eight-candidate pool per source-seed. Latent and random-latent share saved greedy hidden states, not pool-winner states. Each task has 100 sources, seeds 1–3, and 24 additional proposals: exactly 32 candidates within 16,384 generated tokens. Bonus/restart and success/stall early stopping are disabled; this is a diagnostic, not the complete pipeline.
Latent adds raw mean property gains over the pool of +0.4937/+0.3915 on Crippen LogP/ESOL solubility and exceeds it in 173/300 and 150/300 repeated source-seed records. Relative to random latent (noise_std=0.05), paired differences are +0.3794 (95% CI [0.2974,0.4614]) and +0.2852 ([0.1987,0.3792]); both Holm-adjusted p=0.0000399996. We average three seed differences within source before source-level inference (n=100/task; one four-test Holm family; unadjusted CIs).
The requested no-update sampling-and-selection control is now included. Its pool gains are +0.6538/+0.6932, higher than latent's; the data establish an advantage over the tested random-latent control, not over continued sampling. The full-pipeline F/F0 replacement test, new three-seed token-prefix control, and classical optimizer are not included.
Actual LogP/solubility costs are:
Calls/tokens include the shared pool; continuation time excludes its generation, queueing, and model preparation. Shared ceilings do not equal matched actual compute. Resource-aligned descriptive comparisons and coverage accompany the evidence table; shorter latent time does not establish equal-time quality superiority. The eight-start pool is part of the full ChemoGrad pipeline and is counted.
2. Relationship to LatentSeek and implementation
LatentSeek is the direct predecessor of the reward-weighted update. Our contribution is the chemistry-specific search procedure, molecular-model interface, validity-aware verification, and its empirical analysis. We do not claim a new generic estimator or originality for shared prefix/suffix decoding and best-so-far selection.
We will correct the method: the code uses reward-weighted argmax-token log-probabilities and Adam, without the described running-mean baseline. Non-improving outputs do not replace the returned best candidate, but latent/Adam states are not rolled back. The figure must not imply direct differentiation of the chemical reward.
3. Structural quality and verifier scope
All diagnostic final outputs are valid because a valid pool best is retained; proposal validity is 89.90%/89.34% for latent. Its unique final counts are 286/300 and 269/300, not 100%. Latent/source Morgan similarity is 0.5796/0.5174; SA is 2.6669/2.6268. Sampling gives 0.5714/0.4790 and 2.6773/2.6104. These are descriptive structural metrics, not proof of local latent changes, synthesis, or independent-oracle robustness. Three seeds repeat the same 100 sources; held-out-oracle, training-set novelty, and experimental activity remain untested. Reference-based reaction/caption scores will be moved out of the abstract and headline table.
4. Legacy consistency and presentation
The solubility values 1.17 (published Stage 1+2+4), 1.0243 (local stage-4 inference), and about 0.90 (separate base loop) are different protocols, not a paired trajectory. The base-loop initialization was parser-invalid. We will remove the unrecovered Figure 2 point near 0.92 and separate legacy penalized LogP from new Crippen LogP.
Reaction initial 0.00 belongs to Cplus256 (0/100 parsed); B512 has 30/100 parsed and 29/100 valid SMILES. We cannot infer truncation without original stop logs. We will correct Morgan-only versus three-fingerprint metric labels and remove the 0.00→0.85 headline.
We will correct the +51% definition, separate init-to-final diagnostics from competitor tests, remove unsupported legacy seed claims, and fix structure names, “small/local” wording, figure labels, units, citation metadata, and code identifiers. The new three-seed evidence does not retroactively validate the legacy runs.