acceptodds
Under review as a conference paper at ICLR 2027

EpisodeScope: Budget-Aware Evaluation of Diversity Corrections in Molecular Agents

Abstract

Reinforcement learning with verifiable rewards can reduce output diversity, and several corrections restore it within a rollout group or a selected batch. We ask whether local diversity improves discovery per oracle call when a multi-turn agent also controls its oracle spend. We introduce EpisodeScope, an evaluation protocol for multi-turn molecular agents under an unenforced oracle budget. It counts distinct molecules above a baseline-calibrated score threshold per charged call within a validation window and reports realised spend and coverage at common call counts. In four seed-matched Qwen3-4B pairs trained separately under the same per-step scoring cap, farthest-first selection raises within-episode fingerprint spread but lowers qualified discovery per call by – in every pair. Repeated evaluations concentrate on its first-position anchor. A registered, non-concurrent random-anchor control still trails uniform selection in all three same-machine pairs, and determinantal selection gives mixed results. On replayed candidate pools, avoiding already-scored molecules reduces repeats most consistently. Enabling the complete memory configuration raises discovery per call in all five prospectively specified pairs (median difference , one-sided sign test ), although one memory run spends calls per episode. At this operating point, higher local diversity need not improve qualified discovery per call.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, diversity, LLM agents, molecular discovery, evaluation protocol, oracle budget, multi-turn agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hxGzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether improvements in local molecular diversity translate into more efficient discovery in multi-turn molecular agents. The authors introduce EpisodeScope, an evaluation protocol that measures the number of distinct qualifying molecules per charged oracle call while accounting for realised spending and coverage at common call counts. Experiments show that farthest-first (MaxMin) selection improves within-episode structural diversity but underperforms uniform selection in qualified discovery efficiency. Further analyses examine repeated evaluations, anchor selection, and fixed-pool replay. The paper also reports substantial improvements from persistent molecular memory.

Critical strengths
  1. Meaningful evaluation perspective. The paper highlights the distinction between local structural diversity and long-term discovery efficiency. Accounting for actual oracle usage is practically relevant to molecular optimization.
  2. Careful empirical investigation. The authors provide matched comparisons, anchor analyses, fixed-pool replay experiments, and memory ablations. Their transparent reporting of experimental limitations and implementation deviations is also commendable.
Critical weaknesses
  1. Limited novelty and broader scientific significance of the findings. The paper's central observation is that increasing within-episode structural diversity does not necessarily improve the number of distinct high-scoring molecules discovered per oracle call. However, these objectives are not inherently aligned, and the importance of oracle efficiency, avoiding repeated evaluations, and persistent molecular memory has already been extensively explored in molecular optimization. EpisodeScope mainly combines established evaluation quantities rather than introducing a fundamentally new evaluation principle. The most distinctive empirical contribution is the observed underperformance of MaxMin selection relative to uniform selection. However, this reversal is primarily established on a single molecular optimization objective under a specific soft-budget agent configuration. The second molecular objective mainly examines the effectiveness of memory rather than reproducing the MaxMin reversal. Consequently, it remains unclear what sufficiently generalizable insights the study provides beyond demonstrating that local diversity is not always a reliable proxy for discovery efficiency in this particular setting.
  2. Insufficient explanation and experimental isolation of the main performance reversal. The paper reports substantial online performance degradation for MaxMin selection, but the underlying cause remains insufficiently established. First, the main experiments use a fixed first-position anchor rather than the random initialization of the referenced MaxMin-A method. Although the additional random-start control still underperforms uniform selection, the fixed-pool replay in Appendix B.3 shows that random-start MaxMin and fresh uniform selection achieve nearly identical qualified discovery rates, differing by at most 0.010 per call. This contrast raises an important unresolved question: why does MaxMin exhibit substantially worse performance during online training when its direct selection performance on fixed candidate pools is comparable to uniform selection? The observed anchor repetitions explain part of the behavior, but do not fully account for the random-start results. Interactions between selection, policy adaptation, and evolving proposal distributions may be important, yet these mechanisms are not systematically analyzed. Moreover, the realised candidate pools contain only four or five valid proposals at the median, making the selection regime substantially different from the referenced diversity-selection setting. As a result, the paper documents an interesting empirical reversal but does not sufficiently establish its underlying mechanism or the conditions under which it should be expected.
Questions for authors

1.Given that random-start MaxMin and uniform selection perform similarly in fixed-pool replay but differ substantially in online training, what evidence can distinguish the effects of selector choice from those of policy adaptation and changing proposal distributions? 2.What broader conclusions can be drawn from the observed MaxMin reversal beyond the specific molecular objective and experimental configuration? In particular, how might the substantially different candidate-pool sizes and selection ratios affect the comparison with the referenced diversity-selection method? 3.Could the authors further justify the choice of an unenforced oracle budget with zero call-cost penalty? Since training and validation calls differ substantially across methods, how should the reported efficiency results be interpreted in practical molecular optimization settings with limited total oracle budgets?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer hxGz

Thank you for the review. Your first question identified the tension between small fixed-pool differences and consistently negative submitted online contrasts. A frozen 22-cell paired study on Qwen3-4B has now finished, and our companion results comment reports Appendix H's paired results. The corrected study uses stable per-request seeds and fresh paired controls. Random-start MaxMin minus fresh uniform selection changes sign across three seed blocks, so the submitted selector reversal is not reproduced and we withdraw that claim. Isolating policy adaptation from pure selection on evolved policies would still require a fixed-policy selector crossover, which we have not run. Qualified yield, oracle efficiency, persistent memory, and separating local from cross-episode diversity build on established molecular optimization practice rather than introducing a new optimization principle.

1. Fixed-pool and online differences. On the four fixed candidate streams in Appendix B.3, random-start MaxMin and fresh uniform selection differ by at most 0.010 new qualifying molecules per call, whereas the submitted online comparisons differed by −0.289, −0.162, and −0.156. Those two sets of numbers cannot be subtracted to estimate policy adaptation because their source policies, proposal streams, purchase histories, and novelty windows differ (full-stream purchase history in replay versus within-window distinct molecules in the submitted online endpoint). Anchor repetition explains part of the first-start gap, not the random-start gap.

Reconciling episode records with service logs after submission identified a specific competing explanation. Across three complete random-start runs of 3,968 training and validation episode records each, service selection logs contain 673, 923, and 1,344 more selected score positions than recorded episode charges (full-run counts, distinct from the final window, and logged before scoring rather than counting completed backend calls). We reproduced a batched-runner defect where an already-terminated environment can keep requesting scores while another batch member continues, updating shared run memory while omitting those requests from the terminated episode's recorded outcome and cost. Missing historical identifiers prevent assigning every discrepancy to this cause, and adjusting denominators cannot repair altered memory state or feedback. We therefore do not interpret the submitted online-versus-replay gap as evidence of policy adaptation or proposal drift.

2. Generalization and the selection regime. In the logged random-start runs, the median realized pool contains four or five valid proposals with two selected for scoring, which is far less selective than the 640-to-64 regime in the referenced MaxMin-A study. Our first-start rule also used a fixed anchor rather than MaxMin-A's random start, our original uniform control seeded draws from the candidate-list hash, and the random-start follow-up reused earlier controls. Neither the submitted runs nor the corrected 22-cell study reproduces or refutes 640-to-64 MaxMin-A, and no general claim against diverse selection is supported. Likewise, the Perindopril_MPO extension and Qwen3-1.7B replication test memory rather than selector effects, while DPP and memory-plus-IPS remain mixed.

3. Unenforced budget and zero call-cost penalty. Using an unenforced budget with zero call-cost penalty exposes how interventions shift agent-chosen spending rather than fixing expenditure by construction. It estimates a soft-budget operating point rather than performance under a hard total campaign cap. Common-call validation curves adjust for unequal validation spending without equalizing training calls, and omitted post-terminal requests further prevented historical recorded-charge comparisons from showing equal actual spend. For a hard total oracle budget, validation rates must be read alongside cumulative training and validation expenditure. The corrected study logs both, and Appendix H reports the memory arms' earlier evaluation counts.

Corrected runner and open questions. The corrected implementation uses stable per-request seeds, separate selected-position, cache-hit, and backend-evaluation counters, and absorbing terminal states, reconciling all 652 study requests across 384 episodes to 933 actual evaluations in a one-update cache-off Qwen3-4B check. Our companion comment and Appendix H report the completed 22-cell study, while a fixed-policy selector crossover, a selection-ratio sweep, and a second-target selector comparison remain open.

AuthorsOpenReview ↗

Corrected 22-Cell Paired Study Results for Reviewer hxGz

Thank you again for your review. The frozen 22-cell paired study on Qwen3-4B and Osimertinib_MPO (25 GRPO updates) has finished. All 22 cells passed acceptance by reconciling requests, cache hits, and actual backend evaluations, with decision rules fixed before any result was examined. New Appendix H and the revised supplementary material report the full design, per-cell counts, and paired contrasts.

1. Fixed-pool and online differences. Your first question noted that random-start MaxMin and fresh uniform selection had small fixed-pool differences yet consistently negative contrasts in our submitted online runs. The corrected paired study uses stable per-request random draws and trains each block on identical hardware. It does not reproduce the submitted selector reversal. On the primary endpoint (distinct canonical molecules with raw score at least 0.567 first evaluated in the run's final 128 validation episodes, per actual backend evaluation in that window), MaxMin minus uniform is +0.3551, −0.0810, and +0.2085 across three seed blocks with caching off, and +0.0295, −0.0278, and −0.1232 with caching on. On the pre-specified secondary endpoint (final-window qualifying distinct molecules per selected score position), the contrast also changes sign under both caching settings (two positive and one negative in each). These three-block results leave the selector comparison inconclusive rather than proving equivalence, and they do not reproduce the submitted reversal, which we withdraw. A fixed-policy selector crossover would still be needed to separate policy adaptation from pure selection on evolved policies, and we do not claim to have isolated adaptation.

Caching and memory. Enabling raw-score caching raises the primary endpoint in all three blocks for both selectors (uniform +0.6208, +0.3947, +0.5246, and MaxMin +0.2952, +0.4479, +0.1929), partly because cache hits leave the denominator, whereas its secondary contrast changes sign (one positive and two negative under uniform, two positive and one negative under MaxMin). In the five cached memory blocks, at least 99% of the memory-off policy's final-window selections were cache hits, leaving zero actual evaluations in two blocks and one or two in the remaining three. The primary contrast is undefined in two blocks and negative in three (−0.0734, −0.0437, −0.0764). In the latter three, the memory-off rate is 1.0 based on one or two qualifying molecules. This denominator degeneracy makes the primary comparison uninformative about the memory effect. Following the pre-specified rule, the memory claim is restricted to the secondary endpoint, which increases with memory in all five blocks (median +0.5724, range +0.4422 to +0.7143). Directly addressing your budget question, memory also spends substantially more total evaluations across training and validation. Before the final window, the memory-on arms had actually evaluated 5,273 to 7,392 distinct molecules against 681 to 1,953 without memory, so the secondary gain does not show more discovery at a fixed total oracle budget (Appendix H).

Scope. The corrected study covers one model and one objective at 25 updates across three selector blocks and five memory blocks, reporting paired directions without significance tests (with at least 156,014 actual backend evaluations and 99,094 cache hits logged across all campaign stages). It neither reproduces nor refutes the published 640-to-64 MaxMin-A regime, and a fixed-policy crossover, a selection-ratio sweep, and a second-target selector comparison remain open. What the revised paper contributes is the EpisodeScope accounting protocol, which exposed the post-terminal charging defect, together with the corrected online evaluation, including an empirical regime where near-total cache reuse makes per-evaluation validation rates degenerate unless paired with cumulative spend and per-selection reporting. We claim neither a new diversity principle nor a general selector ranking in the revised abstract, introduction, Section 4.1, and conclusion.

Reviewer_hxGzOpenReview ↗

Maintain my Weak Reject rating.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address the experimental issues and improve the reliability of the study. However, the revised experiments also substantially narrow the paper's original findings, particularly regarding the central MaxMin performance reversal. The remaining contributions, which mainly focus on an evaluation and accounting framework, still lack sufficiently demonstrated novelty and broader applicability beyond existing molecular optimization benchmarks. Therefore, I maintain my Weak Reject rating.

Thank you for reviewing our additional 22-run campaign. Rather than merely narrowing our study, investigating the exact discrepancy you flagged in Question 1 (between fixed-pool replay differences of at most 0.010 in Appendix B.3 and the submitted online differences of −0.289, −0.162, and −0.156) uncovered two consequential failure modes in how multi-turn RLVR agents with costly oracles are evaluated, alongside a clean online test of local batch dispersion versus stateful exploration (Appendix H, pp. 47–49 of the 15 November revised PDF via the PDF icon beside the title, changes in blue):

  1. Exposing off-record oracle activity in batched multi-turn RLVR: Replay and online runs already differ in source policies, proposal streams, purchase histories, and novelty windows, and reconciling episode records against service logs further revealed 673, 923, and 1,344 excess pre-scoring selected positions over recorded episode charges across three 3,968-episode runs, together with a reproducible batched-runner bug where terminated environments kept querying the oracle and mutating shared memory. Enforcing absorbing terminal states and exact reconciliation across requests, selected positions, cache hits, and backend evaluations (all 22/22 corrected runs) eliminates an execution hazard that arises whenever variable-length multi-turn RLVR rollouts share a stateful tool or oracle service.

  2. Uncovering a striking failure mode of standard per-call efficiency under caching ("100% precision on 1 call"): Standard cost-aware benchmarks rank agents by campaign-new qualifying discoveries per actual backend oracle call (our primary endpoint, R1R_1). In our corrected GRPO runs with caching enabled (Table H.2, p. 48 and per-cell supplement records), removing memory causes the policy to repeat cached molecules on 99%–100% of final-window selections. Because cache hits leave the paid-call denominator, Memory OFF makes 0 backend calls in two blocks (0/0) and just 1 or 2 calls in Blocks 100, 103, and 104, scoring 1/1 = 1.0000 and 2/2 = 1.0000 and outscoring Memory ON (303/327 = 0.9266, 197/206 = 0.9563, 145/157 = 0.9236) on per-backend-call efficiency despite discovering only 5, 8, and 9 window-qualifying molecules versus 362, 250, and 202. Pairing R1R_1 with per-selection yield (R2R_2, where Memory ON wins all 5/5 blocks by +0.4422 to +0.7143) and prior unique exploration (5,273–7,392 vs. 681–1,953 actual evaluations) is essential to keep cached benchmarks from misranking collapsed policies.

  3. Clarifying what actually drives multi-turn RLVR discovery: Under reconciled online training, enforcing within-turn structural dispersion (MaxMin) gives no consistent gain over fresh uniform selection at this 3-block scale (MaxMin − Uniform flips sign under both caching OFF and ON, Table H.1, leaving the selector contrast inconclusive rather than showing a reversal), whereas online score caching (+0.1929 to +0.6208 on R1R_1 across all 6/6 pairs) and persistent cross-episode memory (5/5 blocks on R2R_2) govern the consistent efficiency and exploration shifts.

Although our 22 prospective runs empirically evaluate Qwen3-4B with GRPO on Osimertinib_MPO (and do not test the 640-to-64 MaxMin-A regime or non-molecular tasks), the protocol requirements demonstrated here (terminal-boundary reconciliation, guarding against cache-induced denominator collapse, and separating window efficiency from cumulative training spend) apply whenever multi-turn LLM agents train against cached or stateful oracles. We hope this highlights the substantive value of the revised contribution.

Reviewer_QunJOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the diversity evaluation of multi-round molecular agents under a soft oracle budget. This paper proposes EpisodeScope, which combines the number of qualified distinct molecules within the window with the actual number of calls. The authors report both the consumption and the number of discoveries at the unified number of calls. Experiments show that farthest preference improves local fingerprint dispersion but reduces per-call discovery efficiency.

Critical strengths
  • The paper focuses on a particle problem. When the agent is able to decide when to evaluate via an oracle evaluator, the local diversity and the discovery benefits of each oracle call need to be measured separately.
  • The paper provides many evaluations and discusses them with numerical support.
Critical weaknesses
  • The writing is not clear. It is really hard to identify what the proposed EpisodeScope is, what the main conclusions are, and how the proposed method addresses the problem described in the introduction. Along with the tables and the figures, there are more like experimental records than publishable results.
  • The discussion of the results is not informative. Most of the content describes the numbers instead of insights gained from the experiment. For example, what is the main takeaway from Table 1 and Figure 2?
  • The paper only conducts experiments on one dataset (Osimertinib MPO), one RL method (GRPO), and one model (Qwen3-4B). The conclusions drawn from these evaluations are limited by the setting.
Questions for authors
  1. Please explain how the EpisodeScope addresses the problem described in the introduction.
  2. Please provide more discussion or results on the model, RL method, or dataset beyond the setting described in the paper.
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer QunJ

Thank you for the review. EpisodeScope is an evaluation protocol, whereas the candidate selectors and memory configurations are the interventions it evaluates. The revision defines the protocol at the start of Section 3, before Table 1 and Figure 2. The abstract and introduction now lead with the completed 22-cell corrected study, detailed in Appendix H and our companion results comment. The submitted selector reversal is not reproduced and is withdrawn. Score caching raises discovery per actual evaluation, and molecular memory raises discovery per selected position in all five pairs while using several times more actual evaluations.

How EpisodeScope addresses the introduction's problem. When a multi-turn agent decides how many molecules to submit for oracle scoring, raw discovery counts can reward higher spending, while within-episode structural diversity ignores repeated molecules across episodes. EpisodeScope pairs the cost-normalized rate Yτ = Uτ/C with recorded charges per episode and cumulative discovery coverage at common recorded-charge counts, where Uτ counts distinct canonical molecules above the baseline-calibrated qualification threshold in the validation window and C counts charges in active-episode records. These quantities have precedents in molecular optimization, and combining them provides an explicit accounting protocol rather than a new diversity principle.

Takeaways from Table 1 and Figure 2. Local within-episode fingerprint dispersion and cross-episode discovery per call measure distinct quantities, and the observed ranking depends on the evaluation horizon and cost measure. Under the submitted record-based endpoint in Table 1, first-start MaxMin is lower in all four matched pairs, random-start MaxMin is lower in three non-concurrent same-machine comparisons, DPP is mixed, and complete molecular memory is higher in all five prospective pairs. Figure 2 shows that first-start MaxMin leads at early checkpoints in three pairs before trailing at the final pass. The corrected study does not reproduce the submitted selector reversal. Table 1 and Figure 2 remain record-based measurements whose causal interpretation the execution defect leaves unresolved.

Post-submission accounting finding. Across each complete random-start run's 3,968 training and validation episode records, selected score positions in the service logs exceed recorded charges by 673, 923, and 1,344 (full-run discrepancies, distinct from the 128-episode final window). Selection is logged before scoring, so these excesses are not counts of completed backend calls. Local reproduction identified an execution defect in which a terminated environment can keep requesting scores while another batch member remains active, updating shared memory without entering the terminated episode's recorded outcome or cost.

Reproducing Table 1 from retained episode records therefore neither validates complete oracle expenditure nor isolates selector or memory mechanisms, and a denominator-only adjustment cannot undo changes to shared memory and subsequent feedback. The revision discloses this defect beside the endpoint definition and reports the frozen 22-cell study collected with our corrected runner, which blocks post-terminal requests and logs selected positions, cache hits, and actual backend evaluations separately (reconciling all 652 study requests across 384 episodes to 933 actual molecular evaluations in a one-update Qwen3-4B check with fresh uniform selection and caching off).

Evidence beyond the principal setting. Beyond Qwen3-4B on Osimertinib_MPO, the submitted paper includes a five-block Perindopril_MPO extension and a nine-run Qwen3-1.7B replication comparing memory, GRPO, and IPS, which yields a consistent ordering without complete separation (U = 17/18). Both address memory rather than cross-target selector replication, retain the Qwen3 family and GRPO under the pre-audit runner, and do not cover independent model families or RL optimizers, for which we have no additional experiments to report.

AuthorsOpenReview ↗

Corrected 22-Cell Paired Study Results for Reviewer QunJ

Thank you again for your review. The frozen 22-cell paired study providing corrected online evidence has finished, and all 22 cells passed acceptance by reconciling requests, cache hits, and actual backend evaluations. Decision rules were fixed before any result was examined. New Appendix H reports the full study, and the anonymized frozen matrix and per-cell counts appear in the revised supplementary material.

Accounting. The corrected implementation prevents terminated environments from requesting scores and logs selected positions, cache hits, and actual backend evaluations separately. The primary endpoint counts distinct qualifying canonical molecules (raw score at least 0.567) first evaluated in the run's final 128 validation episodes, divided by all actual backend evaluations in that window. A cache hit adds no evaluation, whereas an uncached repeat adds one. Across all recorded campaign stages, including preflights, pilots, and stopped attempts, the service logged 156,014 actual backend evaluations and 99,094 cache hits. Seven cell attempts and nine pilot attempts stopped without closing service counters, and two cell attempts were lost with a cloud session, making these totals a lower bound on total campaign cost.

Design and results. Each cell trains Qwen3-4B on Osimertinib_MPO for 25 GRPO updates. Twelve cells cross random-start MaxMin with independent per-request uniform selection and raw-score caching off versus on across three seed blocks (budget eight, per-step cap two, memory on). Ten cells compare memory on versus off with caching on across five seed blocks (budget two). Appendix H reports within-block paired differences by per-block sign, median, and range without significance tests, alongside the submitted secondary endpoint (final-window qualifying distinct molecules per selected score position).

Contrast (first arm minus second)Primary, per blockSecondary, signs
MaxMin − uniform, caching off+0.3551, −0.0810, +0.20852 positive, 1 negative
MaxMin − uniform, caching on+0.0295, −0.0278, −0.12322 positive, 1 negative
Caching on − off, uniform+0.6208, +0.3947, +0.52461 positive, 2 negative
Caching on − off, MaxMin+0.2952, +0.4479, +0.19292 positive, 1 negative
Memory on − off−0.0734, undefined, undefined, −0.0437, −0.07645 positive (median +0.5724)

What the takeaways become. The selector reversal in Table 1 is not reproduced. MaxMin minus uniform changes sign across the three blocks under both caching settings on both endpoints, leaving the comparison inconclusive rather than proving equivalence, and the revision withdraws the claim that farthest-first selection lowers discovery per call. Score caching raises the primary endpoint in all three blocks for both selectors, partly because cache hits leave the denominator, while its secondary contrast changes sign. Without memory, at least 99% of final-window selected positions were cache hits, leaving zero actual evaluations in two blocks and one or two in the other three. Those few molecules all qualified, giving memory-off rates of 1.0 and three negative paired primary differences. The other two differences are undefined. With the memory-off denominator degenerate, the memory claim is restricted to the pre-specified secondary endpoint, which increases in all five blocks (median +0.5724, range +0.4422 to +0.7143). Before the final window, however, the memory-on arms evaluated 5,273 to 7,392 distinct molecules versus 681 to 1,953 without memory, so this secondary gain does not show higher discovery at a fixed oracle budget.

The revised abstract and introduction now lead with this corrected study, while the submitted record-based comparisons follow as historical results that it does not reproduce. Because the corrected study covers one model and one objective at 25 updates (three selector blocks and five memory blocks), it reports paired directions at this scale rather than general cross-domain laws, as stated in the revised abstract, introduction, Sections 3 and 4.1, limitations, and conclusion.

Thank you for the responses.

My concerns about the clarity and generalizability of the conclusions remain. Moreover, the authors also claim that the results in Table 1 are not reproducible.

The memory comparison also remains difficult to interpret because the primary metric is undefined. Please clearly provide what findings are still validated and their supported evidence. In particular, after withdrawing the original claim, what main findings remain, and which corrected experiments support each finding?

I would like to temporarily decrease my score to 1 until further discussion is provided.

Reviewer_1A19OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EpisodeScope, an evaluation protocol for multi-turn molecular agents that determine their own oracle expenditure. It measures distinct molecules above a quality threshold per charged call, alongside realized expenditure and coverage at common call counts. The experiments show that farthest-first selection can increase local structural diversity while reducing qualified discovery per call. A random-start control shows a similar pattern, whereas DPP selection and adding IPS to memory produce mixed results. Five prospectively specified pairs support the benefit of the complete memory configuration.

Critical strengths
  1. The evaluation considers both useful discoveries and oracle expenditure. Qualified discovery rates, coverage at common call counts, and calls required to reach a discovery target provide complementary views of efficiency. Threshold and checkpoint analyses strengthen the main observation.

  2. The main findings have targeted experimental support. Four matched pairs show increased local diversity but reduced discovery efficiency under farthest-first selection. Five prospectively specified memory pairs show positive effects, including under common-call comparisons. These experiments provide useful evidence beyond descriptive comparisons of historical runs.

Critical weaknesses
  1. The practical efficiency conclusion lacks an online deduplication or caching control. Repeated evaluations are charged, and the paper identifies repeats as a contributor to the efficiency gap. Although fixed-pool replay shows benefits from selecting previously unscored molecules, it does not establish the online effect of avoiding repeated evaluations. It therefore remains unclear whether the selector ranking and the additional benefit of memory persist when all methods use pre-scoring deduplication or cached scores.

  2. The central selector reversal remains tied to specific implementation choices. The primary farthest-first implementation uses a fixed first-position anchor, while the random-start follow-up reuses earlier controls. Uniform selection also derives its random draw from the candidate list, so repeated lists receive identical selections. These choices directly interact with cross-episode repetition. Further paired experiments with independent, reproducible random draws are needed to establish whether the reversal is robust beyond the implemented procedures.

Questions for authors
  1. Would the main selector ranking and memory gains persist if all methods used pre-scoring deduplication or historical score caching? An online comparison reporting qualified coverage against actual oracle evaluations, with repeated requests reported separately, would clarify the practical implications.

  2. Can random-start farthest-first and uniform selection with independent draws at each request be compared under the same service configuration and explicitly recorded random seeds? This would help establish whether the reversal depends on the current sampling implementation.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer 1A19

Thank you for the review. A frozen 22-cell paired study on Qwen3-4B and Osimertinib_MPO now tests online score caching and independently seeded selectors. Twelve cells cross random-start MaxMin and fresh per-request uniform selection with caching off and on in three seed blocks. Ten cells compare memory on and off with caching on in five blocks. All 22 cells passed acceptance, and our companion results comment reports the paired findings in Appendix H. The submitted selector reversal is not reproduced. Caching raises discovery per actual evaluation, while memory raises discovery per selected position in all five pairs at several times the actual evaluation cost. This comment explains why the corrected study was needed and how its controls differ from the submitted comparisons.

Why the study was necessary. Reconciling episode records with service logs after submission exposed an execution defect whose effects cannot be isolated in the submitted online runs. Each of the three complete random-start runs contains 3,968 training and validation episode records. Their selection logs contain 22,696, 18,027, and 12,924 selected score positions, whereas the episode records sum to 22,023, 17,104, and 11,580 charges, leaving full-run discrepancies of 673, 923, and 1,344 positions (distinct from the 128-episode final-window endpoint). Selection is logged before scoring, so these discrepancies are not counts of completed backend evaluations. Local reproduction identified a specific cause in the batched runner, which can advance an already-terminated environment while another batch member remains active. Resulting requests can update shared molecular memory even though they are excluded from the terminated episode's recorded outcome and charges. Missing request-to-episode identifiers prevent assigning every historical discrepancy to this defect.

The submitted final-window rates still reproduce from the retained episode records, which do not capture complete service-level expenditure. Because post-terminal requests can also alter shared memory and subsequent training feedback, adjusting denominators alone would not repair the online comparison. We therefore treat the submitted numbers as record-based measurements rather than validated causal contrasts under complete oracle accounting.

Online deduplication, caching, and memory. As you noted, fixed-pool replay does not answer the online question. On its four logged candidate streams, restricting selection to previously unscored molecules raises new qualifying molecules per call by 0.026–0.041 over fresh uniform selection, measuring only direct selection on fixed streams that cannot respond to replayed purchases, scores, or rewards. Likewise, the five submitted prospective memory pairs, including their common-call comparisons, did not test whether memory retains its advantage when both arms use historical score caching. The new online study tests score caching rather than pre-selection candidate filtering. Its cache reuses raw oracle scores while applying the memory-dependent reward transformation separately, because caching transformed rewards would alter the memory intervention. The primary counts distinct qualifying canonical molecules (raw score at least 0.567) first evaluated in the run's final 128 validation episodes, divided by all actual backend evaluations in that window. Repeated requests and cache hits are reported separately, and cache hits add no backend evaluation.

Independent random draws. In the submission, uniform selection derived its draw from the ordered candidate list, giving identical selections to repeated lists, while the random-start MaxMin follow-up reused earlier non-concurrent controls, so its three same-machine differences (−0.289, −0.162, and −0.156) resolved neither candidate-hash coupling nor concurrency. The corrected runner derives draws from explicit, stable per-request identifiers, logs selected positions, cache hits, and actual backend evaluations separately, and prevents terminated environments from issuing further purchases. Before any cell ran, a frozen study matrix fixed each cell's training seed, shared selection seed, and service configuration, now included in the revised supplementary material. In a one-update accounting check on Qwen3-4B with fresh uniform selection and caching off, all 652 study requests across 128 training and 256 validation episodes (384 episodes) reconciled to 933 actual molecular evaluations before the 22-cell comparisons ran.

The revision labels the submitted endpoint as charges recorded for active episodes, discloses the execution defect beside the main comparisons in Sections 3 and 4.1, and reports the corrected study in the abstract, introduction, Section 4.1, conclusion, and Appendix H.

AuthorsOpenReview ↗

Corrected 22-Cell Paired Study Results for Reviewer 1A19

Thank you again for asking for online score caching and independently seeded selector comparisons. The frozen 22-cell paired study has finished, and all 22 cells passed acceptance, reconciling every cell's requests, cache hits, and actual backend evaluations. Decision rules were fixed before any result was examined, and new Appendix H reports the full study. The revised supplement provides the anonymized frozen matrix, per-cell counts, paired contrasts and corrected runner, including its molecular overlay, FP32 streaming actor patch, installer and portable launcher. Configuration, HTTP, runner and recipe checks passed. No live PyTDC service or new GPU training was run during packaging.

Design. Each cell trains the original Qwen3-4B policy on Osimertinib_MPO for 25 GRPO updates. Twelve cells cross random-start MaxMin with independent per-request uniform selection and raw-score caching off versus on across three seed blocks (commanded budget eight, per-step cap two, memory on). Ten cells compare memory on versus off with caching on across five seed blocks (commanded budget two), sharing no arms with the selector grid and testing no selector-by-memory interaction. The primary endpoint counts distinct qualifying canonical molecules (raw score at least 0.567) first evaluated in the run's final 128 validation episodes, divided by all actual backend evaluations in that window. The pre-specified secondary endpoint is the submitted measure, final-window qualifying distinct molecules per selected score position. Appendix H reports each within-block paired difference by per-block sign, median, and range without significance tests, and the table below summarizes per-block primary differences and secondary sign counts.

Contrast (first arm minus second)Primary, per blockSecondary, signs
MaxMin − uniform, caching off+0.3551, −0.0810, +0.20852 positive, 1 negative
MaxMin − uniform, caching on+0.0295, −0.0278, −0.12322 positive, 1 negative
Caching on − off, uniform+0.6208, +0.3947, +0.52461 positive, 2 negative
Caching on − off, MaxMin+0.2952, +0.4479, +0.19292 positive, 1 negative
Memory on − off−0.0734, undefined, undefined, −0.0437, −0.07645 positive (median +0.5724)

Selection. MaxMin minus uniform changes sign across the three blocks under both caching settings on both endpoints. These runs leave the selector comparison inconclusive rather than establishing equivalence, and they do not reproduce the submitted selector reversal. We therefore withdraw the claim that farthest-first selection lowers discovery per call in the revised abstract, introduction, Section 4.1, and conclusion.

Online caching. Enabling raw-score caching raises the primary endpoint in all three blocks under both selectors. Because a cached repeat leaves the actual-evaluation denominator, this gain combines denominator accounting with any change in policy proposals. On the secondary endpoint, which counts every selected position in its denominator, the caching contrast changes sign across blocks.

Memory. Without memory, at least 99% of final-window selected positions were cache hits, leaving zero actual evaluations in two blocks and one or two in the other three. Those one or two molecules all qualified, so the three defined memory-off rates are 1.0 and all three paired primary differences are negative. The other two paired differences are undefined. This degeneracy in the memory-off denominator makes the primary comparison uninformative about the memory effect. Following the pre-specified rule, the memory claim is restricted to the secondary endpoint, which increases with memory in all five blocks (median +0.5724, range +0.4422 to +0.7143). Before the final window, however, the memory-on arms had evaluated 5,273 to 7,392 distinct molecules versus 681 to 1,953 without memory, so higher discovery per selected position does not establish higher discovery at a fixed actual-evaluation budget.

Execution. Each block ran on one machine with identical hardware, software and scoring configuration across paired arms. Seven cell attempts stopped early and two were lost with a cloud session. Each was rerun under a new attempt identifier, and no completed cell was discarded. Nine pilot attempts also stopped without closing service counters. Across all recorded campaign stages (including preflights, gates, pilots, and stopped attempts), the service logged 156,014 actual backend evaluations and 99,094 cache hits, which is a lower bound on total cost because stopped attempts lacked closing service counters.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Post-Revision Discussion Summary for the Area Chair

Thank you for overseeing the review of our submission. Because Reviewer QunJ lowered their score temporarily pending further discussion, we briefly summarize our follow-up clarifications and manuscript update for your reference:

  1. Reproducibility and endpoint definitions: We clarified that every number in Table 1 reproduces deterministically from retained episode records; only the historical MaxMin reversal was withdrawn after our bug-fixed 22-run online study (Appendix H) proved inconclusive on selection. We also provided explicit formulas and per-block counts showing that "undefined" in two Memory OFF cells is arithmetic division by zero (0/0) because 99%–100% of selections hit the score cache (repeating one cached canonical molecule across 264 and 255 selections), whereas Memory ON discovers 137 to 420 campaign-new qualifying molecules per block and improves the pre-specified secondary rate in all 5/5 blocks (median +0.5724).

  2. Scope and revised PDF: Following Reviewer QunJ's final request, we explicitly separated transferable batched-RLVR accounting requirements from our single-setting empirical scope (Qwen3-4B trained with GRPO on Osimertinib_MPO), explained how the pre-audit runner defect affects the historical Perindopril_MPO and Qwen3-1.7B memory extensions, and updated both the abstract and conclusion in the newly uploaded revised PDF (changes in blue) to make this tested scope explicit.

We remain available if any additional clarification would be useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.