- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces RealFlux, a molecular counterfactual explanation framework that performs local chemical editing toward a desired prediction direction. It combines (i) Native-U for forward-utility-based site localization, (ii) CLE for preserving the non-edited molecular context, and (iii) IDG for filtering candidates that are distant from the predictor's training distribution. Localization and subsequent editing share the same directional utility.
Critical strengths
None
Critical weaknesses
-
Eq. (3) averages utility over each region's feasible edits, making rankings sensitive to action-set composition and potentially obscuring rare but highly effective edits. The paper does not justify why mean utility is an appropriate criterion for locating successful counterfactual edits.
-
Native-U scores regions independently using first-step edits. Hence, it likely fails to capture cases where individually weak edits become effective only when multiple sites are modified jointly.
-
The novelty appears limited. CLE resembles existing scaffold-constrained/local editing mechanisms, while IDG is a relatively standard nearest-neighbor representation-space OOD filter.
-
IDG's benefit is lower distribution distance (W) in the same representation space used for gating. w/o IDG yields higher mean targeted success and validity and lower leakage in Table 4, while worsening W. This does not establish that IDG improves counterfactual quality.
-
Scalability with respect to action-space size is unclear. Native-U evaluates first-step candidates across all regions, but the paper reports only latency, without candidate counts, predictor-evaluation counts, or scaling with molecular or action-space size. Therefore, efficiency beyond the tested setup remains unestablished.
Questions for authors
-
What is the definition of Context Retention? CLE is described as a hard invariant that preserves the non-edited molecular context, yet the reported CR is only 0.654 for RealFlux (Full). If CR measures preservation of the locked context, this value seems difficult to reconcile with the hard-invariant statement.
-
The paper gives the high-level definitions of Native-U, CLE, and IDG, but many details remain unclear to me. For example, how is the action library constructed, how many candidates are generated per region, how does multi-step local refinement proceed, what are the stopping criteria, and how exactly is the “unified selector” implemented?
8 comments
Theme 2: Methodological Novelty of Closed-Loop Integration and IDG Manifold Guardrail Trade-offs (Weaknesses 3.4 & 3.5)
-
Methodological Novelty of the Unified Closed-Loop Framework (Response to Weakness 3.4): Existing molecular counterfactual pipelines decouple post-hoc attribution (e.g., Grad-CAM) from predictor-agnostic candidate generation (e.g., SELFIES string mutation in MMACE, database lookup in CReM, or 3D diffusion in DiffLinker). RealFlux's core methodological innovation is reformulating molecular counterfactual explanation as a training-free, closed-loop constrained search that unifies site localization (Native-U), topological invariance (CLE), and representation-manifold gating (IDG) over the same executable action space and directional objective. Moreover, CLE and IDG introduce critical domain-specific advances beyond standard heuristics:
- CLE vs. Standard Scaffold Protection: Standard tools typically freeze a static Bemis–Murcko ring core or require manual atom masks. CLE enforces exact subgraph isomorphism and aromaticity invariance () dynamically across arbitrary single-anchor (L1), multi-anchor linker (L2), and full-ring replacement (L3) complements, preventing topological leakage through message passing.
- IDG as an Active Generation-Time Constraint: Whereas prior XAI benchmarks (e.g., CM, Janisiów et al. 2026) compute embedding distances only as a post-hoc diagnostic, IDG enforces predictor-manifold proximity during candidate generation using the same forward pass as Native-U, preventing the search from drifting into out-of-distribution (OOD) space.
-
Why IDG is Essential & Non-Circular Despite Nominal Flip-Rate Trade-offs (Response to Weakness 3.5):
- Preventing Adversarial OOD Shortcuts: GNN classifiers are highly vulnerable to OOD structural distortions that cross the decision boundary in extrapolated regions of latent space. As shown in the 3-seed per-seed breakdown below, removing IDG (
w/o IDG) expands the unfiltered pool from 8,978 to 9,425 candidates, allowing the optimizer to exploit OOD adversarial shortcuts that slightly inflate nominal Targeted Success (0.622$\to$ 0.644) and Validity (0.979$\to$ 0.991) while catastrophically degrading latent manifold distance W by +90.3% (2.529$\to$ 4.812, worsening by +1.87, +2.31, and +2.66 across Seeds 5, 15, and 25). In XAI, a counterfactual that flips the model only by leaving the training manifold is an adversarial artifact rather than a valid explanation; IDG serves as the essential guardrail that prevents this failure mode. - No Circularity Between IDG Gating and W Evaluation: IDG applies a local pointwise 90$\text{th}\tau_{\mathrm{IDG}}n_{\min}=1,032$).
- Preventing Adversarial OOD Shortcuts: GNN classifiers are highly vulnerable to OOD structural distortions that cross the decision boundary in extrapolated regions of latent space. As shown in the 3-seed per-seed breakdown below, removing IDG (
| Ablation Branch | Seed | E4 Context Retention | E4 Targeted Success (TS) | E2 Prediction Leakage | E2 Validity | E2 Representation Distance W |
|---|---|---|---|---|---|---|
| RealFlux (Full) | 5 / 15 / 25 | 0.743 / 0.627 / 0.593 | 0.717 / 0.550 / 0.600 | 0.253 / 0.242 / 0.274 | 0.975 / 0.977 / 0.973 | 2.442 / 2.388 / 2.756 |
| Mean SD | 0.654 0.064 | 0.622 0.070 | 0.256 0.013 | 0.975 0.002 | 2.529 0.162 | |
| w/o CLE | 5 / 15 / 25 | 0.340 / 0.168 / 0.192 | 0.350 / 0.167 / 0.233 | 0.196 / 0.170 / 0.186 | 0.959 / 0.975 / 0.940 | 3.299 / 3.287 / 2.789 |
| Mean SD | 0.233 0.076 (3/3 ) | 0.250 0.076 (3/3 ) | 0.184 0.011 | 0.958 0.014 | 3.125 0.238 (3/3 ) | |
| w/o IDG | 5 / 15 / 25 | 0.747 / 0.640 / 0.607 | 0.733 / 0.583 / 0.617 | 0.203 / 0.186 / 0.204 | 0.981 / 0.983 / 0.986 | 4.315 / 4.700 / 5.421 |
| Mean SD | 0.664 0.060 | 0.644 0.064 | 0.197 0.008 | 0.983 0.002 | 4.812 0.458 (+90.3% ) |
Theme 3: Formal Definition of Context Retention (CR) and Reconciliation with the CLE Hard Invariant (Question 3.7)
We appreciate the opportunity to clarify this metric. The apparent tension between and the CLE hard invariant (Eq. 5) stems from the distinction between a candidate-level generation filter (CLE) and a task-level full-molecule evaluation metric (). We have added the explicit definition of to Section 4 and Appendix A.3:
where is the parent molecule, is the target region, is the locked context with heavy-atom node set , and is the exact subgraph-isomorphism and aromaticity-invariance indicator. If a task yields no valid candidate within the budget, is recorded in the task denominator.
Why is completely consistent with 100% CLE enforcement:
-
Candidate-Level Hard Invariant () vs. Whole-Molecule Atom Ratio (): During generation, any candidate violating is immediately rejected by CLE; hence 100% of generated RealFlux candidates satisfy . However, because the target region is replaced or edited (), the preserved context accounts for of the parent molecule's heavy atoms.
-
Stratification Across Topological Levels (Table 7): In L1 (single substituent, ) and L2 (multi-anchor linker), RealFlux achieves and 0.833. In L3 (full-ring replacement, where the replaced ring system comprises of parent heavy atoms), the remaining context fraction is naturally .
-
Macro-Averaging & Ablation Proof (Table 4): Macro-averaging across all 60 tasks (20 each in L1, L2, L3, including no-output tasks per seed counted as 0.0) across Seeds 5, 15, and 25 yields 0.743, 0.627, and 0.593, giving . Removing CLE (
w/o CLE) allows edits to invade the locked scaffold (), causing to drop by -0.421 to :
| Breakdown Dimension | Setting / Seed | RealFlux (Full) CR | w/o CLE CR | CM (CReM) CR | CM (DiffLinker) CR | RealFlux (Full) TS | w/o CLE TS |
|---|---|---|---|---|---|---|---|
| By Topological Level | L1 (Single-anchor, 20 tasks) | 0.825 | — | 0.600 | 0.742 | 0.700 | — |
| (Table 7, 3-seed mean) | L2 (Multi-anchor, 20 tasks) | 0.833 | — | 0.800 | 0.742 | 0.583 | — |
| L3 (Full-ring, 20 tasks) | 0.588 | — | 0.350 | 0.658 | 0.583 | — | |
| By Random Seed | Seed 5 (60 tasks, L1–L3) | 0.743 | 0.340 | — | — | 0.717 | 0.350 |
| (Table 4, 60 tasks/seed) | Seed 15 (60 tasks, L1–L3) | 0.627 | 0.168 | — | — | 0.550 | 0.167 |
| Seed 25 (60 tasks, L1–L3) | 0.593 | 0.192 | — | — | 0.600 | 0.233 | |
| Macro Summary | Mean SD (3 seeds) | 0.654 0.064 | 0.233 0.076 | — | — | 0.622 0.070 | 0.250 0.076 |
Response to Reviewer bDSg
Thank you for your rigorous and insightful review. Below, we address all of your technical concerns and questions in four structured themes, supported by new held-out edit-split experiments, 3-seed per-seed ablation breakdowns, formal metric reconciliations, and scalability statistics.
Theme 1: Justification of Mean Utility in Eq. (3) and Handling Multi-Site Synergistic Edits (Weaknesses 3.2 & 3.3)
Why Mean Utility Outperforms Max Utility for Region Localization (Response to Weakness 3.2):
Thank you for raising this fundamental question. In discrete medicinal chemistry optimization, a region's suitability as an intervention site depends on whether its local chemical neighborhood supports generalizable directional editability across subsequent refinement and feasibility filters (CLE, IDG, SA≤4.5), rather than a single fragile outlier edit. Ranking regions by maxc∈Cfirst(r)u(c) suffers from the winner's curse: a region with one extreme mutation but mostly deleterious edits will be selected, only to fail when that single mutation violates downstream manifold/SA gates or requires second-step refinement.
To test this rigorously without in-sample bias, we randomly split each candidate region's first-step edits Cfirst(r) into a selection half A and a held-out validation half B (55 molecules × 3 seeds, 20 random 50/50 splits per molecule). We selected the top region r∗ on A using Mean (Eq. 3) vs. Max (maxc∈Au(c)) and evaluated the chosen region on the unseen edits in B:
(Notes: †All three per-seed 95% bootstrap CIs exclude zero: Seed 5 +0.082[0.007,0.188], Seed 15 +0.056[0.012,0.103], Seed 25 +0.094[0.023,0.184]. ‡All three per-seed 95% CIs include zero: +0.007[−0.008,0.022],+0.017[−0.001,0.037],+0.015[−0.001,0.036].)
As shown above, Mean utility achieves significantly higher held-out utility (0.771 vs. 0.693, 95% CIs >0 in 3/3 seeds) and higher Top-1 region stability (0.846 vs. 0.812) while matching or slightly exceeding Max on held-out flip rate (0.213 vs. 0.200). Furthermore, in E3 (Table 6), Native-U achieves 0.774 sign consistency and ρ=0.458 rank correlation with observed CReM fragment interventions (vs. 0.642 and 0.064 for Grad-CAM).
First-Step Regional Probing vs. Multi-Step Synergistic Refinement (Response to Weakness 3.3):
Enumerating joint multi-site combinations during initial site localization would incur O(∣A∣k) combinatorial explosion. Instead, RealFlux decouples O(1)-batch regional localization from multi-step / multi-anchor synergistic intervention:
CID 3233055, Fig. 2)CID 3240659, Fig. 2)