- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a shape-matched evaluation of SEGO coding trees that preserves hierarchy branching and group sizes while randomizing vertex membership. Using edge-fraction summaries for molecular scaffold-shift detection, it examines whether SEGO’s selected memberships provide information beyond hierarchy shape, finding small gains that depend on the detector and reference estimation.
Critical strengths
-
The shape-matched randomization provides a controlled way to assess whether selected vertex memberships contribute useful information beyond hierarchy branching and group sizes.
-
The paper transparently reports uncertainty and conducts sensitivity analyses across detectors, tree heights, randomization budgets, and repeated pipelines, helping clarify the robustness and limitations of its findings.
Critical weaknesses
-
The main contribution is a shape-preserving randomization control applied to an existing hierarchy construction, which limits the methodological novelty. In addition, the paper needs to justify its conceptual advance over established constrained-randomization evaluations more clearly.
-
The experimental evaluation covers only a limited number of datasets. Additional molecular datasets, including larger datasets such as ogbg-molpcba, and additional hierarchy methods would help establish the generalizability of the findings. Furthermore, the experiments provide limited evidence of a reliable advantage over the shape-matched reference.
-
The analysis uses compact edge-fraction summaries before neural representation learning. It does not test whether replacing SEGO’s coding trees with shape-matched random trees changes the performance of the complete SEGO pipeline. Consequently, the findings cannot directly explain the hierarchy’s contribution to SEGO’s reported success.
Questions for authors
-
What conceptual or methodological advance does the proposed shape-matched control offer over established constrained-randomization evaluations?
-
How well do the findings generalize to larger molecular datasets, such as ogbg-molpcba, and alternative hierarchy constructions? Given the small gains and their sensitivity to reference estimation, what evidence supports a reliable advantage over the shape-matched reference?
-
How does replacing SEGO’s coding trees with shape-matched random trees affect the complete SEGO pipeline? Would the conclusions drawn from compact edge-fraction summaries remain valid after neural representation learning?
3 comments
We sincerely thank the reviewer for their valuable feedback. We hope that our responses have addressed your concerns. As the rebuttal period is coming to an end, we would be happy to further discuss any remaining questions or concerns. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of our clarifications.
Response to reviwer
Thank you for the response. However, my concerns have not been fully resolved. The response only demonstrates performance gains, but the methodological advantage of the randomization control remains unclear. Therefore, I will keep my score.
Response to Reviewer
W1 / Q1: Advance over generic constrained randomization.
Our contribution is to evaluate the gains from vertex membership separately from graph-dependent hierarchy shape. A hierarchy construction determines group sizes, branching structure and vertex membership together. Comparing returned and random trees without matching their shapes mixes these factors. We fix the original graph, every group size and all branching relations, randomizing only vertex membership to establish a membership control within the same shape.
On this basis, we separately evaluate absolute detection performance, gain over a construction's own shape reference, and gain differences between constructions. Each construction uses its own shape reference, allowing comparisons between hierarchy methods with different shapes. The added average-linkage comparison provides a concrete example: its Ledoit–Wolf gain is +0.0545 [0.0291, 0.0804], yet its absolute PC is only 0.5078.
The reference expectation of edge fractions can also be computed exactly from group sizes (Eq. 4), providing an analytically tractable comparison baseline.
W2 / Q2: Larger molecular data, alternative hierarchies and reliable advantage.
We extended the evaluation to the large-scale ogbg-molpcba dataset. Using the exact reference mean, the mean Ledoit–Wolf membership gain is +0.0163.
We also added shortest-path average-linkage as a hierarchy comparator. SEGO, the connectivity proxy and average-linkage use the same fitting samples and matched pairs, each with its own shape-matched reference.
The full results and intervals for all three detectors on ogbg-molpcba are reported below.
W3 / Q3: What changes after neural learning.
We conducted a paired scaffold-retraining extension using the full HCL and HRN architecture across five datasets and three training seeds. Returned and randomized memberships share the same hierarchy shape, initialization and training budget.
With the tree encoder updated during training and the actual tree–graph loss used for scoring, mean PC is 0.686957 for returned memberships and 0.674919 for randomized memberships, an average increase of approximately 1.20 percentage points. The gain changes sign across training seeds.
The two variants respectively retain the optimization and scoring behavior of the inspected source, or enable tree-encoder parameter updates and use the actual tree–graph loss for scoring.