- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces an evaluation framework for edge-level deep graph transformation, comparing ten architectures across thirteen tasks and sixteen experimental settings. It investigates how architectural performance varies with feature availability, structural dependencies, and prediction constraints. Based on these observations, the authors propose an additional task-characterization dimension called the information regime.
Critical strengths
Motivation: There is some effort to establish a standardized evaluation framework for graph transformation. This is somewhat meaningful because existing methods are often evaluated under different task definitions and experimental settings.
Critical weaknesses
I sincerely feel that this work has not been carefully polished, as there are numerous formatting issues and a lack of rigorous presentation of the methodology and figures. For example, Table 1 even uses ellipses to omit experimental results, which I find rather inappropriate for a conference submission. The GitHub repository link is also redacted, making it difficult to directly inspect the claimed open-source framework, although supplementary code is mentioned.
More importantly, I am not convinced that this paper offers a substantial research contribution beyond collecting existing architectures and tasks into a unified evaluation framework. Most of the findings are rather expected, and the analysis does not provide sufficiently convincing insights into why particular architectures perform better under different settings. Given that the main contribution is supposed to be a rigorous and reproducible benchmarking framework, I would expect much greater attention to methodological clarity, experimental completeness, and overall presentation. In its current form, I do not think this work meets the standards expected of an ICLR submission.
Questions for authors
See weakness
4 comments
Thank you for the response, however I don't think your rebuttal is sufficient enough to convince me this manuscript is publishable. For the formatting, i.e., I've never seen a paper use dotted line for experiment results. Authors reported a lot of experiment tables but yet most of them are lack of proper interpretation and analysis. I will keep my scores.
Response to Reviewer t3xp
Thank you for your follow-up and for considering our rebuttal. We appreciate your feedback, but respectfully disagree with your assessment. Our rebuttal addressed the specific concerns raised, and we explicitly requested clarification on the remaining issues. Your follow-up reiterates these concerns without identifying specific shortcomings in our explanations or revisions.
Presentation and reproducibility. We explained that the ellipses in Table 1 are a deliberate space-saving convention, abbreviating a comparison with published baselines. The two displayed leading rows show the best results for AUC and Recall@k, while the final row provides the lowest Recall@k as a reference point. The percentages permit direct comparison with the published GOLD baseline. All intermediate results appear in Appendix Table 11, which is explicitly referenced in the caption and accessible directly through a clickable link in the PDF. No experimental results are omitted from the paper. We also clarified that the complete anonymised source code, including task seeds and per-run results, was submitted as supplementary material. We asked you to identify the other formatting problems mentioned in your original review, but no additional examples were provided.
Methodology and analysis. We substantially expanded the methodological details in Section 3.3 and Appendix A.5, including training configurations, threshold selection, and checkpoint criteria. Sections 4.1–4.4 and Section 5 provide task-specific and cross-task interpretations of the experimental results. While your follow-up states that most tables lack proper analysis, it does not identify which results or explanations are insufficient. We explicitly asked for such examples and the evidence needed to strengthen our interpretations, but received no clarification.
Scientific contribution. Our rebuttal also explained why the work goes beyond collecting existing architectures and tasks: we implemented the tasks, established previously unavailable neural baselines under a controlled DGT setting, and developed a framework for systematically investigating architecture–task interactions. Your follow-up does not address this explanation or identify prior work establishing these contributions.
We respectfully ask you to reconsider your assessment in light of the revisions and explanations provided. If concerns remain decisive, we would appreciate specific examples of the deficiencies that our rebuttal has not addressed.
You clearly did not read through my comments. Please stop using GPT to generate more meaningless comments but rather try to find actual novelty and properly polish your manuscript . My view stands with other reviewers.
Response to Reviewer t3xp
Thank you for the review, and for recognising the motivation for a standardised evaluation framework for graph transformation. We address the specific points first and then the contribution.
Contributions
The review describes the paper as collecting existing architectures and tasks into a unified framework. We see each of the three parts differently:
Findings
The review finds most of the findings expected. An expectation needs earlier evidence, and without baselines there was none: we did not know before running the experiments which architectures would suit which task, or how they would respond to changes in the supplied information. Two outcomes surprised us the most, although many more were identified:
If prior work predicts these outcomes for graph-to-graph tasks, we would be grateful for references so we can cite them and compare against them.
On why architectures differ: where we could test an explanation, we did. The transformations are known, so we know which features carry much of the answer. In three experiments, we withhold those features and report how each architecture's score changes. We would welcome pointers to the explanations you find unconvincing, and to the evidence that would convince you, as this is the part we most want to improve.
Presentation and methodology
The review mentions numerous formatting issues. We would be grateful if you could point them out so we can fix them. The paper was proofread and line-edited by people outside the immediate specialism, and we eliminated any issues identified before submission.
Thank you for pointing out the incomplete methodology. The revision adds:
Please let us know if you find anything else missing.
Table 1
Table 1 is an abridged comparison of selected models with the published GOLD ablations, and the ellipsis marks the rows that continue in the full table. The full table, with all ten architectures and every metric, is Appendix Table 11 (Table 9 in the submitted version), and Table 1's caption points to it. The rows with the most important results were preserved, and the final table row was preserved to accurately show the scale of the results. This is a common convention in computer science papers.
Code
The complete anonymised repository was submitted as supplementary material, with per-run results and task seeds. The public GitHub URL is redacted because the repository identifies the authors, and the ICLR author guidelines explicitly allow code to be submitted as supplementary material.