- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces NetSPE, a method for discovering dynamical equations from network observation data. It leverages prior knowledge of the network structure to organize mathematical expressions, employs residual analysis to identify branches requiring modification, and determines specific editing locations through subtree pruning and coefficient refitting. An LLM generates local replacements at these identified locations, while an SPPO (Structured Policy Optimization) mechanism updates the generation strategy based on the performance of different replacements under the same editing state. Finally, the optimal formula is selected via cross-validation. The approach is evaluated across 25 tasks spanning three benchmarks: synthetic network dynamics, bacterial growth, and epidemic spread.
Critical strengths
- The research problem is clearly defined, and the methodology is well-aligned with the research objectives. The paper addresses how to translate aggregate prediction errors into specific structural modifications, enabling the model to accumulate experience from these adjustments. The process—comprising "branch localization, subtree localization, local editing, and strategy update"—follows a clear logical progression, while type constraints and fixed aggregation structures establish well-defined boundaries for editing operations. As a framework for formula discovery in the context of network dynamics, this design holds significant research value.
- The experiments demonstrate meaningful system-level gains and include analysis at the structural level. Using two different LLM backbones, the authors report results from primary experiments, modular ablation studies, structural recovery tasks, and an analysis of search trajectories. Compared to the baseline method ND², the Llama-based version achieved a reduction in average Normalized Mean Squared Error (NMSE)—from 0.0437, 1.0672, and 0.3819 to 0.0084, 0.8256, and 0.2395, respectively—across synthetic, bacterial, and epidemic-related tasks. Furthermore, the authors present examples of both successful and unsuccessful structural recovery, allowing readers to examine the nuances between model fitting and the underlying structural representation.
Critical weaknesses
-
The paper defines structural accuracy by allowing numerical constants in a returned expression to be reassigned before checking algebraic equivalence. This may help separate operator-structure recovery from coefficient-estimation error, but the manuscript does not clearly explain why this criterion is the most appropriate measure for the claimed scientific objective. The authors should clarify what capability this metric is intended to capture, how it differs from exact recovery of the fitted equation, and how the results should be interpreted when structural accuracy and predictive performance disagree.
-
The contribution of the network-aware representation is not sufficiently isolated from the imposed structural constraints. NetSPE has access to richer network information, including neighboring states, edge attributes, and aggregation operations, while the general LLM baselines primarily operate on local scalar features. Moreover, the proposed method imposes a fixed network-aware expression structure. Consequently, the reported performance gains may partly arise from additional information and a more constrained hypothesis space rather than the proposed search and editing mechanism itself. A fair comparison using matched network information and comparable expression spaces is needed to establish the independent contribution of NetSPE.
Questions for authors
-
Can the five general-purpose LLM baselines access neighbor states, edge attributes, and aggregation operators? Are there results under a shared graph-aware execution environment?
-
Since , what additional benefit does the joint term provide in branch routing? Have you compared it with marginal-score routing under the same history and exploration settings?
-
How is neutral subtree replacement defined for different operator contexts? How are invalid or non-finite ablations scored?
-
Does SPPO remain effective after controlling for patch validity and duplication? How does it compare with DPO, GRPO, or supervised learning on the same patch data?
-
How many candidates and successful edits come from LLM patches, atomic edits, and deterministic templates? Are ablation refitting and final cross-validation included in the reported budget?
-
Can you report how many returned equations are actually equivalent to the ground truth using their fitted coefficients, rather than becoming equivalent only after reassigning constants? Given NetSPE’s and recoveries versus ND²’s , what structural-recovery advantage is supported?
I am open to reconsidering my rating if the rebuttal convincingly addresses the concerns above.
5 comments
Response to Question 3
The same replacement rule applies across parent operators. For example, setting to zero in yields , whereas doing so in yields zero. Therefore, this score measures the impact on fitting performance of disabling the subtree’s output while allowing the remaining coefficients to re-adjust. It does not measure the subtree’s independent contribution while holding the outputs of other operations unchanged.
For ablations that fail structural validation or coefficient fitting, including those rejected because of non-finite predictions, we mark their contribution as unavailable and exclude them from the current target selection. We do not assign them a zero or arbitrarily large contribution. This distinguishes an unsuccessful ablation evaluation from evidence about subtree importance, where a numerical failure alone does not justify prioritizing a subtree for editing. Target selection therefore uses the available contribution scores from successfully evaluated ablations.
We will clarify the zero-valued replacement rules, the interpretation of the contribution score, and the handling of failed ablations in the revised manuscript.
Response to Question 4
To distinguish the contribution of policy learning from that of candidate sampling and selection, we added three ablation variants under matched total sampling and evaluation budgets: Frozen policy, Frozen+best-of-G, and Frozen+random. Frozen policy generates one candidate at a time without updating the generation policy. Frozen+best-of-G generates G candidates per group and selects the best, whereas Frozen+random randomly selects one candidate from each such group.
In particular, Frozen+best-of-G matches SPPO in group size, candidate evaluation, and selection rules, while keeping the generation policy fixed. As shown below, SPPO achieves better overall search performance, supporting an additional benefit from policy learning beyond multi-candidate sampling and selection. (NMSE / ACC_10)
| Benchmark | Frozen+Random | Frozen policy | Frozen+best-of-G | SPPO |
|---|---|---|---|---|
| Synthetic | 0.0631 / 60.98 | 0.0381 / 68.47 | 0.0607 / 67.42 | 0.0084 / 83.68 |
| Bacteria | 0.8509 / 27.80 | 0.8482 / 28.09 | 0.8468 / 28.39 | 0.8256 / 29.24 |
| Epidemic | 0.2679 / 21.95 | 0.2683 / 25.76 | 0.2683 / 24.01 | 0.2395 / 25.56 |
We additionally compared SPPO with DPO and GRPO as alternative policy-update methods within NetSPE. (NMSE / ACC_10)
| Dataset | NetSPE (SPPO) | DPO | GRPO |
|---|---|---|---|
| Synthetic | 0.008411 / 0.8369 | 0.007128 / 0.7434 | 0.024217 / 0.7519 |
| Bacteria | 0.825636 / 0.2924 | 0.898579 / 0.2923 | 0.878835 / 0.2897 |
| Epidemic | 0.239465 / 0.2556 | 0.276176 / 0.2385 | 0.276634 / 0.2349 |
SPPO achieves higher ACC_10 on all three benchmarks and lower NMSE on Bacteria and Epidemic than both alternatives. These results support the effectiveness of SPPO for patch-policy learning in our search framework.
Regarding SFT, our motivation is to learn from the relative utility of candidate patches generated for the same search state. Standard SFT on successful patches instead learns to imitate those patches without explicitly contrasting their relative utility. This motivates our preference-based formulation.
Response to Question 5
We additionally counted the valid, deduplicated candidates and successful edits produced by each source across 25 tasks spanning the three benchmarks.
| Model | LLM patches | Atomic edits | Deterministic templates | LLM share of successful edits |
|---|---|---|---|---|
| Llama | 9,835 / 1,434 | 2,039 / 654 | 12,124 / 381 | 58.1% |
| Qwen | 4,551 / 1,134 | 1,916 / 593 | 11,617 / 369 | 54.1% |
Each source column reports valid, deduplicated candidates / successful edits. The last column reports the proportion of all successful edits contributed by LLM patches.
All three sources produced improvements. LLM patches contributed more than half of the successful edits with both models. Atomic edits generated fewer candidates but achieved a higher rate of local improvement, while deterministic templates provided additional opportunities for improvement.
Regarding computational budgets, we distinguish proposal counts from runtime. The proposal budget counts attempts from LLM patches and atomic edits, including failed and duplicate attempts. Deterministic templates, subtree ablations and their associated coefficient refitting, final cross-validation, and final coefficient refitting are excluded from this count.
Thank you for the detailed response and the additional experiments. The rebuttal meaningfully addresses most of my concerns, particularly through the graph-aware baseline comparisons, the frozen-policy controls, and the clarification of the implementation and evaluation procedures.
However, my main concern about the evidence for equation discovery remains. For a paper aimed at discovering governing equations, I place substantial weight on recovering the underlying symbolic relationships. The proposed SA criterion establishes that a returned expression can represent the ground-truth equation after reassigning its constants, potentially including setting coefficients of redundant terms to zero. For example, the same returned expression can be counted as structurally correct for three different ground-truth laws—, , and —simply by reassigning its coefficients. This remains possible even when the actual fitted expression retains spurious terms. Consequently, SA can award full structural credit to an expression containing several alternative relationships without establishing that the method has identified which relationship actually governs the data. The additional check using fitted coefficients is helpful, yet numerical agreement at a specified tolerance is still insufficient to establish symbolic recovery. Flexible approximations, including polynomial expansions, can achieve very high numerical accuracy without identifying the underlying functional form.
I will maintain my original rating of 2 (weak rejection). If the other reviewers favor acceptance and the AC considers the existing evaluation sufficient for the paper’s intended scope and contributions, I would not oppose acceptance.
We sincerely appreciate your time and thoughtful review of our work. The following is our response to your remaining concerns.
Response to Weakness 1 and Question 6
Our SA metric evaluates whether the returned expression can exactly represent the ground-truth equation through finite reassignment of its numerical constants, thereby assessing structural search capability separately from coefficient-estimation error. It does not imply that the fitted coefficients are already correct. Since coefficients of redundant terms may be set to zero, it also does not guarantee a minimal expression.
We conducted an additional verification of the original 30 runs for each method:
The results show that all NetSPE instances counted as successes under the original SA metric also pass the numerical check above with their actual fitted coefficients. Therefore, these successes do not rely solely on constant reassignment during evaluation.
Response to Weakness 2 and Question 1
In the original experiments, the five LLM-based baselines primarily received local scalar features and did not share NetSPE’s full graph-based execution environment. Thus, those comparisons alone do not isolate the effects of network information, structural constraints, and the proposed search mechanism.
To address this concern, we added comparisons with three strong baselines FunctionEvolve, IGSR, and RESTART augmented with neighbor information and structural priors (denoted “w/ NP”). While preserving each baseline’s original search procedure, we provided information about all neighbor nodes and incorporated NetSPE’s GLV and epidemic readout priors into FunctionEvolve’s generation and evolution contexts, IGSR’s expression-generation prompts, and RESTART’s candidate proposal, refinement, and analysis prompts.
We report task-averaged NMSE and ACC_10 for each benchmark as NMSE / ACC_10. Llama-3.1-8B
Qwen3-4B
In addition to these three augmented baselines, we included SymBOL (TPAMI) as an existing graph-based and LLM-based baseline, using Llama-3.1-8B as its backbone. This provides a complementary comparison with a method explicitly designed to incorporate graph structure. This additional comparison demonstrates that, under the current evaluation setting, NetSPE retains a performance advantage over a method that also incorporates LLMs and network interaction modeling.
Response to Question 2
The joint contribution decomposition accounts for each channel’s standalone explanatory power and its incremental explanatory power given the other channel, while ensuring that Cs+Cz=Vsz. Directly normalizing the marginal scores depends only on Vs,Vz and therefore cannot distinguish cases with identical marginal explanatory power but different joint explanatory power.
For example, when Vs=0.2, Vz=0.1, and Vsz=0.8, directly normalizing the marginal scores gives (0.667,0.333), whereas normalizing the residual contributions gives (0.5625,0.4375). The latter reflects the additional explanatory information provided jointly by the two channels, allocating this information symmetrically and giving the branches more balanced editing opportunities. Direct marginal-score normalization depends only on Vs and Vz and therefore cannot reflect differences in joint explanatory value when these marginal scores remain unchanged.
As described in Appendix D.2 and Algorithm 1, normalized residual contributions are combined with historical editing utility and uniform exploration to determine branch-selection probabilities. The probabilities above illustrate the residual-contribution component before this combination. Therefore, although the joint term does not affect the initial selection based on the largest contribution, it can affect subsequent branch-selection probabilities. We will clarify this distinction in the manuscript, the joint term incorporates joint explanatory information into the allocation of editing opportunities, rather than changing the branch ranking.