- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes CHIRP for recovering network structure from experiments performed at several known coupling strengths. Instead of relying on how strongly two nodes appear related in a single experiment, CHIRP asks how their estimated interaction changes as the coupling strength is varied. The authors provide theoretical justification for why multiple intervention levels can help distinguish structural interactions from nuisance effects, and validate the method on networks of electronic oscillators.
Critical strengths
The central idea of this paper is simple and intuitive: controlled interventions can reveal network structure more reliably than observational relationships alone. The experimental evaluation of the paper is careful designed and executed, with major choices fixed before testing on held-out networks, and the reported performance is very strong.
I also appreciate that the paper includes meaningful controls and negative results. In particular, the paper reveals that performance can degrade substantially when the intervention is confounded or when the response to intervention becomes nonlinear. This helps clarify when the method should and should not work.
Critical weaknesses
For me, the main limitation is how general are the main results of the paper. The method is developed and tested for a rather specific oscillator model with a fixed phase-based interaction form. The strong experimental results therefore do not establish that the approach will work equally well for more general dynamical systems.
The experimental validation also seems narrow to me: all physical experiments use the same family of 28-node oscillator networks and a single type of scalar intervention. The paper does not address directed networks, hidden variables, or substantially different intervention mechanisms.
Finally, I am somewhat uncertain whether the paper fits for ICLR. The main contribution seems closer to experimental design, system identification, and network reconstruction than to a broadly applicable machine-learning methodology. The underlying learning method is relatively simple, while the strongest novelty comes from how the experiments are designed and interpreted.
Questions for authors
See the identified weaknesses. I would like to see how the authors respond to them.
Thank you for the positive reading of the frozen protocol and of the negative results. All of the new analyses described below were done after the reviews came in. The paper marks them as post-review, and none of them touches the frozen protocol or its endpoint (R6–R20, mean AP 0.9980). The frozen estimator still has the same hash. Several answers below refer to the new synthetic checks (Section 7, Table 3; Appendix D). Each one starts from the matched model and changes a single assumption: dense static coupling, edges that do not respond to the dose, directed edges, a phase-lagged coupling function, or hidden nodes. There are 15 networks per scenario, 90 in total. Code, configs, unit tests and hashes are in the reproduction record (Run 5).
Comment: "The method is developed and tested for a rather specific oscillator model with a fixed phase-based interaction form."
-> The sine feature was fixed in advance for this benchmark. The method itself does not need it, since any coupling basis can be interacted with the dose. Section 3 now describes a (sin,cos) variant that scores each pair by the norm of its two slopes. In a new phase-lag scenario with coupling sin(Δϕ−1.2), the sine-only fit falls to AP 0.450 and the (sin,cos) basis brings it back to 1.000 (Table 3). A (sin,cos) single-dose fit also reaches 1.000 in that scenario, so a phase lag is handled by the basis. The slope helps with nuisance terms, which is a separate issue. We do not claim that this covers general dynamical systems. The affine dose response and a known interaction basis are still assumptions.
Comment: "All physical experiments use the same family of 28-node oscillator networks and a single type of scalar intervention. The paper does not address directed networks, hidden variables, or substantially different intervention mechanisms."
-> We agree that the physical evidence covers one device family and one scalar intervention. Section 8 says so, and we do not claim more. We were not able to add a second physical system with a calibrated coupling dose in the time available for the revision. We did add simulations for the two cases the reviewer mentions, both in Table 3 with intervals in Table 10. For directed networks, each edge is oriented one way (42 of 756 ordered pairs), and the unsymmetrized slopes recover the directed graph at AP 1.000. For hidden variables, 4 of the 28 nodes are unobserved, and CHIRP keeps AP 0.960 on the observed subgraph, where a single dose gets 0.393. Section 8 says that these are simulations and not physical results, and that disturbances which move with the dose have not been tested.
Comment: "I am somewhat uncertain whether the paper fits for ICLR. The main contribution seems closer to experimental design, system identification, and network reconstruction …"
-> We think the paper is within ICLR's scope of structure learning and causal discovery. Learning graphs from interventions is an active topic in ML (Hauser & Bühlmann 2012; Mooij et al. 2020; Brouillard et al. 2020), and so is relational inference for interacting dynamical systems (Kipf et al. 2018; Löwe et al. 2022). The revised introduction and related work place CHIRP between these two lines. Interventional structure learning usually intervenes on node variables in static data, and relational inference usually learns from observation only. CHIRP applies a continuous, calibrated intervention to the interactions themselves and reads the graph from how they respond. The learning machinery is deliberately simple, and the paper says so. Our contribution is the choice of what to estimate, along with its identifiability, its limits (Corollary 2, Table 3) and a frozen held-out evaluation.