- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a framework for discovering dynamical systems from natural-language task specifications and partially observed time series. The main contribution of the paper is to jointly discover state variables including latent states and governing equations, guided by task-specific mechanistic requirements. The proposed framework combines LLM-based candidate generation and critique with deterministic verification and numerical parameter fitting. Experiments across different systems, including glucose-insulin dynamics, a chemical reactor, and a synthetic system, show improved predictive accuracy and mechanism compliance over several baselines.
Overall, the paper has a promising methodological idea, but I feel that the current evaluation does not yet establish robustness to limited observations or scalability to complex dynamical systems.
Critical strengths
-
The paper has an interesting new problem formulation. Jointly discovering the state representation and governing equations is a meaningful extension beyond equation discovery with a predefined state space.
-
The proposed solution combines LLM-based proposals with deterministic verification and numerical fitting. This is more reliable and interpretable than relying on LLM generation alone.
-
Its evaluation has an emphasis on mechanism compliance and prediction under unseen interventions. This is valuable, as trajectory fitting alone does not establish mechanistic validity.
Critical weaknesses
I am quite concerned about the evaluation and whether the propose framework works for more general and complex ddynamical systems.
-
The experiments cover only a small number of structured systems. It is unclear to me whether the method can handle high-dimensional, strongly coupled, or multiscale dynamics. No systematic complexity-scaling or computational-cost analysis is provided in the paper.
-
Although the paper focuses on partial observations, it does not systematically investigate sensitivity to observation sparsity, measurement noise, or the choice of observed variables.
-
The framework assumes that the supplied specification is correct and sufficiently informative. It is unclear how robust the method is to incomplete, ambiguous, or overly restrictive specifications.
Questions for authors
The questions are related to the identified weaknesses:
-
How does performance and computational cost scale with the number of state variables, candidate interactions, and latent states? Could the authors provide a controlled complexity-scaling experiment?
-
How sensitive is the method to the amount and quality of observed data? In particular, how does performance change under increased measurement noise, sparser sampling, or fewer observed state variables?
-
How does the framework behave when task specifications are incomplete or partially incorrect? Can it detect inconsistencies between the specification and the observed data?
5 comments
Response 2/3: Sensitivity to Observation Sparsity, Measurement Noise, and Choice of Observed variables
Sensitivity to Observation Sparsity, Measurement Noise, and Choice of Observed variables
W2: Although the paper focuses on partial observations, it does not systematically investigate sensitivity to observation sparsity, measurement noise, or the choice of observed variables. Q2: How sensitive is the method to the amount and quality of observed data? In particular, how does performance change under increased measurement noise, sparser sampling, or fewer observed state variables?
We thank the reviewer for this important question.
Robustness to Measurement Noise
We vary the level of measurement noises on T2-hard of the glucose-insulin benchmark.
| Noise | Retained | Test NMSE ↓ | Response RSE ↓ | Directional accuracy ↑ | Shape error ↓ |
|---|---|---|---|---|---|
| 0% | 3/3 | 1.21 [1.21, 1.17] | 0.91 [1.00, 0.87] | 0.66 [0.59, 0.71] | 0.66 [0.70, 0.64] |
| 1% | 3/3 | 1.24 [1.39, 0.64] | 0.86 [0.93, 0.84] | 0.59 [0.59, 0.78] | 0.68 [0.70, 0.64] |
| 5% | 2/3 | 0.90 [0.90, 0.90] | 0.87 [0.89, 0.84] | 0.69 [0.59, 0.79] | 0.67 [0.69, 0.65] |
| 10% | 3/3 | 0.82 [1.35, 0.35] | 0.82 [0.85, 0.37] | 0.80 [0.57, 0.99] | 0.67 [0.70, 0.33] |
Robustness to Observation Sparsity
We vary the sampling intervals on T2-hard of the glucose-insulin benchmark.
| Sampling interval | Retained | Test NMSE ↓ | Response RSE ↓ | Directional accuracy ↑ | Shape error ↓ |
|---|---|---|---|---|---|
| 1 min | 3/3 | 1.21 [1.21, 1.17] | 0.91 [1.00, 0.87] | 0.66 [0.59, 0.71] | 0.66 [0.70, 0.64] |
| 2 min | 3/3 | 1.23 [1.41, 0.98] | 0.96 [1.00, 0.95] | 0.59 [0.58, 0.80] | 0.69 [0.71, 0.69] |
| 4 min | 3/3 | 0.94 [0.95, 0.93] | 0.85 [1.00, 0.84] | 0.74 [0.59, 0.74] | 0.71 [0.71, 0.66] |
Robustness to Observed Variables
Tasks T1 and T2, each of which has easy and hard tiers, were designed to address the robustness of observed variables.
| Benchmark | Prediction targets | Observed States | Ratio of observed to predicted |
|---|---|---|---|
| T1 easy | 4:1 | ||
| T1 hard | 1:1 | ||
| T2 easy | 4:3 | ||
| T2 hard | 1:2 |
| Benchmark | Retained | Test NMSE ↓ | Response RSE ↓ | Directional accuracy ↑ | Shape error ↓ |
|---|---|---|---|---|---|
| T1 easy | 3/3 | 0.20 [0.21, 0.19] | 0.16 [0.17, 0.15] | 0.99 [0.99, 1.00] | 0.13 [0.13, 0.12] |
| T1 hard | 3/3 | 0.12 [0.19, 0.05] | 0.15 [0.17, 0.13] | 0.99 [0.98, 0.99] | 0.11 [0.13, 0.08] |
| T2 easy | 2/3 | 0.10 [0.10, 0.10] | 1.00 [1.00, 1.00] | 0.41 [0.41, 0.41] | 0.91 [0.91, 0.91] |
| T2 hard | 3/3 | 1.21 [1.21, 1.17] | 0.91 [1.00, 0.87] | 0.66 [0.59, 0.71] | 0.66 [0.70, 0.64] |
Conclusion
The proposed method is robust to measurement noise and observation sparsity. It is robust to choices of observed variables except T2-hard, which is particularly challenging due to the 1:2 observed to prediction ratio.
Response 3/3: Incomplete or Incorrect Task Specifications
Incomplete or Incorrect Task Specifications
W3: The framework assumes that the supplied specification is correct and sufficiently informative. It is unclear how robust the method is to incomplete, ambiguous, or overly restrictive specifications. Q3: How does the framework behave when task specifications are incomplete or partially incorrect? Can it detect inconsistencies between the specification and the observed data?
Rationale behind assuming correct specifications
As we discussed in Sec. 6 of the paper, we assumed that the task specification is correct. We believe that this is a reasonable assumption, because the specification merely requires the existence of certain pathways to fulfill the task, but does not specify the forms of the dynamics. This happens in scenarios where the user has a goal in mind (e.g., insulin therapy design). They can ask for a model including the components to achieve their goal (e.g., insulin pathway), but do not know how the components should look like in an unknown system.
Robustness to incomplete or incorrect specifications
But we do agree that it is meaningful to test how robust the method is to incomplete, over-restrictive, and incorrect specifications. On CSTR-hard, we ran the search under five task specifications while keeping everything else the same:
- Complete: the original specification (Full).
- Over-restrictive: adds a requirement of exactly one latent state.
- Incomplete: removes the jacket heat-exchange requirement.
- Incorrect: requires, falsely, that the reactor temperature not depend on the jacket feed temperature.
We evaluate the performance under unseen intervention with more detailed metrics of response relative squared errors (RSE, evaluating response magnitudes), directional accuracy (evaluting response directions), and shape error (evaluating temporal behaviors). We run three seeds for each experiment.
| Specification | Retained | Test NMSE ↓ | Response RSE ↓ | Directional accuracy ↑ | Shape error ↓ |
|---|---|---|---|---|---|
| Complete (Full) | 3/3 | 0.37 [0.75, 0.17] | 0.14 [0.16, 0.05] | 0.99 [0.99, 0.99] | 0.04 [0.05, 0.04] |
| Over-restrictive | 2/3 | 0.33 [0.43, 0.24] | 0.16 [0.17, 0.15] | 0.99 [0.99, 0.99] | 0.05 [0.05, 0.05] |
| Incomplete | 3/3 | 0.84 [0.88, 0.43] | 0.44 [0.44, 0.17] | 0.72 [0.72, 0.99] | 0.32 [0.32, 0.05] |
| Incorrect | 3/3 | 4.79 [4.94, 0.34] | 0.93 [1.00, 0.16] | 0.62 [0.01, 0.99] | 0.86 [1.00, 0.05] |
Conclusions.
- Incorrect specification resulted in the worst performance. This is because under our assumption of correct specification, we have deterministic validity check and LLM critics to ensure that the specification is met.
- Results on over-restrictive and incomplete specifications give practical guidance in using the system.
- When unsure, we should provide a relaxed, incomplete specification, rather than a incorrect specification.
- Specification on the number of latent states (e.g., for model complexity considerations) has less impact as specification on mechanisms.
Response 1/3: Scaling of Performance and Computational Cost
Scaling
We thank the reviewer for this insightful question. We conduct experiments in a setting similar to the alien device benchmark, which allows us to control dimensionality, coupling, and scales of dynamics without worrying about the physical meanings.
Setup
Given a one-dimensional input u, the states z are generated by z˙i=−dizi+∑j=iSijzj+fi(z)+bitanh(αu), where di is the decay, representing time scales, Sij models coupling between states, and fi is a randomly generated function. A one-dimensional output is generated by y˙=−λy+g(z).
We increase the number of states from 4 to 8 and to 12 (high-dimensional), double the coupling (strongly coupled), and increase the spread of decay rates (multiscale dynamics).
Results
We evaluate the performance under unseen intervention with more detailed metrics of response relative squared errors (RSE, evaluating response magnitudes), directional accuracy (evaluting response directions), and shape error (evaluating temporal behaviors). The definitions of these metrics are in
the first response to Reviewer Ewor. We run three seeds for each experiment.Note: in 2 out of the 6 runs for the 4-state systems, the search did not produce an accepted initial model, therefore skipping the remaining 14 iterations. Thus, 4/6 means four runs produced a retained model and two failed at initial construction.
Conclusion
We can see that our method has stable performance, time complexity, and token usage across most of the scenarios. The two exceptions are the much larger response RSE under wider decay spread, and the two failed runs in the 4-state systems.