- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TimeCoder, an LLM-based framework for intervention-aware time series modeling. The main idea is to use an LLM to automatically generate and refine time series model structures rather than relying on fixed architectures. TimeCoder first generates executable models based on domain knowledge and statistical information, then trains their parameters using observed data. Then, it improves the models by analyzing prediction errors and using feedback from previous iterations. The authors evaluate TimeCoder on four datasets from healthcare, hydrology, and epidemiology. The results show that TimeCoder outperforms existing deep learning and LLM-based methods in forecasting accuracy.
Critical strengths
- S1. TimeCoder achieves strong forecasting performance across four datasets. It consistently outperforms existing deep learning and LLM-based baselines.
- S2. The framework provides a flexible way to adapt model structures to different time series characteristics.
Critical weaknesses
- W1. The authors argue that fixed model structures cannot adequately capture intervention-induced changes, but does not provide direct empirical evidence to support this claim.
- W2. It remains unclear whether structural refinement is necessary for capturing intervention-induced changes. It should be discussed or compared against similar or even simpler approaches, such as intervention-aware training objectives, regularization, or data augmentation, can achieve same improvements.
- W3. Although TimeCoder outperforms existing baselines, the experiments do not clearly isolate the contribution of structural refinement itself.
- W4. Since intervention-aware modeling is the main motivation of the paper, the evaluation should more directly examine how well TimeCoder captures intervention-induced changes. Most results focus on overall forecasting accuracy.
- W5. The paper does not sufficiently clarify what methodological advances TimeCoder introduces beyond adapting ideas in LLM-SR and HDTwin.
- W6. The paper does not sufficiently analyze the computational overhead or performance–cost trade-off compared with existing methods.
Questions for authors
- Q1. Could the authors provide empirical evidence that fixed model structures cannot sufficiently capture intervention-induced changes?
- Q2. Why is structural refinement necessary for intervention-aware time series modeling? Could similar improvements be achieved using fixed architectures with intervention-aware training objectives, regularization, or data augmentation?
- Q3. Could the authors provide an ablation that isolates the contribution of iterative structural refinement?
- Q4. Could the authors provide a more detailed evaluation of intervention-specific performance?
- Q5. What are the main methodological differences between TimeCoder and existing LLM-based model synthesis approaches (e.g., LLM-Sr and HDTwin)?
- Q6. What is the computational cost of TimeCoder in terms of LLM calls, training time, and overall inference or development cost? How does this compare with existing methods?
3 comments
Thank you for the responses. However, I am still unconvinced about the necessity and contribution of structural refinement, as the current ablations do not directly isolate its benefits. I will maintain my rating.
Response to Reviewer mK9W
Thank you for the valuable comments! The motivation for structural refinement is stated in the Introduction and formalized in Section 3.2 and Equation 1. Sparse and heterogeneous interventions make temporal dynamics non-stationary, while limited data make it difficult for a single fixed architecture to represent diverse intervention-response mechanisms. Equation 1 therefore formulates learning as joint hypothesis search over and parameter calibration over , subject to empirical and knowledge alignment. Because is discrete and combinatorial, direct optimization is intractable. TimeCoder approximates this search through iterative structural refinement, where each step proposes a new hypothesis or structural edit and then recalibrates its parameters.
This contribution is directly evaluated in Table 7. Under the same data, calibration procedure, and experimental protocol, restricting TimeCoder to white-box structures increases RMSE from 0.328 to 0.372, 0.092 to 0.110, 0.445 to 0.515, and 0.009 to 0.014 across the four datasets. Figure 2 and Appendix E.8.1 further show that, after the initial SEIR structure is calibrated, introducing latent memory and neural residual mechanisms produces additional reductions in the best historical RMSE. These results demonstrate improvements from explicit structural refinement that parameter recalibration alone cannot achieve.
Author Response to Reviewer mK9W
Thank you for your thoughtful and constructive comments. We appreciate this valuable suggestion and provide additional discussion below.
Q3.1: What shows that fixed structures are insufficient?
A3.1: Table 7 provides empirical evidence through the white box only configuration, which uses the same data and calibration procedure but restricts model construction to white box instructions. Its RMSE is 0.372, 0.110, 0.515, and 0.014 on SF²Bench, Lung Cancer, VitalDB, and COVID-19, compared with 0.328, 0.092, 0.445, and 0.009 for the complete TimeCoder. This corresponds to relative RMSE reductions of 11.8%, 16.4%, 13.6%, and 35.7%. The initial SEIR structure captures the coarse epidemic dynamics, while subsequent refinements introduce latent memory, neural residual corrections, and more expressive intervention response mechanisms, leading to further reductions in the best historical RMSE.
Q3.2: Could simpler objectives, regularization, or augmentation achieve similar gains?
A3.2: As formulated in Section 3.2 and Equation 1, TimeCoder jointly searches over the model structure h and calibrates its parameters . A modified objective or regularizer optimizes within a predefined structure, while data augmentation expands the observed samples without changing the underlying functional form. They therefore cannot directly introduce missing mechanisms such as delayed intervention effects, latent memory, variable coupling, or nonlinear residual dynamics. Empirically, the white box only configuration in Table 7 remains consistently worse than the complete TimeCoder, despite using the same data and calibration procedure. Thus, these simpler techniques are complementary to TimeCoder, but they do not replace adaptive structural refinement
Q3.3: How will iterative structural refinement be isolated?
A3.3: We already isolate the two key mechanisms of iterative structural refinement through controlled ablations with the same backbone, data, calibration procedure, and generation budget. Table 5 removes historical refinement memory while retaining the same number of LLM generations. This increases RMSE by 6.6% to 35.7% across the four datasets, demonstrating the contribution of conditioning structural updates on previously successful hypotheses. Table 7 removes statistical residual descriptions under the same experimental setting, increasing RMSE by 5.7% to 18.2%, which demonstrates the contribution of error guided structural revision. Because the search budget is unchanged in both ablations, these improvements arise from iterative feedback and history conditioned refinement rather than additional LLM calls or candidate evaluations.
Q3.4: Is intervention-specific performance evaluated?
A3.4: Yes. Section 5.3 and Table 2 report intervention-specific results on VitalDB using drug-only test subsets. We evaluate both propofol, a common intervention, and remifentanil, a rare intervention, after removing all training samples involving the evaluated drug. On the propofol-only subset, TimeCoder achieves an RMSE/MAE of 0.972/0.368, compared with 1.034/0.395 for Time-LLM and 1.107/0.512 for iTransformer. On the remifentanil-only subset, TimeCoder achieves 1.061/0.462, compared with 1.126/0.498 and 1.184/0.526, respectively. Appendix Table 3 reports the intervention frequencies used to define the common and rare settings. Appendix Figure 4 further provides a qualitative post-intervention analysis on SF²Bench.
Q3.5: How does TimeCoder differ from LLM-SR and HDTwin?
A3.5: LLM-SR discovers symbolic equations using data fitting feedback, while HDTwin evolves general hybrid digital twins using overall and component level validation losses. TimeCoder is specifically designed for intervention-aware time series. It identifies high loss intervention-response prototypes and uses temporal lag, trend, periodicity, abrupt change, and bias diagnostics to refine stateful models with intervention mechanisms, latent memory, and neural residuals. This time-series-specific refinement is supported by the held out intervention results in Table 2 and the ablations in Tables 5 and 7.
Q3.6: What is the computational cost?
A3.6: Computational cost is analyzed in Section 5.4 and Appendix E.7. Figure 5 reports the mean and variance of token consumption at every refinement generation across all four datasets. Each run uses at most T=10 refinement generations on a single NVIDIA H200 NVL GPU. Token usage is concentrated in the early structural exploration stage and stabilizes as the model converges. Table 8 further shows that successful code repair requires only 1.4 attempts on COVID-19 and 1.1 attempts on Lung Cancer on average. This cost is incurred only during offline model discovery.