acceptodds
Under review as a conference paper at ICLR 2027

TimeEvo: Failure-Driven Self-Evolution of a Time Series Agent

Abstract

Time series agents answer analytical questions by calling external tools, and which tools they carry is decided by people before the agent runs. However, we identify two failures in this setup. **Human–Agent Tool Misalignment**: a library of 21 expert-curated tools helps on some tasks and hurts on others, dropping anomaly accuracy under every backbone we test. **Silent Harm**: one round of generic self-revision changes 147 answers and breaks 56 of them, while the final score moves by less than a point. Both follow from the same gap: whether a tool helps is decided question by question at runtime, while tools are supplied in advance and judged by a single average. To address this, we propose TimeEvo, which clusters an agent's diagnosed failures into capability gaps, plans a measurement for each, synthesizes evidence-only tools that fill them, and admits the candidate library only through a paired admission gate. Experiments on ten time series QA tasks and three backbones show that TimeEvo, starting from an empty library, improves accuracy on every task and every backbone, and that a library grown on a cheap model still gains when it is installed into stronger ones. Code is available at https://anonymous.4open.science/r/TimeEvo-652E.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Self Evolving Agent, Time Series Analysis, Time Series Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tdrDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two failure phenomena in time series agents—Human-Agent Tool Misalignment (expert-curated tool libraries help on some tasks but hurt on others) and Silent Harm (self-revision fixes some answers but breaks other previously correct ones)—and proposes TimeEvo to help the agent evolve its own tool library from diagnosed failures. TimeEvo clusters errors into failure buckets, plans a measurement contract per bucket, synthesizes evidence-only tools, and admits them only after two-level validation: a per-tool prefilter requiring an attributable repair, and a whole-library paired gate that explicitly weighs helped against harmed questions and enforces a positive statistical lower bound (with a one-shot prune-and-retest fallback). Experiments demonstrate the effectiveness of TimeEvo.

Critical strengths
  1. The two failure phenomena are practical problems, which should be addressed in the real-life applications

  2. The two-level validation is well-targeted and reasonable.

  3. The results are strong and complete within the tested setting.

Critical weaknesses
  1. The paper presents Human-Agent Tool Misalignment and Silent Harm as two co-equal motivations, but they concern two distinct problems—"mismatch in externally supplied tools" and "hidden harm from self-revision"—and the paper never establishes a causal or structural relationship between them. This gives the motivation the appearance of being assembled from two unrelated observations.

  2. The evolved library mixes executable tools with prompt patches, yet the paper never reports the separate contributions of these two artifact types to helped/harmed counts, leaving the attribution of Silent Harm unclear.

  3. The paper only tests cross-model transfer within the same task; it does not validate cross-task or cross-domain generalization. Moreover, the method only patches capabilities for the current failure buckets and never defines a global capability space, so it is impossible to judge whether the library is complete or whether it contains redundancy.

  4. The paper claims to cluster by "missing capability," but it never defines how capabilities are represented, how similarity between erroneous questions is measured, or how stable the clustering is and how often it fails; nor does it provide any threshold sensitivity analysis. As a result, this core mechanism is neither reproducible nor verifiable.

  5. The paper does not compare against memory-based methods or tool-selection methods, so it cannot rule out that simply "recording execution experience" or "optimizing selection over a fixed library" would achieve similar results.

  6. The paper calls itself as self-evolving, yet from the experimental study, the evolving is from empty root library with one evolution round

  7. All ten tasks are closed-set multiple-choice questions with automatically scorable answers. There is a lack of open-ended, multi-hop, and real-world domain tasks, raising doubts about the method's external validity.

Questions for authors

Please check the weak points above.

Reviewer_6AogOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TimeEvo, a framework that improves a frozen LLM agent by evolving its external time series analysis tools from labeled training failures. It is motivated by two empirical observations. Expert-curated tools can reduce accuracy on some tasks, and generic self-revision can introduce substantial regressions that are obscured by aggregate accuracy. TimeEvo clusters failures into capability gaps, constructs measurement contracts, and synthesizes executable tools intended to return numerical evidence rather than final answers. Candidate tools undergo individual screening, followed by whole-library admission based on paired counts of repaired and degraded answers, and rejected libraries may be pruned and retested. During the inference, a residual repair mechanism retains the incoming answer unless an applicable tool supports replacement. Experiments cover ten time series QA tasks and three primary backbones, with additional cross-model transfer experiments, and report substantial improvements over the corresponding no-tool agents.

Critical strengths
  1. The motivation goes beyond assuming that more tools or more reflection must improve an agent. Figure 1 is intuitive.

  2. The architecture design of TimeEvo has useful mechanisms for interpretability and controlled execution. Measurement contracts explicitly specify the required computation, inputs, applicability, and anti-scope. Separating numerical evidence generation from final answer selection also makes the intended role of each tool easier to inspect.

  3. The performance improvements are broad and substantial, while cross-model library transfer and case studies are valuable additional experiments.

Critical weaknesses
  1. The contribution of tool evolution is insufficiently separated from supervised calibration. Eq (5) fits a decision tree using training labels and compiles it into the generated tool. Moreover, removing calibration causes both anomaly ablation runs to deploy nothing. This makes supervised prediction a central component of some of the strongest results, rather than a minor implementation detail. Freezing the LLM does not make the overall pipeline free of supervised learning.

  2. TimeEvo uses full training labels to find errors and fit decision trees, and full validation labels for gate selection. In contrast, Chronos-2, TimeART, Self-Refine, and vote@5 use no labels, and ICL uses only four examples. On Anomaly, Table 4 shows that removing calibration reduces the gain to zero. Since this task has series of only 8 to 64 points, the embedded trees act much like a supervised classifier, which is in tension with the premise that tools output evidence rather than judgements. In addition, TimeClaw and SkillOpt are discussed in the related work but not included in the comparison, leaving the empirical advantage over existing self-evolving approaches insufficiently established.

  3. The main experiments use one evolution round. The multi-round evidence in Figure 5c comprises a single Anomaly run, round 1 adds tools, round 2 adds usage guidance, and round 3 is rejected. This demonstrates refinement and stopping in one setting, but does not establish sustained improvement across rounds. In addition, the safe-positive rule in Equation (7) permits admission without a positive lower bound under Equation (6). The same validation split is also reused after attribution-based pruning, retesting, and scope reconciliation. Thus, the statistical interpretation of the complete adaptive admission procedure is unclear: limiting the number of retests does not by itself justify treating the resulting bound as independent confirmation. These issues do not invalidate the untouched test evaluation, but weaken the interpretation of the gate as providing statistically grounded protection against harmful updates.

  4. Computational efficiency, scalability, and deployment trade-offs are not adequately quantified. TimeEvo incurs offline costs from training-set execution, multiple LLM roles, synthesis retries, prefiltering, and repeated validation, together with online costs from tool execution and residual review. Appendix D reports 483 synthesis attempts across thirty runs, but these counts do not provide an end-to-end accounting of tokens, execution time, monetary cost, or inference latency.

Questions for authors
  1. What exactly do the calibrated tools return? Do the compiled decision trees output measurements, class probabilities, decision scores, or encoded labels, and how does the residual reviewer use these outputs?

  2. How dependent are the main results on the supplementary admission rule and adaptive validation reuse?

  3. What are the actual offline and online costs of TimeEvo?

Reviewer_eFYcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies tool augmented LLM agents for time series QA. Using a paired helped/harmed accounting instead of a single average, it documents two phenomena: a 21 tool expert library (TimeART) helps some tasks and hurts others, including the anomaly task it has a dedicated tool for, and one pass of generic self refinement changes 147 answers while moving the mean by less than a point. TimeEvo clusters training failures into buckets, plans a measurement contract per bucket, synthesizes deterministic evidence only Python tools (optionally with a decision tree calibrated on training rows), screens each tool with a prefilter, and admits the whole candidate library only through a paired gate on validation with a lower bound criterion, one prune and one retest. At inference the frozen agent answers first and admitted tools review only in scope questions. On 10 tasks and 3 backbones all 30 deltas are positive (means +7.2 to +8.8 pp), a library evolved on one backbone transfers to three stronger models with +8 to +11 pp, and ablations show each component matters.

Critical strengths
  1. The diagnostic framing is sharp and useful. Counting helped and harmed questions (Eq. 1) is simple, but it exposes effects that aggregate accuracy hides, and Fig. 1b (147 changed, 56 broken, score unchanged) makes the point convincingly. Both phenomena are reproduced on three backbones rather than one.
  2. The admission design takes harm seriously in a way most self evolution papers do not: whole library gating to catch interference, a lower bound rather than a point estimate, exactly one prune and retest to limit fitting the validation split, a report only test split, scope and anti scope with conservative fallback to the incoming answer. Table 5 and the unchanged out of scope sub task in Fig. 4c show the protection empirically. Leakage controls (series level grouping, no verdict outputs, sandbox allowlist, hashed answers) are careful.
  3. Empirical breadth and consistency: ten tasks from six sources, three backbones, five baselines measured under the same paired protocol, and 30 of 30 cells positive. The transfer result (Obs. 3 and 4) is the most interesting finding: a library grown on one model lifts stronger frozen models by about 10 pp and often by more than it lifted the model that grew it, which supports the claim that the library carries task knowledge rather than model specific habits.
  4. Transparency and reproducibility are well above average: every frozen knob is listed in Appendix C, ablation cells were fixed before the runs, Tables 6 and 7 show attrition at each stage of the pipeline, Fig. 5b reports the gate’s own false positive and false negative, and the case studies show exactly which tools were built and what numbers they returned.
Critical weaknesses
  1. The strongest alternative explanation is not tested: inference time code execution. Obs. 8 and Appendix B state that the base agent is not bad at arithmetic but simply never computes, and that one deterministic measurement settles each repaired question. If so, a frozen agent given a Python sandbox at inference (code interpreter style, no evolution, no training data) might recover most of the gain. This is the natural baseline for an agent that supplies its own tools and it is missing. Relatedly, the TimeART comparison would be fairer if the 21 tools were served through the same scope catalog, residual review and gate; the current “w/ 21 tools root” ablation evolves on top of them, which answers a different question. Without these two controls it is hard to separate the contribution of the evolved tool content from the contribution of the inference protocol itself.
  2. Run to run variance is large and unreported. Every cell is a single run with no seeds or intervals, yet the paper’s own tables reveal substantial variance. For Anomaly on the mini backbone the base accuracy is 61.0 in Table 1 but ranges from 58.75 to 63.50 across the ablation runs in Table 4, a spread of 4.75 pp before any evolution. Table 7 deltas for the same task and backbone differ markedly from Table 1 (Classification +26.25 vs +12.08, T1 +9.49 vs +3.16, Anomaly +19.75 vs +14.69). Several headline deltas are smaller than this spread (Classification terra +0.83, Merrill luna +1.53, T4 mini +1.59, TSExam mini +2.01). The claim that TimeEvo improves every task on every backbone needs at least three seeds per cell with a sign count or interval, plus an explanation of why the base agent alone moves by 5 pp between runs.
  3. Calibration is supervised learning and it carries the Anomaly result. Without calibration both Anomaly runs deploy nothing (Table 4), and a calibrated tool is a decision tree fit on 400 labeled training rows and compiled into the tool source. This is in tension with the evidence only, no dataset specific constant characterization (a learned threshold is a dataset specific constant) and with the training free positioning against ChatTS style models. Please report how many of the 108 deployed tools carry calibration and on which tasks; add a direct supervised baseline on the same training split (a small tree or logistic regression over simple statistics of the raw series, or over the evolved tools’ outputs with the LLM removed), since Anomaly is a two way task over 8 to 64 point series and may be easy for a classifier; and clarify whether the error dossier shown to the planner and synthesizer contains gold answers, and if so how the hashes only rule is enforced.
  4. The gate’s guarantee is weaker than the text suggests. The supplementary rule in Eq. 7 admits libraries whose lower bound is negative, Fig. 5b has a separate safe positive category, and the only test negative admission (T1 at −1.27) presumably entered that way. Please report, over the 30 runs, how many libraries passed strictly versus through Eq. 7, with the test delta distribution of each. The validation split is also consulted up to three times within one round (initial gate, prune retest, regate after rescoping), and every additional round reads it again, so the argument that a single prune prevents fitting the validation split is only partial. Two smaller points: sqrt(h+m)/n as the standard error treats changed answers as independent counts and is crude when h+m is small, and comparing only questions that both arms complete can favor the candidate if its tools fail preferentially on hard questions; the 2 percent execution error cap limits but does not remove this.
  5. Cost is never quantified. Evolution runs four LM roles, up to three synthesis attempts per contract (483 in total), a prefilter, and two or three full paired validation runs, and inference inspects up to three tools per question; yet vote@5 is dismissed for spending five times the budget. Please report tokens or dollars and wall clock for evolution per task and backbone, inference cost per question with and without the library, and ideally a budget matched comparison (vote@k or few shot ICL with k chosen to match TimeEvo’s total spend).
  6. The self evolving and transfer claims are under tested. The main protocol is one round, so the chain, the RETIRE and RESCOPE actions, and the tool fired failure partition that motivate multi round evolution are exercised only in one anecdote (Fig. 5c). Does accuracy keep rising or plateau over rounds on the ten tasks, and how are in scope errors introduced by round 1 tools repaired in round 2? Transfer is only from luna; the conclusion that growing tools need not happen on an expensive model would be supported by installing the mini evolved libraries into the stronger models, which is not done. Finally, bare Claude opus 5 and sonnet 5 score 55.6 and 55.9, no better than the mini backbone at 54.6, which suggests the harness (series as text, no computation) is the binding constraint rather than model capability and again points back to question 1. Please also state how the backbone names should be read and whether the baselines run with the same reasoning effort settings as the four TimeEvo roles.
Questions for authors

Please refer to the weaknesses section.

1 comment

Thank you for the authors' response. I have reviewed the rebuttal carefully. The authors have addressed the points I raised, and the clarifications improve my understanding of the submission. I'll consider raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.