acceptodds
Under review as a conference paper at ICLR 2027

Perceive, Solve, Reason: Progressive Capability Formation in Time Series-LLMs

Abstract

Time Series Large Language Models (TS-LLMs) have shown growing capabilities in perception, task solving, and reasoning, yet these abilities are typically studied independently. We organize TS-LLM capabilities into three stages—Perceive, Solve, and Reason, and investigate how they progressively develop under two representative alignment architectures. Experiments show that Perceive facilitates Solve, while the complete Perceive Solve Reason pathway yields the strongest reasoning performance, especially under limited reasoning supervision. We further find that ordered capability accumulation is more effective than mixed supervision, and that the formation process is affected by input format and stage maturity. Cross-layer representation analysis reveals that different capabilities are transferred through distinct network depths rather than simply accumulated. Moreover, local stage convergence does not necessarily indicate sufficient preparation for subsequent capability development. Based on these findings, we derive a progressive training strategy and build strong Pre-align and Post-align TS-LLMs, achieving the best overall performance under fair out-of-domain evaluation.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Reasoning, Large Language Models, Representation Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PuB9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates progressive capability formation in time-series large language models (TS-LLMs) through three stages: Perceive, Solve, and Reason. The authors compare different training strategies and alignment architectures, analyzing how training order, stage maturity, and representation transfer influence temporal reasoning. Experiments across multiple benchmarks demonstrate that progressive training generally improves performance, particularly under limited reasoning supervision.

Critical strengths

The paper systematically investigates the relationships among perception, task solving, and reasoning capabilities in TS-LLMs, with experiments conducted under both Pre-align and Post-align architectures. The results show that progressive training generally improves reasoning performance, with more pronounced gains when reasoning supervision is limited. These findings provide useful insights into training strategies for time-series LLMs.

Critical weaknesses
  1. Limited novelty and insufficient comparisons with alternative training strategies.

The proposed Perceive → Solve → Reason training framework is closely related to existing curriculum learning approaches. Although the authors compare direct training, mixed training, and progressive training, more targeted baselines are still missing, such as mixed training followed by a separate Reason fine-tuning stage. Therefore, the current experiments do not sufficiently demonstrate whether the improvements primarily come from the specific three-stage progression or from general stage-wise optimization. In addition, the authors should clarify whether different training strategies are comparable in terms of total training tokens, optimization steps, and computational budgets.

  1. Insufficient evaluation of reasoning capabilities.

The Reason stage is mainly evaluated using multiple-choice questions, QA accuracy, and metrics such as ROUGE-L and BLEU. However, these metrics may not adequately demonstrate whether the models genuinely understand time series data. I suggest including additional experiments involving masked, shuffled, or counterfactual time series inputs to verify whether the models truly rely on temporal information rather than textual cues or answer choices.

  1. Limited robustness and generalizability of experimental results.

The authors primarily use Qwen3-4B, with additional experiments on Qwen3-1.7B. However, evaluations across different model families and multiple random seeds are still lacking (given the potentially substantial variability in TS tasks, I believe more robust experimental validation is necessary)

Furthermore, progressive training does not consistently outperform Mixed Training across all settings. For example, in Table 17, under the Pre-align architecture with 75% reasoning data, Mixed Training achieves 0.673, while Progressive Training achieves 0.664. The authors should therefore provide a more balanced discussion of the conditions under which progressive training is beneficial.

  1. Insufficient evidence supporting the proposed capability formation mechanism.

The authors use gradient analysis and cross-layer representation analysis to argue that Perceive, Solve, and Reason capabilities are progressively transferred across different network depths. However, these analyses mainly reveal correlations and do not sufficiently establish the causal dependencies among the three capabilities. More direct controlled experiments would be valuable to substantiate the proposed Progressive Capability Formation mechanism.

Questions for authors
  1. Is the training budget fairly controlled?

Are Progressive Training, Mixed Training, and Stage 3 Only strictly controlled for total training tokens, optimization steps, and computational budgets? If the training budgets differ, the observed performance improvements may not be entirely attributable to the training order.

  1. How can the importance of the specific training order be demonstrated?

I suggest adding a Mixed Training → Reason Fine-tuning baseline, where the model is first trained on mixed data and then fine-tuned specifically on Reason tasks. This would help determine whether the improvements stem from the complete Perceive → Solve → Reason progression or simply from concentrating on Reason training in the final stage.

  1. Do the models genuinely utilize time series information?

Many Reason tasks are formulated as multiple-choice questions, and some may be answerable using textual cues or common knowledge embedded in the answer choices. I suggest conducting ablation experiments by removing or shuffling the time series inputs to verify whether the improvements genuinely arise from temporal understanding rather than the language model's existing knowledge.

Reviewer_gB3eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a Perceive → Solve → Reason curriculum for time-series LLMs. It compares isolated, sequential, and mixed training under two alignment architectures and textualized input, mainly using Qwen3-4B, with supplementary Qwen3-1.7B results. Further analyses examine reasoning-data scale, encoder freezing, optimization behavior, and representation transfer. The resulting models are evaluated on MTBench, TimeSeriesExam, and CaTS-Bench.

Critical strengths
  1. The comparison includes a meaningful mixed-supervision baseline. At 15% reasoning data, progressive training improves accuracy over mixed training from 0.479 to 0.528 for Pre-align and from 0.461 to 0.520 for Post-align (Table 17). Evidence across both alignment architectures, alongside smaller-model experiments, supports the practical usefulness of preceding-stage training.

  2. The investigation extends beyond the main reasoning results. Encoder freezing causes only modest overall degradation (Section 5.3), and Tables 7–8 report competitive performance on additional benchmarks.

Critical weaknesses
  1. The benefit of the specific stage order is not isolated. Mixed training uses the same supervision sources, but comparison with P → S → R does not establish that Perceive should precede Solve. An S → P → R control would test this while keeping Reason last. Appendix F also omits stage lengths, sample exposures, and schedule-reset details needed to assess training comparability. The conclusion should acknowledge exceptions: at 75% reasoning data, Pre-align mixed training scores 0.673 versus 0.664 for progressive training (Table 17).

  2. The update-efficiency definition and Figure 6 need reconciliation. Section 5.1 defines normalized quantities ΔU_i = U_i / Σ_j U_j × 100%, whose shares sum to 100%. Figure 6 reports positive progressive-over-mixed gains for every stage; these cannot all be increases in those normalized shares. The figure may instead report gains in unnormalized U_i, but this is unspecified. Moreover, with a common measurement window, U_i shares the same overall parameter-displacement numerator across capabilities, so it does not by itself measure each capability's contribution to useful learning. This limits the claimed mechanistic explanation.

Questions for authors
  1. Can you provide the stage lengths, per-source exposures, and optimizer/scheduler reset rules from existing runs? If an S → P → R result is unavailable, one comparison with matched exposures and trainable modules at a representative setting would address the missing order control.

  2. What exact quantity, raw values, and relative-gain formula produce Figure 6? Please clarify the measurement windows and align the definition and interpretation with the plotted results.

  3. Please reconcile the Stage 3 sample total and A/B naming, and identify the evaluation set underlying Table 17. A corrected data/result mapping would suffice.

Reviewer_avmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how perception, task solving, and reasoning develop in time-series large language models (TS-LLMs). It organizes supervision into a Perceive-Solve-Reason (PSR) curriculum and compares the effects of capability order, supervision strategies, and stage maturity on training effectiveness.. Experiments suggest that preceding perceive & solve stages supervision can improve reasoning, particularly when reasoning data are limited. Optimization and representation analyses provide possible explanations, and the resulting training strategy is evaluated on external benchmarks.

Critical strengths
  1. The paper examines how foundational temporal abilities support higher-level reasoning, extending the focus beyond individual task performance. The PSR organization provides a useful framework for investigating relationships between these capabilities.

  2. The study combines broad empirical investigation. Its results offer useful guidance for training TS-LLMs, especially under limited reasoning supervision, with additional evidence from external benchmark evaluations.

Critical weaknesses
  1. Insufficient positioning relative to curriculum learning and limited evidence for the proposed order. PSR can be viewed as a capability-based curriculum, yet the paper does not adequately discuss its relationship to established curriculum learning or distinguish its specific contributions from the general principle of staged training. The task organization and TS-LLM-specific findings is valuable, but their novelty needs clearer articulation. Moreover, Section 4 mainly compares stage-omission variants and mixed training, without testing stage-permuted variants such as S→P→R or Mixed(P,S)→R. These comparisons therefore do not establish whether the gains specifically depend on the proposed capability order rather than a broader curriculum effect.

  2. The mechanism analyses do not establish the proposed causal progression. Section 5 examines optimization dynamics, temporal attribute ablations, representation transfer and encoder-freezing interventions. However, optimization alignment and representation overlap do not directly establish causal cooperation. The attribute ablations and encoder-freezing experiments support narrower conclusions about the usefulness of temporal attribute supervision and the necessity of updating the encoder. In particular, the results do not demonstrate that improved perception facilitates solving, which subsequently improves reasoning; P and S could instead provide independent benefits to R. The evidence supports an effective training curriculum more strongly than an established mechanism of progressive capability formation.

Questions for authors
  1. How does PSR relate to curriculum learning, and what constitutes its principal novelty: the capability decomposition, the training schedule, or the empirical findings about TS-LLMs?

  2. Could the authors compare other stage-permuted variants, such as P→S→R, S→P→R, and Mixed(P,S)→R, to isolate the importance of the proposed order?

  3. Could the authors report P, S, and R performance after each training stage? Based on these results, could the authors further investigate, through targeted interventions on S, whether S mediates the benefit of P to R, distinguishing this pathway from direct P→R transfer and independent contributions of P and S?

  4. Section 5 provides mechanistic analyses of progressive capability formation. Could the authors test at least one proposed mechanism through targeted interventions? For example, Section 5.1 attributes the advantage of progressive training to reduced competition between capability objectives. Could the authors introduce gradient-conflict mitigation into mixed training, in order to examine whether reducing conflict correspondingly narrows the performance gap to PSR?

Reviewer_XabMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether temporal perception, task solving, and reasoning develop more effectively through sequential training in time-series LLMs. It compares direct reasoning supervision, partial stage sequences, mixed supervision, and the full Perceive → Solve → Reason sequence using Pre-align and Post-align architectures with a Qwen3-4B backbone, together with textualized inputs and supplementary Qwen3-1.7B experiments. The full sequence improves many reported reasoning metrics, with particularly clear gains when reasoning supervision is limited. Additional analyses examine training-data scale, gradient behavior, temporal-attribute ablations, encoder freezing, and representation transfer across layers. The resulting models also achieve competitive results on weather and finance tasks in MTBench and on temporal-perception benchmarks. The evidence supports the usefulness of staged training in several evaluated settings, but does not yet isolate the contribution of the proposed stage order.

Critical strengths
  1. The study offers a useful empirical organization of temporal capabilities and transfer. Evaluating perception, task solving, and reasoning within a common experimental framework makes it possible to examine how earlier supervision affects later tasks. The comparison across two alignment architectures, direct textualization, and a smaller backbone provides more informative evidence than a single-model ablation. The low-reasoning-data results are particularly relevant: at 15% reasoning data, progressive training improves accuracy over mixed training from 0.479 to 0.528 for Pre-align and from 0.461 to 0.520 for Post-align (Table 17).
  2. The paper examines both transfer behavior and downstream utility. The attribute ablations, encoder-freezing study, and layerwise representation analysis offer useful observations about which information may support later-stage performance (Section 5). The additional weather, finance, and temporal-perception evaluations (Tables 7–8) also show that the trained models perform competitively beyond the main stage-specific evaluations. These results make the proposed training procedure practically interesting, even though they do not by themselves establish the claimed mechanism.
Critical weaknesses
  1. The experimental design does not isolate whether the specific stage order is responsible for the gains. The central scientific claim concerns ordered capability formation, but the reported comparisons do not include alternative permutations that test this explanation. In particular, Solve → Perceive → Reason would retain Reason as the final stage while testing the proposed ordering of the two preceding capabilities. Comparisons with Stage 3 Only also change the amount and type of supervision, while progressive versus mixed training can differ in final-stage specialization and optimization scheduling. Appendix F specifies common optimizer settings and initialization, but does not report the epochs, total tokens, total updates, mixture proportions, or scheduler-reset policy needed to establish matched training budgets. Thus, the observed gains remain compatible with several explanations, including additional supervision and a final period devoted exclusively to reasoning. The gradient and representation analyses are useful diagnostics, but do not resolve these alternatives.
  2. The strength and consistency of the conclusions exceed the precision of the reported evidence. Appendix F reports a fixed seed of 2026, without repeated-run variability or uncertainty estimates. This matters for small differences and exceptions: for Pre-align at 75% reasoning data, mixed training reaches 0.673 while the full sequence reaches 0.664 (Table 17), contradicting the statement that progressive training “consistently outperforms” mixed training. Some individual capability comparisons are also non-monotonic; for example, Post-align Inferring and Extrapolating in Table 2 favor Stage (2 → 3) over the full sequence. In addition, Section 3 and Table 11 define evaluation set A as the 4,547-example mixed-format QA set and set B as the 6,163-example five-dimension set, whereas Tables 2–3 and the accompanying RQ2 discussion use the reverse labels. These discrepancies hinder interpretation of the aggregate accuracies and should be resolved before drawing general conclusions about capability formation.
Questions for authors
  1. Decisive experimental control: How does Perceive → Solve → Reason compare with Solve → Perceive → Reason and with mixed training followed by a reasoning-only phase, when the total exposure to each supervision source, optimizer-update budget, trainable modules, and learning-rate schedule are matched? These comparisons would help distinguish a benefit of the proposed order from final-stage specialization and training-budget effects.
  2. Statistical reliability: Can the authors report results across multiple training seeds for the main progressive-versus-mixed comparisons, including the 15%, 75%, and 100% reasoning-data settings? Means and dispersion would help determine whether the smaller gains and observed reversals are reliable.
  3. Protocol clarification: Please reconcile the A/B evaluation-set mapping and specify the aggregation used for Figure 3 and Table 17. Please also clarify whether the encoder-freezing policy in Appendix F applies to all earlier Pre-align comparisons or only the final recipe: Section 3 describes training the encoder and alignment module, whereas Section 5.3 presents encoder freezing as an ablation. These details affect whether the reported comparisons use equivalent training conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.