acceptodds
Under review as a conference paper at ICLR 2027

Similarity Is Not Functional Alignment: What Time-Series–Language Alignment Must Establish

Abstract

Time-series–language alignment is commonly evaluated through retrieval and representational similarity. These scores measure correspondence between paired series and descriptions; they do not by themselves certify whether a statement is supported by the observed history, adds new conditional-mean information about the future, or improves a fitted predictor. We formalize these distinctions as Functional Alignment (FA), which separates representation correspondence, historical support, predictive innovation, and fitted-model assistance. Our central result is an identification limit: with the history–future process held fixed, two ways of selecting true historical statements can induce the same history–text distribution and the same support labels, yet different predictive innovation. No correspondence measure of fixed encoders can therefore tell the two apart, and a history-derived feature with zero innovation can still assist a restricted predictor. Across 20 encoder configurations, retrieval reaches 51.8 to 156.4 times chance, yet the strongest retriever has temporal factual-reversal accuracy of 0.4984. Executable Claim Alignment (ECA), which checks typed interpretations against observations, reaches domain-macro accuracy 0.9874 in controlled verification across 11 held-out CaTS domains. On Time-MMD, task-specific projection training nearly doubles Linear CKA, yet its forecasting increments over matched untrained projections range from −3.53% to 5.67% across six backbones. Functional claims about alignment require evidence matched to the target being claimed.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series–Language Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wLWzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper separates time series and language alignment into representation correspondence, historical support, predictive innovation, and fitted model assistance. It establishes an identification limit for predictive innovation and introduces Executable Claim Alignment for support verification. Experiments across 20 encoder configurations and five forecasting tasks show that correspondence scores alone do not certify factual discrimination or forecasting assistance. Controlled executable verification achieves mean domain accuracy of 0.9874.

Critical strengths
  1. The paper presents its contribution through a clear and coherent structure. Section 3 and Table 1 separate four alignment targets, and Section 4 organizes the experiments around three corresponding research questions. Proposition 1 and Equation 2 further clarify why evidence for one target cannot automatically establish another.

  2. The experimental design uses targeted comparisons to distinguish different forms of alignment success. Section 4.1 and Table 2 evaluate 20 encoder configurations on factual reversals with each history fixed. Section 4.3 and Table 4 additionally compare trained and matched untrained projections, helping distinguish text gains from gains attributable to projection training.

Critical weaknesses
  1. The forecasting analysis does not fully justify the precision of its pooled experimental conclusion. Section 4.3 aggregates 30 cells that share datasets, recipients, and representation states without explaining how their dependence affects the reported 95% interval. This leaves insufficient support for the claim that average increments above 0.401% can be excluded.

  2. The verification experiments do not cleanly isolate the contribution of executable checking. Section 4.1 reports that ECA reuses calibration outside the scorer folds and that the compared systems receive different evidence inputs. These choices leave unclear whether the reported 0.1505 Macro F1 advantage over matching persists when calibration and evidence inputs follow a common evaluation protocol.

  3. The method description leaves the construction of executable interpretations unclear. Section 3.3 introduces up to 20 typed candidate programs without explaining how they are generated or which numerical and temporal claim structures their language supports. This makes the assumptions and reproducibility of the controlled accuracy of 0.9874 in Section 4.1 difficult to assess.

  4. The error analysis does not fully support its explanation of ECA's failures. Section 4.1 uses an oracle with Macro F1 of 0.8951 to attribute much of the error to interpretation selection. A program can produce the correct verdict through an incorrect interpretation, leaving the distinction between candidate generation, binding, and selection errors unresolved.

Questions for authors
  1. How does the pooled interval change when recipient clusters are resampled jointly across all six backbones within each task?

  2. With calibration fitted only within each training fold and identical evidence ranges for every verifier, does ECA retain its reported advantage over representation matching?

  3. How are the typed candidate programs in Section 3.3 generated, and which numerical and temporal claim structures does their language support?

  4. What proportion of Native TSVer claims contains a semantically correct candidate, and does this support the error attribution in Section 4.1?

9 comments
AuthorsOpenReview ↗

Response to Reviewer wLWz — Part 1/4: Joint Statistical Evaluation

Thank you for identifying the shared-sample dependence in the pooled forecasting analysis.

W1 + Q1. How does the pooled interval change when recipient clusters are jointly resampled across all six backbones within each task?

A1. Under the original weights, the joint recipient-cluster bootstrap gives a TASK increment of −0.3535% (95% CI [−0.9172%, 0.3754%]). For each of 10,000 draws, recipient clusters are resampled within each task, with the same draw shared across all six backbones, fitted runs, representation states, and input conditions. The effect and RANDOM numerical-risk denominator are recomputed on each draw.

All TASK aggregation checks.

AggregationMean (%)Joint 95% CI (%)
Original fixed weights−0.3535[−0.9172, 0.3754]
Re-estimated heterogeneity and weights−0.3535[−0.9904, 0.4021]
Equal task/backbone weights−0.3461[−2.0265, 0.9229]
Excluding DONKI/Kp: fixed weights−0.4131[−1.1212, 0.6110]
Excluding DONKI/Kp: re-estimated weights−0.4131[−1.1602, 0.6973]
Excluding DONKI/Kp: equal weights−0.2815[−2.3201, 1.2811]

The original 0.401% upper endpoint decreases to 0.3754% under the joint bootstrap. All six TASK intervals include zero, so none of the aggregation choices establishes a positive pooled increment. The full TASK and CATS sensitivity results are reported in Appendix E.3, Table 16.

Manuscript changes (§4.3, Figure 11(c), and Appendix E.3): We report the joint-bootstrap estimate and complete weighting and task-exclusion analyses.

AuthorsOpenReview ↗

Response to Reviewer wLWz — Part 2/4: Common-Evidence Comparison

W2 + Q2. With calibration fitted within each training fold and identical evidence inputs for every verifier, does ECA retain its reported advantage over representation matching?

A2. Under the common-evidence protocol, ECA obtains Macro-F1 0.4129, compared with 0.3452 for representation matching. The difference is 0.0678, with a 95% source-group bootstrap interval of [−0.0491, 0.1527].

We evaluate the same 304 claims using five source-group-disjoint outer folds. Candidate generation, scorer fitting, checkpoint selection, and calibration are restricted to the outer-training side; training candidates are generated by source-disjoint cross-fitting. Every system receives the same series, entity candidates, and annotated time ranges, while retaining its own scoring procedure.

SystemMacro-F1Supported/Refuted output rate
Representation matching0.34520.9441
Generic verifier0.09090.0000
ECA0.41290.7204

Under matched evidence and fold-specific calibration, ECA retains a positive Macro-F1 advantage of 0.0678 over representation matching. The 95% confidence interval [−0.0491, 0.1527] includes zero, so the observed advantage is not statistically significant. The original retrospective comparison remains in Figure 2 and Table 10(b), while this new evaluation provides a fully nested comparison under matched evidence.

Manuscript changes (Appendix C.3): We add the common-evidence protocol and results alongside the original Native comparison.

AuthorsOpenReview ↗

Response to Reviewer wLWz — Part 3/4: Candidate Generation and Typed Operations

W3 + Q3. How are the typed candidate programs generated, and which numerical and temporal claim structures can their language express?

A3. The two generation paths are different. For controlled CaTS, rules use numbers, dates, and operation cues to construct up to 20 distinct typed programs. A fitted compiler ranks them by mean conditional token log-likelihood and retains up to five. For Native TSVer, accepted programs from training sources outside the held-out groups are organized into structural templates. The generator ranks these templates using claim similarity, a predicted semantic frame, and prototype similarity, fills claim-specific numerical and temporal expressions, deduplicates the resulting programs, and retains up to 20.

The typed language supports point queries and interval aggregates (mean, sum, minimum, maximum, and median); numerical comparisons with tolerances; differences and percentage changes; monotonic trends; extrema and their times; first/last event and onset operations; cross-series/entity operations; and Boolean composition. The DSL specifies executable operations; the Native generator instantiates candidate structures from training-derived templates.

The controlled CaTS result of 0.9874 is domain-macro accuracy over 8,010 binary claims per scoring seed across 11 held-out domains. The verifier receives the correct series, and outer-test examples are excluded from compiler fitting and calibration.

Relevant manuscript locations (Appendix C.1–C.2): These sections describe the two candidate-generation paths, their ranking and retention steps, the typed operations, and the controlled-verification setting.

Reviewer_H5jUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper separates representation correspondence, historical support, predictive innovation, and fitted-model assistance. It presents a non-identification construction and studies these distinctions through factual-reversal ranking, executable verification, and forecasting experiments.

Critical strengths

The paper takes a valuable and fresh perspective on time-series–language alignment by investigating the gap between representation metrics and actual task performance. Understanding whether retrieval and representational similarity reflect factual correctness and forecasting utility is important for choosing models and evaluation criteria that match practical needs. The factual-reversal tests, learned-readout control, and matched untrained projections make this gap concrete and provide useful references for assessing what alignment scores and training gains actually establish.

Critical weaknesses

1. Poor presentation obscures the framework’s motivation, and its theoretical novelty is limited.

The abstract and introduction stack abstract terminology and jump between previews of later results before establishing a coherent problem–argument–evidence progression. The factual-reversal example is not developed into a concrete thread connecting historical support, statement selection, predictive innovation, and fitted assistance. The four targets consequently appear as an asserted taxonomy. Readers must reconstruct why these concepts belong together and how the results support a common contribution, making these sections difficult to follow.

Proposition 1 illustrates that history–text and history–future marginals do not determine their joint law. Its construction lets one statement selector access a future-relevant external variable and applies equally to an auxiliary binary label. The redundant-feature example illustrates familiar feature engineering for a restricted predictor, while Equation 2 is a direct Bayes-risk decomposition. These valid constructions provide limited new theoretical insight into temporal structure or learned alignment.

2. The experiments establish disagreement between evaluation targets but provide limited explanation or new insight.

For Contribution 2, poor factual ranking could reflect missing factual supervision, information loss in the encoders, or the similarity readout. The learned-readout control partially probes the last explanation, but a matched factual-hard-negative training experiment is needed to examine the supervision gap. ECA’s success through direct execution does not identify the source of the matching failure.

For Contribution 3, both projections change, allowing better numerical representations to reduce text’s marginal contribution. Table 10 reports an overall paired-text MSE improvement of about 3.51% but a text-related training increment of only 0.04%. The authors distinguish these quantities, yet do not establish which representation changes produce this pattern. Together, the experiments support separate evaluation of the targets, but offer limited guidance on when or why alignment becomes functionally useful.

Questions for authors
  1. What new alignment-specific theoretical conclusion follows from Contribution 1 beyond the familiar principles identified above?
  2. Does the factual-ranking gap persist when contrastive training includes numerical and temporal hard negatives?
  3. Which changes in the numerical and textual representations explain the near-zero text-related increment alongside improved overall forecasting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer H5jU — Part 1/4: Motivation of the Four Targets

Thank you for highlighting the need to connect the four targets to one motivating problem.

W1. How do the four targets follow from the problem?

A1. Figure 1(a) gives a concrete starting point: “the maximum value from 2002 to 2004 is 8.86.” Changing the interval to 2000–2001 makes the statement false, although both descriptions refer to the same series. Representation matching measures whether the paired inputs correspond; checking the observations tests whether the statement has historical support.

The next question concerns the choice among statements that are both true. As Proposition 1 shows, the reporting choice can reveal a future-relevant signal that is absent from the recorded history. Conversely, a statement computed from history can help a restricted predictor use information it already contains, as expressed by Equation 2. These are predictive innovation and fitted-model assistance. Table 1 summarizes the distinct object and evidence associated with each target. Together, Figure 1(a), the four-target formulation in §3.1, and Proposition 1 establish the progression from correspondence to historical support, statement selection, and fitted-model assistance.

Relevant manuscript locations (§1, Figure 1(a), Table 1, and §3.1): These existing sections provide the motivating temporal-edit example and define the four targets in the progression described above.

AuthorsOpenReview ↗

Response to Reviewer H5jU — Part 2/4: Theoretical Identification Limit

A2. Proposition 1 establishes that the paired history–text distribution and historical support do not identify predictive innovation. It holds the history–future process fixed and considers two true channel descriptions for each history. One selector chooses between them using an independent sign; the other uses a future-relevant sign available to the reporter at forecast time but absent from the recorded history. As stated in the Introduction, both selectors induce the “same population history–text joint distribution and support status, but different predictive innovation.”

The construction gives analytic innovations of 0 and 1.5625, respectively (Proposition 1 and Table 3(a)). Thus, even complete knowledge of the paired distribution cannot distinguish their predictive contributions. This is the alignment-specific identification limit: correspondence and historical truth alone cannot establish what information the selection of a statement reveals about the future.

Equation 2 addresses the complementary fitted-model question. The original paper further shows that a redundant history-derived feature, despite having zero predictive innovation, reduces a restricted predictor's test MSE by 0.0894 (Table 3(b)). This demonstrates why predictive innovation and fitted-model assistance require separate evaluation.

Relevant manuscript locations (§3.2, Proposition 1, Equations 1–2, and Appendix D.1): We use the original identification construction and controlled results to distinguish predictive innovation from fitted-model assistance.

AuthorsOpenReview ↗

Response to Reviewer H5jU — Part 3/4: Factual-Negative Supervision

W2 + Q2. Does the factual-ranking gap persist after numerical and temporal hard-negative supervision?

A3. The original paper shows that factual discrimination is distinct from representation correspondence: the strongest of 20 retrievers reaches temporal PTRA 0.4984, close to chance (Figure 1(b)). We further test whether explicit factual-negative supervision improves this capability.

Under joint numerical and temporal training, verified factual negatives improve numerical PTRA for both encoder pairs (+13.2800 and +7.8400 pp) on the original CaTS reversals, while temporal PTRA point estimates decrease (−3.2800 and −6.7200 pp). The temporal change is uncertain for T5-small + MOMENT-base (95% CI [−9.5168, 2.9659] pp), whereas the interval for Qwen3-Embedding-4B + Chronos-large lies entirely below zero (95% CI [−12.3651, −0.9446] pp, q=0.0291). Thus, the verified factual-negative scheme improves numerical discrimination without corresponding temporal gains in this matched comparison. These results support evaluating numerical and temporal factual discrimination separately, rather than treating improvement on one claim type as evidence of improvement on the other.

We froze the encoders and ran 20 projection fits across two encoder pairs, two negative schemes, and five seeds. Both arms use matched positives, initialization, loss, 50-epoch budget, and validation selection. Scheme A uses ordinary pairing negatives; Scheme B uses edits verified as refuted on the anchor series. The comparison evaluates the verified-negative scheme as a whole.

PTRA on the original CaTS factual reversals.

Encoder pairClaim typeA PTRAB PTRAB−A (pp), 95% CIq
T5-small + MOMENT-baseNumerical0.57520.7080+13.2800 [7.8430, 18.7952]0.0008
T5-small + MOMENT-baseTemporal0.45600.4232−3.2800 [−9.5168, 2.9659]0.3112
Qwen3-Embedding-4B + Chronos-largeNumerical0.65440.7328+7.8400 [3.3755, 12.3556]0.0008
Qwen3-Embedding-4B + Chronos-largeTemporal0.52960.4624−6.7200 [−12.3651, −0.9446]0.0291

PTRA is the five-seed mean fraction of supported claims scoring above their refuted counterparts. Intervals use 10,000 paired source-bootstrap draws; q values are BH-adjusted over the four primary comparisons. On prespecified unseen edits, numerical gains are +15.8400 pp (95% CI [8.7769, 22.6365], q=0.0004) and +11.1200 pp ([5.6922, 16.6394], q=0.0004), in the same encoder order. Temporal changes are −2.4000 pp ([-8.3740, 3.4959], q=0.4160) and +2.4000 pp ([-2.6773, 7.4897], q=0.4160). The frozen-representation binary readout in §4.1 also raises macro accuracy from 0.4973 with thresholded matching to 0.6871 with a linear head.

Manuscript changes (§4.1, Appendix B.5, Tables 7–8): We add the matched factual-negative comparison and unseen-edit results, extending the original factual-reversal and learned-readout findings to separate numerical and temporal supervision effects.

AuthorsOpenReview ↗

Response to Reviewer H5jU — Part 4/4: Forecasting Gains and Representation Changes

W2 + Q3. Which representation changes explain the near-zero text-related increment alongside improved overall forecasting?

A4. On Time-MMD/PatchTST, TASK reduces paired-text MSE by 0.0088288 and numerical-only MSE by 0.0087289 relative to RANDOM. The paired forecast improves by 3.5109%, while the additional text benefit is only 0.0000999 MSE, or 0.0370% of RANDOM numerical-only risk.

BackbonePaired-text improvement (%)Additional text benefit (%)
iTransformer+1.9757+2.3646
TimeMixer+5.5226+5.6676
PatchTST+3.5109+0.0370
MOMENT−5.2453−3.5329
TimesFM+1.2837+3.7592
Chronos−1.4115+2.0911

Both columns use MSE averaged over three fitted runs. The first is normalized by RANDOM paired-text MSE; the second by RANDOM numerical-only MSE. As defined in Appendix E.4, “The different denominators make direct subtraction of its two percentage columns inappropriate.” PatchTST is the only backbone whose numerical-only MSE decreases under TASK. Its paired-text and numerical-only improvements are nearly equal, explaining the near-zero additional text benefit.

We further compare all four existing RANDOM/TASK series- and text-head combinations by changing one head at a time. The complete 24 conditional contrasts are shown below. Effects measure changes in additional text benefit, normalized by RANDOM numerical-only risk; intervals use 10,000 shared cluster-bootstrap draws. Each combination has a separately fitted adapter.

Text-head changes (series representation fixed).

BackboneFixed seriesEffect (%)95% CI (%)q
iTransformerRANDOM+0.9965[−2.2862, 4.8806]0.7351
iTransformerTASK−1.0019[−3.5045, 1.5308]0.7351
TimeMixerRANDOM+6.7946[−23.0271, 21.0207]0.7571
TimeMixerTASK+5.9061[−15.1918, 15.9959]0.7351
PatchTSTRANDOM+2.6501[−17.0342, 12.5962]0.8428
PatchTSTTASK+5.2428[−1.2355, 8.6985]0.3592
MOMENTRANDOM−1.7463[−12.5325, 2.2750]0.7351
MOMENTTASK+6.0527[−0.2829, 9.4144]0.2793
TimesFMRANDOM−2.0222[−11.2808, 4.3565]0.7351
TimesFMTASK+5.7907[−3.0625, 11.2802]0.5732
ChronosRANDOM−3.8201[−20.5080, 3.7806]0.7351
ChronosTASK+8.5788[−13.9350, 21.0485]0.7351

Series-head changes (text representation fixed).

BackboneFixed textEffect (%)95% CI (%)q
iTransformerRANDOM+3.3665[1.3593, 6.6648]0.0080
iTransformerTASK+1.3681[0.7518, 2.3309]0.0072
TimeMixerRANDOM−0.2385[−13.5159, 7.1994]0.8993
TimeMixerTASK−1.1270[−7.3554, 4.1149]0.7571
PatchTSTRANDOM−5.2057[−13.9227, 13.7943]0.7351
PatchTSTTASK−2.6131[−17.7577, 29.8789]0.8942
MOMENTRANDOM−9.5856[−13.1797, −5.4497]0.0072
MOMENTTASK−1.7865[−5.3425, 6.4111]0.7351
TimesFMRANDOM−2.0315[−11.2226, 11.5275]0.8395
TimesFMTASK+5.7814[−4.5375, 20.6810]0.7351
ChronosRANDOM−6.4877[−15.5835, −0.0395]0.2793
ChronosTASK+5.9113[−8.7123, 16.8795]0.7351

As stated in Appendix E.4, “Positive values mean increased marginal text benefit, not necessarily lower overall forecasting error.” Because each representation combination has its own fitted adapter, these comparisons measure “sensitivity across fitted representation combinations rather than an intervention on one branch of a fixed predictor.”

Three series-head contrasts pass correction across all 24 comparisons: both iTransformer contrasts and MOMENT with RANDOM text. No text-head contrast passes correction, and all four PatchTST component intervals include zero.

These results show that improved representation correspondence and overall forecasting gains do not necessarily translate into additional text-related forecasting benefits. The observed changes depend on the backbone and representation component, supporting our distinction between overall forecasting improvement and fitted-model assistance from text.

Manuscript changes (§4.3, Appendix E.4, Tables 17–19): We report the six-backbone decomposition and all 24 conditional comparisons, with the risk definitions and uncertainty intervals.

Reviewer_9ZDXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether similar time-series and text embeddings mean that the text is supported by the data or useful for forecasting. It separates these questions and gives examples showing why they need different tests. It also tests claim verification and forecasting across several datasets and models. ECA works well on controlled claims, but falls behind the direct LLM on Native TSVer. On Time-MMD, higher CKA does not give a clear extra forecasting gain over random projections. The main contribution is a framework for evaluating alignment.

Critical strengths
  1. The paper makes a useful distinction: text can help a model even when it adds no new information beyond the past data. The simple examples make this point clear. This helps avoid reading too much into alignment scores.

  2. The tests cover 20 encoder pairs and include useful controls. The TRUCE results and the learned-readout test help show where the findings apply. The forecasting tests compare trained projections with matched random projections. The authors also report ECA's weaker results on real claims and state the limits of that comparison.

Critical weaknesses
  1. The pooled forecasting interval may be too narrow. The 30 results share data and representations. Appendix E.2 uses the error estimate for each result, but the paper does not explain how it handles their correlation. This makes the claim that average gains above 0.401% can be ruled out hard to justify (Section 4.3). The interval needs to account for the shared data.

  2. The code has a sign mismatch, and key steps for reproducing the results are missing. In src/forecast/bridge.py, relative_effect uses reference minus condition. However, paired_effect calls a bootstrap that uses condition minus reference. For reference losses [2,2,2,2] and condition losses [1,1,1,1], they return +50% and -50%. This does not prove that the tables are wrong: Table 10's arithmetic is consistent, and the table scripts read results from other files. But the authors need to show which code produced the tables. The release also lacks the input files and full steps needed to rebuild the encoder caches, ECA candidate programs, and predictions.

Questions for authors
  1. Can you recompute the pooled interval with a joint bootstrap that keeps shared test samples paired across models? Does the 0.401% upper bound still hold?

  2. Which code produced Tables 4 and 10-12? Did it use paired_effect? Please fix the sign mismatch and provide a small runnable example with the input files and command. Please also give the steps needed to build the candidate programs used by ECA.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9ZDX — Part 1/2: Joint Bootstrap and Statistical Dependence

Thank you for identifying the shared-sample dependence in the pooled forecasting analysis.

W1 + Q1. How does accounting for shared test samples affect the pooled estimate and the original 0.401% upper endpoint?

A1. We recomputed the pooled TASK increment using a joint recipient-cluster bootstrap. Under the original weights, the estimate remains essentially unchanged at −0.3535% (95% CI [−0.9172%, 0.3754%]), compared with the original estimate of −0.353% (95% CI [−1.108%, 0.401%]).

For each of 10,000 draws, recipient clusters are resampled within each task, with the same draw shared across all six backbones, fitted runs, representation states, and input conditions. The effect and RANDOM numerical-risk denominator are recomputed on each draw before pooling.

Joint-bootstrap aggregation results.

StateGridAggregationMean (%)Joint 95% CI (%)
TASKAll 30Fixed original weights−0.3535[−0.9172, 0.3754]
TASKAll 30Updated weights−0.3535[−0.9904, 0.4021]
TASKAll 30Equal task/backbone−0.3461[−2.0265, 0.9229]
TASKWithout DONKI/KpFixed original weights−0.4131[−1.1212, 0.6110]
TASKWithout DONKI/KpUpdated weights−0.4131[−1.1602, 0.6973]
TASKWithout DONKI/KpEqual task/backbone−0.2815[−2.3201, 1.2811]
CATSAll 30Fixed original weights−1.0734[−2.1033, −0.0586]
CATSAll 30Updated weights−1.0734[−2.7521, −0.1502]
CATSAll 30Equal task/backbone−2.2312[−4.4435, −0.6427]
CATSWithout DONKI/KpFixed original weights−1.5031[−3.0605, 0.1329]
CATSWithout DONKI/KpUpdated weights−1.5031[−3.9349, −0.1176]
CATSWithout DONKI/KpEqual task/backbone−2.6748[−5.3348, −0.7371]

Conclusion. The joint bootstrap preserves the original negative TASK point estimate, with a slightly narrower 95% interval and an upper endpoint of 0.3754% instead of 0.401%. All six TASK intervals include zero. This is consistent with our original cell-level finding that forecasting increments vary across tasks and backbones: as reported in the Introduction, “three increments survive correction, two negative and one positive.” The pooled analysis therefore complements the original results by showing that these heterogeneous effects do not establish a positive average TASK increment. For CATS, all three full-grid intervals lie below zero; after excluding DONKI/Kp, two of the three intervals remain below zero.

Manuscript changes (§4.3, Figure 11(c), and Appendix E.3, Table 16): We report the joint-bootstrap estimates and the complete TASK and CATS weighting and task-exclusion analyses. Figure 11(c) presents the updated fixed-weight intervals.

AuthorsOpenReview ↗

Response to Reviewer 9ZDX — Part 2/2: Implementation and Reproducibility

W2 + Q2. Which implementation produced the submitted Tables 4 and 10–12? Does the sign discrepancy in paired_effect affect those results, and how can the experiments be reproduced?

A2. We traced the historical table-generation path. The forecasting contrasts in the submitted Tables 4 and 10–12 were generated by effects and boot_effect in analyze_experiments.py; the separately released paired_effect helper was not used by that table-generation path. In the revised manuscript, these results are presented in Tables 4, 14, 15, and 20, respectively. We corrected the helper so positive values consistently indicate loss reduction and the interval endpoints follow the same convention.

For reference losses [2, 2, 2, 2] and condition losses [1, 1, 1, 1], the corrected helper returns +50.0000%. It also checks negative, zero, and asymmetric-interval cases. The helper uses the observed reference mean as its denominator, whereas the historical table bootstrap recomputes reference risk on each draw.

Runnable CPU reproduction. The supplementary archive contains the five task-level NPZ inputs, saved predictions and targets, per-origin losses, recipient and cluster IDs, the corrected helper, and executable examples. From the extracted functional_alignment_reproduction/ directory, run:

python -m pip install -r requirements.txt
python -m functional_alignment

The command runs the sign checks, a controlled CaTS candidate example, and the forecasting replay. The replay recomputes 300 representation-bridge contrasts, 300 executable-summary contrasts, and 12 joint-bootstrap summaries. It reproduces revised Table 4 from saved predictions and targets; revised Table 14 from the same inputs plus the separately archived Linear CKA result; revised Table 15 from the RAW/CATS controls; and revised Table 20 from per-origin losses for all 150 interface cells.

Rebuilding model-stage outputs. The original pipeline is:

  1. prepare_inputs.py prepares splits, available text, and time-legal donors.
  2. train_base_models.py or foundation_forecasts.py produces numerical forecasts.
  3. prepare_features.py (extract()) builds encoder caches and RAW/RANDOM/CATS/TASK representation states.
  4. forecast_experiments.py fits the state-specific adapters; its --score mode loads validation-selected checkpoints and produces recipient-indexed test predictions.

This stage requires the prepared splits and data readers, encoder and forecast weights, projection and adapter checkpoints, and their model environments. The CPU package starts from the bundled predictions, targets, and losses. For Native TSVer, the generator uses accepted training programs outside held-out source groups to form structural templates, ranks them by lexical similarity, predicted semantic frame, and prototype similarity, fills claim-specific numerical values and time expressions, deduplicates, and retains up to 20 candidates. Rebuilding this stage requires the training-program inventory, fold-specific model artifacts, and the historical ranking and data-loading modules.

Candidate language and controlled accuracy. Controlled CaTS rules use numbers, dates, and operation cues to generate up to 20 programs; the fitted compiler ranks them by conditional token log-likelihood and retains up to five. The typed language supports point and interval queries, aggregations, numerical comparisons, changes and trends, event times, entity sums, and Boolean combinations. The 0.9874 result is domain-macro accuracy over 8,010 binary claims per scoring seed across 11 held-out CaTS domains, with the correct series supplied.

Relevant manuscript and supplement locations (Appendix C.1–C.2 and the Reproducibility Statement): Appendix C.1–C.2 describes the candidate-generation and verification pipeline. The supplementary README identifies the historical model-stage entry points, provides the CPU replay command and bundled inputs, and explains the assets needed to regenerate upstream predictions. The Reproducibility Statement summarizes the included replay materials.

Many thanks to your detailed explaination. I maintain my postive score of 3.

AuthorsOpenReview ↗

Thank You for Your Feedback and Support

Thank you for your positive assessment and for taking the time to review our responses. We sincerely appreciate your constructive feedback, which helped us strengthen the statistical analysis and improve the reproducibility of our work.

Thank you again for your careful review, thoughtful suggestions, and continued support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.