- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ChronoCheck, a training-free layer that audits LLM chain-of-thought rationales about time series. The same frozen model runs a few-shot extractor that turns its own rationale into typed claims from six classes: trend, extremum, periodicity, change-point, comparison and anomaly. Each class compiles to a deterministic statistical checker on the raw series (Mann–Kendall/Theil–Sen, STL + ACF, PELT, robust z-scores).
Critical strengths
-
A verifier on the data side, with measured error. Each claim is compiled into a deterministic check on the raw series. The checkers’ contradiction precision and detection power are then measured on certified claims: 0 false contradictions out of 17,922 true claims (Table 3). Model-based verifiers cannot offer this. The released checkers and claim suite could be a useful auditing tool for time-series QA, and this is the main source of value I see in the paper.
-
Unusually careful and transparent reporting.
Critical weaknesses
-
The headline measurement is not validated on real rationales, and the paper’s own evidence suggests it is biased along the scale axis. The zero-false-positive calibration only covers slot-form claims emitted by the authors’ generator. It does not cover the natural-language→slot extraction, which is performed by the model under audit and differs across tiers. There is no human audit of extracted claims or of contradicted verdicts on real rationales. As a result, the 36.5–42.1% contradiction rates mix model unfaithfulness with extraction and schema error. The extremum class shows the problem. It drives the headline: ablating it removes 16.8 of the 36.5 points. Appendix A acknowledges that the schema turns local-extremum statements (“a trough around index 40”) into global-extremum claims. It also finds that, among refuted extremum claims carrying a value, 13.4% / 23.2% / 25.4% (7B/14B/32B) do name a local extremum within ±10 indices. These are plausibly true statements scored as contradictions, and their share grows with scale. I reclassified only those claims, using the paper’s own counts. The pooled rate moves from 36.5/42.1/39.0% to roughly 34.6/36.4/33.2%. The 7B→14B rise shrinks from +5.6 to about +1.8 points, and 32B ends up below 7B. That shift is the same size as the effect the title reports. The other refutations are checked only coarsely: a value within 2% of the range anywhere in the series, and a fixed ±10-index window. So the overall artifact could be larger or smaller than this. Without a human-validated sample per class and per scale, the claim that scale shifts rationales “toward the claims models get wrong” is not established.
-
The interventions give null or true-by-construction results, and the baselines are handicapped adaptations. With every intervention null, the contribution rests on the measurement in Weakness 1. All results also come from one greedy run per cell. Appendix B shows up to 153 of 887 questions change correctness across hardware.
Questions for authors
-
For a random sample of real claims, stratified by class and scale (a few hundred claims), how often: (a) does the extracted claim faithfully represent its source sentence? (b) would an annotator looking at the series agree with the contradicted or consistent verdict?
-
What are the pooled and standardised rates per scale if local-extremum statements are extracted as windowed claims, or at least treated as unverifiable? Please also state the extremum location tolerance and the index convention used in the rendered series (0- vs 1-based; global vs window-relative). Index misreadings would inflate contradictions specifically for location claims.
-
The rate’s denominator includes unverifiable verdicts. The per-class declines behind the standardised fall include period (40.3→17.6%) and comparison (36.4→27.7%). Do these reflect more consistent verdicts, or more abstentions? Please report the consistent/contradicted/unverifiable split per class and scale. By my arithmetic, extremum and anomaly account for only about 1.2 of the 5.0-point standardised drop at 32B. So this question bears directly on the “per-claim faithfulness improves” part of the title.
Response to Reviewer jw6w
Thank you for the review.
Extraction and verdicts on real rationales (Q1). The revised paper reports one calibration and two automated diagnostics on the real rationales. Table 3 gives 0 false contradictions on 17,922 certified-true synthetic claims. Appendix A reports that, among refuted extremum claims with a value, 97.1%, 98.6% and 98.4% (7B, 14B, 32B) name a value the series attains within tolerance. Among those that also carry an index, 13.4%, 23.2% and 25.4% name a local extremum within 10 indices of the reported index. With the 32B extractor applied to all tiers, the standardized rate is 36.9%, 34.2% and 33.2%. Under the shared extractor, the extremum and numeric class shares keep their order across scales and the anomaly share does not, so the paper states the class-mix sentence under each tier's own extractor. No human annotation of the extracted claims or of the verdicts has been completed, so the paper reports no human accuracy for them.
Local-extremum recoding (Q2). On the stored records, we re-scored the extremum claims. If refuted extremum claims with a value and an index that match a local extremum within 10 indices become unverifiable (83, 291 and 295 claims), the pooled rates are 34.5%, 36.5% and 33.5%, and the standardized rates are 34.5%, 29.4% and 27.6%. If extremum claims are re-judged in a 21-sample window centered on the stated index, the pooled rates are 35.8%, 40.0% and 36.1%, and the standardized rates are 35.8%, 31.6% and 29.5%. The pooled change from 7B to 32B is -1.1 points [-3.2, +1.1] in the first reading and +0.3 [-1.9, +2.5] in the second. The standardized change at 32B in the second reading is -6.3 points [-8.1, -4.6]. In eight of nine readings, the standardized rate falls with its interval below zero at both scales and the composition effect is positive. In the ninth, which treats every refuted extremum claim as unverifiable, the standardized rate still falls at both scales with its interval below zero.
Index convention and tolerance (Q2). The prompt shows the series in order, and a claimed index counts the displayed points from 0 to N-1. The checker subtracts the window start before the lookup. The location tolerance is 5 samples around a near-global extremum. The near-extremum band is 5% of the window range. The value tolerance is the larger of 2% of the range and 1e-9.
Verdict split per class and scale (Q3). The counts are consistent / contradicted / unverifiable at 7B | 14B | 32B-AWQ. Trend 469/74/445 | 516/67/352 | 495/66/304. Extremum 122/720/22 | 350/1284/2 | 303/1213/7. Period 135/147/83 | 277/122/166 | 351/110/164. Change point 403/140/627 | 360/81/442 | 494/71/431. Comparison 102/84/45 | 151/80/40 | 227/113/68. Anomaly 186/396/82 | 229/549/114 | 267/490/110.
Source of the standardized fall (Q3). At 32B against 7B, the standardized fall is 4.9 points. It equals a consistent share up 10.2 points [+8.3, +12.2] and an unverifiable share down 5.3 points [-7.1, -3.6]. Period falls from 40.3% to 17.6% contradicted while its consistent share rises from 37.0% to 56.2% and its unverifiable share rises from 22.7% to 26.2%. Comparison falls from 36.4% to 27.7% contradicted, its consistent share rises from 44.2% to 55.6%, and its unverifiable share falls from 19.5% to 16.7%. Extremum and anomaly contribute 1.2 of the 4.9 points, and the other four classes contribute 3.7. On verifiable claims only, the standardized change is -8.4 points [-10.6, -6.2].
Interventions and baselines. The registered 7B comparison after the reviews (Appendix E) compares the checker-guided repair with two alternatives. After round one, the mean per-question fraction of original contradicted claims with no match in the rewrite is 18.8 points lower under an explicit correction prompt and 19.6 points lower without feedback than under the original repair, and both intervals exclude zero. After two rounds the original repair gives +27 correct answers on the 394 questions relative to the archived chain of thought, against +3 for each alternative. On the three Qwen2.5 tiers, certify-abstain is the only arm that reaches zero contradicted claims, at 9.2-22.9% coverage; it is an offline selection rule. Matching each refuted claim to the rewrite leaves more refuted claims unmatched than restated as consistent (689 against 278 at 7B for ungated repair). The adapted baselines are tool-free adaptations, as stated in Section 4. These are the results of the specific variants under the current pool and budget.
Single greedy runs. Appendix B measures the launch-to-launch floor. Two launches on one box agree on every question (spread 0.0 points). Across boxes, up to 153 of 887 answers change correctness, and the net accuracy spread is at most 2.0 points. The five older comparisons that cross machines are marked with a dagger in the paper; the three new Appendix E arms ran in one generation.