- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the classification of surprising observations in time series as faulty observations, state displacements, or rule changes. The authors contribute synthetic benchmarks in which these three possibilities occur with prior uniform probabilities. They find that a linear probe, LocusProbe, trained on Llama 3.1 8B's hidden states, can make this diagnosis more reliably than the base LLM prompted without thinking on the synthetic benchmarks. Alternatively, they train a LoRa adapter to match the exact posterior on the synthetic data, and they show that the adapter beats the base model on real (recorded-fault) weather data, water-circulation data, computing cluster data, an industrial-control testbed, and aircraft flights with failures (Table 7). The impact of this diagnosis is further measured in revised forecasts based on the updated state level or new rule.
Critical strengths
- The authors develop a synthetic dataset such that training produces an adapter that is more useful than non-adaptation to classify surprise observations with an LLM.
Critical weaknesses
-
No serious baselines are reported. The LLMs are forced to choose a diagnosis in the next token with no chain of thought. At the very least, chain of thought should be compared.
-
Additionally as baselines, a statistical or rule-based method, operating on the values of the time series and implementing the principles used to compute the exact posterior, should be compared. Given the possibility of using the time series values directly, it is not clear why an LLM or a probe should be used to begin with, and hence the impact of the research is questioned.
-
The modeling and synthetic data assume that observation faults or state disturbances operate at a single reading. In practice, future observations are also likely to be corrupted if a sensor becomes faulty, and a state disturbance such as turbulence of an aircraft may displace the dynamics differently at many consecutive observations.
-
The family of rules being considered is never stated. In the synthetic example in Table 1, the surprise reading at time point 4 could be explained by multiplication by 11/6, not just by addition of 5. However many time points are observed, infinitely many rules may exist to explain all the readings, unless the rule family is limited to begin with.
Questions for authors
-
Why is only a PMT adaptor and not LocusProbe tested on the real (recorded-fault) datasets (Table 7)?
-
Layer choices seem inconsistent. Layer 16 is shown to have minimal effect compared to later layers for residual patching (line 275, Table 18), but layer 16 is used for the probe (lines 297, 798, 1215).
-
On lines 144-150, it would be better to say the exact posterior matches the prior. It is uniform only in the setting of the synthetic benchmark. The comments about the next reading depend on weaknesses #3 and #4. A study of the robustness of LocusProbe and PMT when the class priors or fault duration are varied, or with repeated disturbances, or with richer rule families, would strengthen the paper.