- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduce J-FDCS, a framework to test whether LLMs fed with vibration signals represent the fault or just the answer. Models are trained to answer with abstract codes (A/B/C), and hidden states are then read with natural fault words (piston, valve, inner race).
Directions come from an average Jacobian lens or a full-sequence gradient. They are projected to remove overlap with task and answer-code directions. The authors test readout, signal dependence, cross-seed transfer and interventions on LLM models like Qwen, Llama and Mistral, with five public datasets plus an in-house pump dataset (BW250). Most positive evidence comes from one BW250 valve-fault case.
Critical strengths
The question is a good one, does correct diagnosis mean the model represents the fault? Separating answer codes from analysis words is a sensible idea and touch several scientific domains. The authors are also fairly honest about limits (post hoc cases, non-independent seeds, no multiplicity correction). The abstract is well written.
Critical weaknesses
-
The framework is not formalized and the validity is unclear. The method section is about one page. There is no formal problem statement, no definition of what "represents a concept" means, and no result. Causality is cited (causal abstraction) but never formalized, is it causality in the Pearl's sense ? Granger ? or from computational mechanics ? and the intervention is a single-layer centroid shift.
-
There is no positive control (a case where the concept is known to be represented) and no check that a layer-2 direction is not just reading the frozen encoder embeddings.
-
The evidence does not support the claims. Outside one post hoc BW250 case, results look like chance. 75 of 153 states are above 0.5 AUROC, and 9 of 18 medians in Table C.1 are below 0.5. In Table C.1, ridge probes reach about 1.0 while the proposed readout stays near 0.5. Training on the correct task often lowers readout versus the untrained reference, and label-permuted BW250 models read out as well or better in some cases. On MCC5-THU and Paderborn (90 of 153 states) the models barely solve the task, so it is hard to say what is being interpreted. Interventions move answer scores by hundredths of nats per token, and Table C.3 shows they are not better than random or unrelated controls (37 of 54 are significantly worse than the answer-code control). There is also no result on whether the predicted diagnosis changes.
Some minor points:
Sec. 3 introduces CWRU/MCC5 and the encoder before the data and problem are even explained. Citations are missing in the intro. Figs. 2 to 5 are too small and could be merged into one full-width figure. Several of the key results (assignment tests, control readouts) are not reported. This add a lot of overhead to the reader and makes the paper very difficult to grasp.
Questions for authors
-
Can you give a clear problem statement: input shapes, labels, what a "concept" is, and what claim J-FDCS licenses? Using proper notations
-
Is the "answer" in Eq. 4 the code or the fault word? Does the predicted code actually flip under intervention, and at what rate?
-
Where are the results of the K!−1 assignment test and of the random/unrelated readout controls?
-
How do you justify bootstrapping over only 4 operating-condition groups?
-
Why does permuted-label training give equal or higher readout than correct training on BW250?
Some suggestions:
- Describe which causal framework you use (SCM, interchange interventions, or something else) and formalize it.
- Describe the code bottleneck formally (e.g., information bottleneck) and what it forces the hidden state to keep. That would be an interesting theoretical link from an information theory perspective.