acceptodds
Withdrawn from ICLR 2027

J-FDCS: Jacobian Fault Diagnosis Concept Spaces for Interpreting Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

When a language model diagnoses a machine from vibration signals, it remains unclear whether it represents the underlying fault or merely learns to produce the expected answer. Diagnostic accuracy alone cannot resolve this distinction. We introduce Jacobian Fault Diagnosis Concept Spaces (J-FDCS), a framework for examining fault-related representations in signal-grounded language models. Models receive vibration signals with equipment and task context and learn to answer with abstract codes. We analyze their hidden states using natural fault expressions. J-FDCS constructs expression-associated directions through the average Jacobian Lens or a full-sequence gradient formulation. It then removes their linear overlap with specified task-control and answer-code directions. The resulting low-dimensional space supports complementary tests of readability, semantic correspondence, signal dependence, cross-state stability, and local output effects. We systematically evaluate Qwen, Llama, and Mistral on five public rotating-machinery benchmarks and an in-house BW250 drilling-pump vibration dataset. The results distinguish strong linear class decodability from the variable readout of natural fault expressions. Fixed-space perturbations reveal input-responsive readouts, while cross-state transfer and retrospective BW250 interventions show variable semantic retention and local changes in diagnostic answer scores. Beyond predictive accuracy, J-FDCS provides a way to examine what fault-related information a model represents, which signals it depends on, and how changing it affects diagnosis.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, concept representations, multimodal learning, causal interventions, fault diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TQN9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduce J-FDCS, a framework to test whether LLMs fed with vibration signals represent the fault or just the answer. Models are trained to answer with abstract codes (A/B/C), and hidden states are then read with natural fault words (piston, valve, inner race).

Directions come from an average Jacobian lens or a full-sequence gradient. They are projected to remove overlap with task and answer-code directions. The authors test readout, signal dependence, cross-seed transfer and interventions on LLM models like Qwen, Llama and Mistral, with five public datasets plus an in-house pump dataset (BW250). Most positive evidence comes from one BW250 valve-fault case.

Critical strengths

The question is a good one, does correct diagnosis mean the model represents the fault? Separating answer codes from analysis words is a sensible idea and touch several scientific domains. The authors are also fairly honest about limits (post hoc cases, non-independent seeds, no multiplicity correction). The abstract is well written.

Critical weaknesses
  1. The framework is not formalized and the validity is unclear. The method section is about one page. There is no formal problem statement, no definition of what "represents a concept" means, and no result. Causality is cited (causal abstraction) but never formalized, is it causality in the Pearl's sense ? Granger ? or from computational mechanics ? and the intervention is a single-layer centroid shift.

  2. There is no positive control (a case where the concept is known to be represented) and no check that a layer-2 direction is not just reading the frozen encoder embeddings.

  3. The evidence does not support the claims. Outside one post hoc BW250 case, results look like chance. 75 of 153 states are above 0.5 AUROC, and 9 of 18 medians in Table C.1 are below 0.5. In Table C.1, ridge probes reach about 1.0 while the proposed readout stays near 0.5. Training on the correct task often lowers readout versus the untrained reference, and label-permuted BW250 models read out as well or better in some cases. On MCC5-THU and Paderborn (90 of 153 states) the models barely solve the task, so it is hard to say what is being interpreted. Interventions move answer scores by hundredths of nats per token, and Table C.3 shows they are not better than random or unrelated controls (37 of 54 are significantly worse than the answer-code control). There is also no result on whether the predicted diagnosis changes.

Some minor points:

Sec. 3 introduces CWRU/MCC5 and the encoder before the data and problem are even explained. Citations are missing in the intro. Figs. 2 to 5 are too small and could be merged into one full-width figure. Several of the key results (assignment tests, control readouts) are not reported. This add a lot of overhead to the reader and makes the paper very difficult to grasp.

Questions for authors
  • Can you give a clear problem statement: input shapes, labels, what a "concept" is, and what claim J-FDCS licenses? Using proper notations

  • Is the "answer" in Eq. 4 the code or the fault word? Does the predicted code actually flip under intervention, and at what rate?

  • Where are the results of the K!−1 assignment test and of the random/unrelated readout controls?

  • How do you justify bootstrapping over only 4 operating-condition groups?

  • Why does permuted-label training give equal or higher readout than correct training on BW250?

Some suggestions:

  • Describe which causal framework you use (SCM, interchange interventions, or something else) and formalize it.
  • Describe the code bottleneck formally (e.g., information bottleneck) and what it forces the hidden state to keep. That would be an interesting theoretical link from an information theory perspective.
Reviewer_YqsdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes J-FDCS, a framework for analyzing fault-related representations in signal-grounded language models. To address the ambiguity where diagnostic answer words are used both as supervision targets and analysis targets, the authors separate the supervised answers (abstract codes A/B/C) from the analysis vocabulary (natural fault expressions). They construct fault-expression directions via Jacobian lenses, remove linear overlaps with task and answer-code directions, and evaluate the internal representations using five complementary tests: readability, semantic alignment, signal dependence, cross-state stability, and causal influence.

Critical strengths

The separation of supervised answer codes from natural-language fault expressions is a clever design choice that addresses vocabulary confounding in interpretability research. This design reduces the possibility that reading out the “valve” direction merely reflects the model's preparation of its expected answer, rather than genuine fault-related information.

Critical weaknesses

1.The paper places disproportionate emphasis on a favorable BW250 Llama valve-fault case, while the aggregate results are substantially less convincing. Figure 2(c) shows that only 75 out of 153 trained states achieve a full-depth macro-AUROC area above 0.5. Furthermore, Table C.1 reveals that J-FDCS readout performance is close to or below chance level in many configurations.

2.The intervention experiments in Section 4.6 fail to demonstrate reliable target-directed control of the diagnostic outputs. According to the proposed methodology, translating hidden representations along a source-to-target class-centroid direction is expected to increase the model's relative preference for the target diagnosis. However, the experimental results do not support this claim.

3.The authors should better clarify the potential practical use cases of J-FDCS and explain what actionable insights the framework provides beyond existing representation analysis methods.

4.The paper suffers from substantial presentation and organization issues, which make it difficult to follow even after careful reading.

Questions for authors
  1. Given the weak overall readout performance across models and datasets, how representative is the selected BW250 valve-fault case?

  2. Given the unreliable intervention results in Section 4.6, what practical benefits does J-FDCS offer beyond existing interpretability methods?

Reviewer_1VQpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces J-FDCS for studying fault-related representations in signal-grounded language models. The models are trained to predict abstract codes, while natural fault expressions are used to construct Jacobian directions. The paper evaluates readability, semantic correspondence, signal dependence, cross-state transfer, and intervention effects across three language models and six datasets.

Critical strengths

1.The paper asks an important question. Good diagnostic accuracy does not necessarily mean that a model has learned a meaningful fault concept.

2.Using abstract codes for training and natural fault expressions for analysis is a thoughtful design. It reduces the chance that the method only detects preparation of the supervised answer.

3.The authors use several complementary tests rather than relying only on a linear probe. The paper also reports negative and inconsistent results honestly.

Critical weaknesses
  1. The overall evidence for stable fault concepts is weak. Only 75 of the 153 trained states have a J-FDCS readout above 0.5. Many median results are close to chance, and task adaptation sometimes reduces the readout. Signal replacement and cross-state transfer are also inconsistent across datasets.

  2. The strongest result is based on a post hoc selected BW250, Llama, valve case. The appendix states that this is not an independent blind test. This makes it difficult to know whether the same conclusion holds beyond the selected example.

  3. The intervention results do not provide convincing causal evidence. The effects are small and often non-monotonic. None of the 54 direction–state combinations is significantly better than all five controls. The method therefore finds directions that can affect outputs, but it does not show that these directions play a reliable role during normal diagnosis.

Questions for authors

1.Can the authors define the case and layer selection procedure in advance and apply it to all model–dataset configurations, including unsuccessful cases?

2.What conclusion should be drawn from configurations where J-FDCS is near chance or becomes weaker after task adaptation?

3.Can the authors provide a statistical test showing that J-FDCS is consistently better than the matched random and unrelated-expression controls across all configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.