acceptodds
Under review as a conference paper at ICLR 2027

J-Miner: Recovering the Decision Logic of Fine-Tuned LLM Classifiers as Compact Rules

Abstract

Task-fine-tuned large language model (LLM) classifiers acquire task-specific decision knowledge, but this knowledge remains implicit in distributed internal computations, making their decision logic difficult to interpret. We introduce the Executable Decision Compression (EDC) framework and propose J-Miner, which mines vocabulary-named variables from internal readouts and learns rules shared across inputs to produce executable explanations. Analysis reveals that a small set of these variables captures much of the classifier's decision behavior, holding for both varying parameter scales within a family and distinct families. Across six binary tasks, a rule using just one variable reproduces 76.7% of source-classifier decisions on average, rising to 88.8% with 16 variables. Most of the decision information retained by these variables comes from internal activations beyond literal surface matching. A lightweight text reader predicts the variable states, allowing the same fixed rules to execute independently of the source classifier.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Model Interpretability, Executable Explanations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_2TVxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes J-Miner, an Executable Decision Compression (EDC) framework that extracts vocabulary-named variables from hidden representations and learns compact Boolean or linear decision rules. Across nine classification tasks and multiple model families, 16-variable rules achieve 88.8% average fidelity on six binary tasks. However, high behavioral fidelity does not establish causal faithfulness or semantic interpretability. Stronger causal validation and comparisons with competitive interpretability methods are needed to substantiate its claims.

Critical strengths
  1. Recovering interpretable and executable rules from fine-tuned LLM classifiers is important for model transparency and auditing.
  2. The method integrates internal feature extraction, variable selection, and rule learning into a unified approach.
  3. Experiments across multiple tasks and model families demonstrate consistent improvements over surface-word baselines.
  4. Standalone execution and the correction–harm decomposition provide insights into the relationship between internal-state reconstruction and behavioral fidelity.
Critical weaknesses
  1. High behavioral fidelity does not establish that the extracted rules capture the classifier's actual decision mechanism. The deletion experiments provide only indirect causal evidence.
  2. Comparisons with established rule-extraction and concept-based methods are limited. Vanilla logit-lens and SAE features also achieve higher fidelity than J-Lens.
  3. Vocabulary-named variables are not necessarily semantically meaningful or consistent across inputs.
  4. The standalone reader improves prediction fidelity despite lower concept-state reconstruction accuracy, with performance comparable to conventional knowledge distillation.
  5. Robustness under distribution shifts, different fine-tuning seeds, and larger models remains unexplored.
Questions for authors
  1. Can the authors provide stronger causal interventions to validate the recovered decision rules?
  2. How does J-Miner compare with established rule-extraction and concept-based methods under matched complexity?
  3. What measurable interpretability advantages does J-Lens offer over vanilla logit-lens and SAE features?
  4. How is the semantic consistency of vocabulary-named variables validated?
  5. Why does standalone execution improve behavioral fidelity while reducing concept-state fidelity?
  6. How robust are the recovered rules across distribution shifts, training seeds, and larger models?
Reviewer_mYvLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Executable Decision Compression (EDC) framework and J-Miner, a method to recover the implicit decision logic of fine-tuned LLM classifiers into compact, executable rules over vocabulary-named internal variables. By projecting intermediate representations into vocabulary space via the Jacobian lens, J-Miner identifies a minimal set of internal evidence variables without auxiliary feature dictionaries. Experiments across diverse binary and multiclass tasks show that a tiny variable budget reproduces most classifier decisions, primarily driven by non-literal internal activations. Furthermore, a lightweight text reader enables the recovered rules to execute independently with high fidelity.

Critical strengths
  • This approach extracts explicit and inspectable decision rules and further trains a lightweight text reader, enabling the recovered rules to execute independently of the source LLM classifier.、
  • By extracting internal readouts (or internal evidence events), the method identifies variables associated with the source classifier's predictions rather than relying on literal surface keyword matching alone.
  • The experiments provide broad coverage, systematically evaluating alternative internal observation protocols, varying variable budgets, and multiple standalone rule execution schemes.
Critical weaknesses
  • While the rule-guided deletion experiments support the behavioral relevance of the selected readouts, they provide limited evidence about whether the recovered rules reflect how the source classifier combines internal evidence. The authors appropriately acknowledge that each deletion can affect multiple variables. Further validation would help clarify what aspects of the source model’s decision process the rules explain beyond predictive association.
  • Table 2 reports no statistically significant source-classifier fidelity difference between J+Aux and Plain KD. This does not undermine the method’s potential interpretability value, but makes it important to establish the benefits of the explicit rule interface. The current evaluation does not directly show whether these rules help users understand, diagnose, or intervene on model behavior more effectively than direct distillation, particularly given the moderate reconstruction quality of the source internal states.
Questions for authors

If J-Miner does not significantly outperform standard knowledge distillation (Plain KD) in predictive fidelity, and the standalone reader achieves only moderate reconstruction of the source internal states (as shown by concept macro-F1), what concrete advantages do the resulting explicit rules offer in terms of interpretability and practical deployment that unconstrained direct distillation cannot provide?

Reviewer_RJ1jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes J-Miner, a framework for recovering compact decision rules from fine-tuned LLM classifiers. The method extracts vocabulary-named variables from intermediate hidden states using Jacobian-lens readouts, selects informative variables, and fits Boolean or linear rules to reproduce the source classifier's predictions. A lightweight text encoder is further trained to predict these variables, allowing the rules to be executed independently of the original model.

Experiments across multiple classification tasks and model families show that a small number of variables can achieve relatively high behavioral fidelity.

Critical strengths
  1. The recovered rules are compact and easy to inspect. Across several classification tasks and model families, a small set of variables reproduces much of the source classifier's behavior. I find this a useful empirical result: the rules provide a concrete summary of predictions that can be examined across examples.

  2. The standalone execution setup is a useful extension. A lightweight text encoder supplies the variables to a fixed rule, allowing it to run without the original LLM. I also appreciate that the evaluation reports both prediction agreement and concept reconstruction, making it possible to see when matching the classifier's outputs does not mean preserving its internal evidence.

Critical weaknesses
  1. My main reservation is about what has been recovered. High prediction agreement supports the rules as behavioral summaries, but does not establish that they capture the model's actual decision process. The word-deletion experiments help, although a deletion can change several internal variables at once. The paper acknowledges this limitation; I would keep the interpretation at that level. Direct feature interventions would be interesting follow-up work, but I do not see them as necessary for the present behavioral result.

  2. Table 2 raises a related issue for standalone execution. J+Aux has similar fidelity to plain distillation, with no significant difference detected, while concept F1 falls from .686 to .655 relative to J-only. The paper explains how verdict-aligned state changes can improve fidelity. That explanation is useful, but the higher fidelity should not be read as better recovery of the original internal evidence.

  3. I also found the theory mostly explanatory: several results follow from standard properties of binary features and linear classifiers. I see the empirical framework as the stronger contribution.

Questions for authors
  1. What interpretation of "decision logic" do you intend beyond a compact summary of predictions? Please clarify how this is supported by the fidelity and word-deletion results.

  2. Given the lower concept F1 with J+Aux, which standalone configuration would you recommend when preserving the original evidence states matters more than maximizing prediction agreement?

Reviewer_2Nk9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces J-Miner, a method for approximating the decisions of fine-tuned LLM classifiers with compact, executable rules. It projects internal hidden states into vocabulary space, selects a small set of token-named activation variables, and fits Boolean rules or linear scorecards to reproduce the classifier’s predictions. Across six binary tasks, one variable achieves 76.7% agreement with the source classifier on average, increasing to 88.8% with 16 variables. These variables capture internal signals beyond whether their names appear in the input. The authors also train a small text encoder to predict the variable states, allowing the fixed rules to run without the original LLM. The results suggest that much of these classifiers’ behavior can be approximated through a small, inspectable set of internal readouts.

Critical strengths

The strongest result is that a small, fixed set of internal variables can reproduce much of a fine-tuned classifier’s behavior across inputs. The authors test this across several tasks, model families, and readout methods, and compare against equally sized surface-word rules and random variable selections. This makes compact recoverability a well-supported empirical finding.

Critical weaknesses

My main concern is how much interpretability the vocabulary names actually provide. The paper correctly defines them as readout events rather than ordinary word meanings, but this leaves a gap between an executable rule and an understandable explanation of the classifier. The deletion experiments establish behavioral relevance of the selected text positions, but do not isolate the contribution of individual internal variables.

Questions for authors

What evidence supports interpreting a selected variable through its printed vocabulary name, beyond its association with the classifier’s label? Also, how stable are the selected variables and fitted rules across different construction samples or fine-tuning seeds? Substantially different rules with similar fidelity would affect whether I view the result as recovering consistent decision structure or finding one of several predictive approximations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.