- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Fuzzy-LLM, a framework that uses a frozen pretrained LLM to construct a standalone fuzzy classifier from labeled numerical examples. Numerical features are normalized and mapped to five predefined linguistic terms. The resulting descriptions, together with the observed training labels, form IF–THEN rules. When the same linguistic antecedent is associated with multiple class consequents, the authors consider no correction, hard correction (selecting one class), and soft correction (assigning linguistic certainty terms to candidate classes).
The LLM subsequently translates the rules into scikit-fuzzy statements. At inference time, a standard Mamdani system computes feature memberships, rule activations, aggregated class outputs, and centroid-based scores, without further LLM calls. The paper evaluates the approach on five small numerical tabular datasets (Iris, Ecoli, Glass, Seeds, and UKM), and studies correction strategies, membership-function families, and offline construction cost.
Critical strengths
1. Separation between offline construction and explicit inference
The framework separates LLM-assisted rule construction from standalone fuzzy inference. Once constructed, the classifier no longer requires LLM calls, and its predictions can be traced through explicit membership functions, rule activations, and class scores.
This provides a practical degree of execution transparency, rather than relying on post-hoc verbal explanations.
2. Transparent reporting of component behavior
The authors systematically compare different conflict-correction strategies and membership functions, including negative results where correction reduces accuracy.
The acknowledgement of these limitations is valuable, although the experiments do not yet explain the underlying mechanisms or establish when each strategy should be preferred.
Critical weaknesses
1. Insufficient research motivation and limited incremental methodological contribution
My primary concern is that the paper does not sufficiently justify why bridging LLMs and fuzzy logic is necessary or beneficial for the stated classification problem. Connecting two existing paradigms is not inherently a research contribution; what matters is whether their combination addresses a meaningful limitation of existing methods or provides a demonstrable advantage.
In particular, the LLM's role appears largely replaceable by deterministic procedures. The numerical-to-linguistic conversion in Eq. 2 is a predefined nearest-center assignment, while the subsequent code-generation stage translates specified rules into standard scikit-fuzzy statements. The most plausible source of LLM-specific value is therefore knowledge-dependent conflict correction, yet the paper lacks a matched deterministic rule-construction baseline to establish this contribution.
Similarly, the use of fuzzy inference is not sufficiently motivated relative to simpler crisp rule systems or other explicit classifiers. The existing membership-function comparisons do not isolate the benefit of fuzzy reasoning itself. The distinction from existing LLM-assisted offline construction approaches, including FeatLLM, also remains primarily architectural rather than empirically substantiated.
Finally, the claimed interpretability advantage requires clearer characterization. The paper exposes the execution process of the constructed classifier, but this does not explain the LLM's rule-construction decisions or necessarily establish meaningful decision attribution. The contribution would be more convincing if the authors demonstrated how this inspectability provides practical advantages over existing explicit rule-based methods.
Overall, I find the proposed framework to be a relatively straightforward integration of existing components, without sufficiently demonstrated methodological novelty or practical advantages.
2. Insufficient quantitative validation and experimental controls
The evaluation does not adequately characterize the reliability of the LLM-assisted construction pipeline. The authors acknowledge that LLMs may introduce errors during numerical-to-linguistic conversion, but do not quantitatively report violation rates, code-generation and execution success rates, or semantic consistency between linguistic rules and their executable counterparts. The validation harness, retry mechanisms, and possible automatic or manual repairs also require clarification.
The conflict-correction mechanism raises additional concerns. Conflicting consequents may arise naturally from coarse quantization rather than incorrect labels. It is unclear how the method distinguishes these cases, whether class frequencies and numerical evidence are preserved, or why LLM-generated certainty assignments should be preferred to simpler data-driven alternatives.
Moreover, the component ablations may not sufficiently isolate the intended effects. Tables 4 and 5 do not clearly establish whether identical initial rule bases and executable artifacts are shared across configurations. Without these controls, differences attributed to correction strategies or membership functions could be confounded by LLM generation variability.
Finally, the empirical evidence is limited. The reported configurations are selected using test accuracies, repeated-run uncertainty is unavailable, and the direct-LLM baselines do not receive matched supervision. The five small tabular datasets also provide limited evidence of broader applicability. While these limitations are partially acknowledged, they substantially weaken the empirical support for the proposed method.
Consequently, the current results do not convincingly establish either the incremental value of LLM guidance or the reliability of the end-to-end framework.
Questions for authors
Overall
My current recommendation is 1 (clear rejection). The main difficulty is not the clarity of the pipeline or the correctness of standard Mamdani operations, but the limited incremental contribution and the lack of targeted evidence for the value and reliability of LLM-assisted construction.
Q1, Q2, Q3, and Q4 are the most consequential to my assessment. A convincing matched non-LLM comparison, quantitative construction-reliability evidence, clarification of the existing ablation controls, and a properly supported evaluation could substantially affect my judgment. Q5–Q8 address the scope and interpretation of the central claims. I do not expect the authors to undertake an open-ended collection of additional experiments, and I distinguish requests for clarification of existing results from requests for new evidence.
Question
Q1. Evidence/Ablation: What is the actual incremental value of the LLM–fuzzy combination?
Could the authors compare the proposed method with a deterministic constructor that implements Eqs. 2–3 exactly, uses the same training data and Mamdani backend, and handles conflicting consequents using simple data-based rules such as class counts or majority voting? This seems to be the most direct way to isolate the value of LLM guidance from that of the existing fuzzy classifier.
Moreover, what specifically motivates graded Mamdani inference rather than a crisp rule-based classifier or another explicit numerical predictor? I am not asking for proof that fuzzy logic is the only valid option, but for evidence that this design choice is consequential for the stated objectives.
Q2. Quantitative evidence: How reliable is the LLM-based rule-construction pipeline?
Could the authors report the observed rates of incorrect nearest-center assignments, violations of the allowed rule syntax or consequent constraints, code-generation failures, and execution failures? In addition to code execution success, what is the rate of semantic equivalence between intended linguistic rules and generated executable statements?
What validation harness is used at each stage? Which failures trigger retries or automatic repair, is manual intervention allowed, and are failed attempts included in the accounting? Distinguishing first-attempt success from eventual success after a fixed retry budget would be useful.
Q3. Clarification: Do the existing component ablations share identical intermediate artifacts?
For Table 4, do the no-correction, soft-correction, and hard-correction branches start from the same cached initial rule base for each dataset and backbone? For Table 5, are the linguistic rules, chosen consequents, and generated code structures held fixed, with only the membership functions changed?
If any rules or code are regenerated independently, how is stochastic construction variability controlled? This question matters because the conclusions attributed to a single component require the remaining parts of the system to be held fixed.
Q4. Evidence: How robust are the reported performance comparisons under matched supervision and independent model selection?
Could the authors clarify whether direct-LLM/ICL baselines receive the same labeled examples and domain descriptions, and provide a matched comparison if available? Which data were used to choose the backbone, correction strategy, and membership-function family? The current best-per-dataset values were selected using test accuracies, so they should not be interpreted as independent test estimates after model selection.
Have repeated constructions or alternative data splits been evaluated? Uncertainty estimates would be particularly useful because construction involves LLM-generated artifacts, and several reported differences are small.
Q5. Clarification/Evidence: What exactly is being corrected when consequents conflict?
Does the conflict-correction prompt receive per-antecedent class frequencies, original numerical observations, or only the quantized antecedent, candidate class set, and feature/class descriptions? How does the method distinguish a genuine conversion error from multiple valid labels that happen to fall into one coarse linguistic region?
What justifies replacing conflicting labels with a single LLM-selected class, or mapping LLM-assigned certainty terms to fuzzy output sets, instead of retaining the observed data frequencies? Clarifying this would help interpret why correction improves Iris but consistently reduces Glass accuracy.
Q6. Clarification/Evidence: What level of inspectability is actually claimed and evaluated?
Is the intended contribution simply the ability to trace the deployed Mamdani computation, or a stronger form of human interpretability, decision attribution, or auditability of the construction process? The difference matters because the fuzzy execution is inspectable by design, whereas the LLM's choice of linguistic descriptions and corrections is not explained by the output trace.
Could the authors provide quantitative characteristics of the constructed rule bases (size, redundancy, number of conflicting rules, and the number of rules that actually alter aggregated outputs) or a concrete example of how inspection reveals and fixes an incorrect decision? In a max-aggregated system, an activated rule may be redundant with another rule.
Q7. Clarification/Evidence: How is semantic knowledge grounded, and what happens without meaningful feature names?
What information about the original feature scales, normalization, and class descriptions is supplied to the LLM? Does the method distinguish a value that is 'large' relative to the normalized dataset from a value that is large in the domain-specific sense?
Would the correction mechanism still provide value when features are anonymized or engineered and class names are abstract identifiers? An anonymization experiment that leaves the numerical data and labels unchanged would help isolate whether pretrained semantic knowledge is contributing to the performance.
Q8. Characterization: How does the system behave as the rule base or feature dimension grows?
The experiments cover 4–9 features and relatively small training sets. Could the authors describe how rule count, conflict frequency, construction failure rate, query coverage, and prediction cost change with the size and dimensionality of the input? The initial rules contain a term for every feature, while the linguistic vocabulary is fixed to five levels per feature, so both representational granularity and rule-base complexity may matter.
If low rule coverage or undefined output scores occur, how are they handled? The reported cost example also covers only a limited construction workload; are validation, retries, and failed attempts included in end-to-end cost comparisons?
Q9. Clarification: Are the baseline outcomes and preprocessing protocol sufficiently diagnosed?
Table 3 shows identical Ecoli accuracies of 42.86% for seven of the eight fine-tuned Qwen settings, and several Seeds accuracies of 33.33%. This does not by itself establish a bug or prediction collapse, but can the authors provide class-prediction distributions, invalid-output rates, or label-parsing details to clarify these results?
Could the authors also specify whether normalization statistics were fitted only on the training partition, and how train/test splits and prompts were fixed? These details are relevant to interpreting the empirical comparisons, not allegations of leakage or misconduct.