- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies ordinal language-model outputs as reusable operands rather than terminal labels. It introduces the Differentiable Fuzzy Inference Layer (DFIL), which combines a conventional classifier with a supervised scalar and an ordered Gaussian membership bank, and develops formal results on finite-state refinement for proportional composition, margin-separated storage, and exact label reachability under shared parameter uncertainty. The empirical section evaluates algebraic procedures, scalar intervals on FRoG-Hard, matched fine-tuning controls, and several diagnostic studies. The paper is unusually explicit that its learned-bank-specific advantage is unresolved in the reported experiments.
Critical strengths
-
The central conceptual distinction is useful and clearly motivated. Figure 1 gives a concrete example in which two values with the same initial ordinal label yield different labels after multiplication by the same factor. This directly supports the paper's argument that a correct terminal category need not be a sufficient computational interface. The figure also clearly separates the retained classifier path from the scalar composition path.
-
The margin-separated formulation is a meaningful strengthening over standard quantization discussions. Equation (6) gives a concrete recurrence and a state-count characterization under a clearly specified contract, namely correctness for individual threshold queries outside a margin band. The paper is careful to distinguish this from reconstruction, average distortion, whole-label guarantees, and repeated rounding.
-
The paper is commendably candid about empirical limitations. Table 3 reports that the primary Natural-249 nearest-center accuracy comparison is unresolved and that ExactCount fails the prespecified marginal-invariance criterion. Table 4 similarly does not claim that DFIL dominates ordinary fine-tuning. This restraint increases confidence that the authors are not selectively presenting favorable diagnostics.
Critical weaknesses
-
The empirical evidence does not establish that the learned DFIL mechanism improves compositional ordinal reasoning relative to simpler scalar-supervised alternatives. This is the central practical weakness. The paper itself reports in Table 3 that the main Natural-249 nearest-center accuracy difference is percentage points with a confidence interval crossing zero, and that the ExactCount comparison is not a valid test of the intended hypothesis because both methods violate the marginal-invariance requirement. The human-judgment improvement is for a validation-selected probability mixture, which the paper explicitly says is not certified and does not isolate learned-bank effects.
-
The calibration experiment certifies agreement with a model-defined numerical reference, not semantic correctness of nested quantifier reasoning. In Table 2, the reference labels are produced by the deployed bank, the chain is specified externally, and conformal calibration is only applied to the base scalar. This is appropriate for testing the stated Corollary B.6-style numerical coverage contract, but it does not validate that the scalar, the learned centers, or the supplied composition structure correspond to human interpretations of nested natural-language quantifiers. The paper states this limitation, but the abstract and introduction should be correspondingly careful not to let “quantifier reasoning” suggest a validated semantic solution.
-
The experimental comparison protocol is difficult to interpret because the scalar gradient treatment changes across datasets. Section 3.1 states that is a forward-pass identity but preserves its input gradient for the human-probability protocol while stopping it for few-shot and counting controls. Consequently, the MF loss has different optimization effects across the evaluations. This may be justified by task-specific design, but it creates an additional confound when interpreting comparisons of learned-bank effects. The paper should report a consistent-gradient ablation and clearly identify, for every result table, which parameters receive gradients from each term in Equation (1).
Questions for authors
-
For Equation (1), why are used directly as CE logits rather than using , normalized memberships, or a temperature-scaled distance logit? Please provide an ablation comparing these choices, including learned centers, widths, scalar MSE, nearest-center accuracy, and compositional accuracy.
-
What experiment demonstrates that the learned bank, rather than scalar supervision plus a fixed ordinal codebook, improves compositional behavior? Table 3 currently reports no resolved primary nearest-center gain, and the ExactCount comparison fails its validity criterion. A response with a preregistered or held-out nested-expression evaluation would substantially increase my confidence.
-
Are the chains and all uncertainty choices in Table 2 fixed independently of the calibration/evaluation rows? Please report confidence intervals for coverage, certification frequency, and conditional error, and clarify whether calibration-set selection, model checkpoint selection, and mixture-weight selection are cleanly separated from the relevant test examples.
6 comments
Response to Reviewer EAqs (2/4): gradient paths and the requested ablation (W3)
W3: The gradient treatment of differs across protocols
Response. We agree that this confounds comparisons across protocols. The paper therefore labels the affected results diagnostic, and states that a complete claim about detachment requires a matched comparison with selection fixed in advance (Appendix F.1, "Regime dependence"). We ran that comparison, below, and add the requested table. Results from different protocols are never pooled, and no theorem depends on .
With an attached input, the membership loss can shape the extractor as well as the bank; with a detached input, it trains only the bank. Where present, main cross-entropy updates the classifier and LoRA, and scalar MSE updates the scalar head and LoRA. FT+Compose arms freeze the bank and have no membership term.
| Result | MF input (DFIL arm) | MF cross-entropy updates |
|---|---|---|
| Human judgments (Table 3) | attached | bank, scalar head, LoRA |
| Natural-249 (Table 3) | attached | bank, scalar head, LoRA |
| ExactCount (Table 3) | detached | bank only |
| Calibrated certification (Table 2) | detached | bank only |
| FRoG component study (Table 4) | attached (default; not logged per run) | bank, scalar head, LoRA under the default |
| Distillation follow-up (App. C.4) | attached | bank, scalar head, LoRA |
| Few-shot (App. F.1) | detached | bank only |
| Cross-task SST-5/STS22 (App. F.2) | not fully established, as stated there | not established |
| CSMC (App. F.5) | 15 attached, 6 detached reruns | as in the rows above |
| Other appendix DFIL runs (model scales, λ sweep, synthetic composition, IPR, CORAL comparison) | default (attached); detaching needs an explicit setting | bank, scalar head, LoRA under the default |
| Table 1; Human-Grid (App. E.3) | no model is trained | none |
The requested consistent-gradient ablation
On the human-judgment protocol (Appendix D.2), a third arm, DFIL with a detached input, uses the same 20 seeds and settings and is paired with the two reported arms. The criteria were fixed before running. A rerun of one seed reproduced both reported checkpoints bit for bit. With the input detached, LoRA, the classifier and the scalar head receive exactly the FT+Compose loss terms, and every arm selects checkpoints by the same objective.
| Confirmation MSE difference | Mean [95% CI] | Exact | Seeds below 0 |
|---|---|---|---|
| Detached DFIL − FT+Compose | −.000251 [−.000434, −.000068] | .0096 | 15/20 |
| Attached − detached DFIL | −.000213 [−.000494, +.000067] | .13 | 12/20 |
| Attached DFIL − FT+Compose (reported) | −.000464 [−.000756, −.000172] | .0018 | 15/20 |
The first two rows sum to the third, seed by seed. The detached arm passes all six original criteria. Detaching does not change MSE detectably (row 2), and a resolved gain remains without membership gradients into the extractor (row 1).
Post hoc (exploratory). To isolate the bank, we held each trained extractor fixed, swapped only the bank between the uniform one and the detached arm's learned one, and reselected the mixture weight by the same rule. The stored arms were reproduced exactly.
| Learned − uniform bank | Mean [95% CI] | Exact | Seeds below 0 |
|---|---|---|---|
| FT+Compose extractor | −.000086 [−.000206, +.000033] | .15 | 18/20 |
| Detached extractor | −.000064 [−.000168, +.000040] | .22 | 18/20 |
The learned bank lowers MSE in 18 of 20 seeds with either extractor, but both intervals include zero, so its own share is not resolved on these 20 seeds (Appendix D.2). It does improve the membership path itself: membership-only MSE is .049 with either extractor, against .058–.059 with the uniform bank.
Response to Reviewer EAqs (3/4): the requested ablation of the membership logits (Q1)
Q1: Membership, log-membership or distance logits
Response. We ran this ablation (Appendix C.1, new Table 8). The proposed alternatives give two distinct forms besides ours. Normalized memberships equal , so they train exactly like log-membership logits. The temperature-scaled distance logit , with learned, gives the same probabilities with one shared width. Using directly bounds the logits, so no MF class probability exceeds : the MF probabilities stay soft, whereas the log and distance forms can become arbitrarily confident.
The ablation uses the held-out human judgments of Appendix D.2 (Qwen2.5-1.5B, 20 paired seeds, all three forms retrained together, all other settings unchanged). The comparisons and endpoints were fixed before training.
| Membership (paper) | Log-membership | Distance, learned | |
|---|---|---|---|
| Selected-output MSE (Section 6 endpoint) | .00784 | .00972 | .01035 |
| Scalar-head MSE | .0200 | .0147 | .0152 |
| Nearest-center accuracy | 51.2% | 52.1% | 53.5% |
| Composed accuracy† | 74.1% | 75.8% | 75.4% |
| Interior centers (start .26/.50/.74) | .226/.495/.767 | .250/.486/.735 | .243/.487/.743 |
| Widths (start .12) | .09–.18 | .11–.14 | .13, shared |
†The protocol contains single expressions only. Composed accuracy applies one further factor to both and the human target, then scores as nearest-center accuracy does: the target's bin is fixed and shared by all arms, and is decoded with each arm's own centers.
Paired differences from the membership form, with 95% -intervals and exact sign-flip :
- Selected-output MSE: (log-membership) and (distance). The membership form is better in 20 and 19 of 20 seeds ().
- Scalar-head MSE: , , and , .
- Accuracies: log-membership raises composed accuracy by 1.7 points (), and the distance form raises nearest-center accuracy by 2.3 points (). The other two differences are not resolved: (log-membership, nearest-center) and (distance, composed).
The membership form thus gives the better output on the held-out human judgments. Under the unbounded forms the main path is less accurate (MSE .0095 and .0101 against .0089), and their sharper MF probabilities no longer improve it when mixed, as the bounded ones do. The unbounded forms give a more accurate scalar head and slightly better decoding. The theorems and certificates concern decoders applied to and hold for all three forms. The question exposed a trade-off, between the selected output and the scalar head, that is now part of the paper; we thank the reviewer for it.
Response to Reviewer EAqs (1/4): learned bank (W1/Q2) and calibration scope (W2)
We thank the reviewer for a careful and constructive review. Its requests were concrete, and they led to the analyses and changes below, all now in the paper:
The revised introduction opens with the distinction the reviewer highlights (Strength 1) and lists the margin result among its three contributions (Strength 2).
W1 / Q2: Learned bank versus scalar supervision with a fixed codebook
Response. No experiment establishes a learned-bank advantage in composition, and the paper does not claim one; the revised abstract states that this benefit is unresolved.
A preregistered or held-out nested-expression evaluation, as the reviewer proposes, would address this directly; the revision names such an evaluation as the main open empirical question.
Why the contribution does not rest on it. The alternative the reviewer names, scalar supervision with a fixed codebook, is the paper's matched control (FT+Compose): it keeps the same scalar supervision and product rule, and freezes the bank. Both arms thus use the interface the paper proposes; what remains open is whether its centers should be learned. The theorems hold for every ordered bank, fixed or learned, and the margin result the reviewer credits (Strength 2) is tight: no encoder, interval or not, meets its guarantee with fewer states (Theorem A.11). What the interface provides is measured on the trained models:
None of these depends on learning the bank. Learning matters to the analysis where it changes which result applies:
W2: The calibration experiment certifies a model-defined reference
Response. Agreed, and the revision adopts the reviewer's distinction between a numerical coverage contract and semantic correctness. Table 2 tests the numerical contract of Corollary B.6:
It does not show that the scalar, the centers or the chain match human interpretations of nested quantifiers. Accordingly, the revised abstract and introduction replace "quantifier reasoning" with "composition of numerically supplied quantifier meanings". They describe the certificates as numerical guarantees relative to a supplied chain, meanings and uncertainty set.