acceptodds
Under review as a conference paper at ICLR 2027

Beyond Labels: A Composable Ordinal Interface for Language Models

Abstract

An ordinal label can be correct yet insufficient for subsequent reasoning. We study this gap for language-model predictions that admit an underlying scalar quantity, using nested proportional composition as a concrete setting. We introduce the Differentiable Fuzzy Inference Layer (DFIL), a dual-path head that retains a standard classifier while exposing a supervised scalar and Gaussian memberships with learnable ordered centers. Rather than composing intermediate labels, DFIL performs proportional operations in scalar space and discretizes only the final result. For nearest-center decoding, we characterize exactly when intermediate labels suffice for further composition, construct minimal exact refinements for fixed finite proportional multiplier sets, and derive optimal state budgets for margin-separated threshold queries. For a fixed composition structure and a supplied Cartesian uncertainty box that preserves center ordering, we also derive the exact set of attainable nearest-center labels while accounting for parameters shared across composition and decoding. For the analyzed ordered-center families, we further reduce Gaussian label reachability to a two-variable feasibility problem without requiring globally monotone decoding. Experiments on quantifier reasoning and related ordinal tasks, including human-judgment data and matched controls, separately assess predictive accuracy, structural consistency, and the contribution of the learned membership bank. This work frames ordinal prediction as the design of an inspectable, composable interface rather than a terminal category decision.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
compositional reasoning, ordinal prediction, neuro-symbolic learning, large language models, differentiable fuzzy inference, quantifier reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EAqsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ordinal language-model outputs as reusable operands rather than terminal labels. It introduces the Differentiable Fuzzy Inference Layer (DFIL), which combines a conventional classifier with a supervised scalar and an ordered Gaussian membership bank, and develops formal results on finite-state refinement for proportional composition, margin-separated storage, and exact label reachability under shared parameter uncertainty. The empirical section evaluates algebraic procedures, scalar intervals on FRoG-Hard, matched fine-tuning controls, and several diagnostic studies. The paper is unusually explicit that its learned-bank-specific advantage is unresolved in the reported experiments.

Critical strengths
  1. The central conceptual distinction is useful and clearly motivated. Figure 1 gives a concrete example in which two values with the same initial ordinal label yield different labels after multiplication by the same factor. This directly supports the paper's argument that a correct terminal category need not be a sufficient computational interface. The figure also clearly separates the retained classifier path from the scalar composition path.

  2. The margin-separated formulation is a meaningful strengthening over standard quantization discussions. Equation (6) gives a concrete recurrence and a state-count characterization under a clearly specified contract, namely correctness for individual threshold queries outside a margin band. The paper is careful to distinguish this from reconstruction, average distortion, whole-label guarantees, and repeated rounding.

  3. The paper is commendably candid about empirical limitations. Table 3 reports that the primary Natural-249 nearest-center accuracy comparison is unresolved and that ExactCount fails the prespecified marginal-invariance criterion. Table 4 similarly does not claim that DFIL dominates ordinary fine-tuning. This restraint increases confidence that the authors are not selectively presenting favorable diagnostics.

Critical weaknesses
  1. The empirical evidence does not establish that the learned DFIL mechanism improves compositional ordinal reasoning relative to simpler scalar-supervised alternatives. This is the central practical weakness. The paper itself reports in Table 3 that the main Natural-249 nearest-center accuracy difference is −0.98-0.98 percentage points with a confidence interval crossing zero, and that the ExactCount comparison is not a valid test of the intended hypothesis because both methods violate the marginal-invariance requirement. The human-judgment improvement is for a validation-selected probability mixture, which the paper explicitly says is not certified and does not isolate learned-bank effects.

  2. The calibration experiment certifies agreement with a model-defined numerical reference, not semantic correctness of nested quantifier reasoning. In Table 2, the reference labels are produced by the deployed bank, the chain is specified externally, and conformal calibration is only applied to the base scalar. This is appropriate for testing the stated Corollary B.6-style numerical coverage contract, but it does not validate that the scalar, the learned centers, or the supplied composition structure correspond to human interpretations of nested natural-language quantifiers. The paper states this limitation, but the abstract and introduction should be correspondingly careful not to let “quantifier reasoning” suggest a validated semantic solution.

  3. The experimental comparison protocol is difficult to interpret because the scalar gradient treatment changes across datasets. Section 3.1 states that gg is a forward-pass identity but preserves its input gradient for the human-probability protocol while stopping it for few-shot and counting controls. Consequently, the MF loss has different optimization effects across the evaluations. This may be justified by task-specific design, but it creates an additional confound when interpreting comparisons of learned-bank effects. The paper should report a consistent-gradient ablation and clearly identify, for every result table, which parameters receive gradients from each term in Equation (1).

Questions for authors
  1. For Equation (1), why are μq(p^)∈(0,1]\mu_q(\hat p)\in(0,1] used directly as CE logits rather than using log⁡μq(p^)\log \mu_q(\hat p), normalized memberships, or a temperature-scaled distance logit? Please provide an ablation comparing these choices, including learned centers, widths, scalar MSE, nearest-center accuracy, and compositional accuracy.

  2. What experiment demonstrates that the learned bank, rather than scalar supervision plus a fixed ordinal codebook, improves compositional behavior? Table 3 currently reports no resolved primary nearest-center gain, and the ExactCount comparison fails its validity criterion. A response with a preregistered or held-out nested-expression evaluation would substantially increase my confidence.

  3. Are the chains and all uncertainty choices in Table 2 fixed independently of the calibration/evaluation rows? Please report confidence intervals for coverage, certification frequency, and conditional error, and clarify whether calibration-set selection, model checkpoint selection, and mixture-weight selection are cleanly separated from the relevant test examples.

6 comments
AuthorsOpenReview ↗

Response to Reviewer EAqs (1/4): learned bank (W1/Q2) and calibration scope (W2)

We thank the reviewer for a careful and constructive review. Its requests were concrete, and they led to the analyses and changes below, all now in the paper:

  • we ran the requested consistent-gradient ablation, analyzed under criteria fixed before running, and added a table of the parameters each loss term updates in every result (Part 2);
  • we ran the requested ablation of the membership logits (Q1), with comparisons fixed before training (Part 3);
  • we report intervals for Table 2's coverage, certification frequency and conditional error, now also over 11 further training runs, and answer whether calibration, checkpoint and mixture-weight selection are separated from evaluation (Part 4);
  • we made the wording change requested under W2 (below).

The revised introduction opens with the distinction the reviewer highlights (Strength 1) and lists the margin result among its three contributions (Strength 2).

W1 / Q2: Learned bank versus scalar supervision with a fixed codebook

Response. No experiment establishes a learned-bank advantage in composition, and the paper does not claim one; the revised abstract states that this benefit is unresolved.

  • Natural-249. The primary difference is −0.98-0.98 pp [−3.73,1.45][-3.73, 1.45], unresolved.
  • ExactCount. Both arms fail the marginal-invariance criterion, so the comparison does not test the hypothesis. Both recover the direction of the conditional change in about 99% of matched pairs, and their composed estimates beat a marginal shortcut in MAE (0.036 and 0.034 vs. 0.102). We do not use this as evidence about the bank.
  • Human judgments. On single expressions, the complete DFIL treatment lowers held-out MSE against FT+Compose (.007690 vs. .008154, exact p=.0018p=.0018); the scored output is a validation-selected mixture, which the certificates do not cover. The requested ablation shows that about half of this gain persists when the extractor receives exactly the FT+Compose loss terms; an exploratory bank swap leaves the learned bank's own share unresolved (Part 2).

A preregistered or held-out nested-expression evaluation, as the reviewer proposes, would address this directly; the revision names such an evaluation as the main open empirical question.

Why the contribution does not rest on it. The alternative the reviewer names, scalar supervision with a fixed codebook, is the paper's matched control (FT+Compose): it keeps the same scalar supervision and product rule, and freezes the bank. Both arms thus use the interface the paper proposes; what remains open is whether its centers should be learned. The theorems hold for every ordered bank, fixed or learned, and the margin result the reviewer credits (Strength 2) is tight: no encoder, interval or not, meets its guarantee with fewer states (Theorem A.11). What the interface provides is measured on the trained models:

  • Labels lose information. Even the best rule that sees only a correct FRoG-Hard label errs on 6.9–7.8% of one-step composed labels, and on 6.2–10.4% over all 13 trained banks (Appendix A.4, fixed before computing).
  • Certificates separate reliable outputs. Against the deployed bank's labels of the true proportions, certified composed outputs are wrong 0.5–6.5% of the time and the others 36–47% (Table 2); 11 further training runs reproduce these rates within one point (Part 4).
  • Exactness pays. Exact shared-center analysis certifies 0.8–3.3 points more one-step queries than separate interval propagation in every declared setting, under the same guarantee (Part 4).

None of these depends on learning the bank. Learning matters to the analysis where it changes which result applies:

  • The three learned 7B banks fail the exact global-order test (Theorem B.9). The reachability result that needs no global order (Theorem B.13) is therefore the one that covers the trained models.
  • Exactness is proved on DFIL's own center family (Theorem B.15). Its strict gaps change decisions relative to the enclosing closed box (example in Section 5.2).

W2: The calibration experiment certifies a model-defined reference

Response. Agreed, and the revision adopts the reviewer's distinction between a numerical coverage contract and semantic correctness. Table 2 tests the numerical contract of Corollary B.6:

  • the reference label is the deployed bank's label of the true composed proportion;
  • the chain is supplied;
  • only the base scalar is calibrated.

It does not show that the scalar, the centers or the chain match human interpretations of nested quantifiers. Accordingly, the revised abstract and introduction replace "quantifier reasoning" with "composition of numerically supplied quantifier meanings". They describe the certificates as numerical guarantees relative to a supplied chain, meanings and uncertainty set.

AuthorsOpenReview ↗

Response to Reviewer EAqs (2/4): gradient paths and the requested ablation (W3)

W3: The gradient treatment of gg differs across protocols

Response. We agree that this confounds comparisons across protocols. The paper therefore labels the affected results diagnostic, and states that a complete claim about detachment requires a matched comparison with selection fixed in advance (Appendix F.1, "Regime dependence"). We ran that comparison, below, and add the requested table. Results from different protocols are never pooled, and no theorem depends on gg.

With an attached input, the membership loss can shape the extractor as well as the bank; with a detached input, it trains only the bank. Where present, main cross-entropy updates the classifier and LoRA, and scalar MSE updates the scalar head and LoRA. FT+Compose arms freeze the bank and have no membership term.

ResultMF input (DFIL arm)MF cross-entropy updates
Human judgments (Table 3)attachedbank, scalar head, LoRA
Natural-249 (Table 3)attachedbank, scalar head, LoRA
ExactCount (Table 3)detachedbank only
Calibrated certification (Table 2)detachedbank only
FRoG component study (Table 4)attached (default; not logged per run)bank, scalar head, LoRA under the default
Distillation follow-up (App. C.4)attachedbank, scalar head, LoRA
Few-shot (App. F.1)detachedbank only
Cross-task SST-5/STS22 (App. F.2)not fully established, as stated therenot established
CSMC (App. F.5)15 attached, 6 detached rerunsas in the rows above
Other appendix DFIL runs (model scales, λ sweep, synthetic composition, IPR, CORAL comparison)default (attached); detaching needs an explicit settingbank, scalar head, LoRA under the default
Table 1; Human-Grid (App. E.3)no model is trainednone

The requested consistent-gradient ablation

On the human-judgment protocol (Appendix D.2), a third arm, DFIL with a detached input, uses the same 20 seeds and settings and is paired with the two reported arms. The criteria were fixed before running. A rerun of one seed reproduced both reported checkpoints bit for bit. With the input detached, LoRA, the classifier and the scalar head receive exactly the FT+Compose loss terms, and every arm selects checkpoints by the same objective.

Confirmation MSE differenceMean [95% CI]Exact ppSeeds below 0
Detached DFIL − FT+Compose−.000251 [−.000434, −.000068].009615/20
Attached − detached DFIL−.000213 [−.000494, +.000067].1312/20
Attached DFIL − FT+Compose (reported)−.000464 [−.000756, −.000172].001815/20

The first two rows sum to the third, seed by seed. The detached arm passes all six original criteria. Detaching does not change MSE detectably (row 2), and a resolved gain remains without membership gradients into the extractor (row 1).

Post hoc (exploratory). To isolate the bank, we held each trained extractor fixed, swapped only the bank between the uniform one and the detached arm's learned one, and reselected the mixture weight by the same rule. The stored arms were reproduced exactly.

Learned − uniform bankMean [95% CI]Exact ppSeeds below 0
FT+Compose extractor−.000086 [−.000206, +.000033].1518/20
Detached extractor−.000064 [−.000168, +.000040].2218/20

The learned bank lowers MSE in 18 of 20 seeds with either extractor, but both intervals include zero, so its own share is not resolved on these 20 seeds (Appendix D.2). It does improve the membership path itself: membership-only MSE is .049 with either extractor, against .058–.059 with the uniform bank.

AuthorsOpenReview ↗

Response to Reviewer EAqs (3/4): the requested ablation of the membership logits (Q1)

Q1: Membership, log-membership or distance logits

Response. We ran this ablation (Appendix C.1, new Table 8). The proposed alternatives give two distinct forms besides ours. Normalized memberships μq/∑jμj\mu_q/\sum_j\mu_j equal softmax(log⁡μ)\mathrm{softmax}(\log\mu), so they train exactly like log-membership logits. The temperature-scaled distance logit −(p^−cq)2/τ-(\hat p-c_q)^2/\tau, with τ\tau learned, gives the same probabilities with one shared width. Using μq∈(0,1]\mu_q\in(0,1] directly bounds the logits, so no MF class probability exceeds e/(e+Q−1)e/(e+Q-1): the MF probabilities stay soft, whereas the log and distance forms can become arbitrarily confident.

The ablation uses the held-out human judgments of Appendix D.2 (Qwen2.5-1.5B, 20 paired seeds, all three forms retrained together, all other settings unchanged). The comparisons and endpoints were fixed before training.

Membership μq\mu_q (paper)Log-membership log⁡μq\log\mu_qDistance, learned τ\tau
Selected-output MSE (Section 6 endpoint).00784.00972.01035
Scalar-head MSE.0200.0147.0152
Nearest-center accuracy51.2%52.1%53.5%
Composed accuracy†74.1%75.8%75.4%
Interior centers (start .26/.50/.74).226/.495/.767.250/.486/.735.243/.487/.743
Widths (start .12).09–.18.11–.14.13, shared

†The protocol contains single expressions only. Composed accuracy applies one further factor m∈1/4,1/2,3/4m\in\\{1/4,1/2,3/4\\} to both p^\hat p and the human target, then scores as nearest-center accuracy does: the target's bin is fixed and shared by all arms, and mp^m\hat p is decoded with each arm's own centers.

Paired differences from the membership form, with 95% tt-intervals and exact sign-flip pp:

  • Selected-output MSE: +.0019+.0019 [+.0015,+.0023][+.0015,+.0023] (log-membership) and +.0025+.0025 [+.0019,+.0032][+.0019,+.0032] (distance). The membership form is better in 20 and 19 of 20 seeds (p<10−5p<10^{-5}).
  • Scalar-head MSE: −.0053-.0053 [−.0090,−.0016][-.0090,-.0016], p=.003p=.003, and −.0048-.0048 [−.0085,−.0011][-.0085,-.0011], p=.004p=.004.
  • Accuracies: log-membership raises composed accuracy by 1.7 points [+0.1,+3.3][+0.1,+3.3] (p=.036p=.036), and the distance form raises nearest-center accuracy by 2.3 points [+0.2,+4.3][+0.2,+4.3] (p=.023p=.023). The other two differences are not resolved: +1.0+1.0 [−1.3,+3.2][-1.3,+3.2] (log-membership, nearest-center) and +1.3+1.3 [−0.4,+3.0][-0.4,+3.0] (distance, composed).

The membership form thus gives the better output on the held-out human judgments. Under the unbounded forms the main path is less accurate (MSE .0095 and .0101 against .0089), and their sharper MF probabilities no longer improve it when mixed, as the bounded ones do. The unbounded forms give a more accurate scalar head and slightly better decoding. The theorems and certificates concern decoders applied to p^\hat p and hold for all three forms. The question exposed a trade-off, between the selected output and the scalar head, that is now part of the paper; we thank the reviewer for it.

Reviewer_c6Q8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I cannot with confidence provide a precise summary of this papers contributions.

I have specified soundness as 1, though I note that state of the paper makes it hard to properly access the soundness. Similarly, while I am confident about the quality of the written work I am not familiar with the cited lit. or have substantial direct research experience.

Critical strengths
  • This work might contain potentially promising results, but I am unable to evaluate these claims given the current state of the written work.
Critical weaknesses
  • This work is exceedingly hard to read and understand. It lacks an introduction of the motivation and a clear introduction of the problem it attempt to solve and its practical application.
  • Central concepts required for understanding is never introduced, for instance, table 4 introduces the frog component, with the only previous mention of from being in line 374 where it is presupposed.
Questions for authors
  • Can the author clarify the motivation for the work and provide a clear example of use
  • The author states “The authors reviewed the AI-assisted text, references, proofs, implementations and results” , can the author clarify how this work was reviewed?
4 comments
AuthorsOpenReview ↗

Response to Reviewer c6Q8 (1/2): the paper in brief, motivation and an example of use

We thank the reviewer for the review and its concrete examples. We agree that the submitted version was too dense: it used terms such as FRoG before introducing them, as the reviewer points out, and it did not state its motivation and use plainly. The revised PDF opens the introduction with the example below, states the motivation and a use case, lists the contributions, and introduces FRoG and every other dataset before the experiments (changes in blue).

The paper in brief. Language models often answer with an ordered category such as few, some or most, and such answers are then reused in a later calculation. A correct category can lose information that the calculation needs. DFIL is a model head that outputs the needed number alongside the usual category.

Q1: Motivation and a clear example of use

Example.

  • Let few, some and most stand for 10%, 30% and 90%, their centers. Label each proportion by the nearest center, so most means more than 60%.
  • A model reads two reports and correctly labels both most: 65% of residents of town A own a car, and 85% of residents of town B.
  • Now ask: half of the car owners drive daily, and half of those commute downtown. What share of residents commute downtown by car? The answers are 16% (few) for town A and 21% (some) for town B.
  • A system that passes on only the label most receives the same input in both cases, so it must answer one of them wrongly, although the label was correct.

Section 4 develops this example into the first result. It is not a corner case: on the FRoG-Hard benchmark (described below), with the two trained 7B models' own centers, replacing each proportion by the center of its correct category changes at least one of 35 combined answers for 68–69% of items (new Appendix A.4).

What the paper provides.

  1. A test of whether labels suffice for every later update, and otherwise the smallest exact replacement (five states instead of three labels in the example).
  2. How finely the number must be stored when the later update is not known in advance.
  3. When the centers are themselves uncertain, the exact set of possible final labels.

Use. A system that extracts such quantities with a language model and combines them needs outputs that compose, and a way to tell when a combined answer is determined. The method certifies a combined answer when the combined label is the same for every value consistent with the measured uncertainty. On real model predictions (Table 2), certified combined answers disagree with the labels of the true combined proportions, under the model's own centers, 0.5–6.5% of the time, against 36–47% for the others.

FRoG (Li et al., 2024) is a public benchmark built from math word problems. In each problem a percentage is masked and the original answer is given, and the model chooses the quantifier that best fits the mask, with gold labels among eight ordered quantifiers from none to all. Each item records the masked percentage, so it supplies both an ordinal label and a numerical target. Its Easy and Hard versions differ in their distractor options. Table 4's caption, which read "Core FRoG components", now reads "Component study on the FRoG benchmark".

AuthorsOpenReview ↗

Response to Reviewer c6Q8 (2/2): how the AI-assisted work was reviewed

Q2: How was the AI-assisted work reviewed?

This is a fair question. The authors take responsibility for every statement in the paper.

How the authors reviewed the work. Over several revision rounds, the authors checked the theorem statements against their proofs, the reported numbers against the stored predictions and run summaries, and the references against their sources. Claims that the evidence did not support were narrowed or withdrawn; for example, a general monotonicity claim became a counterexample and an exact test for Gaussian decoding (Section 5.2, Figure 2).

Checks that do not rely on the written arguments. Each exact result was compared with a direct formulation on finite instances, in exact rational arithmetic or with a satisfiability solver (Table 1): 18,490 state-transition comparisons, 62,232 threshold-bit checks, 7,740 label-set queries and 61 direct-versus-reduced pairs. Solver "unknown" outcomes are reported as such and never read as answers, and a deliberately incorrect variant confirms that the checks detect errors (Appendix B.8). For this response, Table 2 was recomputed from the stored predictions and reproduced exactly, and 11 newly trained models reproduce its figures to within one point.

On the ethics flag. We understand the concern that a paper must be readable by a reasonable part of the ICLR audience. We disclosed every use of AI precisely so that reviewers could weigh it, and we take readability as the substantive issue. The revised PDF makes the changes listed in Part 1, starting from the everyday example before any formalism, and moves secondary qualifications out of the main text. The appendix is long because it holds complete proofs and verification details; the main text is meant to be read without it.

The reviewer noted that the state of the paper made its soundness hard to assess. We hope the revised Section 1, which now states the motivation, the problem and its use, makes the paper easier to assess, and we are glad to clarify anything that remains unclear.

I must say I feel like this process is quite disrespectful to the time of the reviewers. The authors agree that central aspects were missing and when they respond to reviewers they don't keep with the 5000 character limit. Lacking proper motivation and not introducing central concept are issues that should not be present in an submitted paper.

I thank the reviewer for providing an example and a use-case. It strikes we as contrived and I fail to imagine reasonable use-cases, where you wouldn't simply built the system to not output non-ordinal values. One might argue that LLMs provide such a case, but modern LLMs are reasonably able to provide uncertainty around their prediction and it nothing else can be trained to using well-known methods.

I have additionally also read the review of reviewer FN2x and agree that "the connection between the extensive theoretical results and the actual learned architecture is not sufficiently compelling" and on top of that the gains over simpler alternatives is mixed.

AuthorsOpenReview ↗

Response to Reviewer c6Q8's follow-up: the use case and uncertainty

We thank the reviewer for the follow-up and keep this reply to one short comment.

The alternative. If we read the comment correctly, it is to have the system output numbers instead of ordinal labels, and to use existing uncertainty methods. We agree with both, and the paper does both: DFIL outputs a scalar alongside the label (Section 3), and Table 2 calibrates that scalar with split-conformal intervals. The label is kept because the category is often what the task asks for, as in the FRoG quantifier benchmark, and because the classifier predicts it more accurately than the scalar path alone, by 3.7–6.6 points (Table 4).

What the alternative leaves open. Two questions, and they are the paper's results:

  1. How much of the number is needed. Section 4 shows exactly when a label would have sufficed and, when it does not, the smallest state that does. For the trained models and the 35 chains of Table 2, this is 90 states instead of 8 labels, and even the best rule that sees only a correct label errs on 6.9–7.8% of one-step composed labels (Appendix A.4).
  2. What an interval on the number implies for the final label. The category values enter both the product and the decoding. Section 5 gives the exact set of possible labels. The usual shortcuts are weaker. On the trained models' predictions with declared center uncertainty, separate interval propagation certifies 0.8–3.3 points fewer one-step queries in every declared setting, under the same guarantee (Section 6.2); sampling returns a single label on 6.9% of synthetic instances where more are attainable (Figure 3).

Against the deployed bank's labels of the true composed proportions, certified composed answers are wrong 0.5–6.5% of the time, compared with 36–47% for uncertified answers (Table 2).

On the points shared with Reviewer FN2x. DFIL's centers enter both the composition and the decoding, which is exactly the dependence treated in Section 5, and the results hold for fixed and learned banks alike. The comparison with separate propagation above shows the value of preserving this dependence on trained-model predictions, independently of whether learning the bank adds a benefit.

Reviewer_FN2xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DFIL, a composable ordinal interface for language models that combines classification, scalar prediction, and Gaussian membership functions. It further develops theoretical results on information preservation and exact decision-making under uncertainty.

Critical strengths
  1. Addresses an interesting problem of information loss in ordinal predictions during compositional reasoning.
  2. Provides rigorous theoretical analysis of minimal state representations and exact label reachability under uncertainty.
  3. Includes evaluations on language models, human judgments, and synthetic tasks.
Critical weaknesses
  1. The proposed DFIL mainly combines existing classification heads, scalar regression, and Gaussian membership functions. The connection between the extensive theoretical results and the actual learned architecture is not sufficiently compelling.
  2. DFIL does not consistently outperform standard fine-tuning or matched baselines. The improvement on human judgments is marginal, and the experiments fail to establish a clear benefit from learning the membership bank.
  3. The theoretical results rely on restrictive assumptions, including predefined proportional composition rules, ordered centers, and specified uncertainty sets. Their relevance to general language understanding and compositional reasoning remains unclear.
Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer FN2x (1/2): summary, W1 and W2

We thank the reviewer for the review, which recognizes the problem and the rigor of the analysis. Its first concern, the link between the theory and the learned architecture, led us to a new analysis of the trained models (Appendix A.4; W1 below). In brief:

  • W1. The novelty is the exact analysis, including the minimum number of stored states when any multiplier in (0,1](0,1] may follow and each threshold decision must be correct outside a margin, which a construction attains and no encoder can beat (Theorem A.11), and it applies to the trained models. On their learned banks, even the best rule that sees only a correct label errs on 6.9–7.8% of one-step composed labels, and an exact discrete replacement for Table 2's 35 chains needs 90 states instead of 8 labels. On their predictions, exact shared-center analysis certifies more composed queries than separate interval propagation under the same guarantee.
  • W2. The paper's main empirical claims are exactness (Table 1) and certificates that work on real predictions (Table 2).
  • W3. Order is guaranteed by the parameterization, multiplication is the operation the application specifies, and the base-scalar uncertainty is calibrated on predictions. The scope is ordinal outputs that are reused in a later calculation.

The readability revisions are listed in the general response.

W1: Standard components; the link between theory and architecture

Response. The novelty is the analysis, not the components; we claim none for classification heads, scalar regression or Gaussian memberships. The analysis concerns the interface these parts form when an ordinal prediction is reused in a later calculation, and it applies to the trained models as they are:

  1. What a label loses. We took the two learned 7B banks, the 35 one- and two-step quantifier chains of Table 2 and FRoG-Hard's true proportions. Even a rule that sees only the correct label and, for every label and chain, picks the answer that is right most often on these very rows errs on 6.9–7.8% of one-step and 4.3–4.5% of two-step composed labels. Composing the label's center changes at least one of the 35 labels for 68–69% of rows.
  2. What must be kept instead. Section 4's backward-threshold construction gives the smallest discrete state that answers the label and these chains exactly: 36 states with the one-step chains and 90 with all 35, instead of 8 labels. That encoding is specific to this query family; DFIL keeps the scalar, so a new chain needs no new encoding. Both analyses were fixed before computing (Appendix A.4).
  3. Shared centers. DFIL's centers are both operands and decoding references, and bounding the two roles separately admits labels that no single parameter setting realizes (Section 5.1). On the trained models' predictions, with a declared ±.01 center tolerance and the same calibrated base interval, the exact sets certify 17.9% of one-step queries at nominal 90%, separate propagation 15.4%; both are valid for the declared box. The gain is 0.8–3.3 points in every declared setting (one-step queries; Table 7). Sampling, the other shortcut, returns false singletons on 6.9% of synthetic shared-center instances (Figure 3).
  4. Trained parameters. Ordered centers do not make a Gaussian decoder monotone, and the three learned 7B banks fail the exact global-order test (Theorem B.9). Our reachability result needs no global order (Theorem B.13) and is exact on DFIL's own center parameterization (Theorem B.15), whose strict gaps change decisions relative to an enclosing box (Section 5.2).
  5. Trained predictions. With only the base scalar calibrated, certified composed outputs are wrong 0.5–6.5% of the time against the deployed bank's labels of the true proportions, uncertified ones 36–47% (Table 2); 11 further training runs give the same figures within one point.

W2: No consistent gain over fine-tuning; no clear learned-bank benefit

Response. Both observations are accurate, and the paper reports them itself (Tables 3 and 4). What DFIL adds to fine-tuning is not accuracy but two things a classifier head lacks: an output that composes (W1, items 1–2) and a certificate for the composed answer (W1, item 5).

  • Accuracy. DFIL keeps the full classifier, and its accuracy stays within 1.4 points of FT's: 48.2 vs. 48.0% at 1.5B and 49.9 vs. 51.3% at 7B (Table 4, three seeds).
  • Human judgments. On 22 held-out statements, the complete DFIL treatment lowers MSE by 5.7% relative to FT+Compose and meets every pre-specified criterion (exact p=.0018p=.0018, 15/20 seeds favorable). The gain concerns the complete treatment, not the bank in isolation (Section 6.3).
  • Learned bank. In the matched composition controls, the benefit of learning the bank is unresolved (Table 3). The theory does not predict such a benefit: its results hold for fixed and learned banks alike. The revised abstract states this.

(W3 follows in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer FN2x (2/2): W3

W3: Restrictive assumptions and relevance

Response. Each assumption has a specific role.

  • Proportional composition. Nested selection ("half of most of the group") multiplies proportions, and so do chained conditional proportions, P(A∩B)=P(A),P(B∣A)P(A\cap B)=P(A)\\,P(B\mid A). This is the operation the guarantees target: every finite multiplier set for the refinement result, every multiplier in (0,1](0,1] for the margin result.
  • Ordered centers. The parameterization enforces order for every parameter value (Proposition A.1), so order is a property of the architecture, not an assumption about training. Gaussian reachability does not require monotone decoding.
  • Uncertainty sets. Table 2 obtains them by split-conformal calibration of the base scalar on real predictions, and Corollary B.6 also admits a finite set of candidate chains.

On relevance: the paper does not address general language understanding. Its scope is an ordinal output that stands for a number and is reused in a later calculation. An example is answering "what share of patients responded and later relapsed?" from "most patients responded" and "few of the responders relapsed". Such outputs appear in FRoG's quantifier problems (Li et al., 2024) and in human numerical readings of uncertainty expressions (Belém et al., 2024), the data of our human-judgment experiment. For such outputs, our results state what must be retained, and which labels remain determined, once the number is reused. As Part 1 shows, the question is not academic: on FRoG-Hard, even the best rule that sees only a correct label errs on 6.9–7.8% of one-step composed labels. Following this comment, the revised introduction states this scope explicitly.

The reviewer found the problem interesting and the theory rigorous. Together, the results determine how much a reusable ordinal output must retain: exactly, for a fixed set of multipliers (Section 4), and, when any multiplier in (0,1](0,1] may follow, the minimum number of stored states for which every threshold decision is correct outside a margin, with a matching construction and a lower bound that holds for every encoder, interval or not (Theorem A.11). Part 1 instantiates them on the trained models and shows that exact dependence analysis certifies more composed queries on real predictions than separate propagation. We hope these answers clarify the contribution and its scope, in particular the link between the theory and the learned architecture raised in W1, and we are glad to answer any further question.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what the paper establishes, and the revised PDF

We thank the reviewers. They found the central distinction useful and clearly motivated (EAqs), the margin result a meaningful strengthening over standard quantization discussions (EAqs), the problem interesting and the theoretical analysis rigorous (FN2x), and the reporting candid (EAqs). Two concerns are shared: readability, and whether learning the membership bank improves composition. The revised PDF addresses the first. The paper's theoretical results do not depend on the second, and we state below exactly what the experiments show.

The problem. Let few, some and most mean 10%, 30% and 90%, and label each proportion by the nearest of these values. A model correctly labels two proportions, 65% and 85%, as most. After both are halved twice, the true answers are 16% (few) and 21% (some). A system that passed on only the label most must get one of them wrong, although that label was correct. This is common: on the FRoG-Hard benchmark, with the two trained 7B models' centers, replacing each proportion by its correct label's center changes at least one of 35 composed labels for 68–69% of rows, and even the best rule that sees only the correct label errs on 6.9–7.8% of one-step ones (reply to FN2x). Ordinal outputs of language models are often reused in this way. We ask what a reusable output must retain, and when a composed label is determined.

Contributions. For proportional updates followed by nearest-center or Gaussian decoding, the paper gives:

  1. an exact test of when labels suffice for every later update, and otherwise the smallest exact state, which is finite for every finite set of multipliers;
  2. the exact minimum number of states when any multiplier in (0,1](0,1] may follow and each threshold decision must be correct outside a margin;
  3. the exact set of attainable final labels when the category centers are uncertain and each center is both an operand and a decoding reference. It has a closed form for nearest-center decoding; for Gaussian decoding, an exact two-variable reduction gives it without a monotone decoder, exactly on DFIL's own parameterization.

DFIL is a learned realization. It keeps the standard classifier and adds a supervised scalar and ordered centers that serve both roles.

What the experiments establish.

  • Certificates on real predictions (Table 2). For two trained 7B models, with only the base scalar calibrated, base coverage is 95.0, 89.9 and 80.0% at nominal 95, 90 and 80%. Against the deployed bank's labels of the true proportions, certified composed outputs are wrong 0.5–6.5% of the time, uncertified ones 36–47%. The certificate identifies which composed answers can be relied on under that reference. Eleven further training runs reproduce these rates within one point (reply to EAqs).
  • Exactness and its value (Table 1, Section 6.2). The exact procedures agree with direct formulations on every decided check. On the trained models' predictions with declared center uncertainty, they certify 0.8–3.3 points more one-step queries than separate interval propagation in every declared setting, under the same guarantee (17.9 vs. 15.4% at ±.01 and nominal 90%).
  • Human judgments. On 22 held-out statements and 20 training seeds, the complete DFIL treatment lowers MSE against a matched fixed-bank control under pre-specified criteria (.007690 vs. .008154, exact p=.0018p=.0018).

What learning the bank adds. The theory holds for every ordered bank, fixed or learned. In the matched composition controls, the benefit of learning the bank is unresolved (Table 3), and the human-judgment gain concerns the complete treatment, not the bank in isolation.

Revised PDF (changes in blue)

  1. The introduction opens with the example above, states the motivation and a use case, and lists the three contributions.
  2. Section 6 adds a "Tasks and data" paragraph that introduces FRoG and every other dataset before the experiments.
  3. The abstract and introduction describe the setting as composition of numerically supplied quantifier meanings, and the certificates as numerical guarantees relative to a supplied chain, meanings and uncertainty set. The abstract now reports the main results and states that the guarantees hold for fixed and learned banks alike.
  4. Appendix C.1 adds the requested ablation of the membership logits, Appendix C.3 a table of the parameters each loss term updates in every experiment, and Appendix D.2 the detachment ablation and its decomposition.
  5. Appendix B.3.1 adds partition and training-seed intervals for Table 2 and two further center tolerances, and states whether calibration, checkpoint selection and mixture-weight selection are separated from evaluation.
  6. Secondary qualifications in Section 6 move to the appendix.
  7. Appendix A.4 applies the minimal-state construction to the trained 7B banks (fixed before computing).

We answer Reviewer c6Q8's question on how the AI-assisted work was checked in that thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.