acceptodds
Under review as a conference paper at ICLR 2027

MoMaTune: Model, Match, Then Tune Language Models for Numerical Records

Abstract

Despite remarkable progress in language understanding and generation, decoder language models continue to exhibit striking failures on numerical data. Numerical examples are typically serialized as text by concatenating feature names with their values (e.g., fat:3.4), while conventional task adaptation applies loss only to the downstream prediction target. This leaves two structures weakly supervised: within-value sensitivity, where digit position determines numerical magnitude, because token cross-entropy contains no explicit numerical metric; and across-field structure, where measurements in the same record are statistically dependent, because target-only tuning provides no direct loss on the measurements or their dependencies. We introduce MoMaTune (Model, Match, then Tune), a two-stage post-training strategy organized around these supervision gaps. Stage 1 applies Model, ordinary next-token prediction densely over label-free serialized records, and optionally adds Match, a same-record consistency objective that aligns representations of different partial views of the same example. Stage 2 applies Tune, supervised fine-tuning for downstream target prediction. We evaluate MoMaTune on 169,285 records from eight public datasets, with 339 of 340 input fields numerical. At 1.5B parameters, Model raises the eight-dataset masked-value imputation aggregate from 0.106 to 0.401, while the three-dataset correct-context gain rises from 0.058 to 0.167. On diagnostics, Model also increases sensitivity to numerically consequential digit changes, while Match improves robustness to informative partial views. Adding Tune yields higher target-supervised embedding aggregates for each tested Stage-1 parent, and the two-stage recipes obtain higher aggregate scores than Tune alone. The same pattern is observed at 4B, 9.6B, and 27B parameters when model embeddings are used for downstream prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
numerical records, post training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wjTUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how decoder-only language models handle records that consist almost entirely of numerical measurements. It argues that two structures are only weakly supervised by conventional target-only fine-tuning: (i) within-value sensitivity, i.e. the fact that a digit's position determines its numerical magnitude, which token cross-entropy does not weight; and (ii) across-field structure, i.e. the statistical dependence among measurements in the same record, which target-only tuning never directly supervises. To address these, the authors propose MoMaTune, a modular two-stage post-training recipe. Stage 1 (MODEL) applies ordinary next-token prediction densely over label-free serialized records, optionally augmented with MATCH, a same-record consistency objective that pulls together representations of two partial views of the same example, and with field-order permutation as data augmentation. Stage 2 (TUNE) performs answer-masked supervised fine-tuning in a fixed field order for the downstream target. Validation combines frozen-embedding linear probing (ROC-AUC / R2 with logistic and ridge probes) on eight public datasets totalling 169,285 records (339 of 340 input fields numerical), plus direct-decoder masked-value imputation and digit-perturbation diagnostics, across Qwen2.5-1.5B and larger Qwen3-family checkpoints (4B–27B). The main claim is that MODEL improves embedding quality and numerical-magnitude sensitivity, MATCH adds robustness to partial views, and MODEL→TUNE beats TUNE alone.

Critical strengths

none

Critical weaknesses
  1. Evaluation is not on an established tabular benchmark. The empirical case rests on only eight public datasets (seven from OpenML plus USDA FoodData), scored with a frozen-embedding linear probe and masked-value imputation. This is far from the standard protocol used in the tabular literature, which routinely reports on recognized suites such as TabArena, OpenML-CC18, or TabBench, and which reports model-level predictive metrics rather than a mean of R2/AUC over eight tasks. A single unweighted 'Emb-8' average over eight datasets — two classification and six regression — is a weak basis for claims that the recipe 'improves linear-probe target prediction' generally, and the same eight datasets appear to be reused for every claim (embedding, imputation, and diagnostics), so there is no held-out battery to test generalization. The reported gains (e.g. EMB-8 from 0.688 to 0.753) are also within a range where dataset selection and probe choice can plausibly dominate.
  2. Baselines are incomplete for the stated problem. The comparison omits the most directly relevant numerical-representation and tabular-embedding methods. Specifically: (a) FoNE (Fourier number embeddings; arXiv:2502.09741) targets precisely the within-value / magnitude-sensitivity gap that the paper identifies, and is a natural competitor that should be included or at least discussed; and (b) TabEmbed (a generalist tabular embedding model; arXiv:2605.04962) is a directly comparable tabular-embedding baseline for the embedding-based evaluation. The current baseline set is essentially frozen decoder LMs plus a few general-purpose text embedding models, which does not establish that MoMaTune's numerical-structure objectives are the right tool relative to methods explicitly designed for numbers or for tables.
  3. The paper never acknowledges that its data are tabular. Every 'numerical record' is a row of named numerical features, i.e. a tabular dataset; the field-conditional baseline it reports is a tabular imputation baseline. Yet the words 'table', 'tabular', 'column', and 'feature' are essentially absent from the framing, related work, and claims. This is not merely cosmetic: it (i) prevents the work from being positioned against the large tabular representation-learning literature, (ii) makes the choice of eight datasets and the embedding-probe protocol look arbitrary rather than aligned with an established evaluation culture, and (iii) makes the missing FoNE / TabEmbed comparisons (Weakness 2) and the absent tabular benchmark (Weakness 1) easy to overlook. The work should be reframed as numerical-record / tabular modeling and evaluated as such.
Questions for authors

Q1. Can the authors report results on a recognized tabular benchmark suite (e.g. TabArena, OpenML-CC18, or TALENT) with the same MODEL→TUNE recipe, at least at 1.5B? Concretely: what is the delta over TUNE-only and over a strong tabular baseline (e.g. gradient-boosted trees or a tabular foundation model) measured by standard predictive metrics, not by a frozen-embedding probe average?

Q2. Why are FoNE and TabEmbed not compared or discussed? If they were omitted deliberately, what is the argument that MoMaTune's dense next-token objective dominates Fourier-based number embeddings and tabular-embedding pretraining for the tasks shown? A direct comparison, or a clear statement of scope, is needed before the 'comprehensive evaluation' claim can stand.

Reviewer_QbYLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training decoder-only language models for numerical-record representation and prediction. The method first applies next-token prediction to label-free serialized records (Model), followed by supervised fine-tuning for downstream targets (Tune). An optional consistency objective aligns partial views of the same record (Match). Experiments cover eight datasets, with evaluation through supervised linear probes on frozen representations, numerical completion, and behavioral diagnostics. Record modeling generally improves downstream prediction and numerical completion, while the benefits of Match depend on view construction and Tune can reduce completion performance.

Critical strengths
  1. The experiments are extensive, covering eight datasets and multiple model sizes. Numerical completion, digit perturbations, and mismatched-context tests examine the effects of training from several complementary perspectives.

  2. The paper candidly reports component limitations, including cases where Match hurts prediction and where Tune reduces numerical-completion ability. These observations are useful for understanding when the components may help.

Critical weaknesses
  1. The connection to the motivation concerning within-value sensitivity is not fully established. Model retains ordinary token cross-entropy without introducing constraints on numerical distance or digit significance. The perturbation results show that continued pretraining improves this diagnostic, but do not sufficiently explain how the method specifically addresses the supervision gap identified in the introduction.

  2. Attribution of the two-stage gains remains incomplete. Although the paper includes compute controls for field permutation, it does not clearly present a Tune-only baseline matched to the total training budget of the two-stage recipe. Experiments also mainly use a single training seed, leaving the stability of some smaller gains uncertain.

Questions for authors
  1. With token cross-entropy unchanged, how does Model specifically improve within-value sensitivity? Can the evidence distinguish learning numerical structure from the benefits of general in-domain continued pretraining?

  2. Did the question-and-answer truncation described in Appendix H.2 occur during actual Tune training? For each of the four affected datasets, how many training examples retained valid answer supervision?

  3. Regarding model selection, I acknowledge the experiments with Qwen3.5 and other checkpoints. Could you explain the choice of Qwen2.5-1.5B for the main analyses and whether a BERT-style bidirectional encoder adapted on the same data was considered?

  4. Are results available for a Tune-only baseline matched to the total training budget of the two-stage recipe, and for repeated experiments with multiple training seeds?

  5. These numerical tabular tasks also seem well suited to methods such as XGBoost. Could you further clarify the research value and potential use cases of studying decoder-only language models in this setting?

2 comments

Thank you for the detailed responses. Overall, the rebuttal has addressed most of my concerns. However, the clarification on prompt truncation raises a remaining concern. Table RT14 shows that all Tune training examples in four datasets were truncated, removing the question and a substantial portion of the input fields. Although answer supervision was preserved, could the authors clarify whether the evaluation prompts were similarly truncated and whether the main conclusions hold when complete inputs are retained?

AuthorsOpenReview ↗

Thank you Reviewr QbYL for follow-up

Thank you for your timely follow-up! We are glad that our responses have addressed most of your concerns. Yes, evaluation inputs were also truncated when they exceeded the token budget. The main conclusion still hold when complete inputs are retained. Appendix H.2 evaluates HIGGS, CPU activity and Protein (short enough to fit the token budget) without prompt truncation. Compared with Tune only, Model+Match→Tune improves HIGGS accuracy from .720 to .722, reduces median absolute log error from .013 to .012 on CPU and from .310 to .294 on Protein.

We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!

Reviewer_VQ5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a two-stage post-training framework, MoMaTune, to improve the numerical inference capabilities of language models. The framework combines three steps: 1) next-token prediction on label-free numerical records (MODEL), 2) consistency learning across partial views (MATCH), and 3) supervised fine-tuning for downstream prediction (TUNE). Experiments on mostly numerical datasets demonstrate improvements in masked-value imputation, numerical sensitivity, and downstream prediction, with additional evaluations across multiple model scales.

Critical strengths
  • Addresses a relevant problem in adapting general-purpose language models to numerical data, highlighting limitations in numerical magnitude sensitivity and cross-field context dependencies.

  • Proposes a modular training framework that can be applied to existing language models directly without modifying its architecture or tokenization protocols.

  • Provides experimental evaluation across multiple datasets and model families and scales, in addition to ablation studies and diagnostic analyses demonstrating the effects of different training components.

Critical weaknesses
  • It is not clear in what way this work contributes to numerical data-specific understanding for a language model. It leads to limited methodological novelty, as the proposed framework mainly combines established techniques such as continued pretraining, consistency regularization, and supervised fine-tuning. The specific technical contributions beyond their straightforward combination are not sufficiently articulated in the context of numerical data understanding.

  • There have been numerous ways of working with numerical data other than language models, which is recently more pronounced in multi-modal scenarios but still mostly regarded as alternatives for numerically heavy datasets. Thus, there is insufficient motivation and justification for using general-purpose decoder language models on predominantly numerical data. The evaluation lacks comparisons with numerical representation learning and tabular-specific approaches, making it unclear whether the proposed framework offers meaningful advantages over more specialized alternatives.

  • There remains concers on generalizability retention of language models. There is a lack of validation of generalizable language understanding as well as generalizable numerical understanding. The improvements are primarily demonstrated through in-domain numerical records and downstream prediction tasks, without cross-dataset transfer experiments or sufficient analysis to distinguish numerical understanding from dataset-specific adaptation.

Questions for authors

Please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.