acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Tabular Foundation Models in Early Literacy

Abstract

Tabular foundation models show strong performance across diverse domains and settings, yet few studies examine their use for early literacy prediction-a consequential domain with scarce longitudinal data and heterogeneous, incomplete real-world records. We ask whether this promise carries over to K-4 literacy prediction in realistic settings. To answer this question, we introduce RealLit, a large-scale, longitudinal multi-cohort benchmark comprising 4,582 students across 53 schools and six U.S. states. A large portion of the benchmark comes from real-world school-system data, preserving naturally occurring missingness and heterogeneity in assessment practice, while providing fine-grained student-level demographic attributes across cohorts. Our benchmarking surfaces several key findings while offering guidance for evaluating performance beyond aggregate metrics using student-level metadata. For instance, we find that model ensembling can have uneven effects across student groups, reducing performance for some. Newer model variants can also underperform earlier ones. Surprisingly, in our missing-data analysis, learned imputation also provides little consistent downstream benefit over simpler missing-data handling strategies. By making such behaviors measurable, RealLit provides a standardized testbed for understanding and advancing the reliable use of tabular foundation models in real-world early literacy prediction. Our code and data is available at https://github.com/anonymous-submission-site/real-lit.

Primary area:
datasets and benchmarks
Keywords:
early literacy, foundation models, tabular learning, missing data, imputation, responsible AI in education

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PWgjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

domain-specific benchmark for measuring reliability of tabular foundation models for the nice area of early (upto grade 4) literacy prediction. They introduce a new benchmark task: RealLit. Also, they curated three new longitudinal datasets (D2, D3, and D4) as part of this benchmark suite (+ utilizing one previously published reference dataset (D1))

The theory is “Using longitudinal data, models could learn to help identify students at risk early and, in intervention settings, estimate which supports may benefit them”

Why it make sense to target educational datasets with TFMs?

  • Scarcity: Educational datasets are typically small-scale, site-heterogeneous, and difficult to collect, rendering deep learning or heavy task-specific tuning data-inefficient.
  • Zero/Few Shot: TFMs leverage large-scale pretraining to perform strong off-the-shelf predictions without gradient updates, making them appealing for data-scarce medical or educational domains.
  • these records are naturally tabular in nature
  • Not merely a niche application: evaluating a TFM on RealLit tests its core generalization capabilities in realistic situations: incompleteness (missingness), distribution shifts etc.

they primarily evaluate the ability of existing models to transfer to educational datasets, NOT their ability to learn a transferable education-specific representation

Process

  1. Data Curation & Harmonization: 4 longitudinal datasets through voluntary parent studies and institution records (D1 publicly avbl data + introduced D2, D3, D4) for K->4. they go into good details about subject distributions in these datasets

    • they then convert cohort-specific literacy labels into supervised classification targets for models
    • Preserve normal missingness, align features,labels within each cohort, retain avbl demographic metadata
    • Evals: Compare TabPFN-v2, TabICL-v2, TabDPT, MaskMLP, 6 traditional ML baselines(trees/lstm/…)
  2. Missingness Stress-Testing: different controlled missingness injections (Rubin's classes: MCAR, MAR, MNAR) @ 10%, 20%, 30% on top of -ve missingness.

  3. Imputation: they try learnable generative/flow-based imputers (DiffPuter, CFMI,...) vs statistical baselines (MICE, Mean, Zeros) fitted on training split vs no-imputations 4 . Permutation Ensembling: feature + class-label permutations (K = {1,2,4,8,16,32}) to get in-context predictions & reduce sensitivity to ordering

Critical strengths
  1. Important domain with plenty of tabular data use-cases
  2. Collected datasets represent diverse real-world scenarios and has student subgroup info
  3. Thorough ablations and slice testing: Tested on 3 separate Tab Foundational model families, 6 classical baselines, classfication+regression
  4. Useful findings about newer models not necessarily a lot better; that learned impuations is not always useful, sub-group performance can deteriorate with good macro performance
  5. Clear details of dataset distribution, training & eval methodology.
Critical weaknesses
  1. The benchmark contains ~4500 labeled examples, BUT:

    • D2: only 515 students with valid outcomes.
    • D4: only 285.
    • D1: 946 - despite ~7k originally reported participants.
    • Several subgroups appear in only one cohort.
  2. subgroup analysis unsupported by sample sizes.

  3. MaskMLP reproducibility issues - with metrics not matching

  4. Restricted to English-language outcomes across U.S. school contexts

Questions for authors
  1. Do evals actually measure generalization to unseen schools?

    • Can the authors clarify the cross-validation implementation & compare student-level random splits with school-disjoint splits?
  2. Can you please add subgroup sample sizes, confidence intervals, group-specific false negative rates, etc?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for taking the time to review our paper and for the positive comments. We appreciate your feedback and would like to address your concerns below.

[1] Dataset size and diversity

Yes, D2 and D4 are smaller as we exclude students that do not have a recorded score on the end-of-year assessment (used as the prediction target). Both D2 and D4 originate from longitudinal educational research studies involving individually administered assessments by trained assessors. Such longitudinal data for K-4 students are costly and time-consuming to collect, requiring parental consent, sustained participation, and follow-up assessments, over several years.

For comparison, in D1 (Shangguan et al., 2024), the only prior publicly available early-literacy benchmark, just 14% of the 6,916 reported students have valid end-of-year labels. This puts its labeled sample size on par with D2 and D4 combined, yet D1 lacks in student-level demographic attributes.

Regarding D1 concerns, we would like to clarify that D1 is not a dataset introduced by our work but an existing benchmark included for comparison with prior research. The discrepancy in sample size is not a result of our data collection or filtering choices: our inspection of the original public release revealed only 946 students with valid outcome labels. We explicitly document this observation, along with the substantial missingness in D1, in Appendix A.1.

D3, in contrast to D1, D2, and D4, comes from routine school-system records and provides 3,225 students with valid outcome labels. We see these different data collection settings as complementary strengths of RealLit, where research cohorts offer rich, carefully administered assessments, while D3 offers larger-scale operational data, enabling broader characterization of tabular foundation models under diverse and realistic educational data conditions. Overall, our data is unprecedented in both scale and granularity, with data collected across six U.S. states and 53 schools, and more than four times the labeled sample size of the only prior publicly available early-literacy benchmark.

[2] Subgroup analysis sample size

We have extended Appendix H to include subgroup sample sizes, bootstrap confidence intervals, and additional statistical analyses. Please also see our response to tnEu [2]. We thank the reviewer for helping improve the completeness of our evaluation.

[3] MaskMLP reproducibility

As the original MaskMLP implementation was not publicly available, we reimplemented the method following the methodological details provided in Shangguan et al. (2024), with our implementation choices fully documented in Tab. 18. As shown in Tab. 19 (Appendix I.1), our reimplementation of MaskMLP achieves a comparable (slightly higher) AUC but lower classification accuracy. Despite our best efforts, we are unsure of the source of this discrepancy. We also note that the original study reported 6,916 samples for D1, yet we were able to identify only 6,803 students in the publicly available data. We employed the released CSV file. We have transparently documented these observations and our implementation details.

[4] Geographic and linguistic scope

Although our outcomes are restricted to English-language literacy, the cohorts capture substantial linguistic diversity. English learners comprise 13–20% of labeled students in D1-D3, and D3 records Spanish as the home language of 376 students, alongside 13 additional languages. Our additional analyses reveal significantly higher recall for English learners in both D3 grade bands (+0.13 and +0.27, respectively, after multiplicity correction - Appendix H).

We agree that extending RealLit to other languages and educational systems is an important direction for future work (Sec. 6). We do not claim that our empirical findings generalize beyond the current settings, but hope that RealLit provides a foundation for such investigations.

[5] Generalization

We have clarified our cross-validation protocol (see Appendix C.5) and added experiments comparing student-level and school-disjoint splits (see Appendix L). We have also expanded Appendix H to report subgroup sample sizes and confidence intervals. Appendix L has been added to report additional analysis, including false negative rates (please also see our responses to VUEK [1] and tnEu [2] for further details).

We appreciate the reviewer’s thoughtful engagement with our work and believe that the additional analyses prompted by their suggestions further strengthen our contribution. We hope the clarifications address the reviewer’s concerns and welcome any further discussion.

Acknowledging the responses. Thanks for additional details around generalization concern.

Maintaining initial rating

Thank you, we are glad to hear the concerns have been addressed and appreciate the reviewer's time and consideration of our work.

Reviewer_VUEKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RealLit, a benchmark for early-literacy prediction that combines three newly introduced datasets with the existing ECRI reference. It retains differences across assessments, naturally missing measurements, and student-level demographic attributes where available. The study compares tabular foundation models with conventional methods and MaskMLP. It also examines imputation, ensembling, model versions, fine-tuning, and longitudinal information. The results show that foundation models are competitive, while learned imputation and fine-tuning do not consistently improve performance. The effects of ensembling vary by subgroup.

Critical strengths
  1. A useful benchmark combining research assessment data with operational school records. The introduced cohorts include both research assessments and operational school records. Their longitudinal structure and student-level attributes allow analyses beyond aggregate accuracy. Evaluating each dataset’s targets separately also respects differences in the reading constructs being measured (Section 3 and Table 1, pp. 3–4; Appendix A).

  2. Broad experimental coverage, including useful negative results. The paper examines conventional baselines, native and injected missingness, model versions, fine-tuning, and simple longitudinal references. The finding that newer models or more elaborate imputers do not necessarily improve performance is useful for practitioners. The authors also acknowledge the small subgroups, limited population coverage, and exclusion of MaskMLP runs that did not converge (Table 2, p. 6; Appendices D–I).

Critical weaknesses
  1. The paper gives conflicting descriptions of the evaluation protocol and uncertainty estimates. Section 4 and Appendix C.5 describe three repetitions of five-fold cross-validation. Tables 2 and 8 instead describe three evaluation folds, while Appendix G.1 refers to the shared protocol as 10×5 cross-validation. Appendix C.5 reports standard errors, but the tables report standard deviations (pp. 6, 26, 30 and 36). C.5 also states that joint target/school stratification keeps schools disjoint. Stratification alone does not guarantee this, so the paper needs to explain the grouping procedure. Appendix H reports statistical tests and bootstrap results without identifying the units used for pairing or resampling (p. 40). These details matter because the paper interprets small performance differences and claims protection against school-level leakage. To assess those claims, I need a single, consistent description of the splits, grouping, repetitions, aggregation, and statistical inference, with clear links to the corresponding results. The conflicting descriptions do not, by themselves, show that leakage occurred.

  2. The paper makes contradictory statements about its central subgroup finding. Section 5 states that increasing K from 1 to 8 improves every subgroup and causes no regressions (p. 7, lines 333–337). The next paragraph, however, reports decreases, including an accuracy change of −0.049 for Black students with TabDPT (lines 345–357). The abstract and Figure 2 support the more cautious conclusion that the effects vary across subgroups. The claim of improvement for every subgroup should either be removed or clearly tied to a separate experimental condition. This is a correctable reporting issue, but it matters because subgroup reliability is a central motivation for the benchmark.

Questions for authors

Could you specify the exact splits, school grouping, repetitions, aggregation, and statistical-resampling procedures used for Tables 2 and 8 and the fine-tuning results? Please also explain how that protocol relates to the three-fold, 3×5, and 10×5 descriptions, and clarify the standard-deviation versus standard-error labels.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thoughtful feedback and for recognizing the value of our dataset and empirical findings. We address the concerns raised by the reviewer below.

[1] Evaluation protocol

These are our errors, and we thank the reviewer for the careful read. We have proofread and corrected:

  • The same standardized protocol of three repetitions (i.e., seeds) with five-fold cross-validation (CV) was followed throughout, as described in Sec. 4 and Appendix C.5. For each experimental configuration, we report the mean ±\pm standard deviation across the 15 total fold-level evaluations. Three evaluation `folds’ (repetitions) was a typo, and the one reference in the supplementary to 10 x 5 CV was stale text from an earlier experiment and will be removed (inadvertently retained in the supplementary material). These corrections do not affect the original results in the submitted manuscript, as all the reported results consistently followed a 3 x 5 CV protocol.

  • Our split indeed does not ensure school-disjoint folds, as explained in our updated Appendix C.5. We have added a school split in Appendix L.

  • As suggested, we extended the analysis in Appendix H to include full experimental details (please also see tnEu [2]).

[2] Findings clarification

We thank the reviewer for raising this reporting issue. `Uniform improvement’ overstates our findings and fails to reflect the more nuanced trends observed in our results. We have revised the text to clarify that aggregate performance generally improves with ensembling (statistically supported, please also see tnEu [1]), while subgroup-level trends can be non-monotonic and classifier-dependent (e.g., Fig. 2, and Fig. 9). These revisions concern the wording and interpretation of our findings and do not affect the underlying experimental results.

Thank you for helping improve the clarity and quality of our work. We hope the revisions address the concerns and would be happy to discuss any remaining questions.

Thank you for your explanation and revision. They resolved most of my concerns about the inconsistencies in the original reporting.

I remain cautious about interpreting the smaller performance differences reported in the subgroup analysis, since it is still unclear how the confidence intervals account for repeated predictions for the same student across cross-validation repetitions. Given that the subgroup analysis constitutes a significant part of the paper's contribution, I keep my rating unchanged.

Thank you for taking the time to respond to our rebuttal. We are happy to hear that the clarifications resolved most of your concerns.

Re. our contribution - we emphasize that our primary contribution is the introduction of a novel benchmark that can enable a broad range of future investigations. This includes subgroup-aware model evaluation across hundreds of students in educationally relevant subgroups (e.g., SpEd, ELL, and Section 504), but also the reliability of future foundation models across diverse student populations, cross-cohort generalization, robustness to realistic missingness, and longitudinal modeling.

Re. subgroup analysis - we find subgroup accuracy gaps remain after correction for multiple comparisons:

CohortAxisLevel (minus rest)Total studentsAccuracy gap95% CIqMaskMLPTabDPTTabICL-v2TabPFN
D3 (G1–G2)Advanced/giftedNo2,005−0.181[−0.206, −0.139]0.001−0.193−0.173−0.172−0.184
D3 (G1–G2)Section 504Eligible + plan114−0.100[−0.173, −0.027]0.028−0.085−0.094−0.112−0.109
D3 (G3)Section 504Yes112−0.153[−0.227, −0.081]0.001−0.145−0.147−0.161−0.158
D3 (G3)Econ. disadvantagedYes773−0.096[−0.136, −0.055]0.001−0.116−0.093−0.091−0.085
D3 (G3)ESEYes218−0.083[−0.134, −0.032]0.004−0.123−0.079−0.062−0.069

Moreover, the variation in subgroup sizes is common in real-world datasets, reflecting differences in the prevalence of demographic characteristics, educational needs, and support services. Subgroup sizes in our analysis are generally 100 students or more.

Given the scarcity of comparable benchmarks, RealLit represents an important initial step toward systematic subgroup-aware evaluation in early literacy prediction. We envision RealLit complementing future, potentially smaller-scale datasets with more targeted data collection for specific or underrepresented student populations.

Reviewer_tnEuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of this paper is to study the performance of tabular foundation models for predicting early literacy outcomes of K-4 students using real-world school data. Authors introduce a benchmark named RealLit that consists of four longitudinal datasets. Three of these datasets are new and include student demographic information. Authors compare three foundation models against MaskMLP and six conventional approaches. They also investigate the effects of missing data, imputation, fine-tuning, and ensembling on prediction accuracy. The results show that TabPFN-v2 performs best on three of the four datasets. Interestingly, learned imputation does not provide consistent improvement over simpler techniques. Also, a newer TabPFN version does not improve the performance over v2, while ensembling affects prediction accuracy differently across student subgroups.

Critical strengths

The main strength of this paper is the datasets. Early literacy data for K-4 students are not commonly available in existing tabular benchmarks. Authors provide real-world school district data with missing values, a longitudinal cohort covering Grades 1 to 4, and demographic information of students which is not available in ECRI. I think RealLit could be a useful benchmark for studying tabular foundation models, particularly in settings with limited samples and missing data. I also appreciate that authors openly discuss some negative findings, such as their failure to reproduce MaskMLP results and the discrepancy in the sample count of D1. This adds credibility to their experimental observations.

Critical weaknesses

First, I found some contradictory observations regarding the effects of ensembling on subgroup performance. Authors mention in the abstract and introduction that ensembling reduces the prediction accuracy for some student groups. However, in the permutation ensembling section, they report that accuracy improves uniformly at K=8 without any regression for individual subgroups. Interestingly, in the next section, authors report a decrease in accuracy for Black students using TabDPT and Asian students using TabICL-v2. I am not sure which of these observations supports their main conclusion about ensembling.

Second, I have some concerns about the subgroup analysis. Authors report relatively large variations in prediction accuracy across demographic groups, but do not provide any uncertainty estimates. Some of these groups are represented in only one or two cohorts and may have very small sample sizes. For example, D4 has only 285 labeled students. I am not sure whether an accuracy difference of 0.05 to 0.09 could simply result from sampling variations. It would be useful to provide bootstrap confidence intervals or analyze the variation across different random seeds. Without such analysis, the assertion that small differences in overall AUC could hide subgroup performance gaps that are an order of magnitude larger seems rather strong. Also, since authors emphasize the importance of identifying at-risk students, why not consider false negative rates or recall for different student groups? Missing an at-risk student is arguably more important than the overall prediction accuracy. I think these issues are important because the subgroup equity argument is one of the main contributions of this paper and the current results do not provide sufficient support for this conclusion.

Questions for authors

Which data is actually released? In particular, is D3 available at student level with the demographic, homelessness, discipline and 504 fields, and under what terms? If D3 is restricted, the benchmark is much less usable for others.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful read, constructive feedback, and for recognizing the importance and uniqueness of our large-scale longitudinal benchmark. We also appreciate the reviewer’s recognition of our transparent reporting. We address each concern below.

[1] Ensembling effects

We sincerely apologize for any confusion. We have corrected the vague wording to reflect that overall accuracy generally improves with ensembling, which is statistically supported, for instance for TabPFN on D2 and both D3 cohorts (Appendix H), yet subgroup-level trends can be non-monotonic and classifier-dependent (see Figure 2 and Figure 9).

For instance, as noted by the reviewer, for Black students, increasing K from 1 to 8 improves MaskMLP accuracy by 9.5 percentage points while decreasing TabDPT accuracy by 4.9 points (Fig. 2). As another example, for White students, TabDPT improves at intermediate ensemble sizes but performs worse at K=32 than at K=1 (Fig. 9, MaskMLP as well). Such fine-grained analysis is enabled by the metadata in our large-scale longitudinal benchmark.

We’ve extended the discussion of Figs. 2, 9, and 10 (highlighted in red in the revised manuscript) to sharpen the claims. To better communicate these findings, we have refined figure captions (Figs. 9 and 10) in the supplementary.

[2] Subgroup uncertainty and false negative rates

Our original submission reported variability across three different repetitions (i.e., different seeds) and a five-fold cross-validation, with std reported throughout all the tables. Appendix H also showed bootstrap-based statistical significance for several subgroups and explicit sample-size considerations (Appendix I.2, with full sample sizes shown in Fig. 4, and number of samples per fold ).

As suggested, we’ve revised Appendix H to extend the discussion around performance gaps across student groups and changes induced by ensembling. For each subgroup, we compute the accuracy difference relative to the remaining students within the same demographic or educational attribute, along with a 95% confidence interval using a fold-clustered student bootstrap. To account for multiple comparisons, we apply the Benjamini-Hochberg correction across all subgroup contrasts.

[3] Identifying at-risk students

Thank you for the thoughtful feedback. We have extended our evaluation in Appendix L to include recall (equivalently, one minus the false negative rate) and specificity under both native and injected missingness.

The subgroup analysis in Appendix L now surface how the ability to identify at-risk students varies across student populations. For instance, students receiving ESE services in D3 (G3) exhibit substantially higher recall relative to the remaining at-risk students under both native and injected missingness (+0.287 and +0.282, respectively), with both differences remaining statistically significant after multiple-comparison correction. Under injected missingness, we also observe significantly higher recall for Black students (+0.072, q = 0.012) and lower recall for White students (-0.070, q = 0.010) in D3 (G1–G2), each relative to the remaining at-risk students within the same demographic attribute. We do hope RealLit can facilitate the evaluation of tabular foundation models across multiple metrics, student populations, and realistic data conditions.

We further provide class-conditional accuracy for at-risk versus other students under two missingness conditions. In Tables 23 and 24, we provide breakdown over the at-risk class (recall) and on the remaining students, pooled over the four classifiers, the gap for each classifier, and a 95% fold-clustered student bootstrap interval on the pooled gap (B = 10,000). We find that models are generally less accurate at identifying at-risk students than other students

[4] Data

All three introduced datasets (D2-D4) have been deidentified and are available in the anonymous GitHub repository since the time of submission. We may reserve a private held-out portion to mitigate benchmark contamination and extend the benchmark’s useful lifespan. We also provide student-level assessment scale scores and outcome measures, rather than only binary labels, allowing researchers to define their own prediction targets, alongside the complete evaluation code.

We hope these clarifications address the reviewer's concerns. We would be happy to discuss any remaining questions.

Thank you for the response and for the additional analyses. Adding recall and specificity for each student group is helpful. The observation that models are generally less accurate on at-risk students is useful for anyone who wants to use these models for screening. I also appreciate the clarification that D2 to D4 are available in the repository.

My remaining question is about the ensembling claim, which the abstract still states, that ensembling reduces performance for some student groups. The new intervals in Appendix H compare each subgroup with the remaining students, but this claim is about how ensembling changes the accuracy of a subgroup, for example, the 4.9-point drop for Black students with TabDPT from K=1 to K=8. If intervals for these changes in Figure 2 and Figure 9, after correction for multiple comparisons, are part of the revised Appendix H, could the authors point to them? If not, it would be useful to add them, since they would show whether the drop for some groups is larger than what small group sizes can produce. Could the authors also confirm whether the paragraph in Section 5 saying subgroup accuracy improves uniformly at K=8 has been removed or revised?

Thank you for taking the time to respond to our rebuttal. We are happy to hear that the clarifications helped.

After correction for multiple comparisons, we find the ensembling claim to remain for certain subgroups:

Table 18. Statistically significant subgroup accuracy gaps after correction for multiple comparisons. We report the number of students with valid outcomes, contrasts with 95% bootstrap confidence intervals excluding zero and Benjamini–Hochberg-adjusted q<0.05, alongside classifier-specific accuracy gaps.

CohortAxisLevel (minus rest)Total studentsAccuracy gap95% CIqMaskMLPTabDPTTabICL-v2TabPFN
D3 (G1–G2)Advanced/giftedNo2,005−0.181[−0.206, −0.139]0.001−0.193−0.173−0.172−0.184
D3 (G1–G2)Section 504Eligible + plan114−0.100[−0.173, −0.027]0.028−0.085−0.094−0.112−0.109
D3 (G3)Section 504Yes112−0.153[−0.227, −0.081]0.001−0.145−0.147−0.161−0.158
D3 (G3)Econ. disadvantagedYes773−0.096[−0.136, −0.055]0.001−0.116−0.093−0.091−0.085
D3 (G3)ESEYes218−0.083[−0.134, −0.032]0.004−0.123−0.079−0.062−0.069

where subgroup sample sizes range from 112 to 2,005. The analysis have been included in Appendix H. The abstract and Sec. 5 (Ensembling Effects) have been revised to note that 1) subgroup-level benefits from ensembling remain less conclusive, 2) distinguish statistically supported effects.

Thank you again for your thoughtful and constructive evaluation. We sincerely appreciate the time and effort you invested in helping strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.