- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
domain-specific benchmark for measuring reliability of tabular foundation models for the nice area of early (upto grade 4) literacy prediction. They introduce a new benchmark task: RealLit. Also, they curated three new longitudinal datasets (D2, D3, and D4) as part of this benchmark suite (+ utilizing one previously published reference dataset (D1))
The theory is “Using longitudinal data, models could learn to help identify students at risk early and, in intervention settings, estimate which supports may benefit them”
Why it make sense to target educational datasets with TFMs?
- Scarcity: Educational datasets are typically small-scale, site-heterogeneous, and difficult to collect, rendering deep learning or heavy task-specific tuning data-inefficient.
- Zero/Few Shot: TFMs leverage large-scale pretraining to perform strong off-the-shelf predictions without gradient updates, making them appealing for data-scarce medical or educational domains.
- these records are naturally tabular in nature
- Not merely a niche application: evaluating a TFM on RealLit tests its core generalization capabilities in realistic situations: incompleteness (missingness), distribution shifts etc.
they primarily evaluate the ability of existing models to transfer to educational datasets, NOT their ability to learn a transferable education-specific representation
Process
-
Data Curation & Harmonization: 4 longitudinal datasets through voluntary parent studies and institution records (D1 publicly avbl data + introduced D2, D3, D4) for K->4. they go into good details about subject distributions in these datasets
- they then convert cohort-specific literacy labels into supervised classification targets for models
- Preserve normal missingness, align features,labels within each cohort, retain avbl demographic metadata
- Evals: Compare TabPFN-v2, TabICL-v2, TabDPT, MaskMLP, 6 traditional ML baselines(trees/lstm/…)
-
Missingness Stress-Testing: different controlled missingness injections (Rubin's classes: MCAR, MAR, MNAR) @ 10%, 20%, 30% on top of -ve missingness.
-
Imputation: they try learnable generative/flow-based imputers (DiffPuter, CFMI,...) vs statistical baselines (MICE, Mean, Zeros) fitted on training split vs no-imputations 4 . Permutation Ensembling: feature + class-label permutations (K = {1,2,4,8,16,32}) to get in-context predictions & reduce sensitivity to ordering
Critical strengths
- Important domain with plenty of tabular data use-cases
- Collected datasets represent diverse real-world scenarios and has student subgroup info
- Thorough ablations and slice testing: Tested on 3 separate Tab Foundational model families, 6 classical baselines, classfication+regression
- Useful findings about newer models not necessarily a lot better; that learned impuations is not always useful, sub-group performance can deteriorate with good macro performance
- Clear details of dataset distribution, training & eval methodology.
Critical weaknesses
-
The benchmark contains ~4500 labeled examples, BUT:
- D2: only 515 students with valid outcomes.
- D4: only 285.
- D1: 946 - despite ~7k originally reported participants.
- Several subgroups appear in only one cohort.
-
subgroup analysis unsupported by sample sizes.
-
MaskMLP reproducibility issues - with metrics not matching
-
Restricted to English-language outcomes across U.S. school contexts
Questions for authors
-
Do evals actually measure generalization to unseen schools?
- Can the authors clarify the cross-validation implementation & compare student-level random splits with school-disjoint splits?
-
Can you please add subgroup sample sizes, confidence intervals, group-specific false negative rates, etc?
3 comments
Acknowledging the responses. Thanks for additional details around generalization concern.
Maintaining initial rating
Thank you, we are glad to hear the concerns have been addressed and appreciate the reviewer's time and consideration of our work.
Rebuttal by Authors
Thank you for taking the time to review our paper and for the positive comments. We appreciate your feedback and would like to address your concerns below.
Yes, D2 and D4 are smaller as we exclude students that do not have a recorded score on the end-of-year assessment (used as the prediction target). Both D2 and D4 originate from longitudinal educational research studies involving individually administered assessments by trained assessors. Such longitudinal data for K-4 students are costly and time-consuming to collect, requiring parental consent, sustained participation, and follow-up assessments, over several years.
For comparison, in D1 (Shangguan et al., 2024), the only prior publicly available early-literacy benchmark, just 14% of the 6,916 reported students have valid end-of-year labels. This puts its labeled sample size on par with D2 and D4 combined, yet D1 lacks in student-level demographic attributes.
Regarding D1 concerns, we would like to clarify that D1 is not a dataset introduced by our work but an existing benchmark included for comparison with prior research. The discrepancy in sample size is not a result of our data collection or filtering choices: our inspection of the original public release revealed only 946 students with valid outcome labels. We explicitly document this observation, along with the substantial missingness in D1, in Appendix A.1.
D3, in contrast to D1, D2, and D4, comes from routine school-system records and provides 3,225 students with valid outcome labels. We see these different data collection settings as complementary strengths of RealLit, where research cohorts offer rich, carefully administered assessments, while D3 offers larger-scale operational data, enabling broader characterization of tabular foundation models under diverse and realistic educational data conditions. Overall, our data is unprecedented in both scale and granularity, with data collected across six U.S. states and 53 schools, and more than four times the labeled sample size of the only prior publicly available early-literacy benchmark.
We have extended Appendix H to include subgroup sample sizes, bootstrap confidence intervals, and additional statistical analyses. Please also see our response to tnEu [2]. We thank the reviewer for helping improve the completeness of our evaluation.
As the original MaskMLP implementation was not publicly available, we reimplemented the method following the methodological details provided in Shangguan et al. (2024), with our implementation choices fully documented in Tab. 18. As shown in Tab. 19 (Appendix I.1), our reimplementation of MaskMLP achieves a comparable (slightly higher) AUC but lower classification accuracy. Despite our best efforts, we are unsure of the source of this discrepancy. We also note that the original study reported 6,916 samples for D1, yet we were able to identify only 6,803 students in the publicly available data. We employed the released CSV file. We have transparently documented these observations and our implementation details.
Although our outcomes are restricted to English-language literacy, the cohorts capture substantial linguistic diversity. English learners comprise 13–20% of labeled students in D1-D3, and D3 records Spanish as the home language of 376 students, alongside 13 additional languages. Our additional analyses reveal significantly higher recall for English learners in both D3 grade bands (+0.13 and +0.27, respectively, after multiplicity correction - Appendix H).
We agree that extending RealLit to other languages and educational systems is an important direction for future work (Sec. 6). We do not claim that our empirical findings generalize beyond the current settings, but hope that RealLit provides a foundation for such investigations.
We have clarified our cross-validation protocol (see Appendix C.5) and added experiments comparing student-level and school-disjoint splits (see Appendix L). We have also expanded Appendix H to report subgroup sample sizes and confidence intervals. Appendix L has been added to report additional analysis, including false negative rates (please also see our responses to VUEK [1] and tnEu [2] for further details).
We appreciate the reviewer’s thoughtful engagement with our work and believe that the additional analyses prompted by their suggestions further strengthen our contribution. We hope the clarifications address the reviewer’s concerns and welcome any further discussion.