acceptodds
Under review as a conference paper at ICLR 2027

SCOPE: Self-Calibrated Ordered Principal Encoding for Tabular Foundation Models in High-Dimensional Data

347 views

Abstract

Tabular Foundation Models (TFMs) remain difficult to apply in High-Dimensional, Low-Sample Size (HDLSS) regimes, where thousands of features can exceed practical input limits and dimensionality reduction often requires dataset-specific design or tuning. We introduce SCOPE (Self-Calibrated Ordered Principal Encoding), a deterministic, tuning-free feature compression framework for constructing compact representations of high-dimensional tabular data. SCOPE builds on a global feature ordering produced by LENS (Layout-Enhanced Neighborhood Sequencing), which combines efficient path construction and refinement with a self-calibrated ordering budget. Given this ordering, SCOPE estimates spectral complexity, analytically determines the output dimension, partitions the feature sequence through transition-aware segmentation, and performs local principal projections to obtain a compact representation. Across high-dimensional benchmarks and frozen TabPFN-style backbones, we evaluate the proposed approach on predictive performance, uncertainty-aware ranking, computational efficiency, and sustainability. Results show that SCOPE achieves a favorable compression, prediction, and compute trade-off by reducing representation size, inference cost, fit time, and memory usage while retaining competitive predictive performance on HDLSS benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
HDLSS, High-Dimensional Tabular Data, Tabular Foundation Models, Dimensionality Reduction, Feature Ordering, Representation Compression, Spectral Complexity, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WzPgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCOPE (Self-Calibrated Ordered Principal Encoding), a feature-compression framework for tabular foundation models (TFMs) in high-dimensional, low-sample-size (HDLSS) settings. The method first constructs a global feature ordering using LENS, which combines medoid-based nearest-neighbor initialization, candidate-pruned 2-opt refinement, and a budgeted minimum linear arrangement objective. Given this ordering, SCOPE estimates the effective rank of the data to determine the compressed output dimension, partitions the ordered features according to cumulative transition distances, and applies local principal component projections to obtain a compact representation.

Critical strengths

The paper proposes a tuning-free compression framework for HDLSS tabular data, combining feature ordering, spectral complexity estimation, and local PCA projections. This is consequential because it provides a practical way to adapt high-dimensional data to tabular foundation models without dataset-specific tuning or manually chosen output dimensions. The experiments are also reasonably broad, covering multiple datasets and baselines.

Critical weaknesses
  1. The rationale behind the parameter choices in the LENS module is insufficiently justified. Although the proposed configuration is intended to avoid dataset-specific hyperparameter tuning, it is unclear whether key parameters such as k, b_TSP, b_MinLA, eta, and R are theoretically motivated, empirically selected, or heuristic defaults. Without sensitivity analysis, it is difficult to assess how these choices affect feature ordering, compression quality, and downstream performance, or whether the configuration remains effective across datasets with different characteristics.
  2. The empirical evidence for SCOPE's effectiveness is not fully convincing. In Table 2, the performance differences between SCOPE and the second-best methods appear marginal, and no statistical significance tests are provided to determine whether the reported improvements are meaningful. In addition, the detailed results in Appendix G show that SCOPE is often outperformed by competing methods or achieves only marginal gains. This raises concerns about the consistency and practical significance of its claimed performance advantages.
  3. Writing and readability. The paper would benefit from clearer writing and the removal of redundant content, such as the second paragraph of the Introduction. The layouts of Figures 1 and 2 could also be refined to improve visual clarity.
  4. Reference formatting. The references are inconsistently formatted and should be standardized.
Questions for authors
  1. Parameter settings in the LENS module. Could the authors clarify the rationale for the parameter settings in the LENS module, including the edge-removal parameter k, b_TSP, b_MinLA, eta, and R? Are these values theoretically derived, empirically selected, or heuristic defaults? How sensitive are the resulting feature orderings, compressed representations, and downstream performance to these parameter choices across datasets with different characteristics?
  2. Statistical significance of experimental results. Could the authors provide statistical significance tests for the results in Table 2 to determine whether the reported improvements over the second-best methods are statistically meaningful? Given that the detailed results in Appendix G show that SCOPE is frequently outperformed by competing methods or achieves only marginal gains, how do the authors justify the claim that SCOPE provides consistent and practically significant performance advantages?
6 comments
AuthorsOpenReview ↗

Response to Reviewer WzPg (1/5)

We thank the reviewer for the constructive feedback and for recognizing the practical significance of our tuning-free compression framework for HDLSS tabular data. We particularly appreciate the positive remarks on its ability to adapt high-dimensional data to tabular foundation models without dataset-specific tuning or manually selected output dimensions, as well as the acknowledgment of our broad experimental evaluation.

W1. The rationale behind the parameter choices in the LENS ... different characteristics.

Q1. Parameter settings in the LENS ... different characteristics?

LENS parameter settings and theoretical rationale. We emphasize that these parameters are fixed algorithmic search-budget constants rather than hyperparameters selected through validation accuracy, downstream predictive performance, or dataset-specific optimization. The parameters are primarily theoretically derived. Their theoretical properties and computational rationale are established in Appendix B5, particularly in Proposition 3, which demonstrates bounded-search behavior and scale-equivariant refinement. Specifically, k=2 corresponds to the classical 2-opt local-search operation. The candidate budgets b_TSP=30 and b_MinLA=4 reflect the different computational costs of the two refinement stages: TSP 2-opt updates require only a constant number of distance lookups, allowing a broader candidate search, whereas exact MinLA evaluations are more expensive and therefore use a smaller candidate budget. The parameter η=1 permits one median adjacent-edge unit of TSP relaxation while preserving scale equivariance under uniform rescaling of the feature distance matrix. Finally, R=20 bounds the number of accepted MinLA refinements, limiting computational overhead. Thus, the selected configuration (30,4,1,20) represents a deterministic, computationally motivated design rather than a set of empirically optimized predictive hyperparameters. As stated in the manuscript, all values remain fixed across datasets without validation-based selection, preserving the tuning-free property of LENS with respect to downstream predictive performance. To complement this theoretical and computational justification, we additionally investigate parameter sensitivity through empirical ablation experiments below (Tables C1-C3).

Empirical LENS parameter sensitivity. To complement the theoretical characterization of the fixed LENS budgets, we conducted a "one factor at a time" sensitivity analysis over 12 configurations on 12 Feature Ordering Effectiveness (FOE)-characterized datasets (8 biological HDLSS, 2 text, and 2 face image datasets), using matched 5×5 repeated CV (3,600 predictions). The 2-opt neighborhood k=2 remains fixed because changing k changes the search algorithm rather than a numerical budget. Tables C1-C3 report ordering quality, held-out reconstruction retention, and downstream ROC-AUC. The default (b_TSP,b_MinLA,η,R)=(30,4,1,20) achieves 94.33% mean ROC-AUC; 9 of the 11 alternatives have a lower dataset-balanced mean, and 98/132 dataset-alternative pairs (74.2%) differ by at most 0.5 percentage points. Removing MinLA refinement (R=0) increases mean relative MinLA cost by 0.691% and reduces ROC-AUC by 0.342 pp; likewise, setting η=0 increases MinLA cost by 0.665% and reduces ROC-AUC by 0.339 pp. Increasing the TSP candidate budget from 30 to 60 increases mean ordering time by 46.0% and reduces ROC AUC by 0.403 pp, whereas increasing R to 40 adds 4.8% ordering time for only +0.020 pp ROC-AUC. A smaller TSP budget of 20 slightly improves mean ROC-AUC (+0.158 pp). These results demonstrate that the proposed default configuration provides stable and competitive predictive performance while maintaining a favorable balance between ordering quality and computational efficiency across diverse datasets. No alternative yields a statistically significant improvement under paired dataset level Wilcoxon tests with Holm correction (p_adj≥0.74). Furthermore, the analytical compression dimension M* remains unchanged across all parameter configurations and matched folds, demonstrating the stability of SCOPE's self-calibrated representation budget. Collectively, these findings support the robustness of the proposed default configuration, which delivers stable and competitive predictive performance with a favorable ordering quality-computational efficiency trade-off, without downstream performance-based tuning. Here, D denotes the default configuration (b_TSP,b_MinLA,η,R)=(30,4,1,20). T, L, E, and R identify changes to b_TSP, b_MinLA, η, and the maximum number of accepted MinLA refinements, respectively; all other parameters remain at their defaults.

  • See L412-L418, L3498-3509, L3551-L3563, L3603-L3617, L3662-L3667, and Tables L13, L14, L15, L16 in the updated manuscript.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer WzPg (2/5)

Table C1: LENS ordering quality sensitivity. Dataset-balanced means over 12 FOE-selected benchmarks. Time ratio, TSP cost difference, and MinLA cost difference are computed relative to D within each dataset before averaging; lower ratios/costs are better. Ordering score is normalized independently within each dataset across the same 12 settings (higher is better). Edge overlap denotes adjacency agreement with D where D denotes the default configuration (b_TSP,b_MinLA,η,R)=(30,4,1,20).

SettingScore ↑Time ratio ↓ΔTSP (%) ↓ΔMinLA (%) ↓Edge overlap ↑
D0.6531.000+0.000+0.0001.000
T100.6100.667+0.163-0.0770.869
T200.6800.800+0.039+0.0190.881
T600.6011.460-0.103-0.0040.867
L20.6710.923+0.000+0.0360.994
L80.6331.048+0.000-0.0080.992
E00.5430.824-0.023+0.6650.996
E.50.6740.963-0.012+0.0110.993
E20.6371.029+0.023-0.0780.994
R00.5310.724-0.023+0.6910.995
R100.6810.909+0.000+0.0210.999
R400.6401.048+0.000+0.0000.999

Table C2: SCOPE compression quality sensitivity. Dataset-balanced averages of held-out reconstruction retention (%) across 25 matched folds per dataset. Δ is the alternative minus D in percentage points; |Δ| is its mean absolute value across datasets. The fold-local output budget M* was invariant to all LENS parameter changes (12/12 datasets; 300/300 matched dataset-fold groups).

SettingRetention (%)Δ (pp)Mean |Δ| (pp)Mean M*Invariant M*
D50.41+0.0000.000535.612/12
T1050.12-0.2850.449535.612/12
T2050.48+0.0660.307535.612/12
T6050.49+0.0820.206535.612/12
L250.27-0.1350.145535.612/12
L850.37-0.0410.215535.612/12
E050.37-0.0370.092535.612/12
E.550.50+0.0950.188535.612/12
E250.75+0.3380.367535.612/12
R050.38-0.0250.087535.612/12
R1050.33-0.0810.094535.612/12
R4050.40-0.0130.013535.612/12

Table C3: Downstream predictive sensitivity of SCOPE + TabPFN-3. Dataset-balanced mean ROC-AUC (%) over 12 datasets and 25 matched folds per dataset. Δ is the alternative minus default D, in percentage points. Confidence intervals are unadjusted 95% dataset-bootstrap percentile intervals of the mean difference (30,000 resamples). W/T/L (Win/Tie/Loss) uses practical equivalence defined as |Δ_d| ≤ 0.5 pp. Across 11 paired Wilcoxon comparisons, none achieves p < 0.05 after Holm correction.

SettingROC-AUC (%)Δ (pp)95% CI (pp)W/T/L
D94.33+0.000—0/12/0
T1094.08-0.244[-0.57,+0.01]0/10/2
T2094.49+0.158[-0.13,+0.45]4/7/1
T6093.92-0.403[-0.78,-0.05]1/7/4
L294.18-0.150[-0.54,+0.13]1/10/1
L894.25-0.079[-0.39,+0.23]2/8/2
E093.99-0.339[-0.79,+0.01]0/9/3
E.594.19-0.138[-0.42,+0.12]2/8/2
E294.24-0.086[-0.41,+0.20]1/8/3
R093.99-0.342[-0.81,+0.03]1/8/3
R1094.25-0.075[-0.21,+0.03]0/11/1
R4094.35+0.020[-0.01,+0.07]0/12/0
  • See L3498-3509, L3551-L3563, L3603-L3617, L3662-L3667, and Tables L13, L14, L15, L16 in the updated manuscript.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer WzPg (3/5)

W2. The empirical evidence ... performance advantages.

Q2. Statistical significance ... performance advantages?

Statistical significance and predictive consistency. Appendix G reports ROC-AUC for downstream classification, which serves as one application for evaluating the predictive utility of SCOPE’s compressed representations. SCOPE’s primary objective is to construct compact, informative representations rather than introduce a new classifier. Its compression results should therefore be assessed jointly through dimensionality reduction, representation quality, and downstream computational savings, with ROC-AUC measuring how well predictive information is retained. To address the reviewer’s concern and further validate SCOPE's predictive effectiveness, we conduct comprehensive statistical significance analyses across all 30 datasets extracting results from Tables 2, G1, G2, G3, G4 using paired Wilcoxon signed-rank tests with Holm correction, Friedman tests, and Nemenyi post-hoc comparisons. The Friedman test confirms statistically significant differences over all 10 comparative compressors (χ²=225.17, p<0.001), with SCOPE achieving the best average rank of 2.20 (Table C4). Importantly, paired Wilcoxon tests with Holm correction establish statistically significant improvements (p_adj<0.05) for SCOPE over all ten compression baselines, including supervised PLS. Specifically, SCOPE outperforms PCA on 22 datasets (W/T/L: 22/4/4; p_adj=0.0019) and NSC-pSP on 21 datasets (21/3/6; p_adj=0.0020), achieving mean ROC-AUC improvements of 1.30 and 1.48 percentage points, respectively. SCOPE also demonstrates strong improvements over nonlinear and manifold-based compressors, achieving 30/0/0 W/T/L against UMAP, ParamRepulsor, PaCMAP, and LocalMAP. Domain-specific analyses further reinforce these findings. On extreme dimensional omics datasets, SCOPE achieves a leading average rank of 1.50 and outperforms PCA on all six datasets (Table C5). Notably, on 12 biological HDLSS datasets, SCOPE achieves the best average rank of 1.83 and statistically significant improvements over all nine unsupervised compression baselines, including PCA (11/0/1; p_adj=0.0293) and NSC-pSP (10/0/2; p_adj=0.0293) (Table C8). On the five text datasets, SCOPE achieves the best average rank of 2.60, outperforming PCA and NSC-pSP on three datasets each, while achieving consistent wins against GRP, UMAP, ParamRepulsor, PaCMAP, and LocalMAP (5/0/0 W/T/L for each; Table C6). On the seven face image datasets, SCOPE maintains highly competitive predictive performance, achieving near-ceiling ROC-AUC with small differences among the leading compression methods. It outperforms UMAP, ParamRepulsor, PaCMAP, and LocalMAP on all seven datasets (7/0/0 W/T/L for each), while remaining closely competitive with PCA and NSC-pSP (Table C7). Statistical comparisons are performed using dataset-level mean ROC-AUC scores, treating each dataset as one paired observation; the domain-specific analyses provide complementary evidence across benchmark categories. Collectively, these findings establish statistically supported cross-dataset improvements for SCOPE over existing compression methods, including supervised PLS, while demonstrating consistent predictive advantages across diverse high-dimensional data domains and particularly strong results in biological HDLSS settings.

Table C4: Cross-dataset statistical comparison of SCOPE.

Pooled comparison across 30 datasets and over 10 compression methods, including supervised PLS, with Raw as an uncompressed reference. W/T/L denotes wins, ties, and losses of SCOPE against each comparator. Δ is the mean ROC-AUC difference in percentage points (SCOPE minus comparator). p_W,H denotes two-sided Wilcoxon signed-rank p-values with Holm correction across ten comparisons; p_N denotes Nemenyi all-pairs p-values. Confidence intervals are unadjusted 95% domain-stratified bootstrap intervals. Raw is excluded from the multiple-comparison correction.

ComparatorW/T/LΔ (pp)95% CI (pp)p_W,Hp_N
PCA22/4/4+1.30[+0.69,+1.95]0.00190.786
NSC-pSP21/3/6+1.48[+0.75,+2.23]0.00200.463
AE24/3/3+2.31[+1.34,+3.44]3.64e-50.370
GRP26/2/2+2.58[+1.95,+3.20]5.59e-70.040
Kernel PCA26/3/1+2.54[+1.89,+3.20]4.47e-70.0074
UMAP30/0/0+10.29[+7.99,+12.97]1.86e-83.28e-11
ParamRepulsor30/0/0+11.46[+8.76,+14.42]1.86e-8<1e-12
PaCMAP30/0/0+16.29[+13.53,+19.28]1.86e-8<1e-12
LocalMAP30/0/0+18.18[+15.32,+21.27]1.86e-8<1e-12
PLS (supervised)16/3/11+1.44[+0.71,+2.21]0.0280.786
Raw13/3/14-0.30[-0.68,+0.13]——

→ Please continue to the next comment.

Reviewer_xn28OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SCOPE, a novel dimensionality reduction method with a deterministic, tuning-free feature compression framework for high-dimensional tabular data.

SCOPE consists of two main steps: (I) LENS builds a global feature ordering by combining efficient path construction and refinement based on the Traveling Salesman Problem (TSP) and Minimum Linear Arrangement (MinLA); and (II) SCOPE automatically determines the output dimensionality based on the spectral complexity of the data, partitions the ordered features using transition-aware segmentation, and applies local principal component analysis (PCA) to each segment to obtain a compact representation.

The authors evaluate SCOPE on 30 high-dimensional datasets using TabPFN-3 and TabICLv2, comparing it against various dimensionality reduction methods. The experimental results show that SCOPE achieves the highest average predictive performance among the compared dimensionality reduction methods while largely preserving the performance of the original uncompressed data. SCOPE also reduces inference time and computational resource usage.

Critical strengths

(I) SCOPE automatically determines the output dimension and feature segmentation without tuning hyperparameters for each dataset or using feedback from downstream models. It can also be applied to pretrained tabular foundation models without retraining them.

(II) The authors evaluate SCOPE on 30 high-dimensional datasets from four different domains using two tabular foundation models, TabPFN-3 and TabICLv2. They compare SCOPE with various dimensionality reduction methods. The experimental results show that SCOPE achieves good prediction performance while reducing the inference cost of downstream models.

Critical weaknesses

(I) One of the main goals of this paper is to reduce the computational cost of dimensionality reduction. However, LENS requires O(nm2)O(nm^2) time and O(m2)O(m^2) space to compute the distances between all pairs of features. Since these costs increase quadratically with the number of features, it may be difficult to apply LENS to datasets with hundreds of thousands or millions of features. In addition, Table L.2 shows that LENS takes 370.73 seconds for feature ordering on the SMK dataset. It is not clear whether the reported fit-time speedup includes the computational cost of LENS. Therefore, I have concerns about the scalability and the total computational cost of the proposed method.

(II) SCOPE is very similar to GOTabPFN, which also uses feature ordering and local PCA. However, SCOPE and the NSC-pSP baseline are evaluated with different settings. For example, NSC-pSP uses a 3,000-feature computational limit for GO-LR and limits the output dimension to 512, while SCOPE does not use these limits. Also, the ablation results in Table L.1 do not show statistically significant improvements from each component. Therefore, it is unclear whether the proposed method provides clear advantages over the existing method.

Questions for authors

(I) Does the reported fit-time speedup include the complete computational cost of LENS, including feature distance computation and TSP/MinLA optimization? Since LENS requires O(nm2)O(nm^2) time and O(m2)O(m^2) space, could the authors also clarify how the computational time and memory usage increase with the number of features?

(II) Table L.6 shows that NSC-pSP uses a 3,000-feature computational limit for GO-LR and a maximum output dimension of 512, while SCOPE does not use the same limits. Why are these different settings used? Would SCOPE still outperform NSC-pSP if both methods were evaluated with the same output dimensions and comparable computational budgets?

(III) Table L.1 shows that removing individual components does not lead to statistically significant performance differences. Also, Table L.2 shows that Full LENS takes much more time than Medoid NN, but the improvement in ROC-AUC is very small. Could the authors provide stronger experimental evidence that these components are necessary to improve prediction performance?

(IV) After LENS determines the feature ordering, why is the proposed SCOPE compression method necessary? Could other simpler compression methods, such as mean pooling or local PCA with equal-sized segments, achieve similar prediction performance? Although Table L.1 provides some ablation results on the SMK dataset, it is unclear whether SCOPE has clear advantages over these simpler methods across different datasets.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xn28 (1/4)

We thank the reviewer for valuable feedback and for recognizing SCOPE’s automatic dimensionality selection and segmentation, its compatibility with pretrained TFMs without retraining, and its broad evaluation across domains and backbones. We appreciate the acknowledgment of its competitive predictive performance and reduced downstream inference cost. The reviewer’s constructive feedback has helped us strengthen the controlled comparisons and clarify SCOPE’s computational costs and methodological contributions.

W1. (I) One of the main goals ... proposed method.

Computational complexity of LENS. We respectfully provide a clarification. Our primary objective is to construct compact, informative representations that reduce downstream TFM computation on HDLSS data while maintaining competitive predictive performance relative to raw inputs. This objective is established in prior literature: [1] demonstrated reduced dimensionality and training time while matching classification accuracy; [2] emphasized computational efficiency without sacrificing predictive performance; and [3] demonstrated compact representations under varying computational constraints. Accordingly, we evaluate SCOPE through both downstream predictive utility and computational efficiency. This aligns with TACO [10], which evaluates sample compression through competitive predictive performance and reduced TFM fitting and inference times.

LENS ordering runs entirely on the CPU, keeping its ordering buffers outside GPU memory. Distance construction scales linearly with n and quadratically with m. In our low-sample biological benchmark, ordering finishes within 37 seconds on 9/12 datasets. SMK (187×19,993) requires 332.47 seconds, consistent with the previously observed 320–370-second range; subsequent SCOPE compression takes 0.31–1.27 seconds across all 12 datasets. Mean total preprocessing time is 72.95 seconds, with a median of 28.96 seconds; SCOPE compression averages 0.72 seconds. These measurements quantify the practical preprocessing cost within our evaluated regime.

Ordering is a one-time transductive process following GOTabPFN [11], reused across all 25 CV folds; SCOPE compression remains fold-specific. Increasing n or m incurs additional computation, and dense MinLA buffers require O(m²) host memory. Setting CHECK_HOST_RAM=True checks available host RAM before allocation. This safeguard supports execution on sufficiently provisioned systems but does not change the quadratic storage requirement. See our responses to Reviewer Rq3u regarding timing definitions and Reviewer WzPg regarding ordering sensitivity.

LENS ordering and training-fold SCOPE compression can be performed offline, with compressed representations cached for subsequent downstream use. This follows the broader principle of compressing and reusing representations to avoid repeated computation [12].

Table B1. Total runtime on 12 biological datasets. LENS runs on CPU; SCOPE uses GPU-enabled PC1. Total includes ordering and compression. Each dataset uses capped M=n−1; shared loading and imputation are excluded.

Datasetn×mLENS (s)SCOPE (s)Total (s)
COL62×2,0003.920.314.23
LNG203×3,3129.821.1811.01
GLI85×22,283268.310.60268.91
SMK187×19,993332.471.18333.65
AML72×7,12928.720.4429.15
PRS102×5,96636.120.5036.62
ARC200×10,00030.411.2731.67
TOX171×5,74820.340.9621.30
Lymphoma96×4,02610.060.4310.50
Leukemia72×7,07028.260.5128.77
GLIOMA50×4,43413.690.7414.43
CLL-SUB111×11,34084.570.5585.12
Mean—72.220.7272.95
Median—28.490.5828.96
  • See L528-L531, L3974-L3995, L4040-L4045, L4143-L4157, L4178-L4194, Table L27, Fig. L4, and the supplementary SMK ordering video.

W2. (II) SCOPE ... settings.

Comparison w/ GOTabPFN. GOTabPFN primarily targets predictive performance, whereas SCOPE targets compact, informative representations supporting improved or competitive downstream prediction with reduced dimensionality and computational cost. We directly compared GO-LR + NSC-pSP and LENS + SCOPE using TabPFN-2.5 and matched 5×5 repeated CV across eight classification datasets. GO-LR + NSC-pSP received 20 Optuna trials per dataset; SCOPE used fixed defaults. SCOPE achieved 88.39% accuracy, 85.14% macro-F1, and 96.99% ROC-AUC, compared with 88.40%, 84.20%, and 95.90% for GOTabPFN. SCOPE achieved higher ROC-AUC on six datasets, with one tie and one loss, while reducing total measured runtime, including tuning, from 103.42 to 12.97 minutes (7.97×). Predictive differences remained non-significant after Holm correction, supporting competitive performance without dataset-specific tuning (Tables B2–B3). Tuned baseline scores reuse the tuning CV partitions and are therefore post-selection estimates. Our full-feature, matched-output-budget ablation below additionally addresses unequal baseline settings.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer xn28 (2/4)

Table B2. Aggregate results (unweighted means across eight datasets). Runtime sums dataset-level wall-clock measurements. GO–NSC includes Optuna tuning; SCOPE includes global LENS ordering and 25-fold evaluation.

MetricGO–NSCSCOPEΔ (SCOPE−GO)
Accuracy (%)88.4088.39-0.01 pp
Macro-F1 (%)84.2085.14+0.95 pp
ROC-AUC (%)95.9096.99+1.09 pp
Total time (min)103.4212.97≈7.97× faster

Table B3. Paired dataset-level statistical tests (n=8). Δ is mean SCOPE minus GO–NSC in percentage points; W/L/T counts SCOPE wins/losses/ties. W and p are from two-sided Wilcoxon signed-rank tests on paired dataset means, omitting zero differences; p_Holm adjusts across three metrics. Two-sided exact sign-test p-values provide a robustness check. None establishes significance at α=0.05.

MetricΔ (pp)W/L/TWpp_Holmp_sign
Accuracy-0.023/3/290.8441.0001.000
Macro-F1+0.953/4/1120.8121.0001.000
ROC-AUC+1.096/1/160.2190.6560.125
  • See L3670-L3671, L3622-L3625, L3665-L3673, Tables L17 (dataset-wise results), L18 (summary), and L19 (statistics) in the updated manuscript.

W2. For example, NSC-pSP uses a 3,000-feature computational limit for GO-LR and limits the output dimension to 512, while SCOPE does not use these limits. Also, the ablation results in Table L.1 do not show statistically significant improvements from each component. Therefore, it is unclear whether the proposed method provides clear advantages over the existing method.

Same Budget Ablation. We additionally compare full-feature GO-LR + NSC-pSP against budget-matched LENS + SCOPE on eight datasets. Both order all numeric features and produce exactly the same output dimension in every fold, using TabPFN-2.5 with the same predictor seed and matched 5×5 CV splits. The common budget is the smaller native output width, adjusted where necessary for NSC-pSP's segmentation constraints; SCOPE is capped accordingly. Remaining GO-LR/NSC-pSP settings are retained from the previous tuning experiment. SCOPE achieves higher mean accuracy (87.21% vs 85.56%), macro-F1 (83.66% vs 81.87%), and ROC-AUC (95.77% vs 95.09%), with paired gains of 1.647, 1.786, and 0.677 pp (Tables B4–B5). Thus, its numerical advantages persist without a larger output representation or restricted GO-LR input coverage. This evaluates compression at equal capacity rather than native budget-selection rules. Also see our response to Reviewer Rq3u, W4.

Table B4. Full-feature, matched-budget comparison using TabPFN-2.5, predictor seed 42, and matched 5×5 CV. GO–NSC denotes GO-LR + NSC-pSP; SCOPE denotes budget-matched LENS + SCOPE. Both order all numeric features and produce identical fold-specific output dimensions. M gives the shared output-width range across folds. Values are mean±fold SD (%); bold marks the higher displayed mean, including ties. Mean rows average dataset means.

DatasetPipelineMAccuracyMacro-F1ROC-AUC
GLIOMAGO–NSC7872.80±13.3968.53±16.1094.60±4.58
GLIOMASCOPE7874.80±10.4672.11±12.9995.04±3.72
CLL-SUBGO–NSC172–17562.28±7.1471.15±6.3278.39±6.21
CLL-SUBSCOPE172–17567.23±10.2275.40±7.8283.97±6.14
LymphomaGO–NSC159–16582.53±8.4255.84±13.7796.12±5.62
LymphomaSCOPE159–16590.02±7.9563.07±12.6297.87±4.30
LeukemiaGO–NSC109–11196.11±4.5195.61±5.2098.63±2.89
LeukemiaSCOPE109–11193.68±5.9192.90±6.6797.32±3.67
dbworld_bodiesGO–NSC96–10088.92±8.9388.50±9.5594.68±6.72
dbworld_bodiesSCOPE96–10086.77±8.5986.35±8.9893.14±7.54
Biblical TextsGO–NSC37683.05±3.7276.51±5.4098.35±0.50
Biblical TextsSCOPE37686.41±3.2980.65±5.5998.81±0.47
warpPIE10PGO–NSC94–9799.71±0.7999.73±0.75100.00±0.00
warpPIE10PSCOPE94–97100.00±0.00100.00±0.00100.00±0.00
warpAR10PGO–NSC80–8699.08±2.0199.09±2.0699.97±0.13
warpAR10PSCOPE80–8698.77±2.4198.77±2.4599.99±0.04
MeanGO–NSC—85.5681.8795.09
MeanSCOPE—87.2183.6695.77

Table B5. Aggregate matched-budget results on eight datasets. Means average dataset means; Δ is SCOPE minus GO–NSC in percentage points, computed from unrounded results. W/T/L counts SCOPE wins/ties/losses.

MetricGO–NSCSCOPEΔW/T/L
Accuracy (%)85.5687.21+1.6475/0/3
Macro-F1 (%)81.8783.66+1.7865/0/3
ROC-AUC (%)95.0995.77+0.6775/1/2
  • See L501-L509, L4247-L4258, and Tables L32, L33 in the updated manuscript.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer xn28 (3/4)

Q1. (I) Does the reported fit-time speedup include the complete computational cost of LENS, including feature distance computation and TSP/MinLA optimization? Since LENS requires O(nm²) time and O(m²) space, could the authors also clarify how the computational time and memory usage increase with the number of features?

Computational Complexity of LENS. As clarified under W1, main downstream TFM timings measure model.fit and model.predict_proba on raw versus already compressed representations for all compressors, excluding ordering and compression. LENS ordering time includes feature-distance computation and TSP/MinLA optimization; the order is reused across 25 CV folds. Fold-local compression is recorded separately, and the additional pipeline-runtime field includes compression and downstream prediction costs. Table B1 reports preprocessing costs, including 332.47 seconds for SMK ordering. BeyondArena instead includes fold-local compression/projection.

Dense distance construction requires O(nm²) time and O(m²) storage; doubling m at fixed n approximately quadruples both distance-construction work and dense-matrix storage. CPU execution keeps ordering buffers outside GPU memory, while CHECK_HOST_RAM=True checks available host RAM before allocation. These safeguards do not change the scaling, whose practical cost is quantified within our evaluated HDLSS regime. Our predictive evaluation reaches 67,420 features; the six extreme-dimensional datasets are excluded from the main efficiency analysis because raw TabPFN-3 requires ignore_pretraining_limits=True.

Our intended operational HDLSS regime is m>1000, n<1000, and ρ=m/n>2, consistent with our dataset categorization. The condition ρ>2 has a rank-deficit justification in Appendix D; the absolute thresholds are operational conventions. We distinguish demonstrated applicability within this evaluated regime from unrestricted scalability to millions of features, which is outside our present claims. Improving large-scale ordering remains future work.

Q2. (II) Table L.6 shows that NSC-pSP uses a 3,000-feature computational limit for GO-LR and a maximum output dimension of 512, while SCOPE does not use the same limits. Why are these different settings used? Would SCOPE still outperform NSC-pSP if both methods were ... computational budgets?

GO-LR, NSC-pSP vs SCOPE, LENS. The original limits were computational safeguards informed by GOTabPFN's [11] feature-subsampling and output-budget settings in Tables H.1–H.2, rather than shared constraints. Our additional eight-dataset ablation removes GO-LR's feature restriction and matches both compressors' output dimension in every fold, using TabPFN-2.5, predictor seed 42, and matched 5×5 CV. SCOPE achieves higher mean accuracy (87.21% vs 85.56%), macro-F1 (83.66% vs 81.87%), and ROC-AUC (95.77% vs 95.09%), with W/T/L counts of 5/0/3, 5/0/3, and 5/1/2 (Tables B4–B5). Thus, its numerical advantages persist with matched input-feature coverage and output dimensionality. This controls representation size and downstream evaluation settings, rather than imposing identical ordering-runtime budgets.

Q3. (III) Table L.1 shows that removing individual components does not lead to statistically significant performance differences. Also, Table L.2 shows that Full LENS takes much more time than Medoid NN, but the improvement in ROC-AUC is very small. Could the authors provide stronger experimental evidence that these components are necessary to improve prediction performance?

Expanded LENS stage ablation. We expanded the ordering-stage ablation to 12 biological datasets using identical 5×5 CV splits, matched output dimensions, and SCOPE + TabPFN-3 settings (Table B6). Full LENS achieves the highest mean ROC-AUC (93.892%) and best mean rank (1.667), outperforming MedoidNN on 7/12 datasets and TSPOnly on 9/12, with gains of 0.131 and 0.381 pp. This extends the evidence beyond SMK; additional ordering cost is incurred once and reused across 25 folds. Full LENS is the strongest aggregate predictive configuration in this ablation, while MedoidNN offers substantially cheaper ordering with a small average ROC-AUC reduction. These numerical advantages do not establish that every component is individually necessary; none of the four component comparisons reaches significance after Holm correction (Table B7).

Table B6. LENS stage ablation on 12 biological datasets with SCOPE + TabPFN-3. All variants use identical 5×5 CV splits and matched fold-specific output dimensions. Ordering time is the median across datasets, including global preprocessing and feature-distance construction but excluding compression and TFM computation.

OrderingROC-AUC (%)↑Mean rank↓Ordering (s)↓
MedoidNN93.7612.0000.83
TSPOnly93.5112.33313.40
Full LENS93.8921.66728.72
  • See L4259-L4265, L4302-L4304, and Table L34 in the updated manuscript.

→ Please continue to the next comment.

Reviewer_Rq3uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an unlearned framework for compressing tabular features with the motivation of mitigating the inability of some modern tabular foundation models to deal with wide datasets, specifically in HDLSS setups. The main contributions are:

  1. LENS: a feature ordering mechanism that introduces a cheaper heuristic for arranging features so that related features are placed near each other.
  2. One-shot segmentation: The paper presents a mechanism for dividing the ordered features based on local structural changes, replacing the recursive segmentation used in past works. PCA is then applied to each of these segments separately, with the outputs concatenated to produce the final representation suitable for pretrained tabular foundation models.
  3. Self-Calibrated Compression Budget: The authors derive a closed-form rule for automatically determining the output dimensionality based on the dataset's effective spectral rank. While the framework includes configurable hyperparameters, the authors fix these across datasets rather than performing dataset-specific hyperparameter tuning.

The authors present an extensive evaluation over 30 datasets from 4 different domains in the HDLSS setup.

Critical strengths
  1. A relevant problem and a practical, simple-to-use solution: A large number of features is a bottleneck for modern TFMs, and the community would benefit heavily from ways of representing such input in a compact manner. The paper frames this problem well. The paper also presents an elegant solution to this problem with a mechanism that transforms such input deterministically in a single pass without a need for labels or validation feedback.
  2. A broad empirical study showing consistent returns: The paper runs experiments on 30 datasets and 4 domains with 5x5 repeated CVs, with per-dataset results and standard deviations, not just aggregates. The paper also reports results on two backbones and 11 baselines, including some strong competitors. Throughout these experiments, SCOPE scores consistently well. It ranks as the best compressor across all domains. While its margins over PCA are generally small outside biological and omics data, it often scores lower standard deviations, which makes it a compelling, reliable alternative. The paper also honestly reports the null results and the losses, such as losing to the raw backbone in BeyondArena.
Critical weaknesses
  1. The ablations are incomplete and undermine the thesis that LENS matters for downstream predictive performance. The only ablation used to test the effect of LENS on downstream tasks is on a single dataset (SMK), and it shows statistically insignificant differences between ordering, segmentation, and pooling alternatives. Random feature selection ties with SCOPE, and the supervised ANOVA-k beats it numerically. I acknowledge that this single dataset might be a low-signal experiment with high variance, but then I believe that this warrants expanding the ablation study to a systematic ordering and segmentation ablation across all datasets with statistical tests to isolate and understand the effect of LENS on downstream performance.

  2. The claim that SCOPE preserves or outperforms performance is not fully supported by evidence. On BeyondArena, TabPFN-3 + SCOPE loses to raw by 37.8 Elo points for 1.4x fit /1.64x inference speedups. Improvement over raw models (higher than 100% retention rates) occurs when TabPFN is run beyond its limits with ignore_pretraining_limits=True or when the raw model is already weak, in the case of TabICLv2 on the Biological HDLSS datasets, where even PCA reaches a 104.11% retention rate. The only case where all models are in regime, and SCOPE had better performance than raw, is face images, where SCOPE scores a 100.10% retention rate, which might reflect ceiling effects, with most methods at or near 100 ROC-AUC and vanilla PCA scores 100.01%. The authors should reframe their claims to near-raw performance at a lower cost.

  3. Incomplete comparison with baselines: The paper cites TabPFN-Wide as related work but doesn't add it to the comparison as a baseline. Feature selection methods should also be added to the list of baselines compared against. Random feature selection and supervised ANOVA-k are also valid baselines that are only reported on SMK, as stated in point 1. Random-block PCA (random grouping with the same M* and PC1 per block) also isn't tested on all datasets, and it would serve to isolate the effects of ordering.

  4. The contribution over earlier works is incremental. As acknowledged in Appendix C.7, SCOPE retains the core principle of ordered, contiguous, segment-wise PCA from NSC-pSP, with its main contributions limited to a closed-form budget rule, one-shot rather than recursive segmentation, and a cheaper ordering heuristic. Importantly, while SCOPE is shown to perform better on real data experiments, the synthetic paired Wilcoxon tests (Table K.3) show no statistically significant improvement for SCOPE over NSC-pSP, which was evaluated without tuning. While this is a fair setup for comparing tuning-free methods, it does not establish superiority over a properly tuned version of the prior state of the art. The gains of SCOPE over NSC-pSP on the real data could also be due to capping the GO-LR ordering used by NSC-pSP at 3000 features.

  5. Less critically, the claim that SCOPE requires no tuning is overstated since the fixed constants are still design choices that could be tuned. However, the experiments present results without tuning these constants, so such a claim is practically true but should be less overstated. A sensitivity analysis of the performance of SCOPE against different constants, showing that SCOPE is robust to different choices of these constants, would strengthen this claim.

Questions for authors
  1. Is the LENS ordering computed separately on each training fold, or once on the full dataset including test samples? Eq.1 and Algorithm C.1 apply LENS to X, whereas Section 3.2 and Appendix L state that all reducers are fit on training-fold data only.
  2. Does the 370.7s LENS ordering time on SMK (Table L.2) refer to a single run or to the total across the 25 folds? Is this cost included in the reported reducer times (e.g., 0.283 s in Table L.5) and in the fit speedups in Tables G.5 and H.3?
16 comments
AuthorsOpenReview ↗

Response to Reviewer Rq3u (1/6)

We thank the reviewer for the detailed feedback and for recognizing SCOPE’s practical simplicity, broad evaluation, and consistent performance across domains. We particularly appreciate the acknowledgment of our honest reporting of null results and losses, reflecting our commitment to transparency and reproducibility. The feedback has helped us strengthen the evidence and sharpen the scope of SCOPE’s claims.

W1. The ablations are incomplete ... alternatives.

Expanded component ablation. We expanded the ordering, segmentation, and pooling ablation from SMK to 16 datasets (12 biological, 4 text), using identical 5×5 CV splits, fold-local token budgets, and TabPFN-3 settings. Randomized controls average five fixed ordering seeds. SCOPE achieves the highest mean ROC-AUC (94.252%) and best mean rank (2.188). Against random ordering with matched NSC segment widths, SCOPE gains 0.994 pp and wins on 13/16 datasets (p=0.0155; Holm-adjusted p=0.0620). Gains over RandomEqual and LENSEqual are 0.650 and 0.420 pp; replacing PC1 with mean pooling changes mean AUC by only 0.067 pp. None reaches significance after Holm correction. These results broaden the evidence beyond SMK and favor the full configuration numerically, while indicating limited average predictive benefit from PC1 here. We therefore temper claims about individual components.

Table A1. Matched component ablation across 16 datasets. Δ is SCOPE minus comparator in percentage points; p_H is Holm-adjusted.

MethodAUC (%)Δp_H
SCOPE94.252——
RandomMatched93.2580.9940.062
RandomEqual93.6020.6500.389
LENSEqual93.8320.4200.645
LensNSCMean94.1850.0670.669

Table A2. Component ablation across 16 datasets (12 biological, 4 text). Δ is SCOPE minus comparator in AUC pp. W/T/L counts dataset-level SCOPE wins/ties/losses. Two-sided paired Wilcoxon tests use 16 dataset means; Holm correction covers four comparisons.

MethodAUC (%)RankΔW/T/Lpp_Holm
SCOPE94.25202.1875————
RandomMatched93.25794.06250.994113/0/30.01550.0620
RandomEqual93.60193.56250.650112/0/40.12970.3892
LENSEqual93.83172.81250.420411/0/50.32250.6450
LensNSCMean94.18492.37500.06719/0/70.66850.6685
  • See L4196-L4206 and Tables L28, L29 in updated manuscript.

Table A3. Controlled comparisons evaluating individual components.

ComparisonComponent evaluated and controls
SCOPE vs RandomMatchedLENS vs random feature grouping; NSC block widths and PC1 fixed.
LENSEqual vs RandomEqualLENS vs random ordering; equal-width segmentation and PC1 fixed.
SCOPE vs LENSEqualNSC vs equal-width segmentation; LENS, output budget, and PC1 fixed.
SCOPE vs LensNSCMeanPC1 vs mean pooling; LENS and NSC segmentation fixed.

W2. The claim that SCOPE preserves or outperforms performance is not fully supported by evidence. On BeyondArena, TabPFN-3 + SCOPE loses to raw by 37.8 Elo points for 1.4x fit /1.64x inference speedups.

BeyondArena evaluation. Six BeyondArena datasets satisfy m>n, a broader filter than our intended m≫n setting. They are distinct from our main 30 datasets, with (n,m): clock_protein_toxicity (171,1117), lung_cancer (197,12600), lung_cancer_epithelial_genexp (187,22215), micro_mass (571,1083), pancreatic_cancer_mouse_detection (181,6772), and prostate_cancer_detection (322,15154). Five satisfy our operational HDLSS definition; some have relatively modest feature-to-sample ratios. Only lung_cancer_epithelial_genexp exceeds TabPFN-3's reported feature frontier.

We follow BeyondArena's default task splits, validation protocol, and evaluator in its separate environment, using a fresh Raw TabPFN-3 run for controlled comparison. SCOPE constructs compressed inputs from all original features. Adding PCA, SCOPE achieves 167.2 higher Elo with 13.6% longer median fit and 9.4% longer median inference time (Table A4). We acknowledge the original 37.8-point loss to Raw as a predictive–efficiency trade-off, rather than preservation without loss. Table A4 recomputes Elo across three methods and shuffled schedules; its Raw–SCOPE gap is therefore distinct from the original two-method gap.

Table A4. Controlled comparison on six BeyondArena datasets and 43 shared outer tasks. Three-method Elo averages 1,000 shuffled match schedules. Times include fold-local compression/projection and TabPFN computation, excluding one-time LENS ordering. Speedups are relative to Raw.

MethodEloFit (s)Infer. (s)Fit speedupInfer. speedup
Raw TabPFN-31134.7227.9033.5291.00×1.00×
SCOPE + TabPFN-31016.3162.6032.1491.40×1.64×
PCA + TabPFN-3849.1143.1441.9641.59×1.80×
  • See L3330-L3347, Fig. L2(b), and Table L6 in updated manuscript.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer Rq3u (2/6)

W2. Improvement over raw models (higher than 100% retention rates) occurs when TabPFN is run beyond its limits with ignore_pretraining_limits=True or when the raw model is already weak

TabPFN pre-training limits. We clarify our protocol, which the original submission may not have communicated clearly. TabPFN-3 supports n×m regimes of 1,000×20,000, 100,000×2,000, and 1,000,000×200, where n and m denote training samples and features. Among our 30 datasets, GLI and the six extreme-dimensional datasets exceed the reported feature limit. We enabled ignore_pretraining_limits=True for the six extreme datasets and excluded them from efficiency analyses (Fig. 4, Table G5, Fig. G1). Tables G1–G2 explicitly state: “TabPFN-3 is used as the downstream TFM. Its standard n×m operating limits are followed where applicable; otherwise, ignore_pretraining_limits=True is enabled without applying additional dimensionality reduction or feature selection.”

W2. in the case of TabICLv2 on the Biological HDLSS datasets, where even PCA reaches a 104.11% retention rate.

TabICLv2 gain. Raw TabICLv2 performs below Raw TabPFN-3 on these 12 biological datasets (90.08% vs 94.48% mean ROC-AUC). TabArena and BeyondArena also rank TabPFN-3 above TabICLv2 overall. Both compressors improve TabICLv2 in this challenging regime, but SCOPE achieves higher ROC-AUC (93.61% vs 93.36%) and retention (104.42% vs 104.11%), beating PCA on seven datasets with one tie (Tables H1, G4). With the stronger TabPFN-3 baseline, SCOPE remains close to Raw (93.87% vs 94.48%), retaining 99.32% compared with PCA's 97.89%. SCOPE beats PCA on 11/12 datasets, with 1.39 pp higher mean ROC-AUC. Thus, its advantage over PCA also holds with the stronger backbone, beyond improving a weaker raw-input baseline. We claim competitive predictive utility across backbones, not universal improvement over Raw.

Table A5. ROC-AUC on 12 biological HDLSS datasets (Tables H.1, G.4). Method Avg. is mean±sample SD of 12 dataset-level mean AUCs (%). Retention averages dataset-wise percentages relative to Raw with the same backbone. W/T/L compares against same-backbone PCA using displayed dataset means; self-comparisons are omitted.

Representation + backboneMethod Avg.Retention (%)W/T/L vs PCA
Raw + TabICLv290.08±9.99100.002/1/9
SCOPE + TabICLv293.61±7.11104.427/1/4
PCA + TabICLv293.36±7.38104.11—
Raw + TabPFN-394.48±6.65100.0011/0/1
SCOPE + TabPFN-393.87±7.0799.3211/0/1
PCA + TabPFN-392.48±7.0397.89—

W2. The only case where all models are in regime, and SCOPE had better performance than raw, is face images, where SCOPE scores a 100.10% retention rate, which might reflect ceiling effects, with most methods at or near 100 ROC-AUC and vanilla PCA scores 100.01%. The authors should reframe their claims to near-raw performance at a lower cost.

Competitive predictive utility under compression. Compression need not outperform Raw; its goal is competitive predictive utility with reduced dimensionality and better efficiency. We consistently disclosed the use of ignore_pretraining_limits=True for extreme datasets and excluded them from efficiency evaluation. In TACO's Table 1 [1], TabICL and TabPFN-v2 achieve 0.866±0.103 mean ROC-AUC, whereas TACO reaches at most 0.858±0.100. TACO likewise emphasizes large inference and memory savings with little predictive degradation at 1% of the original context size.

Our SCOPE vs Raw TabPFN-3 mean ROC-AUCs (%) are: biological 93.87±7.07 vs 94.48±6.65; face images 99.30±1.47 vs 99.21±1.61; text 95.64±2.30 vs 96.69±3.44; extreme-dimensional 97.23±1.66 vs 96.76±1.80. Raw is a reference to approach, not a mandatory threshold to exceed. Similarly, tabular generation evaluates TSTR (Train on Synthetic, Test on Real) against TRTR (Train on Real, Test on Real), treating TRTR as the reference [2]. We acknowledge face-image ceiling effects and do not interpret the small gain over Raw as meaningful superiority. We revised our claims to emphasize competitive, near-raw predictive performance with reduced downstream computation.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Response to Reviewer Rq3u (3/6)

W3. Incomplete comparison with baselines: The paper cites TabPFN-Wide as related work but doesn't add it to the comparison as a baseline.

TabPFN-Wide as a baseline. TabPFN-Wide addresses HDLSS prediction through a specialized synthetic prior, continued pre-training, and attention-based feature interpretability, without explicit feature reduction. It primarily targets prediction, whereas SCOPE is a standalone, model-agnostic feature compressor. GOTabPFN combines ordering-based compression with a frozen TFM; TabSwift and TabDPT use PCA internally; TACO targets sample compression. SCOPE follows GOTabPFN's ordering principle and TACO's emphasis on evaluating compression.

Nevertheless, we added TabPFN-Wide comparisons on seven high-dimensional datasets. SCOPE + TabPFN-2.5 and SCOPE + TabPFN-3 achieve +0.78 and +0.14 pp mean ROC-AUC, respectively, with lower recorded aggregate runtimes: 12.01 and 10.59 vs 17.72 minutes.

Table A6. TabPFN-Wide vs default SCOPE pipelines on seven datasets. Wide AUC is mean±SD (%) across 5×5 CV. ΔAUC is SCOPE + TabPFN minus Wide (pp); Δt is Wide minus SCOPE runtime (min; positive favors SCOPE). Columns 2.5 and 3 denote TabPFN-2.5 and TabPFN-3. Δt_3† uses measured stage sums.

DatasetWide AUC (%)ΔAUC 2.5ΔAUC 3Wide time (min)Δt_2.5Δt_3†
GLIOMA95.57±3.49+0.22+0.300.91+0.26+0.18
CLL-SUB89.16±4.74+1.44-0.853.48+1.00+1.57
Leukemia98.98±2.72+0.50+0.331.63+0.39+0.64
dbworld_bodies93.27±7.99-0.54+1.070.86+0.10-0.57
Biblical Texts95.75±0.96+3.20-0.449.59+3.82+5.95
warpPIE10P100.00±0.000.000.000.69+0.11-0.24
warpAR10P99.39±0.70+0.61+0.600.57+0.04-0.39
Mean AUC96.02+0.78+0.14———
Total time———17.72+5.71+7.14
  • ROC-AUC W/T/L: 5/1/1 for SCOPE + TabPFN-2.5; 4/1/2 for SCOPE + TabPFN-3.
  • See L3879-L3887, L3934-L3941, and Table L25 in updated manuscript.

W3. Feature selection methods should also be added to the list of baselines compared against. Random feature selection and supervised ANOVA-k are also valid baselines that are only reported on SMK, as stated in point 1.

W1. Random feature selection ties with SCOPE, and the supervised ANOVA-k beats it numerically.

Feature selection vs compression. Selection retains original features through axis-aligned projections, whereas compression combines information into new representations. Their objectives and constraints differ [3,4], so equal output dimensionality does not make them interchangeable.

Comparison with feature selection. We nevertheless added supervised ANOVA-k and label-free Random-k on 20 datasets (12 biological, 7 face image, 1 extreme), using TabPFN-3, matched 5×5 CV, and identical fold-specific output widths. SCOPE achieves the highest mean ROC-AUC (95.911%), compared with ANOVA-k (95.841%) and Random-k (95.773%), with W/T/L of 12/2/6 and 10/5/5 (Table A7). With HDLSS-UCBT extended to 12 methods, SCOPE ranks first (1258.7), has the highest UCPW (75.68), and lowest average rank (2.40). It ranks first in 72.1% of 1,000 dataset-bootstrap replicates, although leading confidence intervals overlap. These support competitive predictive utility and favorable uncertainty-aware ranking for label-free, tuning-free compression. UCBT means Uncertainty-Calibrated Bradley–Terry; UCPW means Uncertainty-Calibrated Pairwise Win Score (Section 3.4, Appendix E).

Table A7. SCOPE vs feature selection. ROC-AUC (mean±SD, %) with TabPFN-3 under 5×5 CV. Selectors retain the same fold-specific width as SCOPE. Means are unweighted across datasets; SDs describe pooled fold variability, including between-dataset variation. W/T/L counts SCOPE's dataset-level wins/ties/losses. Bold marks the highest row mean.

Domain#SCOPEANOVA-kRandom-kvs ANOVA W/T/Lvs Random W/T/L
Biological1293.870±8.3394.018±8.1593.823±8.156/0/67/0/5
Face-image799.304±1.5398.869±2.3799.089±1.895/2/02/5/0
GBM (Extreme)196.640±1.8296.530±1.7195.960±1.841/0/01/0/0
Overall2095.911±7.0195.841±6.8695.773±6.8812/2/610/5/5
  • See L511-L520, L3816-L3833, L3858-L3876, Tables L22–L24, and Fig. L3 in updated manuscript.

→ Please continue to the next comment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reviewer Response Summary (1/2)

Dear Area Chair and Reviewers,

We thank you for your constructive feedback and the time and effort devoted to reviewing our manuscript. Below, we summarize the main concerns and our additional experiments and revisions.

Reviewer Rq3u

  • Expanded component ablations: Extended ordering, segmentation, and pooling comparisons from SMK to 16 datasets; SCOPE achieves the highest mean ROC-AUC (94.252%) and best mean rank (2.188), with no statistically significant differences after Holm correction (Tables A1–A3).
  • BeyondArena trade-off: Added PCA on the same six datasets and 43 tasks; SCOPE achieves 167.2 higher three-method Elo than PCA, while acknowledging the original 37.8-point loss to Raw alongside 1.40× fit and 1.64× inference speedups (Table A4).
  • TabPFN-3 operating limits: Clarified the documented use of ignore_pretraining_limits=True for the six extreme dimensional datasets and their exclusion from the main efficiency analysis.
  • Backbone-dependent gains: SCOPE’s advantage over PCA also holds with the stronger raw TabPFN-3 baseline: +1.39 ROC-AUC points, 11/12 dataset wins, and 99.32% retention, compared with 97.89% for PCA (Table A5).
  • Predictive claims and ceiling effects: Acknowledged that near-ceiling face image gains do not establish meaningful superiority over Raw and revised the claims to emphasize competitive, near-raw predictive performance with reduced downstream computational cost.
  • TabPFN-Wide baseline: Added comparisons on seven datasets; SCOPE + TabPFN-2.5 and SCOPE + TabPFN-3 achieve mean ROC-AUC gains of 0.78 and 0.14 percentage points, respectively, with lower recorded aggregate runtimes (Table A6).
  • Feature-selection baselines: Added matched-dimensional ANOVA-kk and Random-kk comparisons on 20 datasets; SCOPE achieves the highest mean ROC-AUC (95.911%) and ranks first in the expanded 12-method HDLSS-UCBT comparison (Table A7).
  • Random-block PCA controls: Extended matched-budget ordering and segmentation comparisons to 16 datasets with five random-ordering seeds; SCOPE outperforms RandomMatched on 13/16 datasets, gaining 0.994 ROC-AUC points (p=0.0155p=0.0155 unadjusted; Holm-adjusted p=0.0620p=0.0620; Table A8).
  • Contribution beyond GOTabPFN: Clarified that LENS removes clustering and preliminary feature subsampling, while SCOPE replaces dataset-specific compression settings with automatic spectral budgeting and one-shot segmentation, yielding a deterministic framework without downstream validation-based tuning.
  • Tuned-baseline comparison: On eight datasets, default SCOPE achieves 96.99% ROC-AUC versus 95.90% for tuned GOTabPFN with 7.97× lower total runtime including tuning; jointly tuning SCOPE adds 0.10 ROC-AUC points at 16.51× runtime, without statistically significant predictive gains (Table A9).
  • GO-LR feature-cap control: Removed feature subsampling on eight datasets; SCOPE retains a 1.875-point mean ROC-AUC advantage with 6 wins, 1 tie, and 1 loss, although differences remain non-significant after Holm correction (Tables A10–A11).
  • Tuning-free clarification: Defined the term as no dataset-specific validation search, supported by sensitivity analysis across 12 configurations on 12 datasets and joint-tuning experiments; revised the wording to acknowledge that tuning can improve some average metrics (Table A9; Tables C1–C3).
  • Ordering protocol: Clarified that LENS computes one ordering from the full unlabeled dataset, including test-fold features but never test labels, and reuses it across 25 folds; SCOPE compression and downstream predictor fitting remain training-fold-specific.
  • Runtime accounting: Clarified that SMK’s 370.73 s ordering is a one-time cost, separate from fold-local compression and downstream TFM speedups; BeyondArena additionally includes fold-local compression/projection.
  • Ablation coverage: Expanded component comparisons to 16 datasets and feature-selection comparisons to 20.

→ Please continue to the next comment.

AuthorsOpenReview ↗

Reviewer Response Summary (2/2)

Reviewer xn28

  • Computational cost and scalability: Clarified quadratic LENS scaling, host-RAM safeguards, and offline preprocessing reuse; on 12 biological datasets, mean/median total preprocessing takes 72.95/28.96 s, with SCOPE compression averaging 0.72 s, reported separately from downstream TFM timings (Table B1).
  • Comparison with tuned GOTabPFN: Using the same TabPFN-2.5 backbone and matched CV on eight datasets, default SCOPE achieves 96.99% versus 95.90% ROC-AUC with 7.97× lower total runtime including tuning, supporting competitive tuning-free performance without statistically established predictive superiority (Tables B2–B3).
  • Matched-budget fairness: Added an eight-dataset comparison using all numeric features and identical fold-specific output dimensions; SCOPE retains numerical gains in accuracy, macro-F1, and ROC-AUC (+0.677 percentage points), controlling for feature caps and representation size (Tables B4–B5).
  • Timing and scaling: Clarified that main TFM speedups exclude ordering/compression for all compressors, separately reported complete LENS costs, and explained O(nm2)O(nm^2) distance-construction time and O(m2)O(m^2) host memory, CPU-RAM safeguards, and demonstrated applicability up to 67,420 features.
  • Unequal baseline limits: Explained the GOTabPFN-based safeguards and added a full-feature, matched-output-dimension comparison on eight datasets, where SCOPE achieves 95.77% versus 95.09% mean ROC-AUC; ordering-runtime budgets are not matched.
  • LENS-stage evidence: Expanded the comparison to 12 biological datasets; Full LENS achieves the highest mean ROC-AUC (93.892%) and best rank (1.667), while acknowledging its modest gain over MedoidNN and higher one-time ordering cost (Table B6).
  • Role of SCOPE after ordering: Clarified that LENS organizes features without reducing dimensionality, whereas SCOPE automatically determines the output budget, segments the ordered features, and constructs the compressed representation.
  • Simpler compression alternatives: Added controlled comparisons on 16 datasets; SCOPE achieves 94.252% mean ROC-AUC, versus 93.832% for equal-segment local PCA and 94.185% for mean pooling, with numerical gains that are not significant after Holm correction (Table B7).

Reviewer WzPg

  • Parameter rationale and sensitivity: Clarified fixed search-budget choices and added 12 configurations across 12 datasets (3,600 CV evaluations); 74.2% of dataset-alternative comparisons differ by at most 0.5 ROC-AUC points, output dimensions remain unchanged, and no alternative significantly improves performance after Holm correction (Tables C1–C3).
  • Statistical support: Added paired Wilcoxon tests with Holm correction, Friedman/Nemenyi analyses, and confidence intervals across 30 datasets; SCOPE achieves the best average rank (2.20) and Holm-significant ROC-AUC gains over all ten compression baselines, including +1.30 points over PCA and +1.48 over NSC-pSP (Tables C4–C8).
  • Writing and readability: Streamlined the Introduction, removed redundancy, redrew Figures 1 and 2(b), and improved spacing and visual emphasis for key equations and theoretical results.
  • Reference formatting: Standardized the bibliography and consistently included URLs across entries.

End note

  • All revised or added text, figure captions, and table captions are highlighted in red in the updated manuscript. The rebuttal provides approximate line numbers and exact table and figure references.
  • In keeping with our commitment to transparency, reproducibility, and open-source research, we are sharing most rebuttal experiment code and supporting materials, including .ipynb notebooks with displayed results for verification, after removing identifying information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.