- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies depth-adaptive early exiting for pretrained tabular outlier detection (OD) foundation models. The authors first observe that intermediate-layer representations, when fed into the frozen final prediction head, can outperform full-depth predictions, and that this "oracle early-exit" gain grows substantially when the in-context set is polluted by outliers. They attribute this to context pollution: deeper layers aggregate more heavily from nearby context samples, so outlier neighbors progressively mask query outliers in the representation. Building on this, they propose ROUT, a lightweight post-hoc router (≈4.6M params) that reads frozen backbone representations, layer-wise outlier scores, raw features, and 29 geometric features, and selects a dataset-specific exit layer. ROUT is pretrained on ~211K synthetic datasets, uses query outlier labels as privileged information (LUPI) only during training, and combines a sequential stopping loss with a layer-wise selection loss and a depth penalty. Backbones and prediction heads remain entirely frozen. Experiments span three OD-TFM backbones (OutFormer, TACTIC, ICLAD), three real-world benchmarks (OddBench 690, OvRBench 755, ADBench 47) and a synthetic benchmark (SynBench 800), under clean, held-out pollution, and near-duplicate pollution settings. ROUT reportedly matches or improves full-depth AUROC in 11 of 12 settings while computing 15–63% fewer layers, closing up to 45% of the oracle gap and achieving up to ~1.8× speedup.
Critical strengths
The paper is well organized: motivating observation → mechanistic explanation → method → extensive evaluation.
(1) A genuinely counter-intuitive and well-supported empirical finding.** The observation that early exit can improve outlier-detection accuracy, not just save compute, and that this effect strengthens with context pollution, is the paper's most consequential contribution. It is supported across three architecturally and training-regime-distinct backbones, four benchmarks, and both injected-pollution modes, with consistent directional results (Table 1, Table 4, Figures 9–13, 18). This is the kind of finding that changes how practitioners think about depth in in-context OD models, and it is not an artifact of a single model or dataset.
Critical weaknesses
(1) The method sidesteps rather than solves the train–test distribution shift that defines the problem.** The paper's own framing is that real-world OD datasets exhibit hidden pollution, yet ROUT is never trained on any real data or any human-annotated outlier labels. The authors' justification is that "inference-time datasets lack ground truth labels," but this conflates test-time label availability with training-time data realism: one could label outliers on a held-out pool of real datasets for router training, or at minimum perform a leave-one-benchmark-out study to quantify prior mismatch. As it stands, the training-data ablation (Table 14) shows 2.3–4.4% AUROC swings from removing a single prior source, which is precisely the symptom of an unaddressed shift. The paper acknowledges prior mismatch (" 0.76") but offers no remedy, no diagnostic, and no characterization of when the router should be distrusted. This is consequential because the entire value proposition rests on synthetic priors being adequate proxies for real pollution.
(2) The central causal claim about context pollution on real data is under-evidenced, and the mechanism lacks formal grounding.** The pollution mechanism is convincingly demonstrated only where the authors inject the pollution. On real benchmarks, the existence and magnitude of hidden pollution is inferred from one t-SNE visualization (Figure 2) plus layer-wise AUROC curves — there is no quantitative estimate of pollution rate, no ablation that removes suspected polluted real context rows and measures the effect on oracle exit depth, and no test of whether the real-world oracle gain survives when pollution is controlled for. Since the real-world gain is the headline result (7.3% / 7.0% / 4.7% for the three backbones), the causal story behind it should be nailed down more tightly than a correlation between sibling influence and AUROC loss on synthetic targeted outliers. Relatedly, the paper never formalizes why optimal exit depth should decrease monotonically with pollution; the explanation remains a retrieval/attention analogy plus empirical correlations.
Questions for authors
-
Train–test shift. Why not train the router on a held-out pool of real datasets with human- or rule-annotated outlier labels (or at least a small real labeled set), instead of relying exclusively on synthetic priors? Could you report a leave-one-benchmark-out experiment (train without ADBench, test on ADBench, etc.) to quantify how much ROUT's exit selection degrades under prior mismatch? What is the failure mode when a test dataset's pollution pattern is outside the synthetic prior support — does ROUT degrade to full depth, or does it confidently pick a wrong early layer?
-
Direct outlier identification. ROUT outputs an exit layer, not an outlier score. The pollution experiments use author-injected outliers, and Figure 2's hidden-pollution evidence is qualitative. Can you provide a quantitative estimate of the hidden pollution rate on real benchmarks (e.g., by identifying context rows that a strong supervised detector would flag), and show that removing those rows measurably changes the oracle exit layer? Without this, the claim that real-world early-exit gains are driven by hidden pollution remains correlational.
-
Baseline fairness. Only LA-Entropy is compared, and it is a weak baseline on this task (below Half-depth for two of three backbones). Why were CALM, Router-Tuning, FlexiDepth, or a simple entropy-based exit with the original head not included? Since LA-Entropy retrains per-layer heads while ROUT does not, is the comparison measuring routing quality or head quality? Also, would the authors consider moving Oracle / Best-fixed-layer out of Table 2 into a clearly separated reference table?
5 comments
W2 / Q2. Evidence for hidden pollution on real data, and why the optimal exit moves earlier under pollution.
We thank the reviewer for this constructive suggestion on strengthening the real-data evidence.
Quantifying and removing hidden pollution. Hidden pollution is present in most real-world datasets. Removing it narrows the oracle gain and shifts the oracle layer toward full depth, which reverses the effect of added pollution in Section 3. As the reviewer suggested, for each dataset we train a supervised detector (a random forest) on its labeled query rows and flag the context rows it confidently predicts as outliers: these appear in 66% of the datasets and make up 1.2% of the context on average. We then remove them, compared against removing the same number of random rows:
Table R5: Removing flagged hidden pollution raises full-depth AUROC, shrinks the oracle gain and moves the oracle layer deeper; removing random rows has no effect. Added as Table R5 in Section 3 (873 datasets, OutFormer; first row: values on the original context; other rows: change from it; Wilcoxon signed-rank test, )
| Context | Full-depth AUROC | Oracle − Full depth | Mean oracle layer |
|---|---|---|---|
| Original | .675 | .059 | 6.13 |
| Flagged rows removed | +.019$^{\ast\ast\ast}$ | −.008$^{\ast\ast\ast}$ | +0.20$^{\ast\ast\ast}$ |
| Same number of random rows removed (Control) | −.000 | +.000 | −0.00 |
This simple detector flags only a small share of the context, yet removing these rows significantly narrows the oracle gain and shifts the oracle layer toward full depth. Removing the same number of random rows leaves both unchanged. This intervention on real contexts thus confirms that hidden pollution drives early-exit gains and agrees with the controlled pollution experiments in Section 3.
Controlled intervention on real data. We would like to clarify that the planted-sibling analysis is a controlled intervention on real datasets: for each targeted query outlier we turn one context row into its sibling and keep everything else fixed. Turning the same row into a copy of a random inlier leaves the AUROC of the targeted outliers unchanged (Appendix B.1). The masking is therefore specific to the sibling. Together with the removal experiment above, this establishes context pollution as a major mechanism behind the real-world gains.
Why the optimal exit moves earlier. Here we provide an intuitive decomposition of why more pollution favors an earlier exit, writing the AUROC of layer at pollution ratio as . The clean-context AUROC grows with depth, and so does the sensitivity to pollution (Figures 5, 14 and 15). Because increases with , more pollution makes the cost of depth outweigh its benefit at an earlier layer. Indeed, from clean context to 1:1 the mean oracle layer moves earlier for every backbone and real-world benchmark.
Revision. We added the hidden-pollution experiment to Section 3, with details in Appendix C.2.
Q3. Choice of baselines, routing quality versus head quality, and the reference rows of Table 2.
We thank the reviewer for these helpful suggestions on the baselines.
Why not CALM, Router-Tuning or FlexiDepth? Our focus on early exit follows directly from the mechanism we identify. Methods that skip layers (Router-Tuning and FlexiDepth) or also repeat them (Dr.LLM by Heakl et al. (2026) and PoLaR by Li et al. (2026)) decide which layers to compute. Under context pollution the harm is not tied to a particular layer. Instead, it mainly accumulates with depth because every additional layer aggregates the polluted context once more. Stopping early therefore becomes the most natural remedy. In addition, the three methods the reviewer names make per-token decisions in LLMs and aim to match the full model at lower cost. An OD-TFM instead needs one exit layer per dataset, and its intermediate layers can outperform full depth. These methods also retrain the backbone (CALM) or train with a language-modeling loss (Router-Tuning and FlexiDepth), which a frozen OD-TFM does not allow.
Confidence-based exits with the frozen final-layer head. Following the reviewer's suggestion, we added a CALM-style exit and LA-Entropy with the frozen final-layer head, and rOUT outperforms both for every backbone on every real-world benchmark. The CALM-style exit uses the hidden-state saturation of CALM (Schuster et al., 2022), the cosine similarity between query representations of consecutive layers. We do not use its softmax response because with a two-class head it depends only on the predicted outlier probability, like entropy. LA-Entropy with the frozen final-layer head is the simple entropy-based exit the reviewer suggests. Each exit averages its confidence over the query rows and stops at the first layer where it passes a threshold selected on synthetic validation data.
Table R6: rOUT is more accurate than every confidence-based exit on every backbone and real-world benchmark. Excerpt of the revised Table 2 (AUROC, mean computed layers in parentheses)
| Backbone | Method | OddBench | OvRBench | ADBench |
|---|---|---|---|---|
| OutFormer | Full depth | .740 (10) | .709 (10) | .853 (10) |
| LA-Entropy (retrained heads) | .709 (2.5) | .680 (2.9) | .770 (2.5) | |
| LA-Entropy (frozen head) | .701 (7.1) | .689 (7.6) | .794 (7.4) | |
| CALM-style | .732 (6.9) | .693 (6.4) | .839 (5.7) | |
| rOUT | .761 (5.5) | .730 (5.4) | .860 (5.5) | |
| ICLAD | Full depth | .769 (12) | .745 (12) | .891 (12) |
| LA-Entropy (retrained heads) | .704 (1.6) | .696 (2.0) | .786 (2.1) | |
| LA-Entropy (frozen head) | .744 (8.8) | .735 (10.0) | .883 (10.1) | |
| CALM-style | .761 (7.3) | .728 (6.6) | .843 (5.5) | |
| rOUT | .776 (6.8) | .755 (6.0) | .891 (6.0) | |
| TACTIC | Full depth | .736 (12) | .715 (12) | .832 (12) |
| LA-Entropy (retrained heads) | .693 (2.1) | .682 (2.4) | .780 (2.2) | |
| LA-Entropy (frozen head) | .735 (12.0) | .712 (11.9) | .832 (12.0) | |
| CALM-style | .736 (12.0) | .715 (12.0) | .832 (12.0) | |
| rOUT | .747 (10.2) | .729 (10.0) | .849 (10.1) |
Routing or head quality? To stay aligned with the original method (Küken et al., 2025), the LA-Entropy baseline in Table 2 retrains a head for each layer, and the revised Table 2 also reports it with the frozen final-layer head. The two frozen-head exits instead share the frozen backbone and head with rOUT and differ only in how the exit layer is chosen, so this comparison directly measures routing quality. With the frozen head, confidence proves to be a weak stopping signal: the two exits stop later than rOUT in all but one setting yet are less accurate in every setting. We attribute the advantage of rOUT largely to two core designs that confidence-based exits lack. Sample Attention reads the representations of individual context and query rows and so captures information that a single average confidence cannot. Layer Attention compares all computed layers and lets rOUT look back to return an earlier layer, and removing it significantly lowers AUROC (Table 5).
Reference rows in Table 2. We thank the reviewer for this suggestion. In the revision, the caption of Table 2 states explicitly that Oracle and Best fixed layer use test labels and cannot be deployed.
Revision. We added LA-Entropy with the frozen final-layer head and the CALM-style exit to Table 2, explain the choice of baselines in Section 5, and revised the caption of Table 2.
Finally, we noticed that our paper was flagged for ethics review. Our study uses only publicly available benchmarks and synthetic data, and we are happy to clarify further if the reviewer has a specific concern in mind.
If the new real-data evidence and the added baselines address the reviewer's concerns, we hope the reviewer might consider revisiting the score. We also welcome any further questions during the remaining discussion period.
Response to Reviewer DADc
We sincerely thank the reviewer for the thoughtful and encouraging review. We are glad the reviewer found the paper well organized, and we truly appreciate the recognition of our finding that early exit can improve detection, not just save compute, as "the kind of finding that changes how practitioners think about depth in in-context OD models." The reviewer's questions helped us strengthen the real-data evidence, and we address each of them below. Please see the revisions in blue text in the revised copy of our submission, which is now uploaded on OpenReview.
W1 / Q1. Training on synthetic data only, leave-one-benchmark-out, and the failure mode under prior mismatch.
We thank the reviewer for this important question on the train–test shift.
Why not train on real datasets? Real-world benchmarks are too few to train the router from scratch. Note that one training example for rOUT is an entire dataset, and the three real-world benchmarks together offer only 1,492 such examples, compared to over 200,000 synthetic ones. rOUT therefore follows the standard practice of tabular foundation models such as TabPFN (Hollmann et al., 2023; 2025) and TabICL (Qu et al., 2025; 2026), which are pretrained purely on synthetic data and applied zero-shot to real datasets; the OD-TFM backbones that rOUT builds on are trained the same way. No real-world benchmark is used to train rOUT, and its stopping threshold is selected on synthetic validation data. This is in fact more challenging than the leave-one-benchmark-out setting, and rOUT still recovers up to 45% of the Oracle gain (Table 2).
Leave-one-benchmark-out. Since real datasets are too few to train from scratch, we instead fine-tuned the pretrained rOUT on real labels, in response to reviewer DADc’s request. We fine-tuned on two real-world benchmarks and tested on the third, with early stopping on held-out datasets of those two benchmarks and the stopping threshold selected on synthetic validation data, as in the paper. In our experiments, this yielded a slight gain over zero-shot rOUT, but not a significant one (the 95% CI of the difference includes zero on all three benchmarks):
Table R3: Fine-tuning on real labels does not significantly improve rOUT on the held-out benchmark. Leave-one-benchmark-out fine-tuning, added as Table R3 in Appendix C.3 (mean AUROC at matched depth; Difference = fine-tuned − zero-shot rOUT, with its 95% CI over datasets)
As a sanity check, the router does learn from real labels: on the datasets it is trained on, it reaches 69–88% of the Oracle gain, up from 22–38% before fine-tuning. This fit did not carry over much to the held-out benchmark, suggesting that synthetic pretraining already captures most of what transfers across datasets.
Prior mismatch and failure mode. Outside its training priors, the router becomes slightly more conservative but does not fall back to full depth. To test this, we evaluated the routers of Table 14, each of which has never seen the setting that its removed data source resembles:
Table R4: Outside its training data, a router falls behind rOUT and exits slightly deeper, yet still stops well before full depth. Each router of Table 14 lacks one data source and is evaluated in the setting that source resembles; added as Table R4 in Appendix C.3 (AUROC, mean computed layers in parentheses)
In all three cases, the mismatched router performs below rOUT and exits slightly deeper, while still stopping well before full depth. Prior mismatch thus shows up as weaker layer choices at a slightly deeper average exit, rather than a collapse to very shallow layers or to full depth. On real-world data, these choices still improve over Full depth. On SynBench Full depth is already close to the Oracle and leaves little for early exit to gain even for rOUT. Here the mismatched router falls further behind Full depth than rOUT does. Our remedy is broad coverage: as Table 14 shows, each data source matters most in the setting it resembles, which is why rOUT trains on all of them.
Revision. We added the fine-tuning and out-of-support results (Tables R3 and R4 above) to Appendix C.3, and stated the zero-shot protocol explicitly in Section 5.