acceptodds
Withdrawn from ICLR 2027

HarmCap: State-Aware Consequence Control for Retrieval-Conditioned Tabular Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-conditioned tabular models use neighboring examples or in-context demonstrations to adapt predictions at inference time, but a strong retrieval mechanism does not by itself control how much one realized update may deteriorate an individual prediction. We introduce HarmCap, a post-retrieval controller that treats the retrieval-conditioned output as a proposal and selects the largest query-specific step along its realized logit-space direction whose worst label-wise cross-entropy increase stays within a prescribed budget. The admissible step depends on the full predictive state, making scalar confidence insufficient for exact multiclass control. Across public tabular benchmarks and multiple retrieval-conditioned interfaces, HarmCap satisfies the prescribed pointwise contract while retaining more of the proposed update than a state-agnostic exact-safe envelope. Protocol-aligned TabR experiments further recover the predictive behavior reported in the original work before applying HarmCap to matched same-checkpoint retrieval updates. The experiments also show that admissibility and predictive utility are distinct objectives, supporting consequence control as a separate inference-time component of retrieval-conditioned prediction.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular Learning, Retrieval-Conditioned Prediction, Post-Retrieval Consequence Control, In-Context Learning, State-Aware Control, Safe Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_recrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies post‑retrieval consequence control for retrieval‑conditioned tabular classification models. It targets the failure case where noisy retrieved context can cause severe per‑query prediction degradation. Extensive evaluations are conducted over TALENT, OpenML‑CC18, TabZilla and TabICL/TabR. The paper also builds several internal comparator baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) for controlled ablation analysis.

Critical strengths
  1. The paper demonstrates a finding: scalar update magnitude or only top‑class confidence is insufficient for exact per‑sample worst‑case risk control; full predictive distribution over all classes must be used.
  2. The work cleanly decouples retrieval proposal generation and consequence‑control logic. HarmCap is an inference‑time post‑processing wrapper that requires no retraining of tabular foundation models and no modification to the retriever itself.
  3. The authors design multiple internal baselines to isolate different factors: state‑agnostic bound, moment‑based upper bound, and alternative probability‑space update geometry. Evaluations cover multiple tabular benchmarks.
Critical weaknesses
  1. Most comparison baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) are self-designed variants rather than previously‑published SOTA methods. Empirical improvements are shown against these self‑designed comparators.
  2. HarmCap preserves the original logit update direction produced by retrieval, and only scales its magnitude. It can only limit the damage of flawed retrieval proposals but cannot correct fundamentally bad update directions.
  3. Harm budget h is a manually‑tuned hyper‑parameter. There is no principled way to choose h for real‑world applications.
  4. Even though HarmCap greatly improves update retention under safety constraints, the improvements on standard metrics (NLL, Brier, balanced accuracy) are modest, and confidence intervals often cross zero across many datasets.
  5. The paper lacks validation on real‑world high‑risk scenarios where this worst‑case guarantee is practically demanded.
Questions for authors
  1. Most comparison baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) are self-designed variants rather than previously‑published SOTA methods. Empirical improvements are shown against these self‑designed comparators.
  2. Even though HarmCap greatly improves update retention under safety constraints, the improvements on standard metrics (NLL, Brier, balanced accuracy) are modest, and confidence intervals often cross zero across many datasets.
  3. The paper lacks validation on real‑world high‑risk scenarios where this worst‑case guarantee is practically demanded.
Reviewer_1naMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

HarmCap offers a post-hoc inference-time harness that enables practitioners to control how much retrieved data points can affect the predictions of a given base tabular model. HarmCap offers a mathematical guarantee that the worst-case cross-entropy increase will remain within a certain budget set forth by the practitioner. The paper mathematically derives this constraint and empirically demonstrates that HarmCap satisfies it while preserving more of the retrieval-induced update than conservative baselines, although this does not consistently translate into better predictive performance.

Critical strengths

The paper presents a concrete, mathematically well-founded method for post-hoc prediction control and demonstrates through extensive empirical evaluations that HarmCap successfully maximizes the retained retrieval-induced update along its original logit-space direction while satisfying a constrained bound on the worst-case cross-entropy increase relative to the base prediction.

The paper also provides clear mathematical explanations, connecting its main formulations to established concepts and deriving the proposed method step by step from these foundations.

Critical weaknesses

The following points weigh most heavily in my overall assessment of the paper:

  1. Biased headline metric: The paper shows successfully that HarmCap performs well on retention and uses that as its headline metric, which is what HarmCap was designed to optimize and is analytically guaranteed to do so. However, what matters in practice is whether the controlled predictions are better, and the results suggest they are not reliably so. With native TabR, HarmCap reaches predictive performance that is much closer to the base model than to unconstrained retrieval, so most of the retrieval benefit is discarded. Higher retention also does not track utility: on CC18 and TabZilla, Moment Envelope retains less of the update than HarmCap but achieves a better predictive performance. I believe the paper needs to change this framing or argue for the usefulness of retention more solidly.

  2. Heavy reliance on a trustworthy base model: HarmCap is motivated by having a trustworthy model being augmented with potentially unreliable external data. Basically, it can be understood as a regularization technique that limits deviations from the original predictions of the trustworthy models. However, the paper doesn't consider the case where the base model is trustworthy but incomplete, with retrieval potentially completing it. The constraint of HarmCap keeps every class above a fixed fraction of its base probability, so it specifically restricts reducing the classes the base model is confident about. The paper acknowledges that a wrong base prediction would remain wrong, but it doesn't explain why this is acceptable while full retrieval would help such a model in many cases. In most real-world cases, a strong model would benefit from extra data that demonstrates scenarios it didn't see during training.

  3. Artificial experimental setups: The paper motivates HarmCap with noisy, irrelevant, or outdated retrieved evidence (Figure 1), but no experiment actually tests such a setting by corrupting the retrieved data and testing HarmCap against other baselines in such settings. Instead, the base/retrieval pairs are mostly artificial. The three largest suites use a logistic-regression base blended with kNN predictions at a fixed coefficient, and these boards produce the large retention gains. The TabICL retrieval interface is also author-built. In the native TabR experiments, the base model is obtained by zeroing the retrieval contribution inside a checkpoint that was never trained to work without retrieval. The resulting base is a degraded configuration rather than a trustworthy model, and in this most realistic setup the retention gain over Range Clip drops to 5.8 pp. This contradicts the premise under which HarmCap would be useful. An evaluation with a well-performing base model and realistically corrupted retrieval data would be needed to support the paper's motivation.

There are also a few presentation weaknesses, but I consider these less critical:

a. Figure 1 is uninformative and poorly designed for a scientific paper, and it also contains typos such as "reterived" and "dicarded",

b. Algorithm 1 writes the step as a instead of α\alpha.

c. Table 1 is hard to follow since the rows for each benchmark are not strongly separated.

d. Section 3 is hard to follow and would benefit a lot from clearer definitions of working terms such as “LEVEL-B”, “LEVEL-P2”, “closure”, “fair board”, and “artifact freeze”.

Questions for authors
  1. In what realistic tabular prediction application is it desirable to guarantee that no class probability decreases by more than a fixed multiplicative factor relative to the base model, particularly when the base predictions may be unreliable in the first place? If the authors can explain such scenarios and show that HarmCap is useful in such setups, the argument for HarmCap becomes clearer.
  2. Have you tried setups where the retrieval set consists of, for example, label noise or irrelevant/out-of-distribution rows? If HarmCap can outperform unconstrained retrieval under corruption and still lose only a little when the retrieval data is clean, that would make a clearer case for HarmCap and address some of the concerns about its practical utility.
Reviewer_3AriOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the problem of maximizing retrieval-induced prediction updated in tabular models subject to NLL constraints relative to the non-retrieval based "base" predictor.

It proposes a method to achieve better tradeoffs for this, and tests it with logistic regression + kNN, TabR and TabICL setups

Critical strengths

None

Critical weaknesses

Core weakness: writing is hard to comprehend

The writing in the paper is very hard to understand. It has many lightly/un-edited LLM output feel

e.g. one of the core sentences in the conclusion: "This perspective explains why state-agnostic envelopes can be unnecessarily conservative and why compressed summaries such as scalar confidence are insufficient for exact multiclass contro" - what is a state agnostic envelope?

"LEVEL-B same-checkpoint ablation", "LEVEL-P2 published alignment plus same-checkpoint", "exact-H contract" -- these are just a surface of unexplained newly introduced terms that are all over the text

Motivation & technical issues

Abstracting away from writing and trying to make sense of the contribution leads me to the second weakness.

The papers' core motivation if I understand correctly is that retrieval context could be harmfull in some way (noisy, missleading). I could see this be interesting potentially in some procedures like context filtering, helping in deciding which set is more useful for prediction.

But the paper sole objective is maximizing retrieval prediction update, subject to minimizing the cross entropy diference to the "base" predictor without retrieval and the paper never establishes any usefullness of the procedure (practically results on all benchmarks considered decrease accuracy substantially). It is unclear thus why the method is useful at all.

Furthermore, the paper has no consistent technical experimental setup. Each section has many arbitrary ad-hock decisions in the experimental section and lacks consistency: TabR experiment dataset selection, seemingly motivated only by reproducing the priror work results. Bespoke TabICL pipeline, Logistic regression + kNN on TALENT.

Questions for authors
  • What is the core practical motivation for the proposed method and what is the most concise summary of the problem it is trying to solve?
  • Which results are the empirical highlight?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.