- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript studies post‑retrieval consequence control for retrieval‑conditioned tabular classification models. It targets the failure case where noisy retrieved context can cause severe per‑query prediction degradation. Extensive evaluations are conducted over TALENT, OpenML‑CC18, TabZilla and TabICL/TabR. The paper also builds several internal comparator baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) for controlled ablation analysis.
Critical strengths
- The paper demonstrates a finding: scalar update magnitude or only top‑class confidence is insufficient for exact per‑sample worst‑case risk control; full predictive distribution over all classes must be used.
- The work cleanly decouples retrieval proposal generation and consequence‑control logic. HarmCap is an inference‑time post‑processing wrapper that requires no retraining of tabular foundation models and no modification to the retriever itself.
- The authors design multiple internal baselines to isolate different factors: state‑agnostic bound, moment‑based upper bound, and alternative probability‑space update geometry. Evaluations cover multiple tabular benchmarks.
Critical weaknesses
- Most comparison baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) are self-designed variants rather than previously‑published SOTA methods. Empirical improvements are shown against these self‑designed comparators.
- HarmCap preserves the original logit update direction produced by retrieval, and only scales its magnitude. It can only limit the damage of flawed retrieval proposals but cannot correct fundamentally bad update directions.
- Harm budget h is a manually‑tuned hyper‑parameter. There is no principled way to choose h for real‑world applications.
- Even though HarmCap greatly improves update retention under safety constraints, the improvements on standard metrics (NLL, Brier, balanced accuracy) are modest, and confidence intervals often cross zero across many datasets.
- The paper lacks validation on real‑world high‑risk scenarios where this worst‑case guarantee is practically demanded.
Questions for authors
- Most comparison baselines (Range Clip, Moment Envelope, Probability‑Mix‑Max) are self-designed variants rather than previously‑published SOTA methods. Empirical improvements are shown against these self‑designed comparators.
- Even though HarmCap greatly improves update retention under safety constraints, the improvements on standard metrics (NLL, Brier, balanced accuracy) are modest, and confidence intervals often cross zero across many datasets.
- The paper lacks validation on real‑world high‑risk scenarios where this worst‑case guarantee is practically demanded.