- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses an important problem: inference-time context poisoning in tabular foundation models. The authors propose a training-free approach that partitions the context into multiple shards and aggregates their predictions through plurality voting. Because inserting, deleting, or relabeling a single row can affect at most one shard, the voting margin provides a deterministic guarantee against a bounded number of malicious modifications. The paper establishes theoretical guarantees and presents extensive empirical evaluations supporting the approach.
Critical strengths
- Training-free certification. While partitioning and voting are established techniques, notably in Deep Partition Aggregation (DPA), this paper adapts them from training-time poisoning defense to inference-time context protection. This eliminates the need for retraining and enables certification of frozen foundation models.
Critical weaknesses
-
Modest improvement over existing methods. The improvement over certified -NN is relatively small. In the held-out comparisons (Table 2), the proposed approach achieves median gains of only 0.67, 1.00, and 0.94 percentage points in certified accuracy at radii 1, 2, and 4, respectively. The advantage largely disappears when arbitrary feature edits are allowed without trusted row identifiers. Given the additional computational cost, the practical significance of these improvements is unclear.
-
Trade-off between robustness and accuracy. The experiments reveal a substantial trade-off: increasing the number of shards allows larger certification radii but reduces the context available to each shard, potentially degrading predictive accuracy. For TabPFN v2, using 256 shards leads to a 6.9-percentage-point reduction in accuracy (Table 1). The trade-off can be more pronounced for LLMs that genuinely depend on their demonstrations. For Qwen3.5-2B with arbitrary class labels, certified accuracy at radius 8 falls to 62%, compared with 91.3% undefended accuracy (§9). These findings raise questions about the usefulness of the approach for tasks requiring substantial contextual information.
-
Limited robustness against larger poisoning budgets. The certified radius is relatively small compared with typical context sizes. For example, the median certificate at 16 shards covers only seven malicious row modifications, a small fraction of datasets containing thousands of examples. While this is consistent with the paper's stated threat model, it limits protection against attackers who can control a larger portion of the context. Increasing the radius further comes at the expense of accuracy and computational cost.
-
Significant inference overhead. Evaluating the foundation model independently on multiple shards substantially increases inference cost. For example, at 64 shards, TabPFN v2 incurs approximately 17× the undefended wall-clock latency on the benchmark datasets (§5). Although the authors report memory savings and potential latency benefits for larger contexts, the overhead remains significant in latency-sensitive deployments. Combined with the modest improvement over certified -NN, this raises concerns about practical cost-effectiveness.
-
Limited theoretical novelty. The main robustness guarantee largely reuses the partition-and-vote certification argument from Deep Partition Aggregation (DPA), adapted from training-time poisoning to inference-time contexts. The primary novelty lies in applying this established mechanism to frozen in-context predictors and empirically evaluating its effectiveness, rather than developing a fundamentally new certification method.
-
Presentation. The paper is difficult to follow due to highly compressed prose, dense numerical reporting, and frequent transitions between theoretical claims, experimental findings, and qualifications. A clearer presentation of the central idea and a more structured discussion of the accuracy–robustness–compute trade-off would substantially improve readability.
Questions for authors
1. Does the modest improvement over certified -NN justify the additional inference cost?
The proposed approach outperforms certified -NN by approximately one percentage point in certified accuracy at small robustness radii, while requiring multiple foundation-model inference passes. Moreover, the advantage largely disappears under arbitrary feature edits without trusted identifiers.
Could the authors provide a more systematic comparison of the accuracy–robustness–latency trade-off against certified -NN, particularly under matched computational budgets? In what deployment scenarios would the additional inference cost be justified by the observed gains? It would also be helpful to identify the conditions under which partitioned in-context prediction provides a meaningful advantage over simpler and cheaper alternatives.
2. How well does the approach scale to larger poisoning budgets?
The reported guarantees cover a relatively small fraction of the context, and increasing the number of shards introduces accuracy and computational costs. How does the method scale when an attacker can modify a more substantial fraction of the context, such as 1% or 5% of the support examples? Is there a practical regime in which the method can provide meaningful protection against larger poisoning budgets without excessive loss of predictive accuracy or inference efficiency?
4 comments
Response to Reviewer V52g [Part 2/2]
Q2 / W3. Budgets of 1% or 5% of the table
Setup:
- The budget is ρ = round(f·N), on all 20 tables, under the held-out protocol.
- We extended the partition to m = 1,024 and certified k-NN to k = 4,001.
- We added a context-free reference: the validation majority class, which ignores the context and so is certified at every budget. A certificate is informative only where it beats this reference.
| budget (% of table) | 0.25% | 0.5% | 1% | 2% | 5% |
|---|---|---|---|---|---|
| median radius (rows) | 14 | 28 | 54 | 109 | 273 |
| partition: median certified acc. (tables above context-free) | 86.8 (15) | 79.7 (14) | 73.2 (10) | 65.4 (6) | 0.0 (0) |
| certified k-NN | 81.7 (15) | 78.2 (13) | 74.8 (13) | 68.4 (10) | 61.0 (3) |
| context-free | 62.5 | 62.5 | 62.5 | 62.5 | 62.5 |
- Up to about 0.5%, both certificates are far above the context-free reference and level with each other.
- At 1%, both are still informative, and k-NN with very large k leads by a median 1.5 points.
- At 5%, no method we evaluated beats the context-free predictor in the median.
Why: new Corollary 2 (§3). Under the law's assumption, a fraction f is certifiable for a query only if the learner is right on it from at most 1/(2f) rows: 50 rows at 1%, 10 at 5%. This is an inference-time form of the Lethal Dose Conjecture.
- 5% is out of reach for any partition certificate.
- Sample-efficient in-context learners are what make the reachable range work.
- In absolute rows the radius still grows with the table: at 40,000 rows the partition leads on adult and shuttle at radii 4 and 16 (Appendix L).
- The certificate targets a bounded number of malicious contributors. The abstract and §10 now say so.
W5. Theoretical novelty
The voting bound is DPA's, and we claim no new one (§2, "From training sets to inference-time contexts"). Our contribution is what a frozen model with a per-request context requires:
- Soundness conditions, which fail in practice. TabPFN v2 changes 0.1–0.2% of shard votes when the rows are permuted.
- Explicit pricing of feature edits (Corollary 1). Our attacks show that halving the radius is necessary.
- A predictive theory: Prop. 3, new Corollary 2 and new Algorithm 1. DPA cannot have one, because its models must first be trained.
Under the same certificate, no classical learner beats certified k-NN, while four in-context learners fixed in advance do (Table 3).
W6. Presentation
We revised the whole paper:
- a shorter abstract;
- one paragraph on the relation to DPA;
- Table 3, which gathers the robustness checks;
- an algorithm box and a deployment-guide table (Appendix P);
- one topic per appendix section.
Dear Reviewer V52g,
Thank you again for your detailed review and deployment questions. We wanted to check whether our response addressed your concerns. In brief:
- Cost versus gain (Q1, W1, W4). Capping the shard count at 16 keeps the lead at radii 1–2, significant on all five splits: +0.56 and +0.93 points for the pooled in-context arm, and +0.83 and +0.22 for Mitra v2 alone at 1.4× its undefended latency (Table 10, Figure 4). The partition also keeps a mean 3.7 points more clean accuracy than certified -NN. We now state plainly when certified -NN is the better choice (Appendix P).
- Larger budgets (Q2, W3). We tested budgets of 0.25–5% of the table (Table 19). Up to 0.5%, both certificates are well above a context-free predictor and level with each other. At 1%, certified -NN leads by 1.5 points. At 5%, no certificate we evaluated beats the context-free predictor in the median. The new Corollary 2 explains why this limit is structural, and §10 states it.
- Accuracy trade-off (W2). Algorithm 1 now chooses the shard count from a clean-accuracy floor, and with more context the demonstration-dependent LLM case recovers (82.0% certified at radius 8, against 62.0% before).
- Novelty and presentation (W5, W6). Clarified in §2; the paper was revised throughout for readability.
We would be glad to answer any remaining question. If our response resolves your concerns, we would appreciate your considering an update to your assessment.
Thank you for the additional experiments and clarifications. I appreciate the authors' efforts to address my concerns. However, three questions remain:
- Clean accuracy vs. robustness. While higher clean accuracy improves overall utility, I am not convinced that it directly strengthens the robustness claim, particularly when an attacker can probe multiple inputs and selectively target vulnerable queries.
- Cost-effectiveness. The claimed matched-compute comparison still shows substantially higher inference latency than certified k-NN, while the certified-accuracy improvements remain modest.
- Certification soundness. Given the reported sensitivity of TabPFN v2 to row permutations, how does the implementation ensure that modifying one row cannot affect predictions from otherwise unchanged shards?
Response to Reviewer V52g [Part 1/2]
Thank you for the review and for the deployment questions. We ran new experiments for each, including one that sets a clear limit on the method (Q2). Changes in the revised PDF are in blue.
Q1 / W1 / W4. Is a one-point gain worth the cost? Matched compute? When does it pay?
(1) The certified margin understates the benefit. A certificate is paid for on every query, attacked or not.
(2) Matched compute. Capping the shard count keeps the lead. Margin over certified k-NN, with (tables won) and [significant splits of 5]:
(3) The overhead is per call, not per FLOP. Each shard attends over only N/m rows; what grows is each call's fixed cost. At 20,000 rows the effect reverses:
Appendix G plots accuracy, robustness and latency together.
(4) When it pays (deployment guide, Appendix P):
W2. The robustness–accuracy trade-off, especially for LLMs that depend on demonstrations