acceptodds
Under review as a conference paper at ICLR 2027

ProbeID: Verifying MCP Tool Identity for LLM Agents under Registry Drift

15 views

Abstract

Agents select tools by copyable descriptions, which cannot establish which implementation runs. ProbeID compares canonicalized responses to schema-valid probes with a trusted admission reference and abstains on ambiguity, so a squatter must copy probe-observable behavior. With the gold present, independent probes and a fixed margin, standard concentration arguments give a shared probe budget logarithmic in the candidate count and bound the joint probability of a wrong decision. On cached MCP-IdentityDrift cases, strict choice reaches 0.956 hit@1 while description routers fall near chance, but simpler exact comparisons tie it, MMD better tolerates heavy value noise, and a 32B behavioral judge is similarly accurate at higher cost. Random probes from agent banks and a prospective held-out study of 38 tools test theory-set thresholds. On the agent banks, equality refuses every reachable replacement from 64 probes and the bounded distance from 512, and sharper post hoc gates halve both. Two tools whose drift no bank probe reaches always pass. In an MCP-Bench agent, verification cuts tainted tasks from 15 to 9 of 16 at unchanged task success. Over eleven weeks, 7 of 59 real tools changed responses despite version pins. Verification depends on reference integrity, probe coverage, and response noise.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, Model Context Protocol, tool identity, behavioral verification, registry drift, rug pull, tool squatting, agent security

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JsqmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies behavioral identity verification for MCP tools whose descriptions are cloned or whose behavior changes after integrationl. ProbeID records a tool’s behavior while it is trusted and uses probes to compare candidate tools against this reference at runtime. It accepts a match when the evidence is sufficient and abstains otherwise. Experiments show that ProbeID improves identification accuracy under tool description cloning and reduces agents’ reliance on outputs from tampered tools.

Critical strengths
  1. The paper addresses MCP tool impersonation and behavioral changes after integration, highlighting the limitations of selecting tools solely based on their descriptions. This is a relevant problem for improving the reliability of tool-using agents.

  2. The method combines behavioral comparison with abstention and analyzes the relationship between the probe budget and error probability under explicit sampling and behavioral separation assumptions. This provides a theoretical basis for selecting verification parameters.

Critical weaknesses

1.The additional benefit over existing general-purpose CLI agents remains unclear. Existing general-purpose CLI agents can generate test cases, make trial tool calls, and inspect the results. The paper lacks a direct comparison with such autonomous testing approaches, leaving it unclear how much improvement the proposed verification protocol provides.

2.The necessity and frequency of probe-based verification are insufficiently justified. The paper does not adequately explain when probe-based verification is needed or whether verification must be repeated when the same tool is used multiple times.

3.The paper lacks a systematic evaluation of the execution costs incurred by a large number of probe calls.

4.Legitimate data updates and routine output changes may also trigger rejection, but the paper provides insufficient discussion of the frequency of such false rejections, their impact on task execution, and the recovery mechanisms.

Questions for authors

See Weakness Above.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Jsqm

Thank you for the review. Admission checks cannot see a later behavior-only change, and version pins miss changes under unchanged definitions. Over eleven weeks, 7/59 pinned tools changed responses (Appendix O). In the agent study, 15/16 tasks receive tampered values ungated and 9/16 gated in each of three repeats, despite 1.000 task success (Table P.1). This shows incomplete runtime protection, without establishing how often production verification should run.

1. Comparison with autonomous CLI agents. That comparison is missing. The 7B and 32B behavioral LLM judges evaluate supplied responses rather than autonomously designing and executing tests (Appendix H). ProbeID supplies a trusted-reference protocol, abstention and explicit error-bound conditions, together with evaluation of the values actually delivered to an agent. We claim neither general test-generation capability nor superiority over autonomous coding agents. Such agents could complement the protocol by constructing valid state preconditions.

2. Verification frequency. An admission check cannot detect a later endpoint or backing-data change. Repeated runtime checks address replacement in the agent study, but do not show that every call needs a full-bank check. Reusing a verdict requires an explicit freshness window and state policy. We have not evaluated that cost-risk trade-off (Section 6).

3. Execution cost. Probe calls execute tools and must be counted. The 3.527 ms CPU decision time excludes them (Appendix H.1). Table P.2 gives the following single-run rug-pull configurations on the same 16 tasks.

ConfigurationRuntime probe callsTainted tasksRefused calls
Full-bank structural comparison, m=512126,4289/16102
Structural comparison, m=32 / m=1288,398 / 33,89113/16 / 13/1631 / 68
Retrospective exact equality, m=328,412, alongside 1,180 business calls8/1684

Recording and calibrating the banks adds 152,348 calls. Equality used a budget chosen after earlier outcomes and followed different trajectories, so 8/16 versus 9/16 does not establish superior protection. Full-bank checks took 0.66 s on local held-out tools (B=512) versus 4.2 min on a rate-limited weather server (B=96, Appendix R). The sequential gate's eight-probe average concerns full value drift in bank studies, not an end-to-end agent run. Its protection, refused calls and latency have not been evaluated jointly.

4. Legitimate changes and recovery. A refusal calls for revalidation and does not establish malice. In the held-out study, the original structural gate refuses all 16 tools whose answers change under reworded errors and added fields on 5/7 affected local tools (Table R.2). The finite-population gate refuses added fields on all 7 at the full bank (Appendix S). No refusal occurs across the benign agent arm's 269 structural verdicts or 278 equality verdicts, but static runs do not estimate false alarms under routine production updates. Finite-bank resampling likewise does not certify future stochastic responses.

As stated in the revised Ethics statement, safe recovery requires authorized inspection and a newly trusted reference for an accepted update. Automatically adopting a rejected response would defeat the clean-admission assumption. Recovery latency and its effect on downstream availability have not been evaluated.

5. Ethics. You flagged this submission for ethics review. Probing can expose sensitive responses or change state. The census used whitelisted read-only tools on an isolated host (Appendix O), while write and authentication experiments used controlled, reset sandboxes (Appendix M). Real-world response changes are not classified as malicious. The protocol does not authorize third-party production probing or bypassing access controls. Its risks include misplaced admission trust, service load, false refusals and missed drift. Supported protection concerns only behavioral changes reached by the probes.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Reviewer_vGrqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

When a tool is admitted, ProbeID stores a bank of schema-valid probes together with the tool's canonicalized responses. Later it samples mm probes from that bank, issues the same calls to each candidate, canonicalizes, and compares the mean distance to the reference. With several candidates it returns one only if the nearest is close enough in absolute terms and its margin over the runner-up is large enough; with a single candidate it uses the absolute threshold alone. Theorem 1 gives m>(2/γ2)log⁡(2(n+k)/δ)m > (2/\gamma^2)\log(2(n+k)/\delta) for controlling nearest-neighbour selection error under a positive separation margin γ\gamma and independent probing.

What is verified should be stated precisely: agreement between a candidate's probe-observable behavior and a stored behavioral reference. It is not cryptographic identity, not full security, and not retrieval from user intent. A tool that matches on every probe and misbehaves on other inputs, or through side effects that never reach the response, passes — which the paper states plainly.

The evidence spans cached benchmark cases, controlled noise, real MCP servers, independently reimplemented substitutes, state and auth probes, attested random-probe banks, a prospective held-out study on 38 read-only tools across eight servers, a 16-task agent integration, and an eleven-week longitudinal census of 59 published tools. Two operating points must not be merged: the headline 0.956 hit@1 is forced choice at decision coverage 1.000 with 0.044 wrong, whereas the per-case confidence-radius rule on the same cached cases decides only 0.153 of them (with 0.000 wrong), and those fixed probes meet no sampling condition. The agent study is the most striking result: 15 of 16 tasks receive tampered values ungated and 9 of 16 still do when gated, while ordinary task success is 1.000 in every arm.

Critical strengths

1. The negative results are reported where a reader will see them, and they are the most valuable part of the paper. A submission arguing for its own protocol that also shows two simpler comparisons matching its headline distance, and an entire failure class its guarantee excludes, is giving the community something to build on. Two tools sit at distance 0.000 because no bank probe reaches their drift, so they pass at any budget and account for all of the surviving agent corruption; a set-level MMD beats the proposed structural distance under heavier value noise; a 32B behavioral judge is similarly accurate; exact equality ties the bounded distance on cached cases and needs fewer probes on deterministic tools. The agent-side evaluation method is itself a contribution: replaying the agent's own calls against the trusted tool to check whether the agent received values it should not have, rather than trusting benchmark task success, is what exposes the gap between 1.000 task success and 9 of 16 tasks still tainted. Task completion does not mean the tool's return values were not quietly altered — that finding is worth the paper on its own. The eleven-week census is a second independent empirical contribution: version pins do not fix responses, and three tools changed after a reinstall because an SDK declares a validation-library range spanning two major versions.

2. Claims are consistently scoped to what they support, and the artifact lets a reader check them. The cached rows "meet no sampling condition"; the judge timing comparison "establish[es] neither end-to-end acceleration nor a reproducibility advantage"; among decided cases the error is bounded only by δ/P(decide)\delta/P(\text{decide}), "so an observed error rate among decisions is not a δ\delta-level certificate"; the lower bound is explicitly restricted to the dominant order; the total candidate call volume is written out as m(n+k)m(n+k) rather than being passed off as logarithmic cost. I verified the artifact rather than taking it on trust, and it is the strongest release of the five submissions I reviewed this cycle. verify_public_results.py recomputes 793 numerical checks against values hardcoded from the paper's tables and stops at the first mismatch; it passes, as does the unit suite (100 passed, 2 skipped) including the numerical checks of the probe-complexity and abstention bounds. The census reproduces item by item: census_rollup_t77d.json holds 59 tools over 16 servers with 18 drifted and 41 stable, and its 10 genuine-drift entries decompose into exactly the three weather tools, the one AWS documentation recommender and the three filesystem tools the text names, with the three harness artifacts set aside accounting for the difference from the reported 7. The agent study's three repeats have identical tainted-task sets, not merely counts (15, 15, 15 and 9, 9, 9, the surviving 9 a strict subset of the 15), and Appendix T reports that counting separability-excluded golds as misses moves hit@1 only from 0.956 to 0.953 — so sample filtering is not inflating the headline.

Critical weaknesses

1. For stochastic tools, "closer to the reference" is not a valid identity criterion, and the positive-margin assumption silently excludes the natural attack rather than merely the hard cases. Take a tool that returns 1 with probability 0.9 and 0 with probability 0.1 on a class of inputs, with the reference recorded from that same tool. For scalar outputs the distance reduces to inequality, so the expected gold-versus-reference distance is μgold=2(0.9)(0.1)=0.18\mu_{\text{gold}} = 2(0.9)(0.1) = 0.18, while a replacement that always returns 1 has μrepl=Pr⁡[ref=0]=0.1<0.18\mu_{\text{repl}} = \Pr[\text{ref} = 0] = 0.1 < 0.18. The replacement has changed the tool's output distribution, yet the nearest-distance rule prefers it, and prefers it more reliably as mm grows; it also clears an absolute threshold calibrated as τhi=μgold+ϵ\tau_{hi} = \mu_{\text{gold}} + \epsilon. This is not a counterexample to Theorem 1 — it violates γ>0\gamma > 0 — and that is precisely the problem: the assumption that makes the theorem work rules out distribution-contracting replacements, which is how one would actually attack a stochastic service. The paper supports stochastic tools and notes positive self-distance, but its noise experiments only add perturbation to otherwise-correct responses; there is no substitute that returns the modal answer, reduces output entropy or shifts the conditional distribution. Three things need separating and the paper merges them: a single response being close, the conditional output distribution agreeing, and the implementation or functional semantics agreeing.

The calibration transfer has a matching gap. The theorem needs the threshold related to the true μgold\mu_{\text{gold}}, while deployment uses s^\hat{s} from a few admission runs and sets τ=s^+ϵ\tau = \hat{s} + \epsilon. Three distinct uncertainties are in play — probe sampling, the tool's own response randomness, and estimation error in the reference self-distance — and the headline confidence radius addresses the first. Appendix S's finite-population correction makes this concrete: reading the entire bank drives the sampling radius to zero, which is sound for a fixed per-probe score table as Proposition 3 states, but for an online stochastic tool it only removes uncertainty about which inputs were drawn, not about what those inputs will return next time. If s^\hat{s} and a later d^\hat{d} are two identically distributed estimates from the same undrifted tool, then rejecting on d^>s^\hat d > \hat s with zero radius has rejection probability near 1/21/2, not 0.05. The authors concede that under a complete bank a future gold distance slightly above the admission estimate is rejected, and the held-out interval is short. For a paper whose contribution is a calibrated protocol, this is the core of the deployment story.

A third strand points the same way: the operative security property is bank coverage, which the paper does not characterize beyond type-aware enumeration. Figure 3(a)'s failures are diagnostic — the determinant probe invents a tensor name that never existed, so gold and substitute both return "name does not exist" and sit at distance zero over the whole bank, while the agent calls the tool on a tensor that exists; a batch-conversion probe never covers the modified category. In the real-server survey, 10 of the 35 tools classed deterministic return errors on every probe. Schema-valid is not semantically valid, and a stable error is not a fingerprint.

2. What the experiments establish does not require the proposed machinery, and the adversarial evaluation excludes the threat that matters. On clean cached cases the structural distance scores 0.952 at k≥2k \ge 2 — the same as exact equality on complete responses — and an ablation that reads only top-level type and serialization length "already scores 0.927". Turning the canonicalizer off "raises every comparison of complete responses to 0.974", because several substitutes mutate only the fields the canonicalizer deletes: it drops twelve volatile keys including id, cursor, etag and date at any depth and rounds floats to six decimals. Those are noise in some tools and business semantics in others — {"id": "account_A", "value": 100} and {"id": "account_B", "value": 100} canonicalize to the same thing, which needs no adversary who defeats the distance, only one who works upstream of it. The structural distance has a second scope problem that Appendix N shows: a small critical change inside a large response is diluted, so margin falls with output size. The distance therefore measures the fraction of structure that differs, not the security impact of the difference. Taken together with cost — exact equality uses roughly one fifteenth of the probe calls of the full-bank structural configuration in the end-to-end study (8,412 against 126,428), with 8 of 16 tainted tasks against 9 of 16 — the experiments do not establish that the proposed comparator is the part doing the work, and the paper has already moved its claim from "best distance" to "protocol". That makes the unanswered question sharper: against history-response regression testing with exact comparison and a standard threshold, what is ProbeID's irreplaceable part?

The security evaluation has a structural limitation the authors state: in the adaptive studies the gold remains among the candidates at distance zero, so with a non-negative distance no substitute can be strictly nearer, "certification is excluded by construction", and attacks can only convert identification into abstention. Zero misidentifications over 14,080 runs therefore measures the abstention implementation, not robustness to a general adversary. The deployment-relevant case is gold absent — the original endpoint replaced, with no zero-distance gold to shield the decision and a behavior-matching substitute facing only the single-candidate threshold. Relatedly the adversary must fix its mimicry region before seeing the round's sample; an attacker who observes probes and then adapts is outside the model, while the agent integration draws whole banks at every check, which accelerates probe exposure. And the availability cost is real: the white-box attack pushes abstention to 0.90 at k=8k = 8. Finally, the per-check δ\delta is not an end-to-end task-level failure probability; repeated checks need an error budget, and a union bound grows with the number of checks.

Questions for authors
  1. Does the identification rule survive distribution-contracting substitutes? For a stochastic tool returning 1 with probability 0.9 and 0 otherwise, a substitute that always returns 1 has expected distance 0.1 against the reference while the gold has 0.18, so the nearest-distance rule prefers the substitute and does so more confidently as mm grows, and it clears a threshold calibrated on the gold self-distance. Please add substitutes that return the modal answer, reduce output entropy, or shift the conditional distribution, and report whether the protocol can distinguish single-response proximity from conditional-distribution agreement. If it cannot, the stochastic-tool claim should be scoped accordingly.
  2. How should the finite-population correction be used on an online stochastic service? Proposition 3 is conditioned on a fixed per-probe score table, but a full-bank read removes only input-sampling uncertainty; response randomness remains. If s^\hat{s} at admission and d^\hat{d} at check time are identically distributed for an undrifted stochastic tool, rejecting on d^>s^\hat{d} > \hat{s} at zero radius rejects near half the time. What false-rejection rate do you measure for repeated calls to a genuinely unchanged stochastic tool at m=Bm = B, and how is the admission estimate's own error folded into τ\tau?
  3. Can the security evaluation be rerun with the gold absent? With the gold at distance zero in the candidate set, misidentification is excluded by construction, so the 14,080-run result does not bound the error of the realistic case. Please report false acceptance, false rejection, abstention and downstream task impact jointly for substitutes that exploit observed verification traffic, trigger on context, alter only canonicalized-away fields, or depend on state preconditions.
  4. Given that top-level type and length alone reach 0.927, exact equality ties the structural distance at 0.952, and disabling the canonicalizer raises complete-response comparison to 0.974, what does the structural distance plus canonicalizer add on the cases that matter? A direct comparison at matched reference and call budget against exact-equality regression testing would settle it. Would you also consider per-tool field policies, hard constraints on critical fields, or at minimum a security review of the drop list, given that six substitutes mutate only dropped fields?
  5. How was bank coverage chosen, and can you characterize what fraction of realistic drift a type-aware enumeration bank reaches? The census and the 38 held-out tools are the natural estimation set: over observed real changes, what share would a bank of size BB have caught? Given the two zero-coverage failures and the 10 of 35 tools that error on every probe, state-aware and task-conditioned probe generation that can construct valid preconditions looks more valuable as the paper's core contribution than further tuning the distance.
  6. Would the sequential structural gate (Proposition 1) replace the full-bank check in the agent experiment, and what are the tainted-task and refused-call counts under it? That is the configuration a deployer would run, and the cost comparison should then report security benefit, abstention rate, probe calls and end-to-end latency together, including the 152,348 calls spent recording and calibrating the bank.
  7. Should the main text carry Appendix T.1's rule table, stating for each experiment whether it uses forced choice, the default 0.5 and 0.05 thresholds, the theoretical threshold, or a post-hoc one? As written, a reader is most likely to retain "0.956 against near-random description routing" and least likely to retain that the calibrated-radius run decides 0.153 of cases, which is the more informative result.
  8. Two smaller items. What is the per-check δ\delta composed into over an agent run of many checks, since the paper gives no task-level guarantee? And requirements-tested.txt records the pinned environment as passing "the unit tests (78 passed, 2 skipped)" while the shipped suite reports 100 passed, 2 skipped — which revision do the recorded results correspond to?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer vGrq

Thank you for the review and artifact checks. Your counterexample is correct. Under independent Bernoulli(0.9) reference and gold responses, expected scalar distance is 0.18 versus 0.10 for an always-1 replacement. On an exact 100-point joint-probability fixture, the unchanged selector picks the replacement with gap 0.08 and no abstention, and more probes sharpen that wrong ordering.

1. Stochastic identity and calibration. The counterexample violates Theorem 1's positive-margin premise (γ>0\gamma>0). The theorem holds under its premises, and Section 4, Section 6, and Appendix U narrow the stochastic claim to separate response proximity, distribution agreement, and implementation identity, with no distribution-contract verifier implemented. In Appendix U, a synthetic energy score 2Ed(X,Y)−Ed(X,X′)−Ed(Y,Y′)2Ed(X,Y)-Ed(X,X')-Ed(Y,Y') gives expected 0 for the gold and 0.02 for the replacement. Over 4,000 synthetic trials (ties split evenly), it picks the gold at 0.907 (512 probes) and 0.997 (2,048) versus 0.000 for mean distance, yet needs two candidate draws per probe, lacks abstention calibration, and is untested on real tools.

Proposition 3 (Appendix S) conditions on a fixed score table and admission value. A full bank removes input-sampling uncertainty, not future response randomness. Our 1,000 draws reuse recorded responses. We have not measured the requested repeated-future-gold false-refusal curve or bounded admission self-distance error in the threshold, so we cannot claim live 0.05 control (Appendix U).

2. Gold-absent and adaptive evaluation. The 14,080 gold-present runs cannot establish general attack robustness, because a zero-distance gold excludes strict misidentification by construction. The k=8 white-box attack still causes 0.90 abstention (Table G.2). Gold-absent bank checks are separate. At m=32, Hoeffding equality accepts the eight reachable replacements at 0.006 versus 0.750 for structural distance, with no gold refusal (Table Q.2). Sharper equality has no recorded acceptance at m=32, versus 0.750 for finite-population structural distance, which reaches zero at m=256 (Table S.1). Exact-gate gold refusal is at most 3.2% over reported budgets. Both comparisons refuse reachable replacements at the full bank, while two unreachable replacements pass at every budget. The requested joint probe-observing, context, dropped-field and state-precondition attack study with downstream impact remains unrun.

3. Comparator, canonicalization, and coverage. Structural distance is not indispensable. Equality ties at 0.952 on clean cached cases, versus 0.927 for type and length (Table J.2). At m=32, single agent runs used 8,412 probes with 8/16 tasks tainted for equality, versus 8,398 with 13/16 for structural distance (Table P.2). Different trajectories preclude a reliable task-protection comparison. Unlisted volatile fields reduce equality to 0.083 versus 0.950 for structural distance. At 90% expected coverage, k=4, rho=0.2 and canonicalization on, strict-choice miss rates are 0.0210, 0.2877 and 0.0049 for ProbeID, equality and string MMD (Table U.1). Ties count as misses. These 300-case-per-seed, three-seed comparisons reuse tools and carry no independent-trial confidence claim.

Structural distance does not weight security impact, and canonicalization can erase meaningful IDs. Removing it raises clean hit@1 from 0.952 to 0.974 (Table J.2). We implemented no field contracts or critical-field constraints. Schema-valid probes and stable errors (10/35 deterministic census tools error on every probe) do not establish valid state preconditions. Determinant and convert_batch still have 0.000 bank distance. The held-out structural gate refuses 16/16 changed-error tools and 5/7 local added fields (Table R.2). A fixed equality gate catches 3 of 7 registry-side changes among 59 census tools (Appendix O), without estimating realistic drift coverage.

4. Operating points and cost. The sequential gate (8 probes on local full drift, Table R.2) lacks an end-to-end agent protection and latency run, and 126,428 runtime probes plus 152,348 setup and calibration calls remain material costs (Appendix P). Table 1 separates cached forced-choice 0.956 hit@1 from radius coverage 0.153 (0.000 wrong), neither meeting the sampling premises, and Section 5.3 cites Table T.1. Across TT checks, equal per-check δ\delta bounds give at most a TδT\delta union bound under their premises, and task-level allocation is unevaluated (Section 6).

5. Artifact revision. The note was stale, recording 78 passed and 2 skipped on 17 September. We reran every README section 1 command on a fresh unzip with the pinned versions (Python 3.13.13). The revised supplement records that run, with all 793 number checks and 100 tests passed and 2 skipped, as you found.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Reviewer_g5szOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to verify MCP tools when attackers copy tool descriptions or change tool behavior after admission. It proposes ProbeID, which stores trusted responses to a bank of test inputs. At selection time, it probes candidate tools, removes volatile response fields, and compares the results with the stored reference. It selects the closest match when the distance and separation thresholds are satisfied. The paper derives probe-budget and error bounds under explicit sampling and behavioral-separation assumptions. Experiments cover cached APIs, live MCP servers, and an agent benchmark. ProbeID reaches 95.6% accuracy under forced selection on cached cases and reduces tainted agent tasks from 15 to 9 out of 16. An eleven-week study also finds response changes in 7 of 59 real tools despite unchanged version pins.

Critical strengths
  1. The paper addresses a practical problem.
  2. The verification protocol is simple, and the theoretical assumptions are clearly stated.
  3. The experiments include useful behavioral baselines, an eleven-week drift study, and detailed analysis of failure cases.
Critical weaknesses
  1. The methodological novelty is modest. The protocol builds on reference-based testing, behavioral comparison, and standard concentration bounds. Its application to MCP tools is useful, but the advance over these established ideas is limited. The structural distance also provides no consistent advantage over simpler comparisons: exact matching ties the clean cached results and needs fewer probes on deterministic tools, while MMD performs better under heavy value noise.

  2. In the agent study, the determinant and batch-conversion tools match the reference on every bank probe but still return corrupted values on actual agent calls. Consequently, 9 of 16 tasks remain tainted after verification. Increasing the probe budget cannot fix these cases because the bank never reaches the changed behavior. Better coverage of valid states and nested inputs is therefore central to the method’s practical value.

  3. Verification adds substantial tool-call overhead. The main agent configuration uses up to 512 probes per check, totaling 126428 probe calls across 16 tasks. Exact matching reduces this to 8412 probes, which is still large compared with the 1180 normal calls in that run. The sequential gate reduces probe counts in the bank studies, but its added latency and protection have not been evaluated together in an end-to-end agent run.

  4. The 95.6% accuracy result uses forced selection on fixed cached probes, which do not satisfy the theorem’s sampling assumptions. Applying the theory-inspired thresholds to the cached cases with eight squatters produces decisions on only 15.3% of cases. Although the paper clearly separates these settings, the gap shows that high nearest-match accuracy does not directly establish useful coverage with the proposed error control.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer g5sz

Thank you for the review. Reference-based testing and concentration bounds are established. ProbeID adapts them into a post-admission verification protocol with explicit sampling, separation, and abstention conditions (Algorithm 1, Theorems 1–2), paired with drift and corruption measurements. Gold-server replay shows why value measurement matters. Despite 1.000 benchmark task success in every arm, tampered values reach 15/16 tasks ungated and 9/16 gated in all three repeats (Table P.1), and 7/59 real tools drifted over eleven weeks despite version pins (Appendix O). This is incomplete protection rather than a security certificate, and comparator performance depends on noise.

1. Comparator evidence. Theorems 1–2 hold for the probe mean of any [0,1][0,1]-bounded per-probe comparison with its own margin γ\gamma (Section 3.2, Appendix J.2). A retrospective reanalysis of saved confidence and correctness arrays (Section 5.3, Appendix U, Table U.1) gives strict-choice miss rates among retained cases at 90% expected decision coverage on the 300-gold-per-seed, three-seed cohort of Table J.2 (k=4k=4, canonicalization on).

Value jitterProbeIDExact equalityString MMD
ρ=0.1\rho=0.10.00620.15430
ρ=0.2\rho=0.20.02100.28770.0049

Candidate ties count as misses, and randomization applies only at the coverage boundary. Because tools recur across the three seeds (900 case-runs), these retrospective comparisons carry no independent-trial confidence claim. Equality ties dYd_Y on clean cached cases (0.952 here, 0.956 on the 150-gold cohort) and needs fewer probes on deterministic agent tools, dYd_Y holds 0.950 versus 0.083 for equality under 10 unlisted volatile fields (Table J.2), and string MMD performs best under heavier leaf jitter (+0.063 hit@1 over dYd_Y at k=8k=8, ρ=0.2\rho=0.2, Appendix J.2) as an empirical set statistic outside the per-probe guarantees.

2. Probe coverage. Both surviving agent failure classes sit at 0.000 bank distance (Section 5.5, Appendix P). Tensor-name probes lack prior creation state so gold and replacement return the same error, and convert_batch admits seven schema probes that miss the modified category (likewise, 10/35 deterministic census tools return errors on every probe, Appendix O). These banks remain unpatched, and schema enumeration has no demonstrated coverage fraction for realistic drift. Fixing them requires state-aware, task-conditioned valid inputs rather than more probes on unchanged banks.

3. Execution overhead. The m=512m=512 structural run uses 126,428 runtime probes (9/16 tainted, alongside 135,111 benign probes with 0 refusals, Table P.1). An end-to-end equality run at m=32m=32, chosen after earlier outcomes, uses 8,412 runtime probes alongside 1,180 business calls and leaves 8/16 tasks tainted on differing trajectories, while m=32m=32 structural dYd_Y (8,398 probes) leaves 13/16 tainted (Table P.2). Recording and calibrating the banks adds 152,348 one-time calls (Appendix P). The sequential gate averages 8 probes on local full value drift (Table R.2), without a joint end-to-end agent protection and latency evaluation (Section 6). In the gold-absent bank check at m=32m=32, Hoeffding dYd_Y falsely accepts the eight reachable replacements at 0.750 versus 0.006 for equality with 0 gold refusals for both (Table Q.2), while under the sharper rules (Table S.1) finite-population dYd_Y stays at 0.750 and hypergeometric equality drops to 0.000 wrong acceptance (gold refusal at most 3.2% over reported budgets).

4. Decision coverage and formal scope. On cached k=8k=8 cases, Table 1 separates forced-choice coverage 1.000 (0.044 wrong) from per-case radius coverage 0.153 (0.000 wrong, mi≤12m_i \le 12, Appendix K.3). Neither cached row meets the sampling premises, so forced-choice accuracy does not show certified coverage. Under random bank draws (δ=0.05\delta=0.05, gold present, Table 1, Appendix Q), Algorithm 1 decides 0.784 (m=32m=32) and 0.934 (m=64m=64) on the eight reachable tools under equality with 0.000 wrong (0.750 at m=512m=512 under dYd_Y), abstaining on the two zero-margin tools. Theorem 1 also requires γ>0\gamma>0. A distribution-contracting stochastic replacement can violate this premise and outscore the genuine tool, as in the Bernoulli(0.9)-to-always-1 counterexample (0.10 versus 0.18, Section 4, Appendix U). The supported claim is reference agreement on covered, observed behavior under stated premises, not unrestricted stochastic or stateful identity recovery.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Thank you for the candid response. I appreciate that the paper states its limits clearly, but I will keep my rating.

The response confirms each concern rather than resolving it. The probe banks that miss the changed behavior remain unpatched, so 9 of 16 tasks stay tainted and more probes cannot help. The verification cost is confirmed and the sequential gate is still not evaluated end to end. The new matched-coverage table shows that the structural distance sits between exact matching and MMD, which supports my point that it has no consistent advantage.

The problem is important and the measurements are useful, but the method needs a way to reach state-dependent and nested behavior before it offers practical protection.

Thank you for revisiting the paper and for your careful assessment of what the current probe banks can detect. We appreciate your candor and respect your rating. For clarity on the record, we view ProbeID primarily as a verification and abstention protocol (Theorems 1–2) that accommodates any bounded per-probe comparator rather than claiming structural distance outperforms every empirical set statistic such as string MMD. We agree that reaching state-dependent behavior requires state-aware probe construction beyond schema enumeration. Thank you again for the constructive discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Closing Note Before Discussion Ends

Thank you again to all three reviewers and the Area Chair for your thoughtful feedback and time. As the discussion period wraps up, we remain glad to clarify any remaining question regarding our revised manuscript or updated supplementary artifact.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.