- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
he paper asks whether existing black-box behavioral audits can verify that an API serves the model it claims. It decomposes a deployment into a model specification (checkpoint, learned adapter, numerical representation), a generation policy (decoding, generation-time watermarking) and an interaction layer (system prompt, agent harness), and defines the verification target as rejecting changes to while accepting changes to or . AuditGym then builds reference-target pairs that differ in one known factor, hides the ground truth from the auditor, and runs four auditors — MET, RUT, AgentProv, and a verification adaptation of LLMmap — over 19 checkpoints and seven change families at a nominal .
The reported results are that all four auditors detect checkpoint substitution almost perfectly (181–182 of 182); that MET, RUT and AgentProv miss many adapter (6–7 of 10) and quantization (30, 51, 43 of 62) changes, with INT8 hardest (2/12, 4/12, 1/12 on the twelve shared BitsAndBytes models), while LLMmap detects all of both (10/10 and 62/62) at the cost of 3/14 false detections on the unchanged-deployment control; and that every method falsely rejects unchanged models under some decoding, watermarking, system-prompt or harness change (MET 36/80 and RUT 60/60 under decoding changes, LLMmap 45/45 under system prompts, and all four flagging all five unchanged models under the DeepSeek harness). The claimed contribution is the benchmark itself and the conclusion that current auditors confuse model changes with deployment changes.
I note the framing, because it is the paper's most durable contribution: a detected discrepancy does not establish a change of model, and a non-detection does not establish that the model is unchanged.
Critical strengths
1. The paper fixes a two-sided verification contract before measuring anything, and it is unusually careful that each intervention actually intervened. Prior auditing papers report detection without stating which differences ought to trigger rejection, so a rejection cannot be interpreted. Separating from and , then reporting detection and false detection on the same controlled pairs, is the right primitive, and Section 3.3 draws the distinction honestly rather than scoring a point: when a decoding change moves the response distribution, MET or RUT rejecting may be statistically correct and still wrong for verifying . Few papers that repurpose other people's methods concede this.
The same care shows in the details that are easy to skip. One Mistral medical adapter produces no parameter update and is excluded (Appendix B.2), leaving ten adapter comparisons — that is, the authors checked whether each intervention was real rather than banking a guaranteed miss. The quantization trend is not left resting on heterogeneous model coverage: on the twelve shared BitsAndBytes models the four-bit-over-INT8 ordering still holds (MET 2/12 INT8 vs 11/14 FP4 overall, 10/12 FP4 on the shared subset; RUT 4/12 vs 12/12; AgentProv 1/12 vs 10/12), which is a confound the authors controlled on their own initiative. And the arithmetic is internally exact everywhere I checked: Table 2 sums to Table 1's quantization row for all four methods (30, 51, 43, 62 of 62), Table 3's per-setting counts sum to its totals (36/80, 60/60, 19/80, 5/80), Table 4's watermark rows sum to 42, 17, 28 and 7 of 52, Table 5's column reproduces Table 4's KGW row cell by cell, Table 6's two breakdowns of the same 45 comparisons agree (38, 39, 37, 45), and checkpoint substitution is . Benchmark papers are where arithmetic slips usually surface; here they do not.
2. The watermark asymmetry is a genuinely new empirical result, and it is the kind of result the framing exists to produce. MET flags all thirteen models under SynthID-Text, TextSeal and Gumbel-max but only three of thirteen under KGW, while RUT shows nearly the reverse pattern (Table 4). This is more than "deployment changes perturb auditors": it is evidence that different audit features track different behavioral channels, and it is the finding most likely to seed follow-up work. It also only becomes legible because the benchmark holds fixed and varies one factor, which is the design paying off. I state it as a strength with its limit attached — Appendix B.5 says watermark strengths are not matched across families, so the pattern is real but its mechanism is open.
Critical weaknesses
1. The harness conclusion that the abstract escalates to "more seriously" has no negative control, and the control that would settle it was collected and never reported. Appendix B.7 states: "Ten additional same-model comparisons within matched harnesses serve as diagnostics. The study does not include bare-versus-bare controls." So the measurement that would make the lower block of Table 7 interpretable exists, and its result appears nowhere — Table 21, which disaggregates that block, marks every same-model pair with a dash. Without the matched-harness same-model false-detection rate, 20/20 directed substitutions under Codex Default and 19–20/20 under DeepSeek is also exactly what a rule that always rejects would score. The upper block (bare reference vs same model under a harness, 5/5 and 4–5/5 false detections) is the headline claim, and it is being contrasted against a lower block whose own noise floor is unmeasured.
The same rows are independently confounded by generation that did not work. For RUT, Llama–DeepSeek has 6,375 empty outputs among 7,480 completed reference tasks and 64 among 74 target tasks, with empty outputs scored ; Llama produces no valid native tool call under either harness; available tools "are not forced to execute"; for AgentProv, bare responses are labeled by a fixed LLM judge while harness responses are read off the first schema-conforming tool call, so action extraction differs between the two sides being compared; and MET and RUT "select completed responses under post-collection protocol amendments", which changes the estimand to without reporting completion rates. A rejection computed between a mostly-empty response distribution and a non-empty one is not evidence that a harness invalidates a reference.
This is consequential rather than pedantic because the harness extension is the paper's most valuable direction — it is the condition a practitioner actually faces — and it is the part on which I can place the least weight. It is also the most cheaply fixed weakness in the paper: report the ten diagnostics, add bare-versus-bare controls, hold action extraction fixed across conditions, and report completion, empty-output and tool-execution rates alongside every harness count.
2. The counts that carry every comparative claim have an unmeasured noise floor, do not always describe a common target or a common budget, and cannot be recomputed by a reader, because the artifact contains no measurements. Four independent legs, each sourced from the paper or the supplement:
(a) The one measurement of the noise floor is large and was not propagated. Appendix B.6: MET's "earlier complete collection with the same rendered prompts but different request-seed streams gives 29/45 detections, compared with 38/45 in the primary collection. No individual outcomes are selected between the two collections." That is a same-condition re-run of a headline false-detection count moving by nine of forty-five, about twenty percentage points. I take the authors at their word that no cherry-picking occurred, and I credit them for disclosing it; the problem is what it implies for the rest of the paper. No other cell is repeated, so no count anywhere in Tables 1–7 has a stated noise floor, and the paper's conclusions are drawn from differences between single-run counts that this one re-run suggests can move by that much.
(b) Cross-method comparisons do not always compare the same target. Appendix B.3: "These conversion and precision differences mean that quantized targets are not identical across all auditors." Within a method a pair remains interpretable, but a statement such as "RUT detects quantization better than MET" is then confounded with which quantized artifact each auditor was served. Matching conditions inside each pair does not fix comparisons across methods.
(c) Evidence budgets differ by orders of magnitude, and the decoding totals are not computed over a common set of settings. Per comparison, RUT uses 100 probes with 100 reference responses and one target response each — 10,000 reference and 100 target responses — while LLMmap uses three enrollment traces plus 199 calibration traces at eight probes each, 1,616 reference responses, against a single eight-probe target trace. Keeping each method's native protocol is defensible, but then the table answers "how does each full pipeline do at its own cost", not "which method is better". More specifically, Table 3's Total row compares RUT over six settings against the other three over eight, and the two extra settings are the top-k ones RUT excluded because, per Appendix C.2, "Top-k = 1 does not make the responses deterministic, so we cannot confirm that top-k is applied" — a serving-path problem that would apply to every auditor, not a property of RUT. Restricted to the six settings all four share, the totals are MET 34/60, AgentProv 19/60 and LLMmap 4/60 against RUT's 60/60. The ranking survives; the printed Total column is not a like-for-like comparison and the paper does not say so.
(d) No reported number can be checked offline. I ran the supplement: pip install -e ".[test]" succeeds, the CPU mock quickstart completes, and pytest gives 988 passed, 1 skipped. The engineering is real — 381 files, content hashes on every artifact, reference and target contract hashes compared bidirectionally, independent_sampling_streams asserted, and load_protocol checking the eleven ordered pairs against , seed 42 and repetition 0. But the README states that collected responses, results and logs are excluded, and that is accurate: there are no response banks, no intermediate statistics, and not even per-comparison statistic and -value tables. The files named *_results.py are scoring drivers that read response banks and cannot run without them. For a submission whose contribution is a benchmark, the measurements are the artifact; as released, a reader can inspect the harness and must take all 19 models' numbers on trust. Three of the five submissions I am reviewing this cycle do ship their recorded results, so this is a solvable problem rather than a norm of the area.
These legs compound with a fifth issue specific to the method that looks best. LLMmap's nominal level is not established in the regimes where it posts its lowest false-detection counts. Appendix B.7 reports that some greedy reference conditions contain only one to four distinct calibration traces and that "reference and target collection layouts differ in some conditions, limiting the exchangeability assumption underlying nominal calibration". The supplement pins the mechanism down: paper_settings/llmmap-decoding-v3-protocol.json freezes with 199 calibration samples and one target trace, and records the rule as calibration 199, rank 190, reject on target_distance > threshold, ties not rejected; equality/methods/llmmap.py computes that rank as and takes sorted(distances)[189]. Because the protocol carries an explicit , this is the path Table 3 runs, not the empirical_quantile branch. The consequence is arithmetic: the threshold equals the largest calibration distance exactly when that largest value occurs at least times, and any target sitting on an already-seen distance is then accepted by construction. That is a condition on the multiplicity of the maximum, not on the number of distinct values, and per-condition calibration frequencies are not released — so I do not claim LLMmap's 0/10 cells are caused by a saturated threshold. I claim the nominal level is not established there, and that the paper's cross-method counts assume it holds. I also want to be fair about magnitudes in the other direction: LLMmap's 5/80 is 6.25% at a nominal 5% level and should not be read as a failure comparable to RUT's 60/60; with shared references inducing dependence, an independent binomial test is not available either way.
Questions for authors
Items 1 and 2 are correction requests rather than questions; the rest are the answers most likely to move my assessment.
-
Please correct the abstract. It states that "all four detect checkpoint substitution but miss finer model changes such as adapters and quantization", which Table 1 contradicts: LLMmap is 10/10 on adapters and 62/62 on quantization. The accurate version — MET, RUT and AgentProv miss many adapter and quantization changes, while LLMmap detects all of them in these settings but produces 3/14 false detections on the unchanged-deployment control and up to 45/45 under system prompts — is no weaker as a result, and it is the sentence most readers will carry away.
-
Please report the ten matched-harness same-model diagnostics named in Appendix B.7, and state the bare-versus-bare false-detection rate or that it is unmeasured. Without them, the 19–20/20 substitution figures cannot be distinguished from what an always-reject rule achieves. If those diagnostics show a non-trivial false-detection rate under matched harnesses, which of Section 4.4's conclusions change?
-
How many of the paper's counts move under re-collection? Appendix B.6's 29/45 versus 38/45 is the only same-condition repeat reported. For a representative subset of cells across the seven change families, what are the counts under at least two independent collections, and does any conclusion in Sections 4.2–4.4 depend on a difference smaller than that spread?
-
For the harness rows of Table 7, what do the results look like restricted to models and conditions producing non-empty, parseable responses on both sides, with action extraction held fixed between bare and harness conditions? Specifically, does "unchanged model, bare reference" remain 5/5 for RUT once the Llama–DeepSeek condition with 6,375/7,480 empty reference outputs is excluded? Please also report completion, empty-output and tool-execution rates next to each count.
-
For the LLMmap decoding and watermark cells, how often does the 190th of the 199 ordered calibration distances equal the largest, equivalently how often does the largest distance occur at least ten times? That multiplicity, not the count of distinct values, decides whether the threshold is saturated, and it is what Appendix B.7's "one to four distinct calibration traces" leaves open. What are the false-detection counts restricted to cells where it does not hold? Reporting the distance and threshold distributions, plus same-deployment controls across collection batches, restarts and batching layouts, would settle this directly. I would also suggest naming the method "LLMmap (verification adaptation)" throughout, so that conclusions here are not read back onto the original identification method.
-
Can the quantization comparison be re-run with a target artifact common to all four auditors, and can one cost-matched variant be reported alongside the native protocols — matched target tokens or matched target responses, with reference-preparation, target-audit and auxiliary scoring or judging costs reported separately? This is what would restore the cross-method column as a comparison.
-
Does the benchmark admit an "insufficient evidence" verdict? For a binary verifier, , where and are the audit-record distributions under a permitted redeployment and under a changed model, so when the two are close at the given query budget, low false rejection and low miss cannot be achieved together. Stating the permitted change set, the detectable change scale and the query budget, and allowing abstention, would let the paper separate three failure sources it currently pools: an improvable method, an insufficient budget, and a genuine identifiability limit. Relatedly, would you consider stating explicitly that AuditGym instantiates one strict contract in which numerical representation is part of identity, rather than the only reasonable notion?
-
Two reporting items. Appendix C.2 explains RUT's top-k "N/A" as an inability to confirm that top-k reached the sampler; since that is a serving-path property rather than a method limitation, should the other three methods' top-k columns be marked the same way, and the Total row computed over the six verified settings? And the Qwen3.5-4B AWQ target that "returns empty one-token stops on the audit" is currently counted as a successful quantization detection — could endpoint failures be reported separately from detections of a served quantized model, so that model-identity detection is not pooled with liveness detection?
-
The AgentProv action labels "have not been independently validated by humans". For the adapter, quantization and system-prompt studies, could you report a stratified human spot-check with agreement rates and typical error modes? This matters most for the system-prompt study, where an instruction can change surface form in ways that alter how the judge extracts an action without the underlying tool preference changing.
-
Would you release the collected response banks, or failing that the per-comparison statistics and -values? The protocol freezing and content-hashing machinery already in the code suggests this is a packaging decision rather than new work, and it is what would make the benchmark's central claims independently checkable.
3 comments
W2(a) / Q3: Re-collection variability and stability of conclusions
Our investigation traced 8 of the 9 additional detections to MET’s Phi results. We conducted additional data collections and targeted Phi checks for the other auditors, observing no changes in the detection counts for RUT and LLMmap, and only a single change for AgentProv. Across these additional collections, system-prompt changes continued to trigger false detections, reinforcing the paper’s central finding. For the camera-ready version, we will extend this analysis with a multi-seed variation study across auditors, models, and change families.
W2(b) / Q6: Shared quantized targets and cross-method comparability
The targets are separately generated quantized versions of the same pinned base checkpoints. MET and RUT used different calibration settings and quantization-software versions, producing valid four-bit AWQ/GPTQ targets. We have specified these settings in Appendix B.3. Each audit compares a quantized target with its corresponding unquantized reference. The finding that MET, RUT, and AgentProv can miss quantization changes therefore remains supported.
W2(c) / Q6 / Q8.1: Evidence budgets and consistent top-k reporting
We retain each auditor’s original protocol and evidence budget to evaluate the method as designed; these budgets are specified in Appendix C. Following your suggestion, we removed both top-k settings for all methods. The four original auditors now cover the same six decoding settings and ten models, as reported in Table 3 and Appendix Table 20. The finding that decoding changes trigger false detections remains supported.
W2(d) / Q10: Release of per-comparison results and reproducibility artifacts
Thank you for suggesting this addition. We have added per-comparison statistics, p-values or decision thresholds, audit decisions, configurations, and offline table-generation scripts to our anonymous repository. The package allows readers to reproduce the reported table counts without running model inference.
W2(e) / Q5: LLMmap calibration saturation, controls, and naming
We examined all 23 reference banks used for decoding and watermarking. Each contains 199 distinct calibration distances, so the maximum occurs only once and the rank-190 threshold is always below it. No cases are excluded by the proposed saturation check; the reported counts remain 4/60 for decoding and 7/52 for watermarking. Unchanged-deployment controls yield 1/10 and 0/13, respectively. Decoding uses the same persistent server per model, and watermark collection uses matched software and worker configurations. The low-diversity observation concerns the separate greedy harness study. Appendix C.4 confirms that the calibration thresholds are below the maximum distance. We adopt the name LLMmap (verification adaptation) as suggested.
Q1: Accurate method-specific claims in the abstract
We have revised the abstract to distinguish the methods: MET, RUT, and AgentProv miss some adapter and quantization changes, whereas LLMmap (verification adaptation) detects all tested changes in these settings but falsely rejects 3/14 unchanged deployments, consistent with Table 1.
Q7: Insufficient evidence, identifiability, and the verification contract
AuditGym supports application-specific verification contracts. Our evaluation treats numerical representation as part of model identity, while applications may permit quantization. Table 2 characterizes each auditor’s quantization sensitivity and can guide method selection under either contract. We now make this explicit in Problem Formulation. We retain each auditor’s native decision rule and evidence budget, without adding a calibrated “insufficient evidence” verdict. Such an outcome is a useful extension: non-rejection alone cannot distinguish limited method sensitivity, insufficient evidence, and an identifiability limit. We clarify in Auditing Procedures and Assessment that our results measure sensitivity at the stated budgets, and that non-rejection neither certifies identity nor establishes undetectability.
Q8.2: Separating AWQ endpoint failures from model-change detection
We checked this case, identified a Qwen3.5-specific AWQ conversion defect, and corrected it. The repaired deployment passes usability validation, and all eight recollected audit responses are nonempty. With the original reference, calibration, and decision rule, LLMmap still detects the quantization change. Tables 1 and 2 therefore retain 62/62 overall and 11/11 AWQ detections, with this case now evaluated on responses from the functioning quantized model.
Q9: Human validation of AgentProv action labels
Our evaluation asks whether AgentProv’s complete auditing pipeline distinguishes model changes from permitted deployment changes. Its action-extraction judge is part of that pipeline and is applied consistently across reference and target conditions. Appendix C.3 specifies the action-extraction judge and procedure.
Additional analysis: Watermark sensitivity and utility
We thank the reviewer for highlighting the watermark asymmetry as a promising direction for follow-up work. We investigated whether audit rejection is associated with utility degradation caused by watermarking, evaluating GSM8K accuracy, MBPP pass@1, and generated-text perplexity across thirteen models. MET rejects all thirteen models under selected SynthID-Text and TextSeal settings, despite small average changes in accuracy on both tasks. Thus, watermarking can trigger audit rejection without substantially degrading average task performance. We report this additional study in Appendix D.
If the above answers address the Reviewer's concerns, we would greatly appreciate it if they could consider increasing the rating. We are also happy to address any remaining questions.
We thank the reviewer for recognizing our explicit verification contract, the care of our controlled experimental design, and the distinction between detecting distributional differences and verifying model identity. We also appreciate their recognition of the watermark asymmetry as a new empirical finding with value for follow-up work. Below, we address the reviewer’s comments individually.
Thank you for suggesting these controls. We now report bare-versus-bare controls and all ten matched-harness same-model comparisons for each of the four original auditors in Table 8.
Across the ten matched-harness controls, MET produces one false detection, while the other three auditors produce none. Together with the 19–20/20 substitution detections in Table 7, these results rule out an always-reject explanation. The qualitative conclusion is unchanged: high substitution detection within a matched harness coexists with frequent rejections of unchanged models against harness-free references.
We recalculated the RUT tests after excluding empty responses from both reference and target samples, with calibration based on the retained sample counts. Across all five models, including Llama, this yields 5/5 bare-to-DeepSeek rejections and 0/5 matched DeepSeek control rejections. The revised results are reported in Tables 7 and 8.
Thank you for this helpful suggestion. Following your recommendation, we aligned AgentProv’s action extraction across bare and harness conditions. The bare reference now receives tools through the native API, and both sides use the first delivered tool-call name without an LLM judge. We recalculated the comparisons and report the revised results below.
With the same extraction rule, unchanged models still trigger rejections under both harnesses, while independently collected bare-versus-bare controls yield none. Table 7 reports the revised results, and Table 23 provides per-model decisions.
We thank the reviewer for recognizing the practical importance of the harness extension. We added a controlled Mini SWE study across five unchanged models and four auditors, varying the instruction package, Bash tool interface, and execution feedback while keeping each auditor’s observation rule consistent across conditions.
Entries are false detections across five unchanged models. Tools return fixed acknowledgements unless real feedback is specified. AgentProv retains its probe tools throughout, its feedback comparison is N/A because feedback follows its observed first tool choice. LLMmap denotes the verification adaptation.
Instructions and tool interfaces produce more consistent rejections than execution feedback in this study. AgentProv also shows that tool-interface sensitivity depends on the surrounding instructions. These results extend the whole-harness evaluation by showing which component changes affect reference compatibility in Mini SWE. Table 9 reports the results, and Appendix B.8 provides the setup details.