acceptodds
Under review as a conference paper at ICLR 2027

One Character at a Time: Frustratingly Simple and Cost-Efficient Black-Box Model Equality Testing

Abstract

Users of third-party large language model APIs may have little direct evidence that the served model matches the advertised one. Black-box model equality tests offer a practical auditing mechanism, but existing methods can require many queries, long generations, or a costly query optimization stage. We study whether reliable auditing is possible with minimal resources and find a frustratingly simple yet effective equality-testing strategy that requests only one character per model response. Our method asks both the candidate and a trusted reference model to output one symbol from a fixed alphabet. Repeated responses yield empirical categorical distributions, which we compare using total variation (TV) distance and a paired permutation test. We evaluate our method using a cost-versus-statistical-power protocol, with same-model and different-model comparisons to assess Type I and Type II errors. Comprehensive empirical evaluation shows that our method achieves strong performance at substantially lower cost than existing state-of-the-art methods. We also develop a theoretical analysis of when single-symbol responses suffice for model equality testing.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Equality Testing, Fingerprinting, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WF89OpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an important and timely problem: how to audit reliably with minimal resources. Specifically, the method asks both the candidate and a trusted reference model to output one symbol from a fixed alphabet. Repeated responses yield empirical categorical distributions, which we compare using total variation (TV) distance and a paired permutation test.

Critical strengths
  • The paper studies an important and interesting problem.
  • The experimental results demonstrates that the method achieves strong performance at substantially lower cost than existing SOTA methods.
Critical weaknesses
  • The Methodology in Section 4 introduces numerous definitions and theorem, making it difficult to understand and follow.

  • My main concern is how to construct the input query. Section 4.2 shows that when Δ > 0, there exists a query that induces different output distributions for two models. However, how can such a query be identified in practice? Section directly uses a fixed query instruction: "Choose a random lowercase English letter from a to z. Output exactly that one letter and nothing else". What motivates this particular choice, and how does it relate to the theoretical analysis in Section 4.2?

Questions for authors
  • My main concerns is the connection between the theoretical analysis in Section 4.2, and the practical implementation in Section 4.1. Section 4.2 establishes the existence of a distinguishing query but does not explain how to identify such a query in practice. In fact, the proposed method directly adopts a fixed query instruction. I think it is logically necessary to clarify whether the theory provides any guidance for query design.
1 comment
AuthorsOpenReview ↗

Response to Reviewer WF89

Thank you so much for your encouraging review! We were really happy to see your recognition of the problem’s importance and the empirical results. We also appreciate your thoughtful comments on the theory and presentation. These comments helped us identify where the connection to our practical method needed to be clearer.

W1. Presentation of the methodology and theoretical analysis

Thank you for pointing this out. We have clarified the organization of Section 4.2 by separating the analysis into two explicit questions:

  • Section 4.2.1, “Existence of a Distinguishing Query”, studies whether a short query can distinguish two models in the stated discrete-attention setting.
  • Section 4.2.2, “Testing with a Fixed Query”, analyzes the statistical test when the query has already been chosen.

This distinction separates the structural existence result from the guarantee for the practical testing procedure in Section 4.1. The new result is stated as Theorem 4.5, with its full proof in Appendix B. We hope this makes the purpose of each theoretical result easier to follow.

W2. Motivation for the practical query and how it is chosen

The lowercase-letter instruction is a cost-motivated design choice. It allows a one-token generation cap and produces a small parsed category space consisting of letters and INVALID. The same instruction can also be reused across model pairs without a separate search for model-specific queries.

Our choice is supported empirically by the prompt ablation in Figure 5, which compares the instruction alone with an UltraChat message followed by the same instruction. Under the evaluated settings, the instruction alone provides the better observed cost–power trade-off. We do not assume that the models actually sample letters uniformly; the test compares their induced response distributions.

Accordingly, Theorem 4.4 and our practical prompt now play clear, complementary roles: Theorem 4.4 establishes the theoretical principle that single-token responses can carry strong distinguishing signal, while our empirical ablations motivate the fixed one-character instruction as a simple, zero-search, cost-efficient instantiation.

Q1. Connecting the theoretical analysis to the practical method

To address this connection directly, we have added Theorem 4.5 in Section 4.2.2, with the full proof in Appendix B.

The new theorem analyzes the paired TV test for a fixed prompt, parser, and endpoint configuration, including its finite Monte Carlo implementation. Under independent sampling, it provides:

  • Type-I error control when the two parsed response distributions are equal.
  • A finite-sample power bound when those distributions have a positive total-variation gap.

These guarantees do not require the discrete-attention architecture used in Theorem 4.4. They apply to the fixed response distributions being tested, which directly connects the analysis to our practical procedure.

The power guarantee is conditional on a positive response-distribution gap. It does not assert that the chosen prompt distinguishes every model pair: if two models induce the same parsed distribution at that prompt, repeating it cannot distinguish them. This clarifies both what the practical test can establish and the limitation of using a fixed query.

We sincerely appreciate your feedback, which motivated this additional theoretical analysis. With the new theorem, its proof, and the clearer distinction between the two theoretical questions now included in the revised PDF, we hope you will consider raising your score if these revisions address your concerns. Thank you again for your time and encouragement!

Reviewer_TPqmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a cost-efficient black-box model equality testing method for verifying whether a third-party LLM API serves the same model as a trusted reference. The proposed approach uses a single fixed prompt that asks the models to randomly generate one lowercase English letter. By repeatedly querying both endpoints, the method constructs empirical categorical distributions and compares them using total variation distance and a paired permutation test.

Critical strengths
  • Meaningful research problem.

The paper addresses an important practical challenge in auditing third-party LLM APIs, where users cannot directly verify whether the provided model matches the advertised one.

  • Simple and lightweight approach.

The proposed method is straightforward to implement and does not require access to model parameters, internal representations, or token probabilities. Its simplicity and low query cost make it potentially useful as a lightweight behavioral auditing mechanism.

Critical weaknesses
  • The fundamental assumption underlying the proposed model equality test is insufficiently justified.

The most significant concern is that the entire approach relies on comparing the output distributions of two models under a single fixed prompt requesting a randomly selected character. However, the paper does not establish why the response distribution under this particular prompt should be sufficiently representative or discriminative for general model equality testing. Two substantially different models could produce identical character distributions under this prompt while exhibiting significantly different behaviors across other prompts and tasks. In such cases, increasing the number of queries would not resolve the fundamental indistinguishability. Although a statistically significant difference under the fixed prompt provides evidence that two models are different, failing to detect such a difference cannot establish that they are the same model. This distinction is particularly important given the paper's stated goal of verifying model identity in third-party APIs. Without additional assumptions or evidence supporting the discriminative capability of the fixed prompt, the proposed approach is better characterized as a lightweight, prompt-specific behavioral difference detector rather than a general model equality verification method. The authors should provide stronger justification for this fundamental assumption or more clearly limit the scope of their claims.

  • The theoretical analysis does not sufficiently support the proposed practical method.

Section 4.2 presents a theoretical result showing that, under a simplified discrete-attention model, a two-token query can distinguish models with different response distributions. However, Theorem 4.4 relies on selecting a discriminative query based on the differences between the two models, whereas the actual method uses a predefined natural-language prompt without any query selection or optimization. Establishing the existence of a discriminative query does not imply that an arbitrary fixed prompt possesses the same distinguishing capability. Moreover, the theoretical model excludes several essential components of modern LLM architectures, further limiting its applicability to realistic settings.

  • The experimental evaluation does not sufficiently establish generalizability and comparison fairness.

Although the reported results are promising, the main experiments primarily consider model substitutions involving different model sizes or families, which may exhibit relatively distinguishable response distributions. More challenging scenarios, such as fine-tuned derivatives, closely related checkpoints, and quantized variants, are not systematically evaluated against the baselines in the main experiments. These cases are particularly important for practical model substitution auditing.

Questions for authors
  1. What theoretical or empirical evidence supports the assumption that the output distribution of a single fixed random-character prompt is sufficiently discriminative for general black-box model equality testing?

  2. How does Theorem 4.4, which establishes the existence of a selected discriminative query under a simplified attention model, justify the effectiveness of the fixed natural-language prompt used in practice?

  3. Could the authors provide additional comparisons with existing methods, including RUT, on more challenging model substitutions such as fine-tuned or quantized variants?

2 comments
AuthorsOpenReview ↗

Response to reviewer TPqm (1/2)

Thank you for your detailed review and for recognizing the importance of the problem and the simplicity and practical potential of our approach. We appreciate your suggestions and have substantially revised the paper, including a new theoretical result, additional experiments on closely related models, and clearer statements of the testing objective. We address your three main concerns below.

W1 / Q1. What can a single fixed prompt establish?

Thank you for raising this important conceptual point. We have clarified the exact scope of the hypothesis test in Definition 3.1 (Section 3) and Section 4.2.2, where the null hypothesis tests equality of the response distributions induced by the audit protocol, with a positive total-variation gap δq>0\delta_q > 0 providing the exact condition for finite-sample consistency. Crucially, this statistical boundary is shared by all black-box model equality tests (including MET, RUT, and B3IT): under any black-box query distribution, failing to reject the null hypothesis cannot certify identical weights across all unseen tasks. In practical, non-adversarial substitution scenario, such as model downsizing or quantization, parameter changes broadly perturb next-token probabilities, making exact distributional collisions across a 27-category output space rare in practice.

Within this scope, our contribution is a simple and inexpensive approach to substitution auditing: it uses ordinary text responses, a single-character output, and no separate query-discovery stage. Figures 3-4 demonstrate favorable observed cost-power trade-offs across within-family and cross-family substitutions. The prompt and alphabet studies in Section 5.3, Figures 5-7, together with the response-length study in Appendix E.1, Figure 8, provide empirical support for this cost-motivated design.

We have also added checks on what drives detection. Appendix E.2, Figure 9 shows that restricting the analysis to valid-letter pairs still reaches full observed detection on all eight evaluated pairs at sufficient budgets. Appendix E.3, Figure 10 shows that alternative permutation-calibrated statistics also work well on the same responses. These results strengthen the empirical evidence that the inexpensive response channel retains useful distinguishing information.

W2 / Q2. How does the theory support the practical fixed-prompt method?

We appreciate this question. The original Theorem 4.4 establishes the existence of a distinguishing query in the discrete-attention model; it does not provide a practical query-search procedure or establish that every fixed prompt distinguishes every model pair.

To address the practical method directly, we have added Theorem 4.5 in Section 4.2.2, with its full proof in Appendix B. With the prompt, parser, and endpoint/decoding configurations fixed before sampling, and under the stated independent-sampling assumptions, it establishes:

  • Finite-sample Type-I error control for the implemented paired TV test with Monte Carlo permutations and the tie-inclusive plus-one p-value.
  • A finite-sample power bound when the fixed prompt induces a positive total-variation gap between the parsed response distributions.

This result is stated directly in terms of the response distributions and does not require the discrete-attention architecture. It therefore provides a theoretical guarantee for the practical testing procedure while making its necessary separation condition explicit. We have also separated the presentation into “Existence of a Distinguishing Query” (Section 4.2.1) and “Testing with a Fixed Query” (Section 4.2.2) to clarify their respective roles.

AuthorsOpenReview ↗

Response to reviewer TPqm (2/2)

W3 / Q3. More challenging substitutions and comparisons with RUT

Following your suggestion, we have added controlled comparisons involving NF4 quantization, INT8 quantization, and a rank-8 LoRA fine-tune of Qwen2-1.5B-Instruct, including comparisons with MET and RUT. These results appear in Appendix E.4, Figure 11 and Table 3.

Figure 11 reports cost–power curves under a common token tariff, considering both cold-start auditing and amortization of RUT’s reference-bank cost over 100 audits. The results show particularly favorable observed cost–power behavior for our method on NF4 and the fine-tuned variant.

While near-lossless INT8 quantization is challenging for all evaluated black-box tests, with MET and RUT remaining below 25% power and B3IT finding no validated borders, our method at T=0.5T=0.5 still achieves the highest detection power among all baselines, in addition to substantially outperforming both MET and RUT on NF4 and SFT500.

The paper additionally evaluates the Llama 3.1 70B / Hermes 3 70B fine-tuned pair in Section 5.3 and Appendix E.3, Figure 10, providing complementary evidence on hosted models.

We sincerely hope you will take another careful look at the revised PDF. In light of these substantive PDF changes, we would be very grateful if you could reconsider your assessment and raise your score. Thank you again for your time and for helping us strengthen the paper.

Reviewer_vP6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a low-cost, single-character fingerprinting method for model equality testing by comparing distributions of repeatedly generated characters using total variation distance and a paired permutation test. Experiments across multiple model families and configurations demonstrate its effectiveness and cost advantages.

Critical strengths
  1. The paper proposes a simple, low-cost single-character fingerprinting method, which is practically meaningful.
  2. The experiments demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The experimental setup is insufficiently specified, lacking key details such as API providers.
  2. The paper does not analyze the impact of INVALID responses on detection results, leaving unclear whether detection performance is primarily driven by differences in output-format compliance.
  3. The experimental coverage is limited, as the method’s ability to distinguish practical substitutions, such as quantized versions of the same model, is not evaluated.
  4. The paper does not analyze how sampling hyperparameters, such as temperature and top-p, affect detection performance and cost.
Questions for authors

Would other statistical tests achieve similar results on the same single-character responses?

1 comment

Thank you so much for your thoughtful review and for recognizing the practical value, simplicity, and empirical effectiveness of our approach! Your suggestions helped us identify several useful additions. We have substantially revised the PDF and added experiments and analyses addressing the points below.

W1. API providers and experimental settings

Thank you for requesting these details. Appendix D and Table 2 now identify the serving providers and recorded precision labels for our main endpoints. OpenRouter is the API gateway; the listed downstream providers perform inference.

Section 5.1 describes the evaluation protocol: our primary setting uses temperature 1, top-p 1, a one-token output cap, 25 repeated audits, a nominal significance level of 0.05, and 9,999 paired permutations.

W2. Do INVALID responses drive the results?

Thank you for raising this important question. We now report per-model INVALID rates in Appendix D, Table 2, and directly examine their effect in Appendix E.2, Figure 9.

Figure 9 compares three variants on the same acquired responses: the full 27-category test, a test restricted to pairs where both responses are valid letters, and a validity-only test. INVALID responses are rare or absent for most endpoints. Across most evaluated pairs, the overall cost–power pattern is largely retained after filtering. Some Llama 3.1 70B comparisons do benefit from validity differences at small budgets.

Crucially, the both-valid-letter variant achieves 25/25 detections on all eight pairs at both n = 100 and n = 300. These curves retain the original acquisition costs, including the cost of discarded responses. Thus, valid-letter distributions alone provide strong distinguishing information, and the method’s effectiveness extends beyond differences in format compliance. Together with Figures 3-4, these results support the practical value of this inexpensive auditing approach.

W3. Detecting quantized variants of the same model

Following your suggestion, we have added controlled comparisons of Qwen2-1.5B-Instruct in BF16 against its NF4 and INT8 variants, alongside MET and RUT. These appear in Appendix E.4, Figure 11 and Table 3, together with a LoRA fine-tuning comparison.

On NF4, our method reaches full 25/25 detection (even at the default T=1T=1), substantially outperforming both MET and RUT. While near-lossless INT8 quantization is challenging for all evaluated black-box tests, with MET and RUT remaining below 25% power and B3IT finding no validated borders, our method at T=0.5T=0.5 still achieves the highest detection power among all methods. Figure 11 reports these cost-power trade-offs under a common token tariff, including both cold-start costs and amortized RUT reference-bank costs.

These additions provide evidence that the one-character approach can detect changes within the same underlying model, extending the evaluation beyond model-size and family substitutions.

W4. Sensitivity to sampling settings

We have added a controlled comparison of temperature 1 versus 0.5, with top-p fixed at 1, in Appendix E.4, Figure 11.

The results show that temperature can materially affect detection efficiency. For example, Table 3 reports that, on SFT500 at n = 128, lowering the temperature increases detections from 5/25 to 20/25, while both same-model controls remain at 0/25 under each configuration.

This experiment isolates the effect of temperature and shows that adjusting the decoder can improve sensitivity while preserving the same short prompt and one-character response format.

Q1. Would other statistical tests work on the same responses?

Yes. We have added precisely this comparison in Appendix E.3, Figure 10, evaluating TV, Pearson, and G statistics on identical one-token responses from two model pairs, including Llama 3.1 70B versus its Hermes 3 70B derivative.

All three statistics use 9,999 paired permutations and the same acquired responses and costs. They exhibit similar overall cost–power trends and all reach full observed detection, with Pearson and G providing some intermediate-budget gains on the fine-tuned pair.

These results strengthen the main empirical message: the inexpensive categorical response channel remains useful across multiple test statistics, with TV providing a simple and effective default.

We have put substantial work into strengthening both the evidence and the presentation, and we would be very grateful if you could take another look at the revised PDF. We sincerely hope these additions address your concerns and that you will consider raising your score. Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and a request for re-evaluation

Dear Area Chair and Reviewers,

Sorry for the late update, and thank you very much for your time and thoughtful feedback. Completing the additional experiments and checking the results took longer than we anticipated. We apologize for leaving less time for discussion. For ease of review, all changes in the revised PDF are highlighted in blue.

We have revised the manuscript with the following additions:

  • New theory: Theorem 4.5 in Section 4.2.2 directly analyzes testing with a fixed query, including finite-sample validity and a conditional power bound. The full proof is in Appendix B.
  • Additional experimental evidence: Appendix E now includes an analysis separating format compliance from valid-letter information (Figure 9), a comparison of TV, Pearson, and G on the same responses (Figure 10), and controlled quantization and fine-tuning comparisons with MET and RUT, together with same-model controls (Figure 11 and Table 3).
  • Clearer documentation and presentation: We expanded the API and implementation details, clarified the distinction between query existence and fixed-query testing, and corrected minor grammatical issues and typos.

We were encouraged by vP6s's recognition of the method's practical value and effectiveness, TPqm's recognition of the importance of the problem and the simplicity of the approach, and WF89's positive assessment of the problem and empirical cost advantages. We have taken the concerns accompanying these assessments seriously and worked to address them with additional evidence and clearer explanations.

We would be sincerely grateful for a careful re-evaluation of the revised paper in light of these additions. We hope the new theory, experiments, and clarifications help address the remaining concerns and support a more favorable assessment. We would be happy to clarify anything further.

Thank you again for your time, patience, and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.