- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SCORE, an output-only method for testing whether a target LLM API and a trusted reference endpoint induce the same prompt–response distribution. SCORE constructs paired prompts with and without a control prefix, measures the resulting change in semantic recovery of held-out lexical units, and converts target scores into reference-calibrated randomized ranks. A Cramér–von Mises uniformity test then detects distributional differences. The paper evaluates quantization, supervised fine-tuning, model replacement, and commercial API endpoints.
Critical strengths
- Practical access assumptions. SCORE requires only generated text from both endpoints, avoiding reference-side logits or weights. This meaningfully extends the applicability of rank-based auditing to settings where the reference is also a commercial API.
- Careful statistical calibration and explicit limitations. The randomized-rank construction handles ties and finite reference samples, while the Monte Carlo procedure provides type-I error control under the stated independence and stationarity assumptions. Appendix G also explicitly distinguishes failure to reject from evidence of model equality and discusses alternatives that the score or rank transformation cannot detect.
Critical weaknesses
- The methodological novelty is insufficiently established relative to closely related work. As the paper acknowledges (Section 2, Eq. 14, Appendix G), the score-to-rank-to-uniformity-testing framework comes from RUT, so the novelty rests mainly on the output-only semantic contrast score. However, this scoring design closely connects to ReCaLL (Xie et al., EMNLP 2024), which scores membership by the ratio of a text's log-likelihood conditioned on a non-member prefix to its unconditional log-likelihood, and PETAL (He et al., USENIX Security 2025), which uses semantic similarity between generated and ground-truth tokens under label-only access. These methods address membership inference rather than model equality testing and are not mathematically identical to SCORE. Nevertheless, the paper should explain what nontrivial methodological advance its particular combination provides, and neither ReCaLL nor PETAL is cited or discussed in the submission. Appendix F supports the usefulness of the components, but does not by itself establish their novelty.
- The empirical evidence is difficult to validate because of incomplete budget reporting and an internal inconsistency. Equation (17) shows that SCORE requires multiple generations for every position and prompt condition, but Appendix C does not establish that the baselines use comparable target and reference budgets. Section 3 states that reference requests are reported separately, but these counts do not appear in the experiments. Appendix F.0.1 matches budgets across SCORE variants, but no matched-budget comparison against MMD, PTS, RUT, or KS is given. The budget of N = 10m completions in the commercial-provider audit (Section 5.6) is also insufficiently connected to SCORE's position-wise paired-query protocol. In addition, Table 3(b) reports a SCORE power AUC of 0.130 for Llama 8-bit on MATH, whereas the corresponding power curve in Appendix D.4 integrates to approximately 0.83 (see Question 3), which makes it unclear which configuration the reported results reflect.
Questions for authors
- How does SCORE's semantic contrast score differ technically from combining PETAL-style semantic recovery with ReCaLL-style prefix normalization? What additional insight explains its suitability for model equality testing?
- Please report the actual target and reference request counts for each method, clarify the budget accounting in Section 5.6, and provide a representative comparison against the baselines under matched budgets.
- Table 3(b) reports an AUC of 0.130 for Llama 8-bit on MATH, whereas integrating the corresponding SCORE curve in Appendix D.4 (page 19) gives approximately 0.830. The baseline curves in the same panel also disagree with Table 3(b) (e.g., PTS reaches roughly 0.36 versus 0.062 in the table), while the other three Llama panels match their tabulated AUCs. The panel would also imply higher power for 8-bit than for 4-bit quantization on MATH (0.736), contrary to the statement in Section 5.5 that power decreases in the 8-bit setting. Could the authors clarify whether this discrepancy arises from a table entry, a figure label, or different experimental configurations?
I'll consider raising score if my questions are addressed.