- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Change Type Inference (CTI) to classify previously detected LLM API updates into one of three categories, namely system prompt changes, fine-tuning, and abliteration. CTI uses Border Inputs (BIs) collected from anchor models and trains a classifier on output-probability features from labeled landmark models. At inference time, these features are estimated using tokens sampled from the updated endpoint. The theoretical analysis examines divergence along model changes, distinguishes signature direction from magnitude, and characterizes local sensitivity at low temperature. The experiments evaluate classification performance on open-weight LLMs using held-out instances of the considered change types.
Critical strengths
[S1] Token-Only Change-Type Inference
Although constructing the fine-tuning and abliteration landmarks requires white-box access to the original model’s weights, the paper provides empirical evidence that the considered change types can be distinguished using sampled-token features under simulated token-only access to the updated endpoint. Results on held-out fine-tuning themes and abliteration techniques further support generalization to unseen instances of these change types.
Critical weaknesses
[W1] Limited Practical Applicability
CTI has limited applicability to black-box API auditing because landmark construction requires access beyond that assumed in the stated black-box setting. Section 2 assumes black-box sampling access to the source model and the landmarks (lines 80–81), but also requires correctly labeled landmarks derived from that model and covering every considered change type (lines 81–91). Constructing the fine-tuning and abliteration landmarks requires white-box access to the weights of the original model (Appendix I.1, Tables 5–6). The paper does not establish how an external auditor could obtain these landmarks for a proprietary source model.
Furthermore, the three-class formulation provides no mechanism to reject changes outside the considered categories (Section 4.1, lines 287–290). The three categories do not cover other changes, such as quantization, which is listed among observed endpoint modifications in Section 1 (lines 34–35), or decoding-time watermark insertion. Nevertheless, CTI would assign such updates one of the three labels, even though their underlying causes fall outside these categories. Although change detection is assumed to occur upstream (Section 2, lines 74–83), false alarms would similarly force CTI to assign a change-type label to an unchanged model. The evaluation explicitly considers "unknown updates of known types" (lines 290–291) and does not assess how CTI handles unchanged models or updates outside the three considered types.
[W2] Insufficient Justification of the Proposed Method
The connection between the theoretical analysis and the proposed classifier is unclear. The introduction motivates CTI through BI-based distances and trilateration (lines 49–53), while Section 3 distinguishes the direction of a change signature from its magnitude, interpreting them as indicating which change occurred and whether it is visible, respectively (lines 195–198). However, the paper does not explain how this distinction motivates the use of logistic regression with concatenated token-probability features (lines 297–301). Appendix K.2 describes logistic regression as “a stated choice”, rather than one selected on the basis of the comparative results (lines 1499–1500), and Section I.3 lists its configuration without providing a rationale. The -Span classifier uses the unexplained fraction of a displacement, making its decision invariant to displacement magnitude (lines 1597–1599), and achieves nearly identical mean macro recall, 0.83 vs 0.84 (Table 9, which is in fact a figure), leaving it unclear why logistic regression is preferred over this alternative that more directly reflects the theoretical emphasis on signature direction.
The necessity of BIs for change-type classification is also insufficiently supported. Section 3 claims that BIs are “the only prompts on which changes can be told apart” (lines 263–265). However, Theorem 4 concerns local sensitivity in the low-temperature limit. It does not establish that BIs outperform ordinary prompts for distinguishing change types under the experimental conditions. The ablations do not compare BIs with ordinary prompts under the same landmark set, classifier, and query budget. Consequently, the results do not isolate the benefit of BIs from that of supervised classification using outputs from labeled landmark models.
Questions for authors
I have several questions, and I list them below in descending order of importance to my assessment.
[Q1] Access Requirements
As discussed in [W1], constructing the required landmarks needs access beyond the stated black-box setting. Is CTI intended specifically for endpoints whose original model weights are available? If not, how could an external auditor obtain these landmarks for a proprietary source model? Relatedly, how does performance change when the landmark probability features used to train the classifier are estimated from sampled tokens rather than obtained directly from log-probabilities?
[Q2] Classifier and Anchor Design
As discussed in [W2], the motivation for logistic regression over the similarly performing -Span classifier is unclear. What motivated this choice? Do per-LLM comparisons reveal advantages of logistic regression over -Span that are obscured by the reported averages? In addition, using alone as a single anchor yields nearly the same mean macro recall as fusing 12 anchors (0.83, averaged over the eight LLMs in the column of Fig. 5, vs. 0.84 in Table 9). What does multi-anchor fusion contribute beyond ?
[Q3] Unchanged Models and Other Change Types
As discussed in [W1], CTI does not provide a rejection mechanism for unchanged models or updates outside the three considered types. What does CTI output for these inputs, and can classifier confidence or another available score distinguish them from updates belonging to the considered types?
[Q4] Generalization Across Modification Recipes
The held-out fine-tuning themes share the same LoRA configuration (Table 5), while the held-out abliteration techniques come from the same library (Table 6). How does CTI perform on fine-tuning with a different recipe, such as a different LoRA rank or full fine-tuning, and on abliteration from an independent implementation?
[Q5] Contribution and Construction of BIs
As discussed in [W2], the experiments do not isolate the contribution of BIs. Could you compare BIs with ordinary prompts while keeping the landmark set, classifier, and query budget fixed, including at a higher temperature such as ? Furthermore, the reproducibility statement refers to Section I for the BI search (lines 499–500), but Section I.2 only describes subsampling from an existing BI pool. How is this pool constructed, what criterion is used to identify BIs, and what are the typical logit gaps between their top two tokens relative to ?
[Q6] Relation to Prior Work
Finlayson et al. [1] discussed inferring update types from changes in model outputs and their underlying subspaces, using log-probabilities and logit bias to recover the required output distributions without access to model weights. Oh et al. [2] trained metamodels to infer attributes of black-box models from query outputs using models with known attributes. How does CTI relate to these approaches? Which contributions go beyond these existing formulations, and which specifically address sampled-token access to the updated endpoint without log-probabilities or logit bias?
References:
[1] Logits of API-Protected LLMs Leak Proprietary Information (COLM 2024; Finlayson et al.)
[2] Towards Reverse-Engineering Black-Box Neural Networks (ICLR 2018; Oh et al.)
2 comments
Thank you for the response. As the discussion period is coming to an end and I have not yet received a further response, I am posting this follow-up. [Q1] and [Q2] were the highest-priority questions in my review, and both concerns remain.
[Q1] Access Requirements
The response relies partly on fine-tuning APIs, whose availability can be restricted or withdrawn. For example, OpenAI has already closed self-serve fine-tuning to new users and will stop accepting new jobs from existing users on January 6, 2027 [3]. Even where fine-tuning remains available, it is unclear how an independent auditor would obtain the full set of labeled landmarks, particularly for abliteration, from the same source model version as the endpoint under audit.
[Q2] Classifier and Anchor Design
My original question was about the rationale for using logistic regression in the submitted method. Making -Span the main classifier would require re-evaluating the main results and the query-budget and anchor ablations, as well as revisiting the claims that rely on them. In my view, a revision of this scope would be better assessed as a re-submission.
Furthermore, several other questions remain unresolved, with the relevant experimental results still pending. I am provisionally keeping my rating at 2 and may revise it based on the final results and further responses, even after the discussion period. I encourage you to share the results when available, as they also address questions raised by other reviewers.
[3] OpenAI. Deprecations (https://developers.openai.com/api/docs/deprecations).
Sampled versus logprob landmark features (1rst question)
Please refer to response to reviewer UdoK (4rth question) for that same question, explaining that one can create landmarks without access to weights with the use of fine-tuning APIs.
Logistic regression versus Δ-Span (2nd question)
We chose logistic regression as a standard learner, simple to explain. However, the reviewer's argument convinces us: we will make Δ-Span the main classifier, report logistic regression next to it, and add the per-LLM table. In particular, the theory identifies two structural features that a classifier could exploit: the strength of the displacement and its direction. The experiments show that the direction carries almost all of the information and that the strength adds almost nothing. Indeed, Δ-Span only sees the direction of the displacement and is invariant to its size. It reaches a macro recall of 0.83, against 0.84 for logistic regression, which has access to everything (Table 9). We now see this as a contribution of the paper in its own right. It was undersold in the submission, and we will emphasize it in the revision.
No reject class for other changes (3rd question)
Indeed CTI is evaluated on a closed-set: a change outside the trained types receives a trained label, and we will state this limitation. We are currently running the experiments for unseen changes. We will upload the results before the end of the discussion phase.
We note that CTI is the second stage of a two-stage audit: it never declares a change, it labels a change the upstream detector declared (eg, a proper change-dectection mechanism).
Single LoRA fine-tuning recipe (4rth question)
This is an interesting question which was also asked by reviewer 9nty, please refer to the beginnig of the response.
We can add that unseen abliteration techniques are exactly what our protocol tests. The library we use, OBLITERATUS [1], aggregates techniques developed independently: four of our eight landmarks implement separately published methods (FailSpy's difference-of-means, Gabliteration, Heretic, refusal direction optimisation). Splits hold out two of the five non-anchor techniques (Section 4.1, Table 6), and three of those five are independently published. A held-out technique thus plays the role of a future technique the auditor did not anticipate.
References: [1] OBLITERATUS Contributors (2026). OBLITERATUS: An Open Platform for Analysis-Informed Refusal Removal in Large Language Models. GitHub repository. https://github.com/elder-plinius/OBLITERATUS
BIs versus ordinary or random prompts (5th question)
We agree that the paper has no control that keeps landmarks, features, classifier and budget fixed and replaces the BIs. We are curently running this in the re-run of our pipeline and will publish the results before the end of the discussion phase.
Relation to prior work (6th question)
We thank the reviewer for their very relevant litterature pointers. We were not aware of both works that indeed witness a long running (2018) interest in black box change type inference. Compared to both works, we operate in "hard label" settings, a stringent difference considering that less than 25% of the llms offer logprob access. Compared to [1], we avoid the construction of an intermediary "llm image" which is relatively costly requests-wise. We will include both in the related works of the revision.