- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper fits in the field of auditing remote ML or LLM models.
The authors propose an "access/data/compute" framework, where the interaction of all three means is observed in the success metrics of an audit. Three audit types are considered: demographic parity assessment, manipulation detection, and lie detection for LLMs.
This paper is empirical only (no theory).
The concluding claims made are about the need to consider all three means at all times when a novel audit task instanciation arrives.
Critical strengths
- The overall topic of understanding the conditions under which audits succeed or not is of interest. This paper instantiates such a question with a particular (yet imperfect, in my opinion) focus
Critical weaknesses
-
Artificial framework: Related work on audits places data (here d) as a primary citizen. The current paper adds access and compute as a clear value proposition. Yet I claim that, in the current paper at least, these two additions are artificial. First, the "compute" aspect is in my opinion irrelevant: auditors today are state institutions or tasked private companies with considerable means to perform their audits, so studying the "compute" aspect by solely considering that an auditor can have access to even hundreds of thousands of parameters only for computing is not realistic. The paper claims to be practical, so I think here it is not: this is far from what is occurring even today in practice. Second, the "access" mean is also artificial: I am obviously ok with l=0 and l=1 access. But when l>=2, this means that the model provider will open "part" of her model, which is not realistic. As mentioned by the authors, they keep their model as a trade secret, so this is an all-or-nothing openness that is witnessed in practice. Institutions might force full access to a model (by law, for instance), but not to "several layers". This means that I am fine with the study of the "data" dimension, but we are then back to what most of the related work already does; and because the paper claims practical impact, artificial core assumptions break the whole realism of the proposed framework.
-
There is no theoretical lead-in in this paper. So we are left with a pure numerical assessment of some chosen cases. The three case studies are interesting, and the bibliography is relevant; yet, the reader is then directly forced into an overly technical/practical choice of a totally arbitrary parameter set to instantiate them (Table 1). Then tiny paragraphs very briefly describe some observations. So what about the generalization of these experiments to other parameters, audited models, or tasks? I am left with the impression that, besides the actual numbers presented, I cannot say anything about the stability of these results to any change to any arbitrary metric/parameter/task/target model presented in the paper, and then no knowledge gains beyond a point in the landscape of potential audits.
-
See below for a questionable mapping of related work's use of hypothesis classes (generally on the auditee side) to a mere number of parameters in the auditor's model (auditor side then). In general, I find the related work part to be good, but the choices made to represent their leads in the current framework are far below and totally arbitrary. For instance, for the lie detection task, page 5: "We teacher-force each transcript through its target to extract output probabilities and hidden states over the reply; reply text at ℓ = 0 is encoded as a random projection of GPT-2 token counts (k = 128), the log probs at ℓ = 1 are represented by an 8-dimensional summary, and hidden layer activations at ℓ ≥ 2 are projected to 256 dimensions." This is so complex and arbitrary that I find it impossible to link it to the precise knowledge and conclusions from related work.
details:
-
Some claims such as (Section 6): "Second, access to hidden states does not seem to help in out-of-distribution performance, plausibly because the extra access lets probes specialize on the deception they were trained on" are misleading, as they are talking about what the auditor does with its arbitrary choice of "compute" mean, not about the hardness of the task itself. Overfitting is a problem local to the auditor, not an inherent impossibility to address the audit task. The distinction between both is unclear in the paper.
-
References are misformatted: no mention of the conference in most of them (AI statement claims search for biblio, so I guess this must be fixed next time)
Questions for authors
-
Do you have practical examples of companies giving to an auditor access to layers 2, 3, ..., k, but not the whole model?
-
How do you go from (page 3) from hypothesis classes being central to audits, to flattening all to a "compute" that only considers the size of an auditor's model? I do not see such a straight parallel; hypothesis classes are much more abstract and powerful for what they allow in reasoning about bounds and possibilities of audits in general.