- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper frames behavior discovery in large generative models as budgeted multiple hypothesis testing with false discovery rate control. It shows that wrapping heuristic LLM auditors with eBH lowers the false discovery rate at the cost of fewer verified findings, and proposes DISCO, a bandit-style search procedure in which the hypothesis selector only observes coarse e-process bins so that the final eBH step can use the realized number of tested hypotheses instead of an upper bound . Pre-hoc and post-hoc bounds are derived, and experiments on a synthetic bandit and five text and image tasks report more verified discoveries than eBH baselines.
Critical strengths
-
Framing model auditing as hypothesis testing with false discovery rate control is a useful perspective, since Figure 3 shows that existing heuristic auditors produce high false discovery proportions.
-
Appendix L documents the prompts, task seeds, and spatial-tolerance scoring rules for visual symmetry in detail.
Critical weaknesses
- The paper compares itself with online eBH in Section 4 and Appendix F, but the problem it actually studies is an offline audit with a fixed query budget , where all rejections are made once after the budget is spent. Within this offline setting the paper still requires the agent to spend the budget in rounds of queries and to run a formal test on every round, and this protocol is the only source of the random hypothesis count that DISCO is designed to handle. A practical auditor with the same budget could explore freely, fix candidate hypotheses, and test them on fresh queries with BH, so would be known before testing, while the main benefit of interleaving, reading earlier outputs to propose better hypotheses, is ruled out by Assumption 2. Since the paper neither describes a deployment that needs this round-by-round protocol nor compares against such an explore-then-test split with the same agent, the practical relevance of the setting is unclear.
- Figure 2 claims that all compared methods guarantee and that DISCO achieves the highest power. However, Section 6.1 and Appendix L.3 state that the DISCO guarantees hold with probability at least 96.9%, which corresponds to the conditional certificate of Theorem 2 and only bounds unconditional FDR by with up to 0.031 in Table 8, whereas BUW-eBH and the wrapped baselines control unconditional FDR at . In addition, Section 4.2 notes that DISCO uses boosted eBH while Section 3 uses base eBH, and Section 6 averages runs with and even though Remark 2 admits that does not use coarsening at all. Across the seven task settings in Tables 1 and 2, DISCO has the most verified discoveries among FDR-controlling methods in only three settings.
- Theorem 7 extends the post-hoc certificate to multiple bins, which is the setting of Section 6 because hypotheses are abandoned once . Its proof reuses the equality in Equation 38, which charges each null the exit probability at its realized count , but this equality requires to be independent of the null walk, whereas the rule, which Assumption 2 permits, stops weak walks after one round and keeps only walks already above for later rounds, so need not bound the exit probability and the proof of Theorem 7 does not hold. As a result, the post-hoc certificate behind the 96.9% guarantee in Section 6.1 is not established for the procedure actually run.
- Assumption 3 requires the p-values of each null hypothesis to follow a known distribution , and Algorithm 2 multiplies the per-round e-values of a hypothesis into one e-process. In Appendix L.9, however, the agent writes a new sub-hypothesis and pre-registers a new test type and null baseline in every round, and the system prompt asks for methodological diversity across sub-hypotheses. Section 6 itself states that such sub-hypotheses are only conceptually valid in falsification-based experiments, which behavior discovery is not. The e-values multiplied into one e-process can therefore come from different nulls, so the product is only valid for the intersection of these nulls, and a rejection shows that at least one operationalization departs from its own baseline rather than that the stated behavior holds.
Questions for authors
None
4 comments
Following Up
This note is just to check in before the end of the discussion period - let us know if you have any questions or concerns regarding our response and updated draft. We appreciate your thoughtful feedback!
Thank you for the response. Three points remain.
-
The response still does not explain what practical scenario requires testing in rounds rather than exploring freely first. Without a clear use case for round-by-round testing, the motivation for handling a random hypothesis count remains weak.
-
Why controlling FDR over the reported set is the right goal also remains unclear. Unlike standard multiple testing where the hypothesis family is fixed and homogeneous and the selected set is used as a whole, here the agent generates hypotheses of varying scope and potential duplicates on the fly, and the reported findings read as separate claims. Both FDR and the count of verified discoveries therefore depend on how the agent splits behaviors into hypotheses, so Figure 2 does not show that DISCO finds more distinct behaviors.
-
On Algorithm 1, my review suggested testing fixed candidates on fresh queries with BH rather than e-BH. Because the candidates are fixed before testing and use independent queries, BH applies directly and is less conservative than converting p-values into e-values, so Figure 2 does not rule out this simpler alternative.
Thank you for the followup
We greatly appreciate the reviewer’s thoughtful feedback and clarifying distinctions. We have uploaded a new draft with new Appendices M and N. Appendix M shows that “baselines+BH” does improve over “baselines+eBH” but DISCO still yields far more verified discoveries on average; we will incorporate this more fully into the final draft. Appendix N uses an LLM to cluster hypotheses, collapsing near-duplicates and parent-child relationships; we find all methods (DISCO and baselines) lose 5-12% of verified hypotheses and FDR results are extremely similar; all general trends remain the same.
Details below, including expanded motivation for iterative hypothesis testing and FDR as a metric:
- We argue that iterative hypothesis testing (as opposed to explore-then-test approaches) is an important class of methods for the following reasons, which we will clarify in the next draft. 1) Explore-then-test approaches have a fundamental challenge of query allocation. If we allocate too little to exploration, we risk outputting few discoveries if hypothesis search is hard; allocate too much, and we risk underpowering the testing stage if discovered hypotheses are borderline. Iterative methods like DISCO navigate this tradeoff smoothly by combining the exploration and testing stages, and naturally yield a random, data-dependent hypothesis count. 2) Iterative testing can use previous statistically-calibrated results to inform future hypotheses. We empirically show this is useful - see Tables 4 and 7 in the new draft, where DISCO yields more verified discoveries than baselines even in the case, where model outputs and exact e-values are hidden; 3) iterative approaches are useful for audits run on restricted timelines; they may be interrupted partway through and still yield useful results by running eBH on the hypotheses explored so far.
- We defend FDR as a useful metric on two grounds. 1) We agree with the reviewer’s criticism around accounting for hypothesis duplication, and have added Appendix N (Tables 20 and 21) in the paper, where we use an LLM-based clustering which groups near-duplicate/parent-child hypotheses. We find that average verified discoveries over all tasks drop by 5-12% across all methods, FDR changes are very small, and general trends stay the same: DISCO/BUW-eBH/best baseline+BH have 6.02/3.92/3.31 and 6.81/4.05/2.80 verified discoveries on average on text and image tasks. 2) Conceptually, we argue FDR is an intuitive metric for user trust as it gives a sense of reliability across the output set, even if the claims may overlap or vary in scope. FDR can diverge wildly from alternatives such as Type-1 error; if 30 hypotheses are tested at type-1 level , with 1 true one, then the output set will have FDR .
- Thank you for clarifying the comment about BH; we apologize for the confusion. In the new draft, we add Appendix M (Tables 16-19) where we adapt all “baselines+eBH” methods to “baselines+BH” methods. We find that this does increase verified discoveries, but that on average across all tasks, DISCO still has more than 2x as many verified discoveries as any of the baselines+BH methods. As BH is a strong and valid approach, we will incorporate these results more fully into the next draft, and we thank the reviewer for the suggestion.
Rebuttal response
We thank you for your feedback and are glad that you found our perspective useful; we agree that it is valuable to understand how heuristic methods can produce high FDRs currently.
We respond here to your feedback, and have uploaded a new draft with relevant edits: