- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks how much statistical power is lost when an audit records only whether two responses differ rather than the responses themselves. In a binary two-hypothesis model (each raw response costs one acquisition and is flipped independently with known rate η; pairwise comparisons are noiseless XORs of already-acquired bits), the authors optimize three experiments under a common budget of N acquisitions: (i) fixed designs (allocation + pairing graph), (ii) causal adaptive designs whose actions use only released differences, and (iii) "retained-reference" designs that keep one already-acquired raw response.
Critical strengths
1.The problem is completely solved, not merely bounded. Theorem 2 gives an exact finite-N minimax risk and the exact optimal allocation (including the counter-intuitive "balanced is sometimes strictly worse" effect), and Theorem 3 gives an all-horizon optimal policy rather than an asymptotically optimal one. This level of closure is rare and valuable. 2.A genuinely implementable optimum. Two O(log N)-bit signed counters, one magnitude comparison per acquisition, needing neither η nor the horizon. The clean separation between what the acquisition interface may hold internally (the anchor) and what the auditor receives (differences only) makes it usable in a real logging pipeline. 3.The three-way strict separation is the real contribution, and the authors correctly frame it that way rather than as "information loss can hurt."
Critical weaknesses
-
The motivating application is asserted, never connected. The paper opens with LLM vulnerability evaluation (LLM4Vuln, semantic transformations), but nothing in the model derives from, is validated on, or is calibrated against such an evaluation. There is no measurement of how common paired-only logging is, no instance of a real harness discarding raw responses, and no empirical section.
-
State the novelty delta more sharply. Since the exponent is known, the contribution is: the exact finite-budget fixed-design optimum and β/2; the all-horizon optimality proof under the parity loss; and the three-way comparison. A reader must currently reconstruct this from scattered remarks. Add an explicit "what is new" paragraph/table vs. Nitinawarat et al. and Burnashev (their fixed-design exponent 2β/3 vs. your β/2; different terminal loss; different experiment triples).
Questions for authors
1.Is "at least one control" an assumption or forced by identifiability? With all-informative queries the two orbit laws coincide (risk 1/2), so I believe it is forced — if so, say so; it makes the premise far less arbitrary. 2.What if the retained reference is chosen after acquisition, or more than one raw bit is retained? 3.Under unknown/misspecified η, what is the risk of the η-free greedy policy? A robustness bound would turn a nice property into a strong one.
2 comments
I keep my score
Motivation, controls, reference retention, and noise uncertainty
Thank you for your assessment and questions. We have put the novelty comparison in Table 4 (p. 10), clarified controls in Section 2, and added exact budgets in Table 3.
Motivation and contribution. The vulnerability-evaluation example explains how the two query signatures might arise. You are right that the submission provides no deployment evidence for paired-only logging. The opening now starts from the recording question. The finite program construction in Appendix D realizes the specified signatures; Table 3 quantifies the resulting statistical cost.
The comparison states the contributions you identified: the exact fixed-design risk and allocation, the all-horizon policy for parity loss, and the common-cost reference comparison. We credit Nitinawarat et al. and Burnashev for the shared adaptive exponent. Table 4 compares their three-state examples, with fixed rate 2β/3 and location/message loss, to our β/2 and parity loss. Soloviev and Halpern's noisy Boolean-testing framework contains our latent XOR experiment; Appendix E.3 gives its prior, budget and reward mapping. Our results solve this instance and compare the resulting risks with reference retention.
Requiring a control on every trajectory is our common finite-budget protocol. Once a retained response reconstructs the raw record, a control is no longer needed for identification. Dropping this convention changes retained risk from eN−1 to eN, with the same exponent β. Section 2 now separates the identifiability argument from this benchmark convention. The necessity uses symmetry: Appendix F.2 gives an asymmetric-channel example in which informative-only differences can distinguish the hypotheses.
Disclosure time affects the acquisition policy, not the reconstruction identity. Revealing a response at the end restores the final test's data; earlier decisions still used differences. Section 8 and Appendix F.5 state this distinction. Appendix G also gives a stopping construction attaining the retained-reference leading cost when the reference is disclosed only after stopping. For a control, the acquired response equals its realized flip, so disclosing the response also reveals that flip. This orients the existing differences without another acquisition.
Proposition 6 turns this into a uniform guarantee. For an independently justified bound ηˉ<1/2, infπsup0<η≤ηˉR(π,N;η)=Rad∗(N;ηˉ), attained by the same policy. To prove monotonicity, add independent artificial flips Bv to a lower-rate experiment. Each requested difference becomes (Xu⊕Xv)⊕Bu⊕Bv. This uses only available XORs and independent randomness, and simulates the noisier adaptive history. Pointwise optimality then places the worst risk at ηˉ.
For ηˉ=0.2 and error 0.001, Table 3 gives minimum fixed, adaptive and retained budgets of 48, 34 and 22. Each budget attains the target and its predecessor fails; the adaptive implementation is unchanged. Evaluating the certificate uses no additional audit responses. Without a bound away from 1/2, worst-case finite-budget risk is 1/2.
Appendix F treats changes in the noise law. If every response probability, given the full raw history and action, changes by a factor in [e−κ,eκ], Proposition 7 bounds the unchanged policy's risk by e2Nκ times the new optimum. Exact optimality can change under heterogeneity, as the explicit example shows. The supplementary program reproduces Table 3 and checks both conditional errors against an independent fair-measure calculation.