acceptodds
Withdrawn from ICLR 2027

What Can AI Hide? A Framework and Toolkit for Evaluating Hidden Capabilities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An AI system could hide an ability while being tested and continue to use it when answering ordinary requests. Yet those responses may contain information it withholds during testing. We introduce a framework and toolkit for examining what an evaluator can recover from those responses. To measure how well the system handles ordinary requests, we use a score defined for the task, such as the proportion of answers it gets right. We then ask whether the system can keep the tested answers hidden while meeting a required score on those ordinary requests. Our framework describes when both are possible and when staying hidden requires accepting a lower score. To apply the framework, an evaluator specifies what information it can inspect and which ways of hiding the evaluation covers. We provide tools for choosing requests, scoring responses and determining what conclusions the results support. For repeated testing, we give an upper limit on the chance of rejecting a hiding explanation by mistake, provided the assumptions about sampling and responses hold. Our experiments show why recovering answers and preserving ordinary performance must be checked together. In one model study, an evaluation appeared successful mainly because it recovered answers that the instruction to hide had barely changed. Together, the framework and toolkit provide a way to evaluate what an AI system can keep hidden while still answering ordinary requests to a required standard.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
capability evaluation, AI safety, sandbagging, black-box auditing, utility–information tradeoffs, evaluation robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Eod3OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper asks when an AI system can keep particular answers hidden from an evaluator while still doing ordinary work to a required standard. It defines a hiding class (response policies whose observation distribution is the same across a declared group of tasks) and a hiding value (the best score such a policy can guarantee), so hiding and adequate performance coexist exactly when the required score is at most that value. Worked examples show that hiding can be possible under an expected requirement but not a hard one, and Proposition 1 (checked in Lean) bounds how many interval answers endpoint subtraction still recovers when some inventory entries are wrong. The toolkit finds the cheapest audit menu with a certificate, turns it into fixed-horizon repeated tests under explicit sampling assumptions, and includes software for saved logs, ordinary-request runs and a prospective audit.

Beyond authored examples and control programs, the model studies are mostly negative or unresolved on the hiding question: a registered inventory prediction failed, a prompted concealment pilot (Result 1) recovered answers mainly through a route the instruction barely changed, the prospective program study (TRACE-1) has a broader bound that depends on collection facts no record establishes, and two trained-concealer studies (TC-2 and TC-3) ended not established, with one of eight runs meeting every requirement. A replay of public sandbagging-game logs found that the upstream scorer credited 336 responses containing a wrong value. The empirical part is a set of smaller studies. Authored control programs show how copied prompt wording or a private coin can make an audit look successful or unsuccessful. In an inventory study, one hosted model answered all 32 direct questions wrong, report extraction recovered 24, and a frozen off-by-one correction recovered all 32, so the registered prediction that extraction would beat the direct readers failed. In a prompted concealment pilot (Result 1), the mean recovery score stayed high (23/24 vs. 1 without the instruction) mainly through a configuration that had failed qualification and whose answers the instruction barely changed. A prospective program study recorded 127/128 correct answers, but its broader error bound depends on collection facts no record establishes. Two registered studies that trained concealers on a 27B open-weight model (TC-2 and TC-3) both ended not established, with one of eight runs meeting every requirement. Finally, a replay of 69 public logs (14,969 records) from a sandbagging auditing game found that the upstream scorer credited 336 responses containing a wrong value.

Critical strengths
  1. I think the core framing is useful for sandbagging and hidden-capability evaluations. The key point, which the paper makes well, is that ordinary work doesn't carry more information than a direct question by default; what matters is whether the system is required to keep doing that work accurately while it gives misleading answers to the test. Turning this into a hiding class, a hiding value and a gap makes it precise, and the examples give evaluators concrete design lessons: an expected requirement can allow hiding where a hard one doesn't, the request distribution matters (if every request ends on the corrupted close, one bad balance can make every requested difference wrong), and a recognisable evaluation request with no performance requirement places no constraint on hiding at all. These are the parts I'd expect others to reuse, and they're the main reason I see value in the paper.

  2. Key predictions are registered and failed ones are reported as failed (the inventory extraction prediction, both trained-concealer verdicts), withdrawn claims are kept next to their corrections, Proposition 1 is checked in Lean, the menu certificates are re-verified with a standard-library verifier (10 of 10), and Table 2 cleanly separates observations, rejections of a stated null, further interpretations and claims about mechanism. The replay also produces a concrete finding others can use: the upstream scorer used in the public sandbagging-game logs credited 336 wrong-value responses, and recorded BBH accuracy exceeds strict accuracy by 7.2 to 16.5 points in every log.

Critical weaknesses
  1. I don't think the paper shows that the framework or toolkit works on a model that is actually hiding something, which is the question in the title. None of the model studies shows the approach catching a model that is trying to hide across the whole menu. The inventory study's registered prediction failed. The authors themselves describe Result 1 as a failed concealment induction rather than evidence that the audit can defeat a working concealer. The prospective program study's bound and capability reading are unresolved because no record establishes the required collection facts (one account both held the sealed answer material and ran the collection), and the account holder answered "unknown" to the conduct questions afterwards. TC-2 and TC-3 both ended NOT ESTABLISHED, with one of eight runs meeting every requirement, and the paper's own claim map lists a model that conceals against the full permitted reader as not observed. The closest case is a single-seed development result in Appendix D: an adapter trained to replace arithmetic answers on one narrow interface, whose targets the full menu reader still recovered (32/32). The repeated-test round counts (31 or 61 rounds for dispatch, 812 for ordering with the KL test) come from authored examples.

The model studies also never realize the regime where the theory has teeth. In TC-2 the concealers were trained to give substitutes on evaluation wording and the correct report on ordinary requests, so recovering totals from ordinary reports mostly checks that training didn't break those reports (H3 held on all three runs; it was hiding, H1, that held on only one). A model that tries to make its ordinary work uninformative while staying within the usefulness requirement is exactly the case where the pivot-report argument says recovery is forced. The prompted pilot asked for roughly this, but on the route that supplied most of the recovery, the instruction barely changed the answers. So right now I think the empirical contribution is a set of useful cautionary lessons (copied wording, consistent offsets, scorer bugs, unqualified configurations), not evidence that this approach catches a concealer that hides across the menu.

  1. The theoretical contribution is modest, and the presentation makes the paper very hard to evaluate. By the authors' own account, the core hiding and recovery statements are standard minimax and testing mathematics and not claimed new (Table 3), the menu design is a procedure rather than a new foundation (Table 1), and the concentration bounds and dynamic programming are standard, with the contribution being the reduction and the collection protocol (Appendix H). Proposition 1 is an elementary counting bound. That would be fine for a paper whose value is a clear framing plus a convincing demonstration, but here the framing is buried. The main text and appendix read like a claims ledger: a long list of study codenames (e.g., TC-2, TC-2X, TC-3, TRACE-1, RW-1, UWA-1, HPE-1, S1, P-IMP), three packages with version histories, test pass counts by platform in the reproducibility statement, withdrawn claims and internal corrections interleaved with results, and process details such as a random tape whose hash a custodian holds, studies added after the evidence freeze at the authors' request, and a trained-concealer study (S1) that arrived but wasn't integrated. Table 1 alone packs 19 claims, each with evidence and limits. Working out what is actually established requires going deep into the appendix, and I think most readers won't. I'd want the paper restructured around the framework, the design lessons and one or two studies, with the provenance material moved to the supplement.
Questions for authors
  1. Apart from the single-seed adapter in Appendix D, which was trained to replace arithmetic answers on one narrow interface, is there any model result where a model trying to hide across the menu is caught through its required ordinary work? For example, what did the S1 trained-concealer study find, or what happens when a concealer is trained to minimize recovery from its ordinary reports subject to the usefulness requirement? One end-to-end positive demonstration would change my assessment a lot.

  2. The framework's leverage depends on evaluation requests either not being recognisable or carrying a binding requirement: a recognisable request without one places no constraint on hiding, and one work-framed request defeated the coin control. Given that frontier models can distinguish evaluation from deployment transcripts above chance (Needham et al., 2025, which you cite), how would an evaluator get indistinguishable evaluation and ordinary requests in practice, and can the framework handle partial recognisability?

  3. Appendix H notes that scoring ordinary traffic often requires knowing the target answer. Outside constructed task families, who enforces the performance requirement, and on what kinds of tasks would you expect the framework to apply?

  4. The paper places its contribution in specifying the setting, the menu-design procedure, and the reduction from a usefulness certificate to a repeated test with its collection protocol, while describing the underlying minimax, concentration and dynamic-programming tools as standard. Which of these do you see as the main technical novelty, and how does it change evaluation practice?

  5. Could TRACE-1 be rerun with the sealed material held by a different account from the collector, and with logged access, so that the certified 1/20 bound applies to at least one model study?

Reviewer_KKS5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work discusses a framework and toolkit whose goal is to understand how well can a system continue completing its tasks while also keeping specific information private. They summarize that in a numerical value, the hiding value. The authors introduce the framework by 1) describing the task and the response, 2) describing adequate performance, 3) keeping the observations indistinguishable and 4) finding the highest score that hiding permits (which is the hiding value). They then proceed to apply this recipe in several settings by describing procedures an evaluator can use.

Critical strengths

There is seems to be a significant volume of experiments and investigations done, albeit at toy settings.

Critical weaknesses

The manuscript is incredibly dense, verbose and quite hard to follow, particularly due to being mostly AI written. This makes it very challenging to gauge the correctness, novelty and even how the whole method works. This is the primary reason I am recommending a rejection.

As for some more specific weaknesses

  • The presentation at section 2 is very obscure, with several choices that are not well motivated or explained. For example, how do the authors represent each task by θ\theta alone and how is this grounded in their practical settings later on? What do they mean that the system can “randomize” (line 122)? Randomize what? They are also talking about a τ−τhide\tau - \tau_{\text{hide}} but only τhide\tau_{\text{hide}} has been introduced in the text. Furthermore, they argue about an equality of their two equations for τhide\tau_{\text{hide}} claiming that it follows from standard minimax theory with no citations and only a reference for Appendix B.3 where again the same claim is made without any real proof and references. In the example that follows (line 150 and onwards) things are even more puzzling with new terms being introduced without any real explanation (“regret allowance”) and claims made without a real explanation / proof (why is the “expected hiding value” 5/6 and why is it equal to the “score”? What is the “score”? )
  • Same things apply to most other sections too. Section 3 for example argues about a proposition M in Appendix H (at line 237) but going to that appendix the proposition and its proof is not even there. There is only a reference to a markdown file in the supplementary that an agent wrote, and the proposition is in that file. The other sections are mostly just an exhaustive description of several things tried with no real structure or story to tie everything together.
Questions for authors

Based on the weakness I highlighted above, I would encourage the authors to perform a full rewrite on the manuscript, focusing more on the critical details and the big picture as opposed to providing literally everything done throughout their agent’s investigation.

Reviewer_wR1tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors consider the problem of sandbagging detection. They argue that it's important to measure sandbagging on capabilities the model is "required" to do well. The rest of the paper contains various experiments studying this.

Critical strengths

None

Critical weaknesses

The problem statement is not sufficiently clear for me to understand what the authors are trying to do. The introduction does not clearly and concisely state the problem; the problem statement needs to be inferred by the reader.

The general idea of "measure model performance on realistic responses" is not novel - see https://alignment.openai.com/prod-evals/. Given that, it's unclear to me what novel contribution this paper makes.

The authors introduce a complex formalism to characterise "how much sandbagging" can be present, on the basis of known information. However, it's unclear why this is useful.

In one set of experiments (Sections 4-5), the authors audit capabilities for a model that was not instructed or trained to conceal information. Furthermore, the certification bound depends on the experiment having been conducted in a specific precise way. However, the post-experiment audit checklist that is supposed to verify this shows all questions as being answered "unknown". Thus it's unclear what to conclude from this experiment.

In another set of experiments (Section 5), the authors study sandbagging detection on a model organism trained to conceal information. However, this model organism doesn't reliably conceal information per the authors' own criteria. So it's unclear what was learned from this experiment.

Overall the experiments presented are not convincing and the writing in the paper is dense and difficult to parse.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.