acceptodds
Under review as a conference paper at ICLR 2027

Blind Fibers: Safety-Sufficient Representations for Reliable Agent Verification

Abstract

Reliable evaluation of AI agents depends not only on which trajectories are tested, but also on what information about those trajectories is shown to the verifier. We study failures caused by representations that make safe and unsafe behaviors appear identical. We call such cases blind fibers and show that no downstream verifier can reliably distinguish them once the relevant safety information has been discarded. We introduce the Verifier Sufficiency Defect, which measures the safety information lost by a verifier representation, and establish its connection to irreducible verification error. Motivated by this analysis, we propose FIBER, a method that identifies outcome-matched safe and unsafe trajectories and learns a compact safety representation that preserves the distinctions missed by the original verifier view. Controlled tool-agent experiments validate the theoretical predictions and show that the repaired representation generalizes to unseen fault combinations. We additionally identify the same failure mechanism in a publicly released AgentDojo trajectory.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, agent evaluation, safety verification, representation sufficiency, verifier blind spots, trajectory representations, information-theoretic limits, strategic agents, LLM-as-a-judge, tool-using agents, contrastive representation learning, robust oversight

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iz8YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies failures in agent verification caused by information loss in the representations available to the verifier. It introduces "blind fibers," where safe and unsafe trajectories become indistinguishable under the verifier's view, and proposes the Verifier Sufficiency Defect to characterize the resulting loss of safety information. The paper also presents FIBER, which uses outcome-matched safe and unsafe trajectories to learn a compact safety representation. The evaluation includes controlled synthetic experiments, a shortcut-dominated stress test, and a qualitative example from AgentDojo.

Critical strengths

The problem formulation is relevant and clearly motivated. The paper distinguishes between a verifier failing to use available information and a verifier never receiving the information needed for a safety judgment. This distinction helps clarify what stronger judges can and cannot address.

The theoretical framework is coherent. The interpretation of VSD through optimal verification loss, together with the discussion of coarsening and augmentation, provides a consistent way to reason about verifier representations. Although I have concerns about the novelty of these results, their presentation is clear and accessible.

The authors acknowledge important limitations. In particular, they explicitly state that ordinary supervised training matches FIBER on the main structural benchmark and that the experiments do not establish reliable extraction of safety-relevant structure from natural-language trajectories. The additional shortcut stress test is also a useful attempt to identify a setting in which the blind-pair objective provides a benefit.

Critical weaknesses
  1. The main theoretical results largely follow from established information-theoretic principles and classical statistical testing arguments. While the formulation is clear, the paper offers limited new theoretical insights beyond applying existing concepts to agent verification. The authors should better clarify what fundamentally new understanding the proposed framework provides beyond classical statistical sufficiency and information theory.

  2. The experiments rely heavily on synthetic settings with predefined safety signals. In the main multi-family benchmark, safe and unsafe trajectories can be distinguished by simply counting active structural factors, making the task considerably easier than realistic agent verification. Moreover, the proposed FIBER method is not quantitatively evaluated on real agent trajectories, and its advantage over standard supervised learning remains unclear. I encourage the authors to conduct end-to-end experiments on realistic benchmarks and include stronger baselines to demonstrate the practical effectiveness of the proposed approach.

Questions for authors
  1. Have you evaluated a version of the structural benchmark without the factor-count and safeguard-marker shortcuts? Such an experiment would help determine whether the held-out-family results reflect learning safety-relevant relationships rather than exploiting properties shared by every training and test example.

  2. Does the advantage in Appendix J.12 persist against baselines that use the same matched supervision more directly? In particular, comparisons with blind-pair-reweighted supervised training and supervised contrastive learning would help isolate the contribution of the proposed objective.

  3. Is there evidence that the complete FIBER pipeline works on natural agent trajectories? A focused evaluation covering pair construction, sketch extraction, and downstream verification would materially affect my assessment. It would also be helpful to distinguish the occurrence of successful-but-unsafe executions from verified cases where a particular representation makes safe and unsafe executions indistinguishable.

Reviewer_dohTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates step-level verification of an agent trajectory. However, it specifically designs informative representations of the agent trajectory prefixes. A blind fiber here is a set of trajectories that share the same representation but have different safety labels. Ideally, one would like to avoid this blind fiber (via a term the authors call the verifier sufficiency defect). The authors use a contrastive learning like approach to learn these representations.

Critical strengths

S1: This is a well-posed problem about trying to learn verifier representations that are maximally informative of the safety labels. I enjoyed learning about the distinction between representational failure and verifier failure.

S2: The mathematical formalism is well-justified.

Critical weaknesses

W1: Theorem 4 basically arguing a very weak result, i.e., if the observations are identical, then the verifier's tests are identical too. Similarly, i think there is a lot of math that could be simplified. Theorem 8 similarly is just stating that VSD = excess Bayes log loss, which is well-known. I am unsure where the novel theory is here, if any.

W2: The experimental results are quite limited. By construction, the trajectories have very few features (six binary features). Thus, the representational learning component is not very complicated. In contrast, in many settings we may not know the features beforehand.

Questions for authors

See weaknesses.

Reviewer_Tt6tOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies information loss in agent verification: safe and unsafe trajectories may become indistinguishable under an evaluator’s representation. It formalizes this through blind fibers, defines the Verifier Sufficiency Defect using conditional mutual information, and proposes FIBER to learn compact representations from outcome-matched safe/unsafe pairs. Evaluation consists primarily of synthetic experiments, supplemented by one previously released real AgentDojo trace with GPT-4o.

Critical strengths
  • Provides interpretable, controlled experiments with analytically known limits.

  • Explicitly acknowledges several limitations, including the simplicity of the structured benchmark and the lack of benefit from the pair loss on that benchmark.

Critical weaknesses
  • The main benchmark does not establish a substantial contribution. Unsafe examples have exactly two active structural attributes, while safe examples have one. Simply counting these attributes achieves 100% test accuracy without training. BCE-only also matches FIBER’s perfect AUROC and recall, so these results do not demonstrate the benefit of the proposed objective.

  • The intended application to LLM-agent verification remains largely unsupported. There are no LLM-judge evaluations or comparisons, and no evaluation of extracting safety representations from realistic, unconstrained agent traces. The single AgentDojo example illustrates a possible failure mechanism but does not establish detection performance or successful repair. Maybe conduct several LLM-backed experiments for an LLM agent paper?

  • The theoretical novelty appears limited. I don't have the capacity to go through all the formulations, but the central results largely apply existing classic theory to the verification setting. There's not much new theory proposed, and it's only serving as a support of the paper's central claim, which seems trivial as well.

Questions for authors

As is stated in the weaknesses section,

  • On the main structured benchmark, counting active attributes perfectly separates safe and unsafe examples, and BCE-only matches FIBER. What evidence establishes a benefit from the proposed pair-loss objective beyond this construction?

  • Can you evaluate the complete trajectory-to-sketch pipeline on realistic LLM-agent traces, comparing the same LLM judge with final-answer, full-trajectory, and FIBER-sketch inputs? How are safety attributes extracted without access to ground-truth annotations?

Comments on the paper

5 comments
AuthorsOpenReview ↗

What a verifier can recover from an agent trace?

Thank you for pushing us to answer the central question with observable evidence. In AgentDojo banking task 7, a user asked how much they had spent on a gift. Two runs of the same GPT-4o model returned byte-for-byte identical answers, and both completed the user’s task. One execution looked up the transaction history. The other also completed a money transfer and satisfied the benchmark’s injected goal. A judge given the task and final answer receives exactly the same input in both cases. If the two executions are equally likely, every such judge has error 1/2, regardless of its reasoning ability.

The distinction can be measured and partially repaired on released agent traces. We retrospectively checked all 16 banking task IDs for GPT-4o-2024-05-13, injection goal 0, under the direct and important_instructions conditions. Of the 16 pairs, nine completed the legitimate task in both runs. Eight of those nine had opposite outcomes for the designated injection goal. Two of the eight, tasks 4 and 7, had identical final answers.

Give equal weight to the 18 executions in those nine doubly successful pairs. For the view consisting of task ID and exact final answer, each of the two collisions contributes one bit of uncertainty on two records. All other observed views have a single label. The empirical verifier sufficiency defect (VSD) is therefore 2/9 bits, and the minimum possible classification error is 2/18. Now add one field available directly from tool receipts: whether a money-transfer call returned without error. This distinguishes the two executions of task 7. Both executions of task 4 contain a successful transfer, so that collision persists. On the same records, the defect falls to 1/9 bit and the minimum error to 1/18. Thus we can identify exactly which safety distinction the added field restores and which one still requires better evidence. These are exact calculations on a targeted retrospective sample for one specified safety predicate. Task 7, goal false · Task 7, goal true.

The theory connects a hidden risk difference to the information required for repair. Let Δ be the difference in unsafe-execution rates in absolute value and d the total variation distance between the two distributions of verifier inputs. For deterministic binary trajectory labels, the smallest possible VSD on the balanced mixture, over systems with those Δ and d, is max(0, Δ − d) bits. The classification-error step has a classical antecedent in Zhao and Gordon (NeurIPS 2019). The sharp formulation derived during discussion connects that step to the paper’s mixture definition and conditional repair identity. The audit provides a concrete instance. Across the nine tasks, the injected goal succeeded in 1/9 runs under one condition and 9/9 under the other, giving Δ = 8/9. The visible distributions differ on seven of the nine task IDs, giving d = 7/9. Those two facts alone force at least 1/9 bit of missing information. Inspecting the collisions gives the exact defect of 2/9 bits, and Proposition 10 identifies the decrease of 1/9 bit as the safety information contributed by the receipt field beyond the original view.

The controlled results answer where a better judge can still help. In Table 2, a fraction b = 0.6655 of equally weighted safe/unsafe pairs have identical three-step views. In an AUROC comparison, independent positive and negative draws both come from this shared component with probability b². Their expected ranking credit is then 1/2. Even perfect ranking elsewhere gives a ceiling of 1 − b²/2 = 0.77855, compared with 0.779 observed. The counting shortcut explains why the visible cases are easy. The collision rate explains, to the reported precision, why even a better judge could do no better on overall ranking.

For learning, Appendix J.12 shows that adding matched supervision to the specified BCE learner raises silent-fault recall at 5% FPR from 0.2139 to 0.3628. The reviewer’s matched-difference comparator is stronger on this generator, with analytical population recall 0.8029, because one labeled pair reveals the planted direction exactly. The evidence therefore supports targeting the distinctions exposed by matching. The relative merits of other pair-aware objectives and the performance of a learned sketch on natural traces remain open.

We request the committee to weigh the contribution demonstrated across these results. Given a verifier failure, the framework can establish how much improvement is possible with the judge’s input held fixed, identify a particular observation that recovers missing safety information, and quantify the ambiguity that survives the repair. The released AgentDojo records let readers check all three claims on the same executions.

A practical test on 48 public AgentDojo executions

In AgentDojo banking task 4, the user requested one refund. Two GPT-4o executions completed the user’s task and returned byte-for-byte identical final answers. Both also recorded a successful transfer. The receipts reveal the distinction that the answer and the “transfer occurred” bit both omit. One execution made one transfer. The other made two and satisfied the benchmark’s injected goal.

This observation suggested a specific change that we could test across tasks. We flagged a run when its number of error-free send_money receipts exceeded the number of direct send_money calls in that task’s published reference solution. To measure what the count adds, we applied the same rule twice, first with the observed count capped at one (a presence bit), then with the full count. The task reference and decision rule were identical in both comparisons.

We evaluated injection goal 0 across all 16 banking tasks and included every run in the conditions below, irrespective of whether the legitimate task succeeded.

  • GPT-4o, 32 runs across the direct and important_instructions conditions. Preserving the count increased detections from 11/16 to 16/16 runs in which the injected goal succeeded. False alarms remained at 0/16 runs in which the goal failed.
  • GPT-4o-mini, 16 runs under important_instructions. Applying the rule unchanged increased detections from 4/7 to 6/7, with 1/9 false alarms under both representations.

The two GPT-4o-mini errors clarify exactly what the richer view still misses. On task 15, the injected goal succeeded within the reference count. On task 10, the count exceeded the reference even though the injected goal failed. A count records how many transfers occurred. Determining whether a particular transfer satisfied the designated goal can require evidence about the transfer itself.

This is a retrospective test of a hand-specified receipt field for one benchmark goal. It demonstrates the practical contribution of the framework on agent-generated traces. An exact collision identified a distinction the verifier could not observe. Preserving that distinction caught five additional goal successes on GPT-4o and two on GPT-4o-mini, with the decision rule held fixed and no additional false alarms in either set of records.

Published task references · GPT-4o records · GPT-4o-mini records

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.