acceptodds
Under review as a conference paper at ICLR 2027

Gatekeeper Arena: Diagnosing Perception and Memory for Multimodal Policy Decisions

Abstract

Policy decisions can fail when relevant state is absent from the current encounter. We present Gatekeeper Arena, an offline testbed with 100 fictional entrants, 222 Unreal Engine clips, a disclosed nine-role policy, and a three-stage lifecycle: profile registration, unscored familiarization, and frozen-memory evaluation. Models return a fixed record; event-level balanced accuracy becomes the 0–100 Arena Score, accompanied by decision coverage. We evaluate the current- evidence setting with three open-weight VLMs across all five text/visual conditions, and ten hosted families join them on a shared text-plus-eight-frame leaderboard (13 models; 2,925 scheduled decisions). All cluster near chance (49.78– 54.01) and are strongly deny-biased. FlyWire-v783 recurrence retains delayed synthetic input, yet its Arena Score is 45.25 versus 50.53 for a degree-preserving rewire. Additional frames and generic recurrence therefore do not supply the profile and encounter state required by a longitudinal gatekeeper.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, multimodal evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qQzROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Gatekeeper Arena, an offline benchmark for evaluating multimodal models on policy decisions that require latent longitudinal state beyond the current visual evidence. The benchmark includes fictional residents, Unreal Engine clips, a nine-role policy, and a three-stage lifecycle. Across 13 models, performance is close to chance and strongly deny-biased. FlyWire retains delayed synthetic input while trailing its rewired control on the gate task.

Critical strengths
  1. The benchmark clearly separates current perception, longitudinal memory, and generic recurrent capacity.

  2. Balanced accuracy, FAR, FRR, and unresolved rates are reported to reveal failure modes.

Critical weaknesses
  1. Although the paper motivates longitudinal memory, the main model evaluation uses an empty-context setting, while the recurrent track uses synthetic Rademacher inputs and frozen features rather than an end-to-end registration–familiarization–evaluation protocol.
  2. The task deliberately withholds the state needed for some decisions, so near-chance performance may primarily reflect task underdetermination rather than limitations in multimodal reasoning.
  3. The benchmark is narrow, with 100 fictional identities, 222 synthetic clips, one Chinese prompt, and one fixed policy, which limits external validity.
  4. The paper mentions an oracle-state ceiling but does not report empirical results for it, and does not include retrieval-based memory, profile-matching, or human baselines.
Questions for authors
  1. Can the authors evaluate the proposed registration–familiarization–evaluation lifecycle end to end?
  2. Why does the paper not report retrieval, profile-matching, human, or the stated oracle-state baselines?
1 comment

Thanks for the rebuttal. Some of my concerns are addressed in this rebuttal. However, the rebuttal does not clearly report the end-to-end results of the evaluated models on the new track, and the retrieval, profile matching, and human baseline results are still missing. Furthermore, the synthetic scope of this benchmark is relatively narrow, which also limits its external validity. I therefore maintain my current rating.

Reviewer_8XV8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Gatekeeper Arena which is an offline benchmark for a simulated residential gate with fictional residents. A model can observe the entrant using Unreal Engine clips and apply admission policy along with returning a role guess and its confidence.

Several models are evaluated and placed on a text-plus-eight-frame leaderboard. All the models score near chance and are also found to be mostly deny-biased.

The authors also study whether any memory is useful or does it have to be the right memory. They use a fly connectome to check whether the recurrent network helps with improved decisions or not.

The authors conclude that the bottleneck is task-relevant longitudinal state rather than more frames or generic recurrence.

Critical strengths

The benchmark is unique and is significantly more purpose built for the ReID and decision problem here. Curation including detaisl such as registration, familiarization, frozen evaluation lifecycle all show good design.

Critical weaknesses

The first question is whether any task shown is solvable? The tracks P and C show what is not possible. What are the conditions under which the policy can perform above chance? The Track L study is not run where the models are run with registered profiles plus familiarization notes, etc.

As it is, all models are performing near chance, so there is not much to distinguish models' abilities through the benchmark performance.

Minor: No comparison with constant baselines like always allow or always deny shown.

Questions for authors

The first question is whether any task shown is solvable? The tracks P and C show what is not possible. What are the conditions under which the policy can perform above chance? The Track L study is not run where the models are run with registered profiles plus familiarization notes, etc.

What is the connectome probe? What exactly are its inputs and its 1,600 events, and how is each clip turned into reservoir input? Why is the MLP well below chance?

Provide profile cards and the frozen snapshot, and report Arena Score.

Show the oracle, the constant baselines, role macro-F1, and ideally a small human baseline. These would show whether roles are recognizable and whether the benchmark is solvable.

Reviewer_AdZXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper present a benchmark "Gatekeeper Arena" which decides whether an entrant at a residential gate should be allowed to enter in or denied. Each entrant is rendered in Unreal Engine and decisions follow nine-role policy. The paper summarized that no existing VLM model can achieve satisfactory results on this benchmark (max is 54).

Critical strengths
  1. The motivation that real-life decision often depends on information that can not be observed is very intuitive and the research direction is very promising.
  2. This paper designs a balanced accuracy to avoid bias in always-allow and always denied situation.
Critical weaknesses
  1. My main concern is the main claim cannot be supported by evidence presented. The paper claims that current VLMS fail at multimodal gatekeeping decisions (L041). However, unidentifiable tasks by construction leads near-chance scores which are expected rather than informative. In this way, the status existing VLMs cannot perform well is not sound. Specifically the label depends on 2 variables, the role r and the identity-claim state h. The model sees neither: the role is not visually annotated and the claim is withheld. Even a perfect model would therefore be expected to score near 50 under this condition. Furthermore, the paper also reports no human baseline.

  2. Another concern is longitudinal memory condition is never evaluated. The contribution centers on 3-level memory protocol. Yet every results in Table 3 uses the non-context view. The claim that longitudinal state is the bottleneck (L473) is therefore never directly tested.

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.