acceptodds
Under review as a conference paper at ICLR 2027

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

Abstract

Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's true memory capability. To address this, we introduce OPIS, an input-grounded benchmark that strictly anchors the assessment to a fixed set of object instances from the initial observation for evaluating multi-object memory in video world models. The OPIS dataset comprises 500 cases across real-world, embodied-robotic, and game-world domains, providing dense object-level annotations for 12,672 rigid, articulated, and deformable instances. Our object-centric evaluator combines association and explicit visibility reasoning to hierarchically measure Object (O) Presence (P), Identity (I), and Structure (S), utilizing static or dynamic evaluation tracks based on object kinematics. Across eight image-to-video or camera-conditioned world models, our proposed OPIS scores range from 48.65 to 56.01. As the reference inventory grows from less than 20 to more than 40 objects, the Presence, Identity, and Structure scores show an overall decline, with the average Identity score falling from 40.22 to 23.11. The results demonstrate that preserving the particular object instances in the input is considerably harder than generating plausible visual elements.

Primary area:
datasets and benchmarks
Keywords:
world models, object-centric memory, image-to-video models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ff6rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPIS evaluates whether a generated video preserves the objects established in an initial image. It constructs a fixed list of objects along with their geometric references, then measures their presence, identity, and structure, distinguishing the absence of an object from cases involving occlusion or insufficient evidence. The benchmark consists of 500 examples spanning real-world settings, specific scenarios, and gaming environments. Eight video generation systems were evaluated using a rigorous process that included frame-level aggregation, diagnostic analysis of evidence coverage, bootstrapping for confidence intervals, and manual review of evaluation decisions. The study found that strict identity scores were significantly lower than presence scores, with identity scores appearing even lower in scenes characterized by high object density.

Critical strengths
  1. The fixed-input-based inventory addresses meaningful evaluation challenges. Generated videos can remain visually plausible even when the identity or structure of specific objects is altered. By distinguishing between presence, appearance, structure, and evidence that cannot be evaluated, this benchmark serves as a more precise diagnostic tool than a single video quality score. The inclusion of 500 diverse cases, along with the provided code and sample data, represents a valuable contribution.

  2. The paper makes a significant effort to shed light on the limitations of the evaluators. Appendix G distinguishes between repeatability and accuracy, reports inter-rater agreement across four types of decisions, and reveals that "structure" is the dimension with the lowest reliability. Through coverage diagnostics, the removal of aggregation, and the use of bootstrapped confidence intervals for cases, the reported scores are rendered more interpretable.

Critical weaknesses
  1. The validation does not yet establish the reliability of the final strict metric. A single below-threshold object invalidates an Identity or Structure frame score (Section 4), so individual evaluator errors can have disproportionate effects. Appendix G reports 86.0% identity agreement and 83.3% structure agreement with manual review on 150 decisions per dimension, but these overall agreements do not reveal false-failure rates, density dependence, or how disagreement propagates to rollout scores and model comparisons. The 90.5% repeatability result cannot resolve this issue, as the authors correctly acknowledge. Controlled preservation/deletion/identity-change cases or a human audit of complete scored rollouts would provide more direct evidence that the benchmark separates generator failures from evaluator failures. This is consequential because the benchmark's principal contribution is its measurement, rather than a new generation method.

  2. The density result is difficult to separate from the chosen aggregation rule. Section 5.3 reports strict Identity declining from 40.22 to 33.10 to 23.11 as reference-object count increases. Under an all-object gate, the chance that a frame passes decreases with the number of assessed objects even if per-object fidelity stays constant. Appendix H explicitly acknowledges that the score does not establish a decline in per-instance fidelity; that qualification is appropriate, but leaves the substantive scaling insight underdetermined. The aggregation ablations do not replace a count-controlled analysis. Reporting per-object failure rates and fixed-count subsamples from the archived measurements would show whether density adds difficulty beyond the intended stricter joint-preservation criterion. These measurement-validity concerns lead me to weak rejection despite the useful benchmark design.

Questions for authors
  1. Table 13 shows overall agreement rather than the direction of errors. Could you report the false-positive and false-negative rates for manual judgments, including details on the sampling unit and the criteria used for handling ambiguous cases? (Weakness 1)

  2. Sections 4.2–4.4 employ a strict "frame gate." In cases where audited frames or rollout data are available, please explain how often corrections to human judgments alter the outcomes of these gates, rollout scores, or model comparisons. Please distinguish this from the level of agreement in the human judgments themselves. (Weakness 1)

  3. Section 5.1 specifies acceptance thresholds and dimension weights. How were these values ​​determined, and were the calibration cases kept separate from the evaluation cases? Furthermore, do the main conclusions remain valid if these thresholds are varied within a reasonable range? (Weakness 1)

  4. Table 5 shows a reversal in the close point estimates for Seedance/H3-Max-Turbo when equal weights are applied. Could you provide pairwise bootstrap confidence intervals for the comparisons highlighted in the paper, while maintaining the evaluation hierarchy? (Weakness 1)

  5. Appendix H appropriately distinguishes between joint preservation and instance-level fidelity. Could you report the failure rate (falling below the threshold) per object, as well as the number of valid objects per density level, to allow for an assessment of scaling performance independent of the overall object-count requirements? (Weakness 2)

  6. As the number of evaluated instances increases, it becomes increasingly difficult to pass the overall object-count verification criteria. Could you recalculate the identities for a fixed number of object subsamples, while controlling the domain configuration, and report which frames remain valid? (Weakness 2)

Reviewer_eAm6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OPIS, a benchmark designed to test whether video world models actually hold onto the objects present in an input image over time. OPIS creates a fixed object inventory from the starting frame and tracks each item throughout the rollout. It even builds in visibility reasoning so that objects walking off-screen are not unfairly flagged as failures. The dataset spans differnent cases and annotated objects pulled from real world, embodied, and gaming scenes, all scored using frame-gated metrics for Presence, Identity, and Structure. When tested across eight models, the results showed that while models are great at keeping objects present, they struggle to maintain their identity.

Critical strengths
  1. Grounding the evaluation in the input image is a great idea. The reference does not inherit drift from generated frames and does not punish valid trajectory that different from a reference video.
  2. Object level scoring reveals what is obscured by scene level scoring.
  3. For this task, the dataset is large enough in scale, its annotation process incorporates SAM 3.1, DINOv3, MoGe-2, and manual review.
Critical weaknesses
  1. DINOv3 is been proven that it performed worse on some downsteam tasks, DINOv2 is still a more robust choice, have the author identified any issue with the dinov3 compare to dinov2?
  2. The factor of density is to a large extent that already embedded within this evaluation metric. If even a single object falls below the identity threshold, the entire frame is considered a failure, so consequently, a higher number of objects often leads to a greater number of failed frames. So it is necessary to calculate the per-object failure rate based on object density separately for each domain.
Questions for authors

see weakness

Reviewer_MNaNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OPIS, a benchmark for multi-object memory in video world models. Each case has an input image, a camera-motion instruction, and a fixed inventory of the objects in the input image, built with a VLM, SAM 3.1, DINOv3 and MoGe-2 and then checked by humans. The evaluator matches objects in the generated frames to this inventory, decides which objects should be visible using a camera fit, and computes three scores: Presence, Identity and Structure. The paper evaluates eight recent world models and reports that keeping object identity is much harder than keeping objects present, especially in scenes with many objects.

Critical strengths

The problem is important, and anchoring memory evaluation to a fixed inventory of the input objects is well motivated. Separating presence, identity and structure, with explicit visibility states, is a good design for a memory benchmark.

  • The dataset covers three domains with a balanced design, and the paper evaluates recent open and closed models with bootstrap intervals and several analyses of its own scoring rules.
Critical weaknesses

My main concern is that the score may not fully measure memory yet, though I think this is fixable. Nothing in the score depends on how much the camera moves, so a model that barely moves the camera (or simply repeats the input image) could score well, even though the instruction asks for motion. Presence also seems lenient. The paper assesses moving objects only through direct observation, so it is unclear how a moving object that disappears is counted, and in the paper's own case studies (Table 14), Presence is 100 for rollouts where objects are described as replaced or multiplied. In addition, the density finding is presented as a limitation of current world models, but App. H itself says lower scores in denser scenes do not establish a decline in per-instance memory. The human check (Table 13) is done on single decisions, not on whether the final scores rank videos or models the way people do.

Questions for authors
  1. What score does a video that just repeats the input image get? Could the authors report the camera motion of each model?
  2. How are moving objects that disappear counted in Presence? How often is an object's state unknown rather than missing, for each model?
  3. Could the authors add a human study at the video or model level?
  4. Could the authors report a paired test between models, and explain why the ranking changes under the scoring variants in Table 5?
  5. Can OPIS separate the same model with memory on and off (e.g., Matrix-Game)?

I like the question this paper asks, and the input-grounded design is a good idea. Existing memory protocols cannot tell whether a model remembers the input, and OPIS gives the community a useful and well-motivated way to test this. My concerns are about validity: the score has no motion control, Presence may miss some failures, and the human check is not at the video or model level. I think these can be fixed with cheap experiments, so I currently lean toward a weak acceptance. My score depends on the authors showing the static-video baseline, a memory on/off test, and a human agreement study at the video or model level. If the static video scores as high as the evaluated models, I would lower my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.