acceptodds
Withdrawn from ICLR 2027

Morph-V: Autonomous Growth of Reusable Observation Programs for Long-Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video reasoners adapt their observations to individual questions, yet repeatedly plan observation from scratch when similar requirements recur across videos. This prevents successful observation procedures from being retained and reused, limiting the accumulation of observation capability. To this end, we propose MORPH-V, a two-phase framework that couples coverage-guided program acquisition with applicability-guided reuse through a persistent program library. During coverage-guided acquisition, MORPH-V identifies recurring observation requirements unsupported by the current library, composes candidate programs from typed primitives, verifies them structurally without answer outcomes, and stores the verified programs for reuse. Each newly stored program expands library coverage, redirecting subsequent acquisition toward the remaining unsupported requirements. During applicability-guided reuse, MORPH-V selects an applicable program, grounds its observation requirement in the current video, and executes it to construct evidence for the reasoner. Across five video evaluation populations and two frozen reasoners, MORPH-V improves over the corresponding observation controls. Further analysis reveals rising macro-averaged accuracy across reloaded checkpoints and that fixed deployment programs benefit both reasoners.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Reasoning, Observation Programs, Autonomous Library Growth

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ckb6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MORPH-V, a closed-loop framework for long-video question answering. Without updating the parameters of the underlying reasoning model, the framework distills observational capabilities, such as locating cited speech, tracking essential state slots, and organizing event timelines, into reusable programs. These programs are managed through a system involving coverage-guided acquisition, structural verification, and a persistent program library. For new inputs, the framework routes, grounds, and executes relevant programs to construct evidence for a frozen reasoning model.

Critical strengths

The framework establishes a clear, closed-loop pipeline, integrating program acquisition, structural verification, persistent storage, applicability-based routing, and grounded execution to support a frozen reasoner’s predictions.

By selecting programs based on structural trajectories, coverage, and declarative checks, the approach mitigates the risk of answer-label leakage and adheres to the design constraint of accumulating observational capabilities without parameter updates.

Critical weaknesses

The method relies on a fixed taxonomy of observation types, a registry of typed primitives, and predefined structural objectives. Since the program space and applicability conditions are externally defined, the generalization boundaries in open-ended scenarios remain unclear.

Program acquisition and routing are highly sensitive to factors such as the monitoring of question structures, recurrence thresholds, applicability predicates, and priority settings. Changes to these thresholds or the underlying taxonomy could significantly alter coverage and reuse behavior.

Evidence for the growth of the program library is derived primarily from limited deterministic replays and open-ended compositional runs; the study lacks an assessment of variance and scalability that would involve multiple independent data-collection cycles, varied random seeds, and larger-scale program libraries.

Questions for authors

Please refer to the weaknesses.

Reviewer_S4ziOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MORPH-V, a framework designed to enable frozen long-video question-answering (QA) models to accumulate and reuse evidence-gathering capabilities through a persistent library of observation programs. The system first identifies recurring observation requirements not yet covered by the library. It then tasks the frozen model with composing typed, primitive operations—which are subsequently compiled and structurally verified—to form new candidate programs. During inference, the system selects appropriate programs based on the specific requirements of a given question, executes them to locate and organize evidence, and feeds the resulting information into the frozen QA model.

The study evaluates the framework across three types of tasks: locating cited dialogue, covering multi-state evidence, and organizing events chronologically. Experimental results demonstrate that specialized observation programs improve accuracy on these tasks. Furthermore, experiments involving library checkpoints show that the macro-average accuracy increases as more programs are accumulated. The primary contribution of this work is the integration of observation program acquisition, persistent storage, and cross-input reuse into a unified framework.

Critical strengths

This paper addresses the challenge in long-video question answering where observation processes are difficult to accumulate across different inputs. It establishes a pipeline that integrates the acquisition, verification, persistent storage, and subsequent reuse of "programs." Each program encapsulates both its applicable conditions and an executable procedure, enabling the system to expand its evidence-gathering capabilities while keeping model parameters frozen—a design with notable engineering value.

The localization experiments employ controls based on pixel-level and textual-context matching; the state-coverage experiments maintain a constant number of frames; and the temporal experiments keep the input visuals identical while varying only the relational information. These design choices help confirm that performance gains stem specifically from the methods used for localization, coverage, and organization.

The authors evaluated the system by reloading checkpoints from the program library, observing an increase in macro-average accuracy from 51.99% to 59.77%, which demonstrates that the stored programs offer tangible utility for downstream tasks. Furthermore, the fact that these fixed programs yield performance improvements across two different frozen inference models provides evidence that such programs can be reused across different models.

Critical weaknesses

Section 4.3 specifies that in the open-ended composition experiment, programs are acquired following a pre-specified sequence: "interval → set of slots → event timeline." Consequently, the incremental performance gains shown in Table 3 primarily demonstrate the value of incorporating effective programs; they do not, however, establish that coverage-guided dynamic objective selection is superior to a fixed acquisition sequence. Furthermore, the paper lacks comparisons against baselines—such as random acquisition, exhaustive search, or a predefined program library—under an equivalent computational budget, making it difficult to assess the necessity of the proposed, more complex acquisition mechanism.

Candidate specifications can target structural objectives, with the system subsequently ranking candidates based on their average scores across these objectives. While the paper justifies the comparability of these scores by noting that each objective yields a value in the [0,1] range, an identical numerical range does not imply a consistent evaluation scale. Objectives may differ in terms of difficulty, score distribution, and saturation levels; thus, a candidate might gain an unfair advantage simply by prioritizing objectives that are easier to score highly on. The paper should clarify how these objectives are calibrated and whether this ranking approach reliably selects more effective programs.

During the acquisition phase, an input is marked as "covered" primarily based on the existence of an applicable entry; however, the reuse phase further requires a successful binding—otherwise, the system falls back to the "Direct" approach. The paper does not clearly explain how the system identifies capability gaps or refines existing programs when binding fails or when the available evidence is of insufficient quality.

Questions for authors

Under the same candidate evaluation budget, what gains do coverage guidance and model-based proposal generation each provide over random search, enumerative search, and a preconstructed program library? Do any of the existing experiments directly isolate the contributions of these factors?

Reviewer_KHiAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes MORPH-V, a framework for acquiring and reusing observation programs for video reasoning. The method starts from a program library with only Direct answering, and identifies recurring requirements without an applicable program. A frozen LLM proposer then composes candidate programs based on a predefined primitive registry, and candidates are selected based on program graph compilation and structural verification. The resulting programs are then stored and applied to frozen video reasoners. MORPH-V is evaluated on five datasets with three predefined observation requirements: internal localization, slot-set coverage, and event-timeline reasoning. Experiment results demonstrate performance improvement over the corresponding observation controls.

Critical strengths
  1. The motivation of the paper is useful. It aims to reuse observation procedures as programs/skills across videos to improve frozen video reasoners and repeated planning.
  2. The idea of the paper is interesting. It aims to autonomously grow the program library based on current capability coverage.
  3. Experiment results demonstrate performance improvement over the corresponding observation controls and base video reasoners.
Critical weaknesses
  1. The writing of the paper needs substantial revision to improve the clarity.
  2. While the main idea of the paper is to autonomously "identifies recurring observation requirements unsupported by the current library" and generate the missing programs, the experiment setting is too restrictive. The method predefines three required capabilities (quoted speech, required states, and event order), and the goal is essentially to generate one program for each capability. It is unclear why the proposed coverage-guided program acquisition is needed. An alternative could be just generate the three programs in parallel or sequentially.
  3. The main results (Table 1) do not compare with recent state-of-the-art video reasoning and agentic video understanding methods.
Questions for authors
  1. Can the authors explain why the proposed coverage-guided program acquisition is needed and why it is beneficial in the experiment setting?
  2. Can the authors add recent video reasoning and agentic video understanding methods in the evaluations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.