- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ObserveAct, which investigates how to choose among "continue observing," "modify the video," or "retain the status quo" under a shared budget in video production. The method employs a versioned workspace to maintain assets and evidence, predicts modification benefits, content degradation, and costs via an action-value model, and then trains an observation router tailored for specific action selection. Its core decomposition explains that the value of observation depends not only on information gain but also on deducting the loss of repair options caused by observation overhead. Experiments report a local task completion rate of 77.6%, outperforming Plug-in VOI's 74.9%.
Critical strengths
Strength 1: Information value modeling under budget constraints has clear practical significance.
Decomposing query value into information gain, lost repair opportunities, and query cost accurately captures the phenomenon that "more thorough observation may render effective repairs unaffordable." The appendix supplements assumptions, derivations, error bounds, and finite-state verification. Within its explicitly defined single-step scenario, the theoretical logic holds, and it clearly distinguishes between optimal information value and the actual, imperfect selector's utility.
Strength 2: Supervision and evaluation strengthen the evidence for local validity.
Freezing the selector, independently supervising execution outcomes to verify query gains, partitioning data by semantic families, and conducting blind evaluations on complete videos are all sound designs. The newly introduced NFR metric also avoids the limitations of focusing solely on the conditional harmful-revision rate.
Critical weaknesses
Weakness 1: There is an unexplained inconsistency between the ablation of the core "lost repair options" mechanism and the publicly disclosed cost settings.
Appendix F explicitly states that the main experiment involves executing one modification after limited observation; Table F.1 lists an initial cost of 1.08, a maximum action cost of 1.02, and a maximum atomic query cost of 0.085. If these values represent the complete costs used for feasibility judgments, queries should not crowd out any candidate repairs under the main experimental budget of 2.5. However, in Table 2, the "Ignore lost repair options" ablation still reduces the completion rate from 77.6% to 75.2%. Appendix F.2 describes this ablation merely as replacing the post-query action set with the pre-query action set.
Weakness 2: The primary capabilities of the complete production system remain unverified by multi-step closed-loop experiments.
I believe the current experiments cannot yet answer the multi-stage operational capabilities claimed by the paper: whether evidence invalidation across multiple shots after modifying a reference is handled correctly; whether repair chains where the first step yields no obvious benefit but the second completes the goal are prematurely abandoned; and whether observation at the current node exhausts the budget necessary for subsequent nodes.
Questions for authors
Question 1: Could you explain which actual decisions lose repair options due to queries under the costs in Table F.1 and a budget of 2.5? Does the budget-blind ablation in Table 2 use a fixed predictor or is it retrained? If my understanding is not flawed, could you provide the actual reserved costs, the proportion of changes in the feasible set before and after, or an explanation or experiment that solely switches the feasible set?
Question 2: Could you provide real online closed-loop results involving cross-shot dependencies and at least two modifications, particularly regarding re-validation and subsequent generation after modifying a reference? Can improvements in local query gains still translate into final overall task success rates, or do they vanish due to short-sighted budget allocation?